Agile Axiom #2 – Without automation it is mathematically impossible to stay releasable at the end of every iteration.

Based on my experience, without automated testing it is mathematically impossible to stay releasable at the end of each iteration. To prove this point, I will walk through an over-simplified example. The key take away here is as follows. Having an automated nightly build process and a functional automated testing strategy are prerequisites for staying “releasable” when executing even a small Agile project.

Assume the following…

  • You are executing an Agile project that will take 6 iterations to complete
  • You have to implement 20 user stories, for purposes of illustration assume they are all the exact same size
  • You will be design, building, and testing 4 user stories per iteration
  • There is one hardening iteration at the end
  • Your team is made up of 5 people – 3 developers and 2 quality assurance engineers
  • For simplicity in this example assume
    • each iteration is 2 weeks long
    • 6 ideal days of velocity each iteration for development
    • 4 ideal days of velocity each iteration for quality assurance
    • Each story takes 1.5 ideal days for development and .75 days for testing
    • Once built, each story requires .5 days to regression test in subsequent iterations

Iteration 1

4 stories are built taking all 6 IDDs for development and all 4 IDDs for QA

Iteration 2

4 more stories are built taking 6 IDDs for development and all 4 IDDs fro QA

Team absorbs the regression tests requiring two days to complete. The product remains releasable given the fact that the team is confident regression testing has been complete.

Iteration 3

The development of 4 more stories is complete, this is now a total of 12. The QA engineers have tested the 4 new features.

This time the team must absorb 2 days in order to execute regression tests and ensure the product is releasable. The quality of the regression testing becomes compromised given that the work required to complete the testing for the current iteration requires all available velocity.

Iteration 4 and 5

The story is the same. The completion of user stories continues and iteration demos look good. However, the quality is questionable given the lack of velocity spent on regression testing.

Iteration 6

Full regression testing now requires 10 days of QA velocity. The developers are not available to perform the testing because their time is spent fixing bugs. Each fix that goes into the code must be retested and more regression tests must be run. It is likely the team will enter into a series of mini (test, fix, retest) cycles during this iteration. The developers are challenged because many of the problems that were found in hardening are in features that were complete over a month ago. Fixing them takes time. At this point, either the schedule will slip or the quality of the release will be compromised.

The Bottom Line

While this example is extremely over-simplified, the point is the same. The more stories that are developed, the more time is required to perform regression tests. There are two ways to address this.

  1. Acknowlege during release planning that velocity will shrink as the iteration progress. In this scenario, you would simply plan fewer stroies in the later iterations. This approach does not optimize the productivity of the team
  2. Have a good automated test strategy which enables the team to run automated regression tests every day without requiring the team to waste their precious velocity executing repetitive regression tests. In this scenario, the teams would adopt criteria that includes (Design + Build + Test + Automate) before a user story is “accepted”.

It is true that good development techniques which secure the stability and reduce the turmoil of the code (object oriented programming) as it is developed can reduce the risk caused by the problem identified above. However, these techniques combined with a successful automation strategy will enable teams to deliver high-quality, feature rich, code on time.

5 Responses to Agile Axiom #2 – Without automation it is mathematically impossible to stay releasable at the end of every iteration.

  1. Geert says:

    What you say here is very true and it will indeed be visible.
    Another facet that is often forgotten is the burden of not having an automated build (continuous build).
    It is the sad truth that in a lot of cases, QA has to complete their tests for each iteration but only receive weekly drops from development.

    The easiest “solution” to the problem is by starting to use a nightly build system, which “only” makes the QA team run at a schedule “day – 1”. Even when nightly builds are in place, but there is no checkin validation (making sure that checkin will not break the build), QA will often find themselves running at “day – 2” or even worse.

    Add those contraints to the schedule of the example above and it will look even worse. And while there are pretty good models out there “like sandbox checkins and only integrate if the automated build and tests succeed”, they are rarely implemented in bigger software companies. Maybe a topic for your next blog entry ?

    — Geert

  2. pabeavers says:

    There is no question…an automated nightly build process will greatly increase the success of being releasable at the end of each iteration. Having all the developers integrating their code each day reduces the amount of disconnects between them. My experience has been it is releasability at the end of the iteration is not achievable without a successful automated build process.

  3. Daryl Kulak says:

    This is a good posting.

    The first sentence isn’t quite right, I don’t think. You cannot say “Based on my experience, this is mathematically impossible.”

    Something is based on experience (empirical) or it is based on a formula. It can’t be both. Am I off-base here?

    Anyway, great post. I’ve lived several iterative projects where we decided not to do automated testing, and we paid dearly. Regression kills.

  4. pabeavers says:

    I would not say you are off base but it is possible for something to be based on experience and non-empirical. The experience provides the data that is used to conduct the proof of the formula. In this case, the data strengthens the hypothesis. Therefore, if this is an unproven axiom perhaps it is quite an axiom. However, I don’t think Agile Hypotheses sounds as good🙂

  5. […] Agile Axiom #2 – Without automation it is mathematically impossible to stay releasable at the end of… […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: