Based on my experience, without automated testing it is mathematically impossible to stay releasable at the end of each iteration. To prove this point, I will walk through an over-simplified example. The key take away here is as follows. Having an automated nightly build process and a functional automated testing strategy are prerequisites for staying “releasable” when executing even a small Agile project.
Assume the following…
- You are executing an Agile project that will take 6 iterations to complete
- You have to implement 20 user stories, for purposes of illustration assume they are all the exact same size
- You will be design, building, and testing 4 user stories per iteration
- There is one hardening iteration at the end
- Your team is made up of 5 people – 3 developers and 2 quality assurance engineers
- For simplicity in this example assume
- each iteration is 2 weeks long
- 6 ideal days of velocity each iteration for development
- 4 ideal days of velocity each iteration for quality assurance
- Each story takes 1.5 ideal days for development and .75 days for testing
- Once built, each story requires .5 days to regression test in subsequent iterations
4 stories are built taking all 6 IDDs for development and all 4 IDDs for QA
4 more stories are built taking 6 IDDs for development and all 4 IDDs fro QA
Team absorbs the regression tests requiring two days to complete. The product remains releasable given the fact that the team is confident regression testing has been complete.
The development of 4 more stories is complete, this is now a total of 12. The QA engineers have tested the 4 new features.
This time the team must absorb 2 days in order to execute regression tests and ensure the product is releasable. The quality of the regression testing becomes compromised given that the work required to complete the testing for the current iteration requires all available velocity.
Iteration 4 and 5
The story is the same. The completion of user stories continues and iteration demos look good. However, the quality is questionable given the lack of velocity spent on regression testing.
Full regression testing now requires 10 days of QA velocity. The developers are not available to perform the testing because their time is spent fixing bugs. Each fix that goes into the code must be retested and more regression tests must be run. It is likely the team will enter into a series of mini (test, fix, retest) cycles during this iteration. The developers are challenged because many of the problems that were found in hardening are in features that were complete over a month ago. Fixing them takes time. At this point, either the schedule will slip or the quality of the release will be compromised.
The Bottom Line
While this example is extremely over-simplified, the point is the same. The more stories that are developed, the more time is required to perform regression tests. There are two ways to address this.
- Acknowlege during release planning that velocity will shrink as the iteration progress. In this scenario, you would simply plan fewer stroies in the later iterations. This approach does not optimize the productivity of the team
- Have a good automated test strategy which enables the team to run automated regression tests every day without requiring the team to waste their precious velocity executing repetitive regression tests. In this scenario, the teams would adopt criteria that includes (Design + Build + Test + Automate) before a user story is “accepted”.
It is true that good development techniques which secure the stability and reduce the turmoil of the code (object oriented programming) as it is developed can reduce the risk caused by the problem identified above. However, these techniques combined with a successful automation strategy will enable teams to deliver high-quality, feature rich, code on time.