One of the key benefits of Agile is the reduction in the difference between the level of quality for which we strive and the level of quality we actually achieve. Realizing this benefit requires careful attention to iteration acceptance criteria. Driven by the fact that most Agile methods rely on “inside the iteration” testing as their primary test cycle, the impact of ambiguous or not followed acceptance criteria can have a devastating impact on the quality of a release. Over the past several years, there has been much discussion around what it means to be “releasable” at the end of an iteration. Often, teams will set fairly ambiguous acceptance criteria for marking a story or requirement as complete. The level of discipline here is directly related to the level of quality of a final release. So what is good acceptance criteria? “Good” acceptance criteria needs to meet the following in order to be effective in ensuring a given story or requirement is ready for its consumers.
- Can the user or system consumer of a story or requirement complete the task for which the story is to enable? In other words, does it work? This is fairly binary criteria. If the user or consumer can complete the action, then this one is passed. One should keep in mind that if the result of a given story or requirement cannot be measured with this binary view, the story itself may need revision. “Testability” is one of the key characteristics of a “good” requirement.
- Are the scalability aspects of the story defined and met? This point is likely the most critical and least understood. An example of a scalability aspect of a requirement might be the number of objects a given UI needs to be able to manipulate. Scalability is often a separate consideration than “inside the iteration” testing usually provides. One could argue whether or not it should be (a good topic for another post). However, regardless of how the overall scalability of a release is managed and tested, it is critical that the scalability needs are considered at design and construction time, within the iteration. Therefore, the acceptance criteria needs to include the scale levels for which a given story must operate. Without an understanding of the scalability requirements of a given story, there is significant risk of needing to rework code after the end of an iteration.
- Are the non-functional requirements for a given story met? Other non-functional requirements include but are not limited to platform support (operating systems etc.), foreign language support, coexistence requirements, compatibility and interaction with third-party software, etc. Understanding how each of these impact a given story and to what level they need to be tested within an iteration will improve the overall quality of the release.
- Does the story meet the overall quality objectives of a release? This is more of a general criteria but must be examined. It is possible to be build the code to support a story that passes the criteria above but still causes the product quality to be poor. Lets take a simple example. If there is a story that says “Create a file selection dialog which allows a user to select which files to open for editing”. It could be the dialog exists, works with lots of files, and meets the defined non-functional requirements but does not function properly when the user presses the cancel button. Maybe the user sees a stack trace when pressing cancel. This case may not be captured as a specific acceptance criterion. However, it will likely need to be addressed (or should be). I would suggest there is an ambiguous criterion which requires that products must be at a “reasonable” level of quality beyond the defined acceptance criteria. This notion does not imply a level of quality less than the traditional criteria states, it implies the quality criteria will likely not cover all of the cases which should be tested. Agile helps with this situation as the product is being tested as it is being coded and these issues will be addressed within an iteration before the code is committed or causes other problems. In waterfall methods, these issues would be found much later in the process and would certainly be much more costly to address. In either case, success with this ambiguous level of quality requires a team that establishes a high threshold for quality and simply refuses to deliver code which does not meet the basic quality objectives of a given release.
When teams transition to Agile, the lack of a full testing cycle at the end of the release creates a substantial risk to the overall quality of a release unless the stories and features are tested at the end of the iteration. This is done through the proper defining and testing of acceptance criteria for the stories and requirements that are marked complete for a given iteration. Additionally, stories and features should be at a level of quality which meet the objectives for a given release. For highly successful Agile teams looking to evolve into test driven development, the need and ability to define of detailed acceptance criteria “the tests” is a prerequisite.