The more things change…

I have often heard it said the more things change the more they stay the same.  While I am sure it is not completely true all of the time, it is interesting how many times our industry regenerates certain concepts, only better and better with each incarnation.  There are many examples of this.  When I started my career, I was responsible for several large mainframe applications for a large oil company.   These were the days before the client server revolution.  At that time, application developers had the luxury of having their application hosted in a single instance.   Their individual users did not deploy any software.  Any time an enhancement or bug fix was deployed, the entire user community could instantly take advantage.

Today, there is a big trend toward SAAS based applications.  This allows applications to have the luxury of having their applications hosted on a single instance where all of their users get the benefit of any deployed bug fixes immediately.  Only this time, the platforms are cloud based, elastic, and can be deployed on virtual systems that allow for low cost scaling to user demands.  It is a very exciting time to be in the field of software development.

As the world was moving to client server in the 1990s and early 2000s, there became a need to transform the way we build software to be much more agile.  At that time, there was a tremendous push toward transforming R&D organizations to be able to deliver software in a more predictable manner with high quality.   As an R&D leader at the time, I was a part of a major Agile transformation.  Today, I am involved in building a SAAS platform for our company.  What is enlightening to me is the demands and pressures for agile delivery have surfaced again.  Only this time, with more rigor and even more focus on continuos delivery.

On a flight from Boston to Houston, after a meeting with several colleagues regarding our SAAS direction, the similarities between what we need to do now and what we needed to do a few years ago gave me cause to take a look back at our first agile transformation and contrast that to the demands of today’s SAAS based development projects.   They are quite similar, only this time around we must be faster, stronger, more disciplined and more agile.

Here is a paper I wrote back then.  The similarities between then and now are intriguing. So much so that I have actually decided to start blogging about Agile software development again.  I hope you enjoy.  Here is the link to the paper…

Managing a Large “Agile” Software Engineering Organization

 

 

Question: Statistical Analysis to Predict Outcome of Software Projects

Recently, I was wondering if there has been any meaningful work done to apply linear regression analysis to determine the correct balance between various dependent variables used in an Agile process.  For example, what is the impact of architectural runway, number of developers, hours per day, length of an iteration etc. on the quality and output of a software project?   There is no question many ideas and opinions exists on the right value for these variables.  However, I would like to know if there is “statistical proof” on how each of these variables impacts a software project?  If anyone out there has a work they I can reference, I would appreciate it?

Discipline in an Agile Team Reduces Stress

Recently, I was reviewing some of the Agile projects in my area.   One of the things I noticed was there is a direct correlation to the discipline a team has managing its day to day activities and the overall stress level of the team.  Agile without discipline is chaos.  This has always been my belief.  However, I believe there are some intangible benefits to being disciplined during requirements definition, release planning, iteration planning, daily standups, demos etc.  This intangible benefit is the teams tend to feel a lot more in control when these items are in order.  For scrum masters, stress is reduced simply because they can speak intelligently about the meaningful functionality that has been completed.  For the team, they are less stressed because they believe they are a part of an organized project that has a clear probability of success.  For management, a team that has the proverbial “stuff” together is a pleasure to work with and the antidote to many other stressors.

Agile Axiom #5 – Understanding and ahdering to iteration acceptance critieria is critical to the overall quality of a release.

One of the key benefits of Agile is the reduction in the difference between the level of quality for which we strive and the level of quality we actually achieve. Realizing this benefit requires careful attention to iteration acceptance criteria. Driven by the fact that most Agile methods rely on “inside the iteration” testing as their primary test cycle, the impact of ambiguous or not followed acceptance criteria can have a devastating impact on the quality of a release. Over the past several years, there has been much discussion around what it means to be “releasable” at the end of an iteration. Often, teams will set fairly ambiguous acceptance criteria for marking a story or requirement as complete. The level of discipline here is directly related to the level of quality of a final release. So what is good acceptance criteria? “Good” acceptance criteria needs to meet the following in order to be effective in ensuring a given story or requirement is ready for its consumers.

1. Can the user or system consumer of a story or requirement complete the task for which the story is to enable? In other words, does it work? This is fairly binary criteria. If the user or consumer can complete the action, then this one is passed. One should keep in mind that if the result of a given story or requirement cannot be measured with this binary view, the story itself may need revision. “Testability” is one of the key characteristics of a “good” requirement.
2. Are the scalability aspects of the story defined and met? This point is likely the most critical and least understood. An example of a scalability aspect of a requirement might be the number of objects a given UI needs to be able to manipulate. Scalability is often a separate consideration than “inside the iteration” testing usually provides. One could argue whether or not it should be (a good topic for another post). However, regardless of how the overall scalability of a release is managed and tested, it is critical that the scalability needs are considered at design and construction time, within the iteration. Therefore, the acceptance criteria needs to include the scale levels for which a given story must operate. Without an understanding of the scalability requirements of a given story, there is significant risk of needing to rework code after the end of an iteration.
3. Are the non-functional requirements for a given story met? Other non-functional requirements include but are not limited to platform support (operating systems etc.), foreign language support, coexistence requirements, compatibility and interaction with third-party software, etc. Understanding how each of these impact a given story and to what level they need to be tested within an iteration will improve the overall quality of the release.
4. Does the story meet the overall quality objectives of a release? This is more of a general criteria but must be examined. It is possible to be build the code to support a story that passes the criteria above but still causes the product quality to be poor. Lets take a simple example. If there is a story that says “Create a file selection dialog which allows a user to select which files to open for editing”. It could be the dialog exists, works with lots of files, and meets the defined non-functional requirements but does not function properly when the user presses the cancel button. Maybe the user sees a stack trace when pressing cancel. This case may not be captured as a specific acceptance criterion. However, it will likely need to be addressed (or should be). I would suggest there is an ambiguous criterion which requires that products must be at a “reasonable” level of quality beyond the defined acceptance criteria. This notion does not imply a level of quality less than the traditional criteria states, it implies the quality criteria will likely not cover all of the cases which should be tested. Agile helps with this situation as the product is being tested as it is being coded and these issues will be addressed within an iteration before the code is committed or causes other problems. In waterfall methods, these issues would be found much later in the process and would certainly be much more costly to address. In either case, success with this ambiguous level of quality requires a team that establishes a high threshold for quality and simply refuses to deliver code which does not meet the basic quality objectives of a given release.

When teams transition to Agile, the lack of a full testing cycle at the end of the release creates a substantial risk to the overall quality of a release unless the stories and features are tested at the end of the iteration. This is done through the proper defining and testing of acceptance criteria for the stories and requirements that are marked complete for a given iteration. Additionally, stories and features should be at a level of quality which meet the objectives for a given release. For highly successful Agile teams looking to evolve into test driven development, the need and ability to define of detailed acceptance criteria “the tests” is a prerequisite.

Is it possible to be half Agile?

With the all the hype in recent years regarding Agile software development, one does not have to look very far to find a team or organization claiming to be Agile when in fact their methodology is anything but Agile. Recently, I was discussing this phenomenon with a colleague. The conversation focused how Agile should be defined. What is the true test as to whether or not an engineering team is Agile? What does it mean to be Agile? I thought I would take a few minutes and share with you what it means to me.

There are many Agile-like characteristics which can be evaluated as to a team’s level of agility. However, I believe there are three key principles that must exist in order for a team to be Agile.

When your Agile project starts have you committed to a fixed set of features using a fixed set of resources to be completed on a fixed date? If the answer to this is “Yes”, then it is impossible to be Agile when making changes to the project scope to ensure a high-quality on time release. Sure to be controversial, it is my belief that projects with these three variables fixed simply cannot be Agile.

Does your project include a series of increments or iterations that require the completion of “releasable” or “working” software” (defined, developed, and tested) at the end of each increment? Completing working software in an incremental approach enables the team to re-evaluate the project scope at the end of each iteration. This is the foundation for having the ability to make decisions at the end of each iteration. If none of the features are complete and tested until the end of the project, it is too late to re-asses and ensure an on-time successful release.

Does your project allow for the team members to operate in an empowered fashion in which each team member has some ability to influence the methodology and engineering practices? One of the key principles in Agile is believing the people who manage and build the project day to day know the most about the changes that need to be made. If the team does not have the empowerment to self-adjust then there is no mechanism to be Agile.

While it is true most Agile methodologies call for much more than these three items, these represent (in my opinion) the foundation for operating in an Agile way. Adherence, to these key principles as one transitions to Agile is critical to the long term success of the transformation. In short, “Variability”, “Incremental Completion”, and “Empowerment” are the most critical Agile characteristics. All other Agile concepts support at least one of these. For example, automated testing enables incrementally releasable software.

So, is it possible to be half-Agile? It may very well be. However, projects not containing the three principles above will struggle to be successful with their transformation to this new and exciting methodology.

Agile Axiom #4 – To be successful with Agile, failure is more than an option, its necessary.

I am sure most of you have heard the movie quote “Failure is not an option”. You may have even used it as a motivational technique. With Agile development, I would suggest this statement be modified. It works best like this “Failure from time to time is expected – trying, assessing, and adjusting are not optional“.

Most Agile methodologies call for some type of ongoing assessment of the development process. The assessment is intended to produce explicit process changes that are applied to the next iteration or release. Many Agile teams struggle with this in the beginning – particularly when their roots lie in a traditional waterfall management structure. However, it is one of the most important concepts with “Agile”. It is critical that teams be allowed to fail in the process, bring creative ideas to address problems, and determine how improvements are made. There are two primary reasons for this.

• The individuals actually working the process are likely to have the best sense of where the problems are and how to address them.
• People who have influence over how they work are typically more satisfied than those who have a process forced upon them and will demonstrate more passion. This results in an improvement in quality and productivity.

On one of my first Agile projects, the team was struggling with the acceptance rate of their stories at the end of each iteration. For us traditional managers this was of significant concern. It seemed that no matter what we told the team to do, things would not get better. The management team tried measuring the work completed and determine where things were breaking down. This resulted in great debate over the metrics – thats about it.

We even would drive the retrospectives from the management level. If you want to call them retrospectives – these meetings were more like managers listing out what the scrum teams needed to change. Somewhere along the way, we shifted our thinking (with a little influence from Agile thought leaders) to a mindset that allowed us to emphasize the importance of completing and accepting the stories to the teams and then empower them to make adjustments. As managers, we needed to back off and let the teams work the Agile process. The only management directive was that the retrospectives will occur.

Slowly the teams started to increase their story acceptance rate. Each iteration they would make small adjustments to the process. Some would change their iteration plan to accurately reflect the velocity they were able to contribute. Some improved their testing processes to have the Quality Assurance engineer working side by side with developers. The average acceptance rate improved by over 25%.

The challenge with empowering teams to this level is to find the right amount of influence each independent team has on their processes. For example, if there are five teams contributing to a major release of a single product, it would not make sense to allow each team to define its own iteration length. Its important that managers develop a “feel” for how much to control. Managers will likely find themselves in a similar try, assess, and adjust cycle for the how much control they exert over the teams.

My belief is most managers error on the “heavy management” side in the beginning. It is not until they truly understand what is meant by “People over Process” that they begin to see productivity gains on a large level.

Agile Axiom #3 – An automated build system is critical to the success of most Agile software projects.

Recently, I was having a discussion about automated build systems with a colleague. Primarily, we were talking about how important they are for Agile projects. As most of us have, I have witnessed varying levels of usage when it comes to automated build processes.

Looking at the software projects I have been involved with one message is clear…. the better and more frequent run the automated build system the higher the quality in the delivered product. Additionally, the risk is a signficantly lower as well.

In this discussion, I will not get into the specifics of a particular automated build system. I will focus on the reasons to have one.

An automated build system, gives the team a consistent daily goal – complete the work you agreed to do in your daily stand-up, get it unit tested with the testers by your side, commit the code, and make sure it does not break the team’s daily build. Perhaps, this does not need to happen every single day, but certainly many times per iteration.

Each engineer typically works on their parts of the product independently. This is almost always done in their own version of the checked out code. Since each engineer does not test the entire product every day. There is a certain amount of work that must be done when integrating the code. The smaller the amount of code to integrate into the main source tree, the easier it is. Its kind of like when you write a few hundred lines of code without compiling and you are presented with thousands of errors with the first compile. Just as compiling more frequently helps developers pinpoint problems faster, building the entire product helps the teams solve problems faster.

One of the most risky and painful parts of a waterfall project is that time when the code is all written, management has been told the code is 90% done (been that way for weeks), the only thing left to do is pull the pieces together and get the product into QA. However, when all of the individual code pieces are integrated together, a severe problem requiring weeks of refactoring is discovered. At this point, both the date and quality have been compromised. It is very difficult to achieve the true benefits of Agile if this same challenge occurs. Code integration must be frequent.

Recently, I heard of a situation where one of the teams was “accepting/completing” stories that were build on developer machines. The problem with this is the code that is “accepted” may not function the same way when combined with the code from other developers’ machines. The risks here are two-fold. First, teams could find themselves slipping dates as they are going through hardening or even worse having “dead on arrival” defects actually reach their consumers.

To me the best possible situation is to have a nightly build system that does the following:

1. Builds all the target images automatically
2. Creates an installable image
3. Runs and automated installation
4. Automatically conducts some basic tests
5. Notifies the “entire” team if a problem occurs anywhere along the way.

As I write this, it occurs to me I could go on for days about the benefits of having an automated daily build system. However, I believe the point is made here. If you have an Agile project (actually any software project) and are not integrating and building your code every day, I recommend you do so. I was in charge of one project where when we added automated nightly processes, the change in team mindset and quality of committed code was earth shattering.

One closing comment on “build engineers”. Find a good one. Often times teams will assign members with less experience to be the “build engineers”. Having a solid build process is critical to the success of your project and should be staffed with the appropriate capabilities.

Ubuntu Linux – Example of the passion.

As you may have read in previous posts on this blog, the existence of free open-source software is a phenomenon that amazes me. To be quite honest, it is quite satisfying to work in a field where people love it so much , they are willing to do it for reasons other than financial reward. While I am sure this passion exists in other fields, it is rapidly growing in the software business. For desktop users, individuals can obtain the operating system pre-loaded with virtually all of the software needed for home and business. Many people believe they have to know a lot about operating systems and hacking etc. to take advantage of it. This was true at some point. However, today I would contest there are operating systems available for free that are as complete and easier to use than any commercially available solution.

Checkout Ubuntu Linux. If you take the time and install it you will find the experience delightful.

Straight from their website….

“Ubuntu is a community developed, Linux-based operating system that is perfect for laptops, desktops and servers. It contains all the applications you need – a web browser, presentation, document and spreadsheet software, instant messaging and much more.”

Its worth looking into.  Then ask yourself, what drives them to do this and to do it so well?

Customer support or customer success?

Have you ever called a help desk line for a product or service and ended up feeling like the primary objective of the person on the other end of the phone was to get rid of you? Sure, they were helpful but there were just enough underlying behaviors to make you feel like the person truly did not care if you were successful with their product or service. While this particular topic is not directly related to Agile product development, I believe it to be relevant for anyone developing products with some form of customer. The problem runs deeper than telephone support. Face to face interaction with an individual often produces the same feelings – this individual does not really care if I am successful or satisfied. They simply would like the me to go away – successful or not.

Often times providers of service will do their best to generate a state of perceived success in order to satisfy a mis-used measurement system. My recent experience with a car dealership is the perfect way to demonstrate this. Here is the story. Again, it does not directly apply to Agile development but it does apply from the perspective that customer success is a key to the success of any product.

The Dropoff

It seems to me if a business has the objective of retaining long term customers, the primary focus must be making them successful with their products. I recently made a trip to Momentum Mini here in Houston, Texas to have my car serviced. The car did not have a major problem. I had noticed a few times the engine had died while it was running and wanted them to take a look at it. In addition, there were some other maintenance items to be completed. They were fairly simple requests. I dropped the car off on a Monday morning after making an appointment. They have a program where you can pick up a rental car from them. It took an hour and fifteen minutes to get the rental car. Additionally, the first place I needed to visit was the gas station as they had given me a car running almost out of fuel.

The Callback – Not

The service manager told me they would call me the next day. On Tuesday, I did not receive a call. Wednesday morning I left a message on the voice mail of the service representative. Still no answer or call back. Another day passes. This time its Thursday and I am furious. I leave another message for the service representative and the service manager. The message was fairly strong and asked them if they were having technical difficulties with outgoing calls. A few hours later after leaving the messages, I finally get a call back. Basically, there was nothing they could do about the problem and all they had done was rotate the tires.

Picking It Up

When I went to pick up the car, the service representative knew I was upset.  This particular service representative actually seemed to care that I was unhappy.  When they brought the car around to me she noticed there was tree sap all over it and offered to wash it for me.  I accepted the offer even though I was in a hurry.  While they were washing the car, she said the service manager (her boss) wanted to speak to me. She indicated they would be sending me some coupons for dinner at a local restaurant.  This helped a little.  When her manager came out, he apologized for the experience and asked me how they can improve.  When I was finished giving my feedback, I was actually becoming more satisfied.  Then as he left, he destroyed the experience for me.  He said “Now you know you will be getting a survey in the mail.  Feel free to write any comments regarding your experience but it only helps us if you rate us a five (the best rating) on all the questions.”  I told him he had some nerve asking me that and went on about my business.  By the way, I never received any dinner coupons.

Reflections

By now you may be asking yourself, why is this relevant to either Agile development or management, hasn’t everyone had similar experiences?  The primary reason to discuss it here is simply to highlight the fact that measurement when implemented the way the dealership above has done can cause employee behaviors which actually harm the business.  My feeling as I left the dealership was that the only thing the service representatives and management cared about was getting a good score on their survey.  When it was clear they would not, they gave up.  This is why I believe the sales manager never followed up with the dinner coupons.

The lesson here is to carefully choose the mechanism you will use to measure your employees.  Make sure that it truly encourages them to achieve the results you desire.  In this example, having the service manager actually analyze and observe the experience of the customers would have been a much better approach. Or perhaps, reading the comments from the survey and using them to gain a qualitative understanding of how well the service department was doing would have been better.   Nonetheless, in this instance, I am not a successful or satisfied customer and I truly believe the measurement system used by the dealership greatly contributed to my dissatisfaction.

Agile Axiom #2 – Without automation it is mathematically impossible to stay releasable at the end of every iteration.

Based on my experience, without automated testing it is mathematically impossible to stay releasable at the end of each iteration. To prove this point, I will walk through an over-simplified example. The key take away here is as follows. Having an automated nightly build process and a functional automated testing strategy are prerequisites for staying “releasable” when executing even a small Agile project.

Assume the following…

• You are executing an Agile project that will take 6 iterations to complete
• You have to implement 20 user stories, for purposes of illustration assume they are all the exact same size
• You will be design, building, and testing 4 user stories per iteration
• There is one hardening iteration at the end
• Your team is made up of 5 people – 3 developers and 2 quality assurance engineers

• For simplicity in this example assume
  • each iteration is 2 weeks long
  • 6 ideal days of velocity each iteration for development
  • 4 ideal days of velocity each iteration for quality assurance
  • Each story takes 1.5 ideal days for development and .75 days for testing
  • Once built, each story requires .5 days to regression test in subsequent iterations

Iteration 1

4 stories are built taking all 6 IDDs for development and all 4 IDDs for QA

Iteration 2

4 more stories are built taking 6 IDDs for development and all 4 IDDs fro QA

Team absorbs the regression tests requiring two days to complete. The product remains releasable given the fact that the team is confident regression testing has been complete.

Iteration 3

The development of 4 more stories is complete, this is now a total of 12. The QA engineers have tested the 4 new features.

This time the team must absorb 2 days in order to execute regression tests and ensure the product is releasable. The quality of the regression testing becomes compromised given that the work required to complete the testing for the current iteration requires all available velocity.

Iteration 4 and 5

The story is the same. The completion of user stories continues and iteration demos look good. However, the quality is questionable given the lack of velocity spent on regression testing.

Iteration 6

Full regression testing now requires 10 days of QA velocity. The developers are not available to perform the testing because their time is spent fixing bugs. Each fix that goes into the code must be retested and more regression tests must be run. It is likely the team will enter into a series of mini (test, fix, retest) cycles during this iteration. The developers are challenged because many of the problems that were found in hardening are in features that were complete over a month ago. Fixing them takes time. At this point, either the schedule will slip or the quality of the release will be compromised.

The Bottom Line

While this example is extremely over-simplified, the point is the same. The more stories that are developed, the more time is required to perform regression tests. There are two ways to address this.

1. Acknowlege during release planning that velocity will shrink as the iteration progress. In this scenario, you would simply plan fewer stroies in the later iterations. This approach does not optimize the productivity of the team
2. Have a good automated test strategy which enables the team to run automated regression tests every day without requiring the team to waste their precious velocity executing repetitive regression tests. In this scenario, the teams would adopt criteria that includes (Design + Build + Test + Automate) before a user story is “accepted”.

It is true that good development techniques which secure the stability and reduce the turmoil of the code (object oriented programming) as it is developed can reduce the risk caused by the problem identified above. However, these techniques combined with a successful automation strategy will enable teams to deliver high-quality, feature rich, code on time.