Agile Axiom #3 – An automated build system is critical to the success of most Agile software projects.

March 30, 2008

Recently, I was having a discussion about automated build systems with a colleague. Primarily, we were talking about how important they are for Agile projects. As most of us have, I have witnessed varying levels of usage when it comes to automated build processes.

Looking at the software projects I have been involved with one message is clear…. the better and more frequent run the automated build system the higher the quality in the delivered product. Additionally, the risk is a signficantly lower as well.

In this discussion, I will not get into the specifics of a particular automated build system. I will focus on the reasons to have one.

An automated build system, gives the team a consistent daily goal – complete the work you agreed to do in your daily stand-up, get it unit tested with the testers by your side, commit the code, and make sure it does not break the team’s daily build. Perhaps, this does not need to happen every single day, but certainly many times per iteration.

Each engineer typically works on their parts of the product independently. This is almost always done in their own version of the checked out code. Since each engineer does not test the entire product every day. There is a certain amount of work that must be done when integrating the code. The smaller the amount of code to integrate into the main source tree, the easier it is. Its kind of like when you write a few hundred lines of code without compiling and you are presented with thousands of errors with the first compile. Just as compiling more frequently helps developers pinpoint problems faster, building the entire product helps the teams solve problems faster.

One of the most risky and painful parts of a waterfall project is that time when the code is all written, management has been told the code is 90% done (been that way for weeks), the only thing left to do is pull the pieces together and get the product into QA. However, when all of the individual code pieces are integrated together, a severe problem requiring weeks of refactoring is discovered. At this point, both the date and quality have been compromised. It is very difficult to achieve the true benefits of Agile if this same challenge occurs. Code integration must be frequent.

Recently, I heard of a situation where one of the teams was “accepting/completing” stories that were build on developer machines. The problem with this is the code that is “accepted” may not function the same way when combined with the code from other developers’ machines. The risks here are two-fold. First, teams could find themselves slipping dates as they are going through hardening or even worse having “dead on arrival” defects actually reach their consumers.

To me the best possible situation is to have a nightly build system that does the following:

  1. Builds all the target images automatically
  2. Creates an installable image
  3. Runs and automated installation
  4. Automatically conducts some basic tests
  5. Notifies the “entire” team if a problem occurs anywhere along the way.

As I write this, it occurs to me I could go on for days about the benefits of having an automated daily build system. However, I believe the point is made here. If you have an Agile project (actually any software project) and are not integrating and building your code every day, I recommend you do so. I was in charge of one project where when we added automated nightly processes, the change in team mindset and quality of committed code was earth shattering.

One closing comment on “build engineers”. Find a good one. Often times teams will assign members with less experience to be the “build engineers”. Having a solid build process is critical to the success of your project and should be staffed with the appropriate capabilities.


Ubuntu Linux – Example of the passion.

March 23, 2008

As you may have read in previous posts on this blog, the existence of free open-source software is a phenomenon that amazes me. To be quite honest, it is quite satisfying to work in a field where people love it so much , they are willing to do it for reasons other than financial reward. While I am sure this passion exists in other fields, it is rapidly growing in the software business. For desktop users, individuals can obtain the operating system pre-loaded with virtually all of the software needed for home and business. Many people believe they have to know a lot about operating systems and hacking etc. to take advantage of it. This was true at some point. However, today I would contest there are operating systems available for free that are as complete and easier to use than any commercially available solution.

Checkout Ubuntu Linux. If you take the time and install it you will find the experience delightful.

Straight from their website….

“Ubuntu is a community developed, Linux-based operating system that is perfect for laptops, desktops and servers. It contains all the applications you need – a web browser, presentation, document and spreadsheet software, instant messaging and much more.”

Its worth looking into.  Then ask yourself, what drives them to do this and to do it so well?

Customer support or customer success?

March 22, 2008

Have you ever called a help desk line for a product or service and ended up feeling like the primary objective of the person on the other end of the phone was to get rid of you? Sure, they were helpful but there were just enough underlying behaviors to make you feel like the person truly did not care if you were successful with their product or service. While this particular topic is not directly related to Agile product development, I believe it to be relevant for anyone developing products with some form of customer. The problem runs deeper than telephone support. Face to face interaction with an individual often produces the same feelings – this individual does not really care if I am successful or satisfied. They simply would like the me to go away – successful or not.

Often times providers of service will do their best to generate a state of perceived success in order to satisfy a mis-used measurement system. My recent experience with a car dealership is the perfect way to demonstrate this. Here is the story. Again, it does not directly apply to Agile development but it does apply from the perspective that customer success is a key to the success of any product.

The Dropoff

It seems to me if a business has the objective of retaining long term customers, the primary focus must be making them successful with their products. I recently made a trip to Momentum Mini here in Houston, Texas to have my car serviced. The car did not have a major problem. I had noticed a few times the engine had died while it was running and wanted them to take a look at it. In addition, there were some other maintenance items to be completed. They were fairly simple requests. I dropped the car off on a Monday morning after making an appointment. They have a program where you can pick up a rental car from them. It took an hour and fifteen minutes to get the rental car. Additionally, the first place I needed to visit was the gas station as they had given me a car running almost out of fuel.

The Callback – Not

The service manager told me they would call me the next day. On Tuesday, I did not receive a call. Wednesday morning I left a message on the voice mail of the service representative. Still no answer or call back. Another day passes. This time its Thursday and I am furious. I leave another message for the service representative and the service manager. The message was fairly strong and asked them if they were having technical difficulties with outgoing calls. A few hours later after leaving the messages, I finally get a call back. Basically, there was nothing they could do about the problem and all they had done was rotate the tires.

Picking It Up

When I went to pick up the car, the service representative knew I was upset.  This particular service representative actually seemed to care that I was unhappy.  When they brought the car around to me she noticed there was tree sap all over it and offered to wash it for me.  I accepted the offer even though I was in a hurry.  While they were washing the car, she said the service manager (her boss) wanted to speak to me. She indicated they would be sending me some coupons for dinner at a local restaurant.  This helped a little.  When her manager came out, he apologized for the experience and asked me how they can improve.  When I was finished giving my feedback, I was actually becoming more satisfied.  Then as he left, he destroyed the experience for me.  He said “Now you know you will be getting a survey in the mail.  Feel free to write any comments regarding your experience but it only helps us if you rate us a five (the best rating) on all the questions.”  I told him he had some nerve asking me that and went on about my business.  By the way, I never received any dinner coupons.


By now you may be asking yourself, why is this relevant to either Agile development or management, hasn’t everyone had similar experiences?  The primary reason to discuss it here is simply to highlight the fact that measurement when implemented the way the dealership above has done can cause employee behaviors which actually harm the business.  My feeling as I left the dealership was that the only thing the service representatives and management cared about was getting a good score on their survey.  When it was clear they would not, they gave up.  This is why I believe the sales manager never followed up with the dinner coupons.

The lesson here is to carefully choose the mechanism you will use to measure your employees.  Make sure that it truly encourages them to achieve the results you desire.  In this example, having the service manager actually analyze and observe the experience of the customers would have been a much better approach. Or perhaps, reading the comments from the survey and using them to gain a qualitative understanding of how well the service department was doing would have been better.   Nonetheless, in this instance, I am not a successful or satisfied customer and I truly believe the measurement system used by the dealership greatly contributed to my dissatisfaction.

Agile Axiom #2 – Without automation it is mathematically impossible to stay releasable at the end of every iteration.

March 22, 2008

Based on my experience, without automated testing it is mathematically impossible to stay releasable at the end of each iteration. To prove this point, I will walk through an over-simplified example. The key take away here is as follows. Having an automated nightly build process and a functional automated testing strategy are prerequisites for staying “releasable” when executing even a small Agile project.

Assume the following…

  • You are executing an Agile project that will take 6 iterations to complete
  • You have to implement 20 user stories, for purposes of illustration assume they are all the exact same size
  • You will be design, building, and testing 4 user stories per iteration
  • There is one hardening iteration at the end
  • Your team is made up of 5 people – 3 developers and 2 quality assurance engineers
  • For simplicity in this example assume
    • each iteration is 2 weeks long
    • 6 ideal days of velocity each iteration for development
    • 4 ideal days of velocity each iteration for quality assurance
    • Each story takes 1.5 ideal days for development and .75 days for testing
    • Once built, each story requires .5 days to regression test in subsequent iterations

Iteration 1

4 stories are built taking all 6 IDDs for development and all 4 IDDs for QA

Iteration 2

4 more stories are built taking 6 IDDs for development and all 4 IDDs fro QA

Team absorbs the regression tests requiring two days to complete. The product remains releasable given the fact that the team is confident regression testing has been complete.

Iteration 3

The development of 4 more stories is complete, this is now a total of 12. The QA engineers have tested the 4 new features.

This time the team must absorb 2 days in order to execute regression tests and ensure the product is releasable. The quality of the regression testing becomes compromised given that the work required to complete the testing for the current iteration requires all available velocity.

Iteration 4 and 5

The story is the same. The completion of user stories continues and iteration demos look good. However, the quality is questionable given the lack of velocity spent on regression testing.

Iteration 6

Full regression testing now requires 10 days of QA velocity. The developers are not available to perform the testing because their time is spent fixing bugs. Each fix that goes into the code must be retested and more regression tests must be run. It is likely the team will enter into a series of mini (test, fix, retest) cycles during this iteration. The developers are challenged because many of the problems that were found in hardening are in features that were complete over a month ago. Fixing them takes time. At this point, either the schedule will slip or the quality of the release will be compromised.

The Bottom Line

While this example is extremely over-simplified, the point is the same. The more stories that are developed, the more time is required to perform regression tests. There are two ways to address this.

  1. Acknowlege during release planning that velocity will shrink as the iteration progress. In this scenario, you would simply plan fewer stroies in the later iterations. This approach does not optimize the productivity of the team
  2. Have a good automated test strategy which enables the team to run automated regression tests every day without requiring the team to waste their precious velocity executing repetitive regression tests. In this scenario, the teams would adopt criteria that includes (Design + Build + Test + Automate) before a user story is “accepted”.

It is true that good development techniques which secure the stability and reduce the turmoil of the code (object oriented programming) as it is developed can reduce the risk caused by the problem identified above. However, these techniques combined with a successful automation strategy will enable teams to deliver high-quality, feature rich, code on time.

Indicators of Agile Maturity

March 20, 2008

Consider this a guest posting. Recently a colleague and I were discussing the best indicators of Agile maturity. He came up with an interesting indicator. Below is the email he (Geert) sent on the subject. The approach requires close observation of the behaviors and attitudes of the team. This is valuable because such observation will uncover other strengths and weaknesses. I think you will find the perspective interesting.

Excerpt from Geert’s email to me….

I think you can measure the maturity of an agile development team by looking at how they want to use hardening iterations and where they would like to put them.

The text in red identifies actions one can take to help move to the next level … the levels go from 1 (novice) – 4 (mature). After that I tried to explain where unified backlog management fits into this and why there is really a need for it after a team reaches the fourth level.

  1. A beginning agile team will prefer to schedule 6 hardening iterations after a 12 iteration development plan. This is “agile” to the hard core “waterfall guy”. A lot of people will actually think they are doing agile, just because they renamed the phases in the development process. People will say “whatever” when they are asked if they like “the new agile approach”. This is still a safe mode to operate in, the “waterfall” process is still there but especially if the discipline and goals of agile are not understood people will really find it cumbersome. Getting past this stage is essential in growing in agile (this is the “painful” stage – because benefits are not clear to anyone). The only thing people have remembered from the meetings is “cool, we don’t have to write specs anymore”. Instead they understand they are now writing cards and should be accepting them (but they don’t have the discipline to do so, especially because they don’t “get it” yet). What will typically happen here is that development management will tell the people the process needs to be followed, cards have to be accepted, etc … A next step is typically that there should be an explanation of “known bugs” in the iteration… not because it’s useful to anyone, but to tell people that this is exactly what we are trying to prevent from happening! Make them realize that a card can’t be accepted with outstanding bugs on it.
  2. As time goes by, the team will mature a bit and you will see the seasoned agile team will shrink the number of required hardening iterations at the end, just because they understand they need to “fix” the high severity bugs as they go and QA understands they need to test closer and better early up in the release cycle. More matured agile teams have been told to accepting cards will be followed more strictly and processes need to be followed. The team will start understanding the benefit of agile and although some will still see it as a corporate tax, they “feel” there is something different about how they do things. Here you have to do whatever you can to encourage there is a trust relationship between Dev/QA and documentation. They have to learn they are all essential in the process and all understand how it works. This can be done in a variety of ways. For example why not let a development demo done by someone from Documentation (if they feel up for it), or let development explain the open bugs to everyone present during the demo, while let QA explain how a certain problem was addressed… this is somewhat controversial, but it works.
  3. Further down the road the team will notice that by adding a hardening iteration in the middle of the development cycle (and flushing out even lesser priority bugs earlier on in the process), it will help them to maintain cadence later on. Usually this proposal comes from the team itself. If you get to this stage you can say you have a team which is starting to “live” the agile way. Some members of the team will say “I don’t care if it’s waterfall or agile, I can do whatever”, but they will have been transformed in their mind and really never want to go back. From this point on, the team starts growing and is self maintained… and they get better at it. Now is the time to encourage this team to train other teams and share experience with others. The team can still grow further but expose them to the experience of others. Product management better be ready at this point understanding that this is not the team they were working with before and it’s the goal of product management to come closer to the development process (closer than they are now – which is typically during demo time).
  4. The final step of maturity is there when the team starts understanding “hardening is not required anymore”, because they made fixing bugs part of their daily routines. They understand bugs will need to be fixed anyway, so there is no point waiting for the hardening iteration to do that. They have completely nailed it down, in most case if catastrophy strikes code can be release almost immediately (there is always the need for some QA testing and getting to the desired image). In this context “immediate releasability” is in relative context. It means : “There was no way we could have done it faster”. For some projects it may be determined that untested story cards are removed, or sometimes when there is a really large feature being implemented in the codebase, you can leave it “inactivated” if possible. The team is now a 100% agile team, and product management has to be brought into the mix on an almost daily basis because now you can start leveraging the whole team as a software factory.

After teams have hit stage “4”, the product management team will really start appreciating the benefit of having a backlog available at all times. That way “smaller items” can be added easily if development is done early. Release planning is defining the “must have” backlog for the release and iteration planning is just a run-time orchestration of feeding the backlog to the development teams.

In very mature teams you will see that bugs end up on the backlog as well, with a high priority, and the backlog is nothing but a set of card decks, where everyone in the team takes a card from their stack when they finished the previous card. Bugs found by QA are added on top of the development deck (because QA has to be able to complete testing and signoff the card as soon as possible). Development completes cards and adds them to the bottom of the QA desk.

Product management always has access to the development cards that have not been started yet, so they may reshuffle the stack as they want. Development can really only see the top card from the stack anyway.

You will start to notice that you need an agile development team before you can leverage the power of backlog management like described here. I think this growth is natural (and can be boosted with proper management), but will require time.

Of course such teams are not created by putting bodies in a room and telling then “you are a team”. This requires trust in each other a lot of trust … and experience … a lot of it… so bottom line, it requires time. Time for people to learn to trust each other, time for people to grow in the maturity curve and time for people to understand and learn to accept this has true benefit to them.

Agile Axiom #1 – Agile development actually requires more discipline that traditional waterfall methods.

March 8, 2008

Recently, I was browsing the Rally database for one of the Agile teams I manage. Rally is the tool from Rally Development we use to track the work we do as we move through the development iterations of our projects. Much to my surprise, the team had not been taking the time to keep Rally up to date. Looking at the database, it was not possible to tell which requirements had been completed, which ones had been tested, and what work was remaining. It was already the third iteration of a seven iteration project and all we had was a list of stories which had been elaborated from the requirements.

As a manager this is quite alarming. When discussing with the scrum master, I asked the simple question “how releasable are we after the first two iterations?”. The response was we had tested the code but would not be certain of the quality until we reached the two hardening iterations at the end. My response was the trouble with that is we are now attempting to execute a waterfall style QA cycle in two weeks. In waterfall, this cycle would take up to six months. Essentially, if we do not address the issue, we will significantly jeopardize the quality of the product.

In this example, it is my belief the team had actually been doing the testing and the code developed within each iteration was releasable. However, because we had not taken the time and exercised the discipline to do the housekeeping on the stories, it was impossible to be confident in the quality.

We should have been doing the following within the iteration…

  • defining stories for the iteration,
  • developing the code for the stories
  • testing the stories
  • automating the tests
  • demonstrating the functions
  • accepting the stories
  • pushing the stories that were not completed to the next iteration

Without doing this and being disciplined about it, it is impossible to tell progress and quality of the work that has been completed. Why is this discipline more important with Agile than it is with waterfall? Agile simply does not provide the end of cycle evaluation of engineering discipline found in waterfall. In Agile, this evaluation and discipline must occur throughout the iterations. You cannot procrastinate the discipline to the end of the release.

Agile Axioms

March 8, 2008

According to most dictionaries, an Axiom is defined as “a self-evident truth that requires no proof” or “a universally accepted principle or rule”. What a perfect name for the forthcoming series of blog entries on the simple rules and principles one must follow to be successful with Agile. Beginning this weekend you will see special purpose posts called “Agile Axioms” . The intent is to share basic rules and principles I believe must be followed for an agile project to be successful.