Yesterday I was at a .NET User Group meeting starring Kent Beck. He did a presentation on the different considerations you need to make when increasing the speed of deployment.
Most companies deploy their software on a yearly to quarterly basis. The trend is going towards shorter intervals. Why would you deploy on a more regular basis? The reason is somewhat obvious. It makes the business much more flexible. If you can deploy faster, you can respond faster to the changing business requirements. Also I believe that companies that are ably to deploy on short intervals, are companies with a small amount of overhead.
Moving from annual to quarterly deployments
Add: Automated acceptance tests, Refactoring, Continuous Integration, and Subscription based model
When moving from annual to quarterly deployments you need to add automated acceptance tests. Making software ready for production by manual acceptance tests takes too much time (if you do it the right way). If you move to quarterly deployments, the time it takes to manually test the software will easily overshadow the development time. However if you are doing annual deployments the manual testing is not an overwhelming part of the development effort, and you can live with it.
You need to practice code refactoring at all times when developing for quarterly deployments. In the business world the term refactoring is (mis)used to mean changing the entire code base in one go. This is not what I mean by refactoring. Refactoring is changing the structure of the code one small step at a time. This is a continuous effort that ensures that the code base always is optimized for changeability.
The above considerations are technical. From a business perspective you can consider moving to a subscription based model for your software. If your software is released on a annual basis you would send your sales team out to sell “the big upgrade”. This model does not work as well if you are deploying every three months. Moving to a subscription based model makes more sense. This model also ensures a steady income.
Moving from quarterly to monthly deployments
Add: Developer testing, “Stand up meetings”, “Cards on a wall” and Pay per use
Remove: QA Department, Multiple deployed versions, Design document, Change request, Analysis team and the Build team
When moving to monthly deployments the handoffs between the developers and the QA department starts to become a bottleneck. You need developer testing to prevent bugs moving further down the production line. You could still have a QA team, but this team is much closer to the development team, and the software they are testing is of a much higher quality.
The communication bandwidth needs to increase between the team members. This can be done by adding visual aids like cards on the wall and by frequent short meetings.
Also you need to remove the heavy weight handovers that occurs between the developer team, the test team, the build team and the analysis team. The overhead caused by these handovers will start, again, to overwhelm the development effort. The team needs to be cross functional.
Having multiple versions of your software deployed will easily become maintenance problem. You cannot afford this, so you must have a smooth upgrade process for your software. This will ensure that you only have one version deployed at all times.
From a business perspective you have now the ability to introduce the “pay per use” model. With shorter deployment cycles you have the ability to adjust and change your software as the revenue varies. With longer deployment cycles you are stuck with your software longer and a “pay per use” model would not work very well because it takes a long time to change and adapt.
Moving from monthly to weekly deployments
Add: Live 2-way data migrations, Defect zero, Short lived branches, Keystoning, Kanban, Bootstrap financing
Remove: Test team, One-way data migration, Release branch, Patches, Up-front usability design, Venture capital
When moving to weekly deployments you are possibly deploying 52 times per year. This means that there is a good chance that some of the deployments will fail. You need to be able to roll out and roll back your changes automatically. Doing this manually is too much effort. In other words, you need, among other things 2-way data migrations.
Defect rates must be close to zero. Due to the sort deployment intervals, smaller defects can be fixed in the next deployment. Ideally they don’t occur at all.
Having code that live for weeks on a separate branch is a no-go. The cost of merging is too high. Branches must be short and sweet and usually merged into the main branch during the day.
Weekly iterations will not work. You need to be able to continuously change the priority of the work and work in a continuous manner. This can be done by applying Kanban. Developers will pull a task from the task list when the previous task is completed. There is no hard limit to the amount of work that should be done within a week. Developing the product is a smooth continuous effort.
Note: The idea of Kanban is much more involved than what I have outlined here. Do the Google dance if you want to know more. I strongly recommend this approach for developing software.
When deploying weekly and possibly using a subscription based model you are much closer to the revenue stream. For this reason alone this approach if very tempting to bootstrapped businesses.
Moving from weekly to daily deployments
Add: Immunization, A/B Testing
Remove: Staging, Operations team, Stand-up meetings
Is it getting hot in here? Did you just say daily deployments? Well. Yes :)
Immunization is the idea of automating rollout and rollback by using metrics. For companies with a constant revenue stream you can monitor the money flow on the servers you have upgraded. If you see drops in performance you automatically roll back to the previous version. There are also other metrics you can look at, e.g. the response time of the server.
Daily deployments enable a high level of A/B testing. By deploying different versions of your interface and monitoring the performance of each version, you can easily find the better solution.
“Inside the building there are only opinions”
…you need to test your software our there to really know what is the best solution.
Considerations
If you look at what is happening as you move to a shorter deployment cycle, you will see that all we do is removing friction. An organization that is only able to deploy on a yearly basis has a lot of waste in the system.
Moving to a shorter deployment cycle means learning new technical skills and having organizational backing. If your sales team is selling “big-upgrades”, having weekly deployments makes no sense. The technical skills also need to improve. The old software needs to support the new technical practices. This is not always trivial. You should move slowly to the next level, and not plunge into it.
I will finish this with a somewhat unrelated Kent Beck gem
“Your relationship with your customer is your most important asset”
