It’s Friday at 3pm…are you ready to push the release button? In this blog series, we’ve been looking at software’s top release strategies and the risks they bring. In our last post, we talked about blue-green deployments as an all-or-nothing approach. We also touched on canary releasing as less risky but requiring a lot of manual labor and guesswork. Does that make software releasing doomed to be the “pager-duty-panic” zone? Our answer to that is a firm no! In this post, we’ll examine how building canary releases around release policies can transform a manual, subjective and error-prone process into a repeatable, hands-off, data-driven release process.
Canary releasing and its benefits
Canary releasing is a methodology that reduces the impact of a failed release. It limits the number of users exposed to a bad release (known as limiting the blast radius) by gradually rolling out a new version of a service to a small, representative subgroup of real user traffic before rolling it out to the entire user base. The benefits of this approach are that you can carry out a gradual release, observe its behavior and then scale up or roll back.
As a methodology, canary releasing has a few advantages over blue-green deployment:
1. With blue-green deployments, most testing must be done in non-production to limit the blast radius if something goes wrong. With canary releases, testing can be done in production while still limiting the impact of faulty releases.
2. With canary releases, the conditions that determine which traffic is sent to the new release are highly configurable and granular, allowing testing to be carried out on specific segments of user traffic before releasing to the majority of users.
3. In blue-green deployments, the entire production environment needs to be switched over. With canaries, individual microservices can be released and tested in production, allowing multiple teams to simultaneously test releases.
4. A side benefit of these per-microservices resources, is that they don’t require double the resources for an entire additional environment as with blue-green deployments.
5. Canary releases allow running multiple versions in production, giving teams maximum freedom, flexibility and velocity to test out new releases.
Disadvantages of canary releasing
Having said that, the downside is that performing a “classic” canary release is a manual and subjective process, with huge variation across release operators and teams. Bespoke decision-making and subjectivity are a major reason teams are stuck at re-inventing the release management wheel in isolation. They have little other choice than to create their own release automation code, which unfortunately means toil and sub-optimal code quality. This highly customized approach reduces re-usability of release automation code across teams.
This siloed approach results in each team manually guesstimating, using metrics from monitoring systems to decide to scale up or roll back the release. Every engineer and team will do it differently; obviously this is not a scalable, repeatable and consistent way of doing releases.
Scaling your release management
So how do we go beyond having to custom-code everything, guesstimating what works and hand-holding our software releases?
The answer is to move towards a policy-based release methodology. Policy-based release management separates setting conditions – both technical and business – from enforcing the conditions. The key is to determine how you want your services to perform up-front, with input from all relevant stakeholders. You translate that input into release objectives and metrics (Service Level Objectives and Service Level Indicators), and implement these as code in a release policy. As a codified set of rules that manages the release, the policy lays the foundation for repeatability, observability and scale.
Advantages of policy-based canary releasing
There are a couple of distinct advantages of moving towards policy-based release management:
1. Decoupling decision-making from the releasing process by settings objectives (SLOs) in advance helps to remove manual monitoring of each release, resulting in releases without human errors. Making releases consistent and repeatable.
2. Codifying the conditions around which to set the release test makes the release objective; add intelligent automation and the release process autonomously decides what to do next based on real-time metrics and pre-set conditions. You have the foundation for self-driving releases.
3. No more untraceable decisions; release policies give you a data trail for incident response, post-mortems or compliance audits.
4. Less toil and rework. Policies are reusable across teams, removing duplicate work so teams can spend more time on creating new code and features.
5. Policies can be customized for each service or even a specific release to test for specific conditions (such as a bug fix that targets a very specific user segment).
The bottom line is….
It’s easy enough to automate a rollback or a canary release for one service. But without a structured and standardized set of conditions on which to base your release test, multiple teams testing multiple services across dependent services will run into roadblocks. That’s where release policies make the difference! They lay the foundation for a robust automated release management process that scales – allowing you and your release team to push the release button – even on a Friday afternoon.
If you would like to learn how policy-based release management can help you release on that proverbial Friday, book a Guided Tour with one of our experts!
Or join us for a deep-dive into all the release strategies in our on-demand webinar 5 Release Strategies You Should Know for Continuous Delivery.