We all know it - long lived feature branches and the resulting "merging hell’. The more features are in parallel development, the bigger these features become (scope creep anyone?), the more developers and teams are added to the project, the bigger the chances are of having several independent feature branches living inside their own universes. And bringing these all together in due time, to release these features to their intended audiences, is more often than not super-painful. Which means a lot of debugging, resolving merging conflicts, stress, and business and the customers not getting what they are asking for in a realistic timeframe and with the intended quality.
Move Faster With An Always Releasable Codebase
To solve this problem lots of solutions have been thought of over the years, including merging strategies like f.e. Gitflow.
With the advent of Continuous Deployment/Delivery it became clear that having an “alway releasable” stable codebase, without having to carve out “release branches”, is a very powerful and valuable thing to have.
If you combine this with the modern and “cloud native” “digital” mindsets like agile, experimental enterprise and data-driven development, the advantage of having this “always releasable codebase” become even more evident.
When you can slice your software features into the smallest possible chunks, to be able to release often and early, and learn as fast as possible about what works and what not, the thought of manually solving merging conflicts, cutting deployment branches, and taking weeks or even months to deliver features to clients, becomes really unattractive.
So, how can we improve this?
Meet Trunk Based Development
A solution for the above described situation can be Trunk Based Development, which is:
"A source-control branching model, where developers collaborate on code in a single branch called ‘trunk’, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge hell, do not break the build, and live happily ever after.”
Yes, this means we need to enable a way of working with the corresponding discipline, processes, and tooling. An approach that helps us to work with smaller feature slices, not having long-running feature branches, and always (automatically) make sure that the “trunk” aka “Main" is always buildable, stable, and ready to deploy.
Sounds easy, eh? Of course, nothing is ever easy, but luckily there are ways and solutions to achieve this Trunk Based Development way of working.
Real-World Examples and Implementations of Trunk Based Development
If our goal is to keep the Main (Trunk) always green and deployable, it starts making sense to be smart about our PR’s and merging strategy. As often, automation AND process are key here.
If we look at successful “Digital Cloud-Native” companies, like Github and Shopify, a clear pattern emerges. GitHub, as well as Shopify, are using a combination of merge and deployment queue’s, and automated Canary Validations.
Typically, when a developer wants to release a feature or a change to the public, she will activate a Pull Request (PR). Now the automation kicks in, it will create a “pull request branch” and add this to the deployment queue. When it’s our turn, the system will automatically build the deployment-branch based on our PR and the latest Main, and do all the CI tests.
Only after the CI tests have been successfully run, the deployment-build build will be canary released into production. This will also locks the environment, so no other branches can be deployed to it. So it’s really sequential, to make sure we always keep a releasable Main trunk.
The deployment-build is running as a canary-released version in production, it can be configured to f.e. get a certain percentage or volume of traffic, based on certain segmentations, like mobile devices or geo-location. You can also chain together a sequence of steps of different segmentations, ramping up f.e. the percentages and/or the segments over time to have even more control over you canary validation.
You can also include automatically triggered synthetic automated tests in this process, like f.e. test order flows with Selenium, Puppeteer or Cypres, to validate that in production, together with all the other services (including third party external ones, like a public SaaS service f.e.) your core functionalities are behaving as expected and not breaking.
Only after you have validated that the deployment-build is checking all the boxes, it will automatically be merged into the Main trunk, again running all the CI tests, and then the Main Trunk will be deployed to production. You could even included a few canary steps inside this update process if you’d like.
Now that the PR has been successfully build into a deployment-build and passed our canary-tests, merged with the Main Trunk, and updated into production, the deployment queue will pick up the next PR deployment request from the queue and unlock the designated environment.
The Secret Sauce To Keep Trunk Based Development Fast and Nimble
You might wonder how you can keep this process fast & nimble, and not block the speed of deployments because of this “sequential deployment queue with canary tests” way of working. The answer is in slicing your features and changes into smaller chunks, so you can more quickly develop, test, validate and release them.
Also for the canary-testing part of this process, there is often the (wrong) assumption that this will need either A) a lot of traffic volume, or B) a long time, to get to significant results. But often a quick “sanity check” aka “smoke test” in your production environment is already super valuable. This can be done with or without synthetic automated test-flows (f.e. using Selenium or Cypres).
It turns out that more than 80% of production issues is caused by a wrong configuration setting, or a dependency-issue between different services, f.e. like a breaking API “contract”. To validate these doesn’t need a lot of traffic or time, it simply needs a deployment being pushed to the goal production environment, because this is were the real test happens, as we all know, having it contained with the canary-release traffic shaping features, to reduce the blast radius, and validating for your core functionalities and features if they behave as expected and everything works together as designed.
One of the methods to decrease the size of features and move faster, is what Martin Fowler from ThoughtWorks calls “Back-to-front” or “UI-last” working:
"If you really must hide a partly built feature, then the best way is to build all of it save the UI entry point and add that UI in a single release cycle. This way the non-ui code is fully integrated with everything else, but nothing is visible or used until you add the last bit at the end. The problem with this approach is that you can't do any testing that requires the UI, such as BroadStackTests through the UI or, more importantly, exploratory tests with the UI, until the final release cycle, just before the feature goes live. You can (and should) still use other tests, such as subcutaneous tests or perhaps a backdoor into the UI.”
With Vamp it’s easy and straightforward to create such a “private VIP entrance” or backdoor into your new feature, without having to instrument any code, and without a feature flag that can be forgotten and create real big damage.
Trunk Based Development and Vamp
I hear you thinking "Yeah, but we are not Shopify or Github. We’re not even close, we first need to focus on other things before we can start doing this!”. This is where Vamp can help you big time, enabling f.e. the Github Flow in your delivery (https://guides.github.com/introduction/flow/).
We call this the “leapfrog approach” where you basically can skip a few expensive steps in between, and end up at a valuable and mature delivery pipeline which can include the approach described above quiet easily.
Vamp’s automation workflows integrate out of-the-box with all CI/CD solutions (like f.e Jenkins, Azure DevOps, CircleCI, Gitlab, Codefresh and ArgoCD) by adding annotations and metadata to the deployment artefacts. When Vamp picks up such a deployment-build, it can automatically Canary-release and validate it in your designated goal environment (this can be production, but can just as wel be UAT, QA, staging etc.), with the specific steps, settings and validations that you have configured. We even provide best-practice “controlled Go-Live policies” out of the box for you to get started right away.
VIP Entrance For Your Software Teams
Vamp also automatically adds a “developer endpoint” or “QA/Test exclusive VIP entrance” where only you (using f.e. your IP address and/or secret headers) have access to the new version and feature running.
Vamp automatically validates all kinds of technical metrics out of the box (f.e. like minimal percentage of healthy instances, number of restarts, HTTP 5xx error codes, latency, CPU and Memory usage), to assess and compare the quality of the new version to the current running one. It’s also straightforward to extend the validations with “business” metrics from f.e. your APM or Analytics data, so you can check if there are no anomalies on f.e. shopping-basket value, click-through rates in essential online flows, or client-side (mobile or web-app) anomalies reported by f.e. Sentry.io.
All of these can be easily included in Vamp's Canary validation process, and only if the results are good, Vamp will remove the canary from production again and feed the resulting information back to your CI/CD solution so it can start building the deployable Main trunk to be delivered to production, all under Vamp’s direction.
Vamp - The Orchestrator Of Your Release Process
This is why we call Vamp an “orchestrator”. If you want to learn more on how easy and quick it is to setup such a modern delivery pipeline for “controlled Go-Live” delivery of features and updates with the highest possible velocity and quality, get in touch with us! You can schedule a 1-on-1 session with me right here. No strings attached.
Hopefully this post gives some insights in how we can increase our software delivery quality and velocity both at the same time, with some novel and valuable learnings from successful online platforms and companies.