In a microservice architecture and organizational structure, teams can work in relative isolation during the development cycle. In part one of this series, we looked at how that approach works well throughout the development lifecycle, but that you need a strategy for it to continue working successfully once you start testing-in-production.
In this edition, we’ll continue to look at how testing-in-production touches on a second misconception of moving to microservices, namely that each service put out by an independently-functioning team will perform perfectly all the time.
We know that there’s only one trustworthy touch-point between services: in production. But if you only meet each other at that point in the development process, what does that mean for building in resilience and quality?
Misconception #2: teams can depend on each other’s services
Reality tells us that services experience all kinds of operational issues, ranging from outright downtime or performance issues to business-logic level failures. If multiple teams are testing in production, this situation becomes complex, quickly. You know that your service can experiences a failure because of a bug in another team's new, faulty, release - just as you know that your own team's releases can never be completely free of defects.
There's always a balancing act between speed of execution and quality of a release, and you optimize for one or the other depending on the lifecycle of the service you put out. Is it an MVP? Better to get quick feedback on what works or not. Is it the umptieth release? You should probably focus on scalability and performance, too.
Still, false assumptions about developing in a distributed, microservice landscape abound. To truly design for resilience and quality, microservice development requires writing code that does not assume other services are available, performant or even there at all. That way, a service will work, as much as it can, even if dependencies fail. In other words, it's a good idea to build services to withstand issues across dependencies by designing for failure.
But when the time comes to go to production, how do you test for all variants of failure scenario? Any simulations you create in non-production environments will never match the reality of how services behave when exposed to live user traffic in a microservices environment.
Insight into dependencies while testing-in-production
The answer is to test continuously in the only way that works for a microservice environment. By continuously testing-in-production and having telemetry on not just your service, but also the services you depend on, you can start analyzing the behavior of your service at times when dependent services start acting a little wonky.
That means creating a strategy around how you manage releases - one that allows you to safely test-in-production. That way, you can analyze the world around you, making the telemetry of other services available to you and helping you decide if you can go forward with a new release, or not.
A good strategy for testing-in-production is adding automated, data-driven controls to each release. That requires a framework where golden signals and service observability are combined with easy to use policy-based controls to test and scale releases in production. Creating policies around how new releases ought to perform gives you insight into the state of your dependency ecosystem. In addition, by adding contextual release state information (like ‘production’, or ‘testing’) to the telemetry of each service, you know if observed errors are due to your own code or another team's.
In the end, services will never perform perfectly all of the time, but by having a strategy around testing and observing them in production, you can mitigate the risk to your production environment.
If you are interested in seeing how Vamp Cloud-Native Release Orchestration can help your organization cut through the complexity of today's IT landscape to innovate more frequently with near-100% reliability, book a guided tour, with or without your own containers!