TL blog

Maintaining testing environment for microservices

December 24, 2021


Containerizing applications is almost the new norm in modern software development due to the increasing popularity of cloud infrastructure. Although it has always been a challenge to ensure stability in mircroservices architecture, the advantages of containerizing apps has overtaken its counterpart.

Having a stable testing environment plays a crucial role in ensuring the functionalities of such systems, which also helps maintaining a good development velocity. In my experience working with large and complex microservices systems, I’ve seen a number of approaches that can be used to achieve that goal. In this post, let us go to the details of these approaches

Suitable level of testing

In a typical microservices architecture where many components connect to each other like a mesh of services, it is almost a common sense to ensure each service has enough test coverage. It can span from unit to integration test, and at the highest level, system tests with other microservices. Unsurprisingly, the most complex level of testing is system tests.

A dilemma that are often faced by developers when designing an end to end testing infrastructure is whether they should test frontend applications like websites, mobile apps together with all backend dependencies. This topic is apt for debate as it depends the context of applications. However, I’ve found that most of the time, frontend systems do not need to connect to all backend components in testing end to end. It is usually enough to only connect to an abstraction component like api gateway, then all the microservices behind that api gateway can be replaced with some kind of static mock data. Having said that, there is also some benefits to have a few test cases that run against the complete system, or in other words, end to end tests with real containers. This layer of testing is extremely complex and costly to maintain, thus, keep the number of tests in bare minimum to avoid spending too much time in investigating and fixing tests.

Maintaining stable QA environment

To develop good system tests that run against all backend dependencies, having a stable QA environment is a prerequisite. There are many reasons that can affect the stability of the environment, e.g. a micro service fails to promote new version, new version of the service has bug, container instance suddenly crashes, servers have issues with cpu and memory… Regarding the first sort of root cause which affects the healthiness of a service, one way to mitigate is to create a centralized monitoring for all the components’ status. It is very convenient to check if a node is unhealthy, then developers can quickly isolate which service needs attention.

The second kind of reason which causes problems when introducing a new version in microservice is related to bugs in implementation, it would be harder to investigate issues that make tests failed in these cases. The first resort to remedy this situation is implementing appropriate logging messages, which not only helps debugging the failed tests, but also facilitates investigation for production issues. Logging message is the first thing developers have to look into in order to find potential problems when tests are failing, however it is best to prevent promoting new problematic service in the first place.

Consumer driven tests

As we discussed before, a microservice should have system tests to ensure its functionalities when run against all of the dependencies. In addition to that, before deploying a new version of a service, it is important to have consumer’s tests run against the new version. This is especially needed in api services used for mobile applications, to ensure backward compatibility.

Pact test is a popular framework in the consumer driven tests approach, which allows developers to specify required fields in requests, responses and the rules to validate them. Another subtle aspect in this testing direction is keeping the test script in a separate repository from the microservice codebase, which helps avoid test scripts to be modified carelessly without knowing the impact of all consumers.

To run these tests easily, your infrastructure has to support the capability to spawn all backend components as needed. There should have a single source of truth to register all of the components and their versions such that it will be easy and quite simple to modify the version for a service.

Mock data in external systems

In applications that require interaction with third parties, relying on third parities testing environment is not really an ideal direction. The instability of external systems is hard to control and difficult to monitor. Thus eliminating this dependency by replacing with mock data is necessary to maintain stability in your main testing environment.

There are systems with very complicated domain logic when communicating with external systems, so much that sometimes it is required to generate dynamic mock data, i.e. mock data generated by configurations to support varieties of testing scenarios. I have worked in systems that employ this kind of dynamic mock and my advice is avoid using them unless absolutely necessary. When there is the logic in mocking data, then you would need to answer the question: "Do I have to have tests for tests ?"

With a good mock data, it is tempting to overuse system tests in CICD because one can argue that all the systems are only connected to mock data, then there is no reason that could cause the environment to be unstable. In my experience, it usually turns out differently. Because as we analyzed, there are many reasons that affect the healthiness of containers, which is why we should always limit the usage of system tests and keep the number of these tests at bare minimum.


Having stable QA environment for end to end tests which verify all connected components is useful to ensure all microservices function properly. In order to maintain the environment, the systems should have: monitoring for healthy nodes, consumer driven tests and simple mock data for third party applications. There are radical notions that every component in micro services should be tested independently, but in my personal views, keeping a balance between integration and end to end tests is key to have quality software and at the same time ensure development velocity.

Happy learning :)

Written by Thang Le, who likes to learn new things.