Enhancing continuous delivery with blue-green deployment

In my role as an Equinox IT senior consultant specialising in solution architecture, I am often asked by clients to help with technology decisions and implementations.

In a recent assignment, as part of an Equinox IT team working with a client, we identified an opportunity to improve the client’s deployment process to better fit a continuous delivery approach. The Equinox IT team comprised of an enterprise architect, two very experienced Java software development specialists, and myself as solution architect.

I guided the development team to implement blue-green deployment. In this article I cover the work that we as an Equinox IT team performed to achieve this implementation.

Client situation

The client’s previous release process involved a significant amount of upfront planning, ceremony and bureaucracy before system components could be deployed and new features were made available to users. When a deployment was performed the existing system components were taken offline, replaced and restarted, causing considerable user and business disruption. As a result releases were performed infrequently and outside of business hours.

Decision to implement blue-green deployment

To get over these problems the Equinox IT team made a recommendation to update the release process, so that new releases could be deployed as required with minimal business impact and without requiring a full system outages. The recommendation was agreed by the client.

With continuous delivery in mind and an existing continuous integration (CI) pipeline in place, the strategy we agreed on was a blue-green deployment approach, where two production environments (blue and green) would be hosted, with either being capable of servicing production users. At any given time, one environment would be active and receive production traffic while the other would be available as a staging area for deployments. Traffic could then be redirected to the staging environment when ready with the newly deployed changes. If anything were to go wrong, then the DevOps team would simply switch back to the previous environment.

Implementing the blue-green deployment change

Before embarking on building a blue-green deployment platform the Equinox IT team had to get the client’s environment and applications into a state that would support this approach.

We limited our scope to focus only on the services layer of the system as it was much simpler to deal with deploying individual stateless components rather than an entire system.

Establishing two production environments

First, we needed a means to run two production environments, a blue and a green. The Equinox IT team was already migrating the system to a microservices based architecture and this enabled us to deploy and run components as individual services. Each service instance was self-hosted, running under its own embedded http server and did not rely upon any external dependencies or configurations. This meant we could host several instances of a service on the same server listening on different ports. Essentially this gave us our two environments, a service instance representing the blue environment and another instance running as the green environment.

Other approaches we could have taken was to stand up more virtual machines or use application containerisation.

Using a shared database for both service instances

One of the issues we grappled with during the development of the blue-green deployment implementation was how to handle the database layer. We had been using a mixture of standalone databases and embedded application databases. Each approach had its drawbacks and complexities. We finally settled on service instances using a shared database and carefully managed our database migrations so that schema updates would support both the old and new service versions during the deployment transition period.

Routing traffic between both service instances

We also needed a way to route traffic between the two environments (or in our case: service instances).

We decided to leverage an existing reverse proxy server that was currently used for terminating HTTPS connections. The reverse proxy would host the service endpoint URLs that client applications would connect to and it would then route traffic to specific service instances.

Continuous delivery pipeline

With the environments and traffic routing in place we updated our continuous delivery approach to fit the new arrangements.

The updated deployment and release process is based upon two distinct actions:

1. Deploying software – when a deployment is initiated the DevOps role selects the environment instance (e.g. green) to target. A build and deploy job on the continuous integration server deploys the component artefacts and configuration files for the green target. The deploy job also instantiates the web service to listen on the port designated for the green environment. The new deployment is smoke tested and any fixes implemented and redeployed through the same mechanism.
2. Releasing software - when the new version of the service is ready to be released to users, the DevOps role initiates an activation job on the CI server. The activation job updates the proxy configuration to point to the green service instance. A graceful restart is sent to the proxy to reload its configuration and redirect traffic to the newly deployed green instance. The roles of the service instances are now reversed and production traffic is routed to the updated instance.

The complete process from deployment to release is performed without taking down the existing service and without causing disruption to users.

Result

The goal of our blue-green deployment initiative was to make releasing software a ‘non-event’ - something that just occurs as part of the development process and doesn’t require major outages or intensive upfront planning to perform. Using a blue-green deployment approach reduced the risk and overheads associated with performing deployments and enabled the client to deploy alongside a current live system during business hours. The result was an environment and approach that better supported the client’s continuous delivery direction, enabling faster delivery of new features and enhancements to users.

Kosta is a Senior Consultant based in Equinox IT's Wellington, New Zealand office.