Canary Deployment: What It Is and Why It Matters

Advanced technology is not always immediately effective and well received. There’s a risk that users will not adapt to or ultimately accept every new software release due to functionality complications or bugs that affect system software. To increase the likelihood of any product or program’s immediate success upon release, billion-dollar companies utilize canary deployments that allow testing across a limited market segment.

Canary deployments offer an outlet for releasing new features staggered to mitigate risks such as breaks in services, outages, and non-compliance. This approach also allows for a swift and safe rollback to previous working versions as potential issues are navigated and solved. Canaries are typically shared with the most tech-savvy users initially, while others utilize older, stable product versions. They are ideal for trailing new versions of an entire application that include specific features, upgrades, or configuration changes. Multiple canaries can be used to release different program aspects to millions of users to effectively expose all or most of one’s user base at a more cost-effective rate. Unfortunately, canary deployments are not applicable in all circumstances. It’s important for industry professionals to understand the challenges and considerations involved when developing a canary deployment.

Origin and Purpose of the Canary Deployment

The meaning behind the rationalization for canary deployments is quite literal. During the 19th century, coal mining became more common and dangerous as miners travelled deeper into mines with the help of motorized tools. Canary birds would be brought into mines to have their singing and chirping monitored. If they became less audible, it indicated that carbon monoxide and other poisonous gases were present because canaries are more sensitive to these gases than humans.

Today, a canary release can “warn” developers and designers of potential flaws and inefficiencies in a new application version that consists of specific new features, upgrades, or configuration changes. A canary includes all the necessary application code and dependencies and is released to a target environment.

In a canary deployment, two application versions run simultaneously—the current “stable” version and the new canary version. Canary deployments generally follow a standard set of steps:

1. New Software Deployment. All servers within the target environment are set to run the current software version and receive 100 percent of the user traffic.
2. Rolling or Parallel Deployment. In a rolling deployment, some servers can be upgraded to run the canary in small stages, or the DevOps team can create a whole new clone of the target environment to deploy the canary. This is called parallel or side-by-side deployment.
3. Smoke Tests. DevOps teams can conduct smoke tests to evaluate the systems’ most critical functionality aspects, monitor upgraded servers, and check the logs for errors and performance issues.
4. Deployment or Rollback. If the canary version appears to be good at that point, a decision is made to proceed with gradually deploying it to additional servers with more users in small phases or roll back the upgraded servers to the original state if there are significant issues with the canary release.
5. Successful Deployment. When the deployment is complete, 100 percent of the traffic is directed to the new version, with the old version no longer in use.

Benefits of Canary Deployment

There are key benefits to implementing a canary deployment model configured for a specific infrastructure:

• Version Comparisons. Testing in production with real users and use cases lets developers compare the capacity required for both canary and the current version by testing a small subset of users.
• Easier Rollback. Switching the traffic back makes reverting to a previous version more easily accomplished than the traditional approach, which was complex, time-consuming, and risky.
• Early Feedback. Canary tests can be designed to observe potential errors or bugs and obtain user feedback. Canary deployments shorten feedback loops on new features delivered to target environments. Users are exposed to new features, and experiences are improved because vendors tackle problems as they occur.
• Real-world Testing. Canary deployments are an excellent strategy to test new features or versions without imposing the significant risks of pushing a new application to production in front of the entire user base.
• Reduced Risk. Deploying canary versions in incremental phases offers a safer way to test in production with minimum user impact. Companies like Facebook, Google, Firefox, and Netflix leverage canary deployments to validate changes to backend services by rolling out new software to small, controlled groups. This mitigates the risk of negative consequences that may arise during rollouts.
• Improved Morale. Because the stakes are lower, new features can be deployed with more confidence and less stress. Informed decisions about the readiness of new releases result in less need to roll back major releases that could have consequences from a public relations standpoint and a need to rework the codebase.

Drawbacks of Canary Deployment

Canary deployments also contain certain limitations:

• Added Complexity. Teams have two environments to maintain, as well as additional application code, services, and components.
• Less Control. The deployment timing could be difficult to control if the software is deployed remotely on customer devices or other sites. Developers will spend additional time to ensure customers have updated and evaluated the software before any version changes. The more software runs on the client side, the less amenable it is to canary deployments.
• Observability. Deployments that appear good can fail when rolled out to a more extensive user base. To determine if the deployment is successful, it’s essential to constantly measure relevant metrics from user behaviours and system and application issues.
• Technical Challenges. Routing to subsets of users for new versions can be error-prone, time-consuming, and complex. An automated process and a reliable deployment pipeline are essential to performing canary deployments using feature flags.

Canary Deployments and Orchestration Engines

Orchestration engines are frequently used to deploy, scale, and manage containerized applications that contain the dependencies required to run the software, such as configuration files, binaries, libraries, and frameworks. Kubernetes is a particularly popular orchestration engine used in tandem with Docker to deliver application software in container packages. Although Kubernetes does not provide canary deployment functionality out of the box, there are several ways to achieve this.

A common way to enable or disable a new feature in software development is by using feature flags to change the runtime behaviour of an application without restarting it. This strategy gives DevOps teams more granular control over deployment to eliminate rash and taxing rollbacks. The canary release is now deployed to all nodes in production, but the new features are hidden behind feature flags. Each flag can be turned on or off to control the rollout of a new feature to a subset of users. The feature flag handles the canary deployment; if the canary test fails, the feature flag can remain off. Conversely, if all is well with canary tests, code can be deployed to all nodes, and feature flag rollout can begin.

Planning a Canary Strategy

Consider these factors when planning a canary deployment:

• Users. Select a canary user group. Users can be a random sample or assigned based on geographic region. Another option involves releasing the canary to internal users and employees first.
• Capacity. Running production release and canary release side by side will cause performance issues and require increasing capacity.
• Stages. A canary deployment is probably not necessary with a small overall user base. If the user base is large, it is necessary for the organization to decide the number of users to direct requests to from the canary (and in how many stages).
• Architecture. The architecture can be complex, with many clients that need to be updated before results can be assessed. This could be resource-intensive and take several weeks.
• Metrics and Monitoring. It is vital to collect application health metrics to decide whether the canary deployment should be rolled out or rolled back. It is better to monitor application performance and error reports proactively. Prometheus and Grafana are popular tools for collecting and displaying data on dashboards.
• Confidence. To ensure the success of a deployment, it can be redirected to one percent of the user base and make sure there aren’t any serious problems before slowly ramping up to more users until all the user base has been migrated.

Software delivery is always challenging. The real test is when users begin using a new release in production. Canary deployments allow DevOps teams to conduct controlled trials with real users and help them have a consistent experience. They are more suitable for projects that leverage Kubernetes’ flexibility and closely monitor its performance. By conducting comprehensive deployments, organizations can implement sophisticated deployment strategies that enhance an application’s reliability and user experience. The future of canary deployment looks promising as a suitable option for modern challenges where new technologies emerge at an unprecedented rate. It has an essential advantage compared to other deployment models where automation is a priority in cloud-based and distributed applications, increasingly frequent deployments, microservice-based architectures, and layering of multiple deployment teams. More recently in the field of AI and machine learning, a canary deployment strategy can be used to validate new AI models and updates online under real-world conditions. Teams that possess an agile development practice will especially experience the benefits.

About the Author

Dinesh Chacko is an enthusiastic IT evangelist with a knowledge and passion for AI, cloud computing, information technology, and online security. He has seen the IT landscape change, been a part of organizational and cultural change during large company mergers and has been involved and affected by many first and second-generation IT outsources. Dinesh understands the personal and professional challenges and opportunities created during organizational transitions. He has held multiple roles for small and large organizations, including the public sector, oil and gas, banks, telecoms, financial services, EU institutions, and directly for IT solution integrators. For more information, contact dinesh.chacko@ieee.org.