Here at GitLab we understand the importance of safe deployment practices.
Progressive delivery is continuous delivery with fine-grained control over who sees the change. This ensures that all code and configuration updates go through the CI/CD stages to catch any regressions or bugs before they reach customers. If something does make it past those gates, progressive delivery makes sure any negative impact is as small as possible.
We have recently added several features that add safeguards to your deployment process, which we will review in this blog post.
Protected Environments
It is important that deploy jobs are restricted to only those who are authorized to deploy in that environment, and we call this restriction by roles "protected". While this feature has been around for a while, it is important to remember that this should be the first step to take when thinking about safe deployments.
Take a deeper dive into protected environments.
Sequential Deployment (or Safe Continuous Deployment)
If your project follows the Continuous Deployment practice that deploys the master branch to the production environment with GitLab CI/CD pipelines, you may encounter the following problems due to the asynchronous nature of pipeline jobs:
- Multiple deployment jobs run concurrently, targeting the same environment. This can make the environment unstable because the deployment script could conflict and finish in an incomplete state.
- An older deployment job could overwrite the latest deployment, resulting in an unintentional rollback. Some users could be exposed to old feature sets on the production website even though the pipeline shows that the latest deployment job successfully finished.
- A pipeline might deploy to production at the worst time, such as on a holiday or over the weekend, when there is limited staff available to solve potential problems.
To address these problems, GitLab provides the following options:
Limit job concurrency
You can limit deployment concurrency by adding a resource_group to any .gitlab-ci.yml jobs that should run one at a time. For example:
- Pipeline-A starts running with SHA-A
- Pipeline-B starts running with SHA-B (newer)
- Pipeline-A starts a deployment
- Pipeline-B waits for Pipeline-A's deployment to finish
Prevent deployment of old versions
The execution order of pipeline jobs can vary from run to run, which could cause undesired behavior. For example, a deployment job in a newer pipeline could finish before a deployment job in an older pipeline. This creates a race condition where the older deployment finishes later, overwriting the "newer" deployment.
You can ensure that older deployment jobs are cancelled automatically when a newer deployment runs by enabling the prevent deployment of old versions feature.
- Pipeline-A starts running with SHA-A
- Pipeline-B starts running with SHA-B (newer)
- Pipeline-B finishes. Now SHA-B is on the production environment
- Pipeline-A is canceled automatically because it was going to deploy SHA-A to production
Deployment Freeze
To prevent deployments for a particular period, such as during a planned holiday when most employees are out, you can set up a deploy freeze. During a deploy freeze, no deployments can be executed. This is helpful to ensure that deployments do not happen unexpectedly.
Find more detailed information about deployment safety.
Read more about GitLab and safety:
-
How app sec engineers can use GitLab to improve security
-
Wondering how secure GitLab is?
Cover image by Mathew Schwartz on Unsplash
