GitLab 12.1 released with Parallel Merge Trains and Merge Requests for Confidential Issues
GitLab 12.1 released with Parallel Merge Trains, MRs for Confidential Issues, Automated Let's Encrypt certificates for GitLab Pages and much more!
With GitLab 12.1 we are excited to announce new features
that are unique to GitLab, such as Parallel Merge Trains
and Merge Requests for Confidential issues - along with
long-awaited features like managed Let's Encrypt certificates
for GitLab Pages. Read on to learn more!
Keep the master branch green at high velocity
Keeping the master branch green is critical for Continuous Delivery. When the production build breaks, it means your new code isn't going live, which impacts users and revenue. The only way to be 100% sure the master branch stays green when new code merges is to run the pipeline using the latest version of the master branch. For teams that have a high volume of merges, this can be difficult or even impossible. In the time it takes the pipeline to complete on one code change, other changes can get merged to master with the potential for conflict. The only way to mitigate this is to queue and sequence the changes so that once a production pipeline starts, other code doesn't get merged ahead of that change. This is why we came up with Merge Trains, and shipped the first iteration
in Gitlab 12.0.
Merge Trains make use of GitLab's "Pipelines for Merged Results" feature. With Pipelines for Merged Results enabled, GitLab automatically creates a separate ref that contains the result of merging the source and target branch. The Merge Request pipeline runs on that result ref rather than just running on the source branch. This way, you know if your code is safe to merge into master or not without needing to rebase continually.
Building on this functionality, Merge Trains then make sure that merges are put in the proper order once they are set to merge into the target branch. The merge button in the MR is replaced with a start/add to merge train button that adds that merge to the Merge Train queue. Merges are processed in the proper order even if they are coming in at a rapid pace.
Initially, Merge Train pipelines executed sequentially. The previous MR
in the train needed to complete before the pipeline for the next
MR would start, this kept master green, but could slow
things down, especially for projects with long-running pipelines.
Now with GitLab 12.1, we’ve improved Merge Trains to
run in parallel.
Up to four pipelines execute simultaneously including the ref from the previous Merge Request in the train based on the assumption that all
previous Merge Request succeeds. This way, the pipelines only need to re-run if a previous Merge Request fails. Executing Merge Train pipelines in parallel significantly speeds up the process. You can get started with Merge Trains today by enabling Pipelines for Merged Results and enjoy CD with an always-green master branch.
Iterate privately on sensitive updates
Working publicly in the open is a powerful vehicle for collaboration.
Open source projects with public code, and issue trackers know the benefits of enabling people to contribute regardless of their location. At GitLab we think transparency grants so many benefits that we don’t just make our issues and code public,
but our business processes
as well.
Alongside all the things that work great when they are out in
the open are a few things that are better kept private. A typical scenario for many public projects is the need to keep security vulnerabilities private while they are being worked on to mitigate risk. Previously, GitLab supported
Confidential Issues in public projects but if you wanted to
create a confidential merge request you needed to coordinate
in a separate, private code repository.
Now, Merge Requests for Confidential Issues is making this a lot easier. Clicking on the
Create Confidential Merge Request button on a Confidential Issue allows users to select a private fork within which to create the new branch and merge request, this keeps everything under wraps until
it’s appropriate to reveal it by merging the code from the private
fork back into the upstream public project.
Get automatic HTTPS certs for Pages using Let’s Encrypt
GitLab Pages is a great way to publish web content, from landing
pages and documentation to project artifacts and reports. Of course,
securing your traffic with HTTPS is a must, but keeping track of
the process to procure, provision, and renew certs can be daunting.
It's even more challenging if you are managing a large number of domains. Not having built-in, supported certificate management meant that many organizations operating at scale wanted to use GitLab Pages, but couldn't.
Many of you in the GitLab community have told us how valuable
automated Let’s Encrypt certificates for GitLab Pages would be.
Today, we’re happy to say this highly-requested feature is
available for use. Now, getting secured web traffic to all of
your GitLab Pages custom domains and subdomains is as simple as
clicking a toggle in your Pages settings.
And much more!
There are such a lot of great features shipping with GitLab 12.1 that
we couldn’t possibly highlight them all. Git object deduplication, and Deploy Boards for Kubernetes clusters
at the Group
and Instance level are just a few of the improvements in this release. Keep reading below to get details on every feature along with links to the docs.
Guillaume’s contributions to GitLab 12.0 AND 12.1 bring some much needed (and requested) enhancements to our support for AsciiDoc in both repository and wiki views.
Thanks so much to Guillaume for your contributions!
When you are rapidly deploying changes at a high volume, master or target branches should stay green with accelerated validation and a high degree of accuracy. To meet these needs, we have enhanced merge trains to use a parallel strategy for pipeline execution. Parallel execution accelerates validation by sequencing merge requests in order and kicking off controlled, parallel pipelines. Accuracy is achieved by automatically removing failed merges in the train and refactoring validated merge results in the new order for your target branch.
When discussing, planning and resolving confidential issues, such as security vulnerabilities, it can be particularly challenging for open source projects to remain efficient since the Git repository is public.
As of 12.1, it is now possible for confidential issues in a public project to be resolved within a streamlined workflow using the Create confidential merge request button, which helps you create a merge request in a private fork of the project.
In 12.1, Pages users that add a new custom domain can enable “Automatic certificate management using Let’s Encrypt”. With this setting configured, GitLab automatically procures certs from Let’s Encrypt, provisions them to your custom domain in GitLab, keeps track of the expiration date and automatically renews your certs.
Your selections can be set on a per-domain basis to accommodate varying users and requirements.
Forking workflows makes it easy to contribute to any project by creating a copy of the upstream project to work on before opening a merge request to merge your changes into the upstream project. For popular projects, the server-side storage requirements needed for thousands of copies accumulate quickly and increase hosting costs.
In GitLab 12.1, creating a fork of a public project creates an object pool (if one doesn’t already exist) and uses objects/info/alternates to reduce the storage requirements of forks.
Object deduplication requires hashed storage to be enabled, and for the parent project to be using hashed storage. Existing forks are not currently migrated to object pools automatically – for updates follow gitaly#1560.
In an upcoming release, we will also implement fast forking by directly creating the fork in a deduplicated state. Currently, the fork is first created, then deduplicated.
When migrating Git repositories to GitLab and other systems that need to access them, it is useful to use the same name between systems; This is already possible in most cases, but a few Git tools like Gitolite allow projects to be accessible without a namespace. In GitLab, every project exists within a namespace, making it difficult to migrate these projects.
In GitLab 12.1 instance, administrators can use the new Project Alias API to map a short project alias to a project in GitLab, reducing the risks of migrating to GitLab.
Instance administrators can specify a required include: that runs on every pipeline created in that instance, this can be used to ensure adherence to compliance or standardized processes that should run with no exceptions, helping to enforce consistency across your company’s CI/CD process.
You can now upgrade your GitLab Plan via the customer billing portal! Previously, upgrading your GitLab Plan required reaching out to GitLab manually to request an upgrade which was often frustrating and time-consuming.
Adding the ability to upgrade a plan without manual intervention is an exciting first step towards providing you with a fully self-serviced billing and licensing platform.
For more information on what the Fulfillment team’s roadmap looks like, please check out the direction page!
After removing Rugged and enabling Gitaly for all Git operations, we observed severe performance regressions for large GitLab instances when using NFS. To address these performance problems we added and backported feature flags to re-enable the Rugged implementations that improve performance for GitLab instances using NFS.
If GitLab detects that Rugged can safely be enabled, GitLab automatically re-enables Rugged to improve performance for GitLab instances using NFS.
Performance improvements are being made every release to improve Git access patterns, improve Git performance and implement caching. You can read more about this in our recent blog post, “What we’re doing to fix Gitaly NFS performance regressions”.
To improve the organization and clarity of top-level configuration options in gitlab-ci.yml, we are moving the following configuration options under a separate keyword called default: before_script, after_script, image, services, variables, and cache. The change is fully backwards compatible, and existing configuration options outside of the default context are still honored.
Deploy Boards offer a consolidated view of the current health and status of each CI environment running on Kubernetes, displaying the status of the pods in the deployment.
Starting in GitLab 12.1, deployments made using an instance-level cluster supports the deploy boards feature.
Deploy Boards offer a consolidated view of the current health and status of each CI environment running on Kubernetes, displaying the status of the pods in the deployment.
Starting in GitLab 12.1, deployments made using a group-level cluster supports the deploy boards feature.
Data integrity is crucial when replicating data from a Primary node to a Secondary node. The Geo team is currently working to improve how we verify replicated data. In 12.1, Geo now calculates checksums for attachments, LFS objects and job artifacts on Secondary nodes after transfer compares it with the stored checksums and rejects transfers if mismatched. Please note that Geo currently does not support an automatic way to verify these data if they sync before 12.1.
The GitLab Container Registry allows you to build, push and share Docker images/tags to GitLab. However, if you have special characters (e.g. leading underscore, trailing hyphen/dash, double hyphen/dash) in either your group, project or branch name and navigates to your projects Container Registry, you receive a 500 error and no explanation why.
In 12.1, we have updated the landing page for the Container Registry in the case where you can’t connect to the Docker registry to include an explanation of the error and a link to documentation on how to remediate the issue.
You are now able to leverage the data contained in the Dependency List via an API so that you can create automated processes or to feed other external tools. This API should make the generation of dependency data needed for compliance reports easier and faster.
This API is public so you can retrieve information for this list in a JSON format.
Security Teams use the dashboard as the primary tool for their work. They should be able to set it as the default view for a group, so going to the group shows it immediately.
Instances using smartcards for authenticating with a hardware token have been able to log into the GitLab UI since 11.6. In 12.1, we’re extending smartcard support into the command line by allowing instances to enable the requirement to log in with a smartcard before allowing git activity.
With this change, organizations using smartcards can now use them as a complete authentication solution for both UI and git activity - especially when combined with LDAP.
Issue boards can have many lists, making them difficult to navigate horizontally. In 12.1, we added the ability to collapse an issue board list, allowing you to hide lists. Additionally, you are now able to drag issues from open to closed more easily by collapsing board lists in-between the open and closed lists.
Not all parts of the world have a 40 hour work week, leading to time tracking in GitLab being inaccurate for those areas that may have a 35-hour workweek. In 12.1, an instance-level setting is introduced to configure the number of hours in a week and always display absolute hours for time tracking purposes.
GitLab 12.1 includes
Mattermost 5.12, an open source Slack-alternative whose newest release includes infinite scrolling, bot accounts, deeper Jira integration, and much more. This version also includes security updates and upgrade from earlier versions is recommended.
In the 12.1 release, we’ve introduced a powerful new way to extend the GitLab Runner with custom behaviors which can be used to support different provisioning platforms, set up a secure environment/implement compliance processes, configure secure variables or to run any custom code during a GitLab Runner execution sequence.
This feature is usable with gitlab.com, but because the configuration maps to the runner rather than the .gitlab-ci.yml, it requires you to register your custom runners.
Discussing and requesting changes proposed in a merge request is central to the merge request workflow, but a single line change may need multiple independent discussions. For example, a discussion of the correctness of a line of code may be separate to a discussion of it’s broader implications elsewhere through the codebase.
In GitLab 12.1, it is now possible for multiple discussions to exist on the same line in a merge request diff, making it possible for different discussions to occur in parallel and reducing noise in diff discussions.
It is helpful to some people to create and communicate a Release before it is shipped. You can now create an Upcoming Release through our Releases API by modifying and editing the released_at value with a future date, this makes the Upcoming Release viewable on the Releases page in chronological order with releases and display the “Upcoming Release” label for future dated releases.
If you do not modify the released_at date then the created_date is assumed by default.
AsciiDoc implements several formatting marks above and beyond those supported by Markdown, including underline, line-through, highlight, small and admonition (alert boxes) marks. These formatting marks are now rendered and visible when viewing AsciiDoc files in repositories and wikis.
In GitLab, there are interfaces where the details of arbitrary commits are shown, like merge requests discussions or CI pipeline lists. Fetching commit information one commit at a time is slow since there is an overhead to launching the Git process each time.
GitLab now shares the same Git cat-file process for each Rails request session. After deploying this change to GitLab.com, we observed significant decreases in median latency for FindCommit and TreeEntry RPCs.
You can create custom dashboards based on important metrics needed to observe the state of your applications. Starting in 12.1, you can source control those dashboard definitions right alongside your application code. In 12.1, we started with support for area and single-stat chart types.
When an incident occurs, usually at the least convenient time, it can be difficult to hone in on what is causing the problem, and where you need to start remediation. In 12.1, we enable annotation of GitLab incident issues using Prometheus alert attributes. This implementation is an MVC, and we are excited to continue to iterate on customizing incidents in future milestones.
The GitLab NPM Registry allows node.js developers to publish and share NPM packages. However, many people were blocked from using the feature due to a lack of support for sub-groups. We are excited to announce that with 12.1, we’ve updated the NPM Registry package name format to @root_namespace_path/any-name to allow for authentication and pulling packages at the group/sub-group level.
The GitLab Package stage includes the Container registry, the Package Registry and the [Dependency Proxy](/direction/package/#dependency-proxy. Until 12.1, all of these features were nested under different sections of the top-level navigation, creating discovery and usability issues. As we launch more features and integrations for the Package stage, we must ensure that they have a consistent, easy-to-find navigation path.
With 12.1 we have grouped all of the above features under a new top-level navigation Packages to improve discoverability and ensure we have a place to nest future features such as metrics and policies related to the Package stage.
Today we don’t include all the paths/URLs that were scanned and tested with Dynamic Application Security Testing (DAST) in the MR, this leaves the user unaware of what was scanned and if a full-scan was completed.
Git tags are useful for referencing specific points and are typically used for tagging specific versioned releases. To make git tags easier to use by development teams, we’re adding the ability to allow Developers to rewrite and remove non-protected tags. Protected tags still require Maintainer or Owner permissions.
User can change the milestone for many issues at the same time in a specific project. In GitLab 12.1, we are releasing the capability to bulk edit milestones for many issues at the group level, allowing for easier management of milestones and issues.
We’re also releasing GitLab Runner 12.1 today! GitLab Runner is the open-source project that is used to run your CI/CD jobs and send the results back to GitLab.
Remove ‘app’ label matching from Kubernetes deployment selector
As part of GitLab 12.1, we have removed the app label matching for the Kubernetes deployment selector. As part of GitLab 11.10, we introduced a new matching mechanism which uses app.gitlab.com/app and app.gitlab.com/env to show deployments on deploy boards.
To see these deployments in your deploy boards, all you need to do is push a new deployment, and GitLab uses the new labels for your deployments.
Deprecation of node[‘gitlab’] settings for monitoring components in omnibus-gitlab
In omnibus-gitlab, settings handling monitoring components like
Prometheus, Alertmanager, Grafana, various exporters etc. have been moved
to be under the monitoring cookbook. This means, in gitlab.rb,
instead of referring them like node['gitlab']['prometheus']['username'],
they should be referred using node['monitoring']['prometheus]['username].
Check the update notes
for details.
Removal of Sentry settings for error reporting and logging will come from the UI
Sentry settings for error reporting and logging are removed from the UI in GitLab 12.1. These settings have been available within gitlab.yml since GitLab 11.11. Also, you can able to define a Sentry Environment to differentiate between multiple deployments like development, staging, and production. See gitlab-ce#49771 for more details.
GitLab 12.1 includes a Rails upgrade to 5.2, this may cause issues with zero-downtime upgrades.
If zero downtime is required, we recommend using a load balancer and taking nodes out gradually to upgrade.
We want to hear from you
Enjoyed reading this blog post or have questions or feedback? Share your thoughts by creating a new topic in the GitLab community forum.
Share your feedback