Managing Go versions

Overview

All Go binaries, with the exception of GitLab Runner and Security Projects, are built in projects managed by the Distribution team.

The Omnibus GitLab project creates a single, monolithic operating system package containing all the binaries, while the Cloud-Native GitLab (CNG) project publishes a set of Docker images deployed and configured by Helm Charts or the GitLab Operator.

Testing against shipped Go versions

Testing matrices for all projects using Go must include the version shipped by Distribution. Check the Go version set by GO_VERSION for:

• Linux package builds.
• Cloud-Native GitLab (CNG).

Supporting multiple Go versions

Individual Go projects might need to support multiple Go versions because:

• When a new version of Go is released, we should start integrating it into the CI pipelines to verify forward compatibility.
• To enable backports, we must support the versions of Go shipped by Distribution in the latest 3 minor GitLab releases, excluding the active milestone.

Updating Go version

We should always:

• Use the same Go version for Omnibus GitLab and Cloud Native GitLab.
• Use a supported version.
• Use the most recent patch-level for that version to keep up with security fixes.

Changing the version affects every project being compiled, so it's important to ensure that all projects have been updated to test against the new Go version before changing the package builders to use it. Despite Go's compatibility promise, changes between minor versions can expose bugs or cause problems in our projects.

Upgrade cadence

GitLab adopts major Go versions within eight months of their release to ensure supported GitLab versions do not ship with an end-of-life version of Go.

Minor upgrades are required if they patch security issues, fix bugs, or add features requested by development teams and are approved by Product Management.

For more information, see:

• The Go release cycle.
• The Go release policy.

Upgrade process

The upgrade process involves several key steps:

• Track component updates and validation.
• Track component integration for release.
• Communication with stakeholders.

Tracking work

1. Navigate to the Build Architecture Configuration pipelines page.
2. Create a new pipeline for a dry run with these variables:
   • Set COMPONENT_UPGRADE to true.
   • Set COMPONENT_NAME to golang.
   • Set COMPONENT_VERSION to the target upgrade version.
3. Run the pipeline.
4. Check for errors in the dry run pipeline. If any subscriber files throw errors because labels changed or directly responsible individuals are no longer valid, contact the subscriber project and request they update their configuration.
5. After a successful dry-run pipeline, create another pipeline with these variables to create the upgrade epic and all associated issues:
   • Set COMPONENT_UPGRADE to true.
   • Set COMPONENT_NAME to golang.
   • Set COMPONENT_VERSION to the target upgrade version.
   • Set EPIC_DRY_RUN to false.
6. Run the pipeline.

Known dependencies using Go

The directly responsible individual for a Go upgrade must ensure all necessary components get upgraded.

Prerequisites

These projects must be upgraded first and in the order they appear to allow projects listed in the next section to build with the newer Go version.

Component Name	Where to track work
GitLab Runner	Issue Tracker
GitLab CI Images	Issue Tracker
GitLab Development Kit (GDK)	Issue Tracker

Required for release approval

Major Go release versions require updates to each project listed below to allow the version to flow into their build jobs. Each project must build successfully before the actual build environments get updates.

Component Name	Where to track work
Alertmanager	Issue Tracker
Docker Distribution Pruner	Issue Tracker
Gitaly	Issue Tracker
GitLab Compose Kit	Issuer Tracker
GitLab container registry	Issue Tracker
GitLab Elasticsearch Indexer	Issue Tracker
GitLab Zoekt Indexer	Issue Tracker
GitLab agent server for Kubernetes (KAS)	Issue Tracker
GitLab Pages	Issue Tracker
GitLab Shell	Issue Tracker
GitLab Workhorse	Issue Tracker
LabKit	Issue Tracker
Spamcheck	Issue Tracker
GitLab Workspaces Proxy	Issue Tracker
Devfile Gem	Issue Tracker
GitLab Operator	Issue Tracker
Node Exporter	Issue Tracker
PgBouncer Exporter	Issue Tracker
Postgres Exporter	Issue Tracker
Prometheus	Issue Tracker
Redis Exporter	Issue Tracker

Final updates for release

After all components listed in the tables above build successfully, the directly responsible individual may then authorize updates to the build images used to ship GitLab packages and Cloud Native images to customers.

Component Name	Where to track work
GitLab Omnibus Builder	Issue Tracker
Cloud Native GitLab	Issue Tracker

Released independently

Although these components must be updated, they do not block the Go/No-Go decision for a GitLab release. If they lag behind, the directly responsible individual should escalate them to Product and Engineering management.

Component Name	Where to track work
GitLab Browser-based DAST	Issue Tracker
GitLab Coverage Fuzzer	Issue Tracker
GitLab CLI (glab).	Issue Tracker

Communication plan

Communication is required at several key points throughout the process and should be included in the relevant issues as part of the definition of done:

1. Immediately after creating the epic, it should be posted to Slack. Community members must ask the pinged engineering managers for assistance with this step. The responsible GitLab team member should share a link to the epic in the following Slack channels:
   • #backend
   • #development
2. Immediately after merging the GitLab Development Kit Update, the same maintainer should add an entry to the engineering week-in-review sync and announce the change in the following Slack channels:
   • #backend
   • #development
3. Immediately upon merge of the updated Go versions in Cloud-Native GitLab and Omnibus GitLab add the change to the engineering-week-in-review sync and announce in the following Slack channels:
   • #backend
   • #development
   • #releases

Upgrade validation

Upstream component maintainers must validate their Go-based projects using:

• Established unit tests in the codebase.
• Procedures established in Merge Request Performance Guidelines.
• Procedures established in Performance, Reliability, and Availability guidelines.

Upstream component maintainers should consider validating their Go-based projects with:

• Isolated component operation performance tests.

  Integration tests are costly and should be testing inter-component operational issues. Isolated component testing reduces mean time to feedback on updates and decreases resource burn across the organization.

• Components should have end-to-end test coverage in the GitLab Performance Test tool.

• Integration validation through installation of fresh packages and upgrade from previous versions for:

  • Single GitLab Node
  • Reference Architecture Deployment
  • Geo Deployment