--- title: "GitLabOps" slug: gitlabops category: delivery term_status: vendor-specific evidence_status: partially_verified summary: "GitLabOps is this Atlas's name for operating software delivery on the GitLab platform: designing .gitlab-ci.yml pipelines, running runner fleets, governing deployments and scanners, and keeping a GitLab instance healthy and current." audience: [Beginners, CI/CD engineers, GitLab administrators, Platform teams, Engineering leaders] last_verified: 2026-10-16 key_takeaways: - "GitLabOps is a vendor-specific label, not a standard discipline: it means operating delivery on one platform, GitLab." - "Most of the work is runner capacity and isolation, pipeline efficiency and safe upgrades, not writing YAML." - "Segment runners by trust: protected, ephemeral runners for deploys and privileged builds; never shell executors for untrusted code." - "Measure wall-clock feedback time, queue time, success rate and cost per pipeline before tuning anything." - "GitLab recommends Flux with the agent for Kubernetes production. Whoever can move the tag Flux follows, or obtain the signing identity, can deploy: protect the tag, verify signatures, and give the signing identity to the deploy job only." - "Self-managed instances must upgrade every month or two to stay in the security-fix window, passing required stops and letting background migrations finish." related: [gitops, devops, devsecops, releaseops, platformops, testops] --- ## What GitLabOps is ### GitLabOps in 30 seconds **GitLabOps** is the work of running software delivery well on GitLab: designing `.gitlab-ci.yml` pipelines, operating the runners that execute jobs, governing environments and deployments, wiring in GitLab's scanners and registries, and, on GitLab Self-Managed, keeping the instance healthy, backed up and upgraded. Every push or merge request becomes a **pipeline** of **jobs**; a **runner** with matching tags runs each job in an **executor** (container, VM or Kubernetes pod), and deploy jobs record **deployments** to **environments**. GitLabOps keeps this loop fast, cheap, safe and dull. ::: note Honest term status "GitLabOps" is **not** a recognised discipline, job family, standard or certification; we found no independent definition in October 2026. GitLab's homepage presents "agentic software engineering" and a "DevSecOps platform"[^gitlab-home]; its topic page "How to Use GitLab for Ops" means ops teams using GitLab for monitoring, Kubernetes and incidents[^gitlab-topics-ops]. We use the plain sense, operations on one vendor's platform: a **vendor-specific** label. ::: **Where to start.** Beginners: the [glossary](#glossary-and-mental-model), [Lab 1](#hands-on-labs). Engineers: the [playbooks](#core-practices-and-playbooks), [reference implementation](#reference-implementation). Leaders: the [decision framework](#decision-framework), [judgement call](#a-judgement-call-one-shared-fleet-or-a-separate-build-fleet). **Last verified** 2026-10-16 against GitLab 19.x, when 19.5 was current and 19.4 and 19.3 received security fixes only[^maint]. ### At a glance | Aspect | Summary | |---|---| | Goal | Fast, trusted feedback; gated, reversible deployments; a supported, affordable platform | | People | Developers (the users), CI/CD engineers, runner operators, GitLab administrators, security engineers, environment owners | | Prerequisites | Git, YAML, containers and Linux; for Self-Managed administration also PostgreSQL and backups | | In scope | Pipelines and components; runners; environments, deployments and the agent for Kubernetes; scanners as configured; registries; self-managed upgrades | | Out of scope | Principles ([GitOps](../gitops/)); culture and DORA ([DevOps](../devops/)); release strategy ([ReleaseOps](../releaseops/)); scanning policy ([DevSecOps](../devsecops/)) | | Outputs | A pipeline skeleton and pinned components; runner fleets by trust; a guarded promotion path; an upgrade calendar; a monthly scorecard | | Term status | Vendor-specific (note above); product facts are dated because GitLab ships monthly[^maint] | ### The problem it solves Starting is easy; trouble arrives with scale and time: - **Waiting, not building.** Jobs stay `pending` because no runner has the tags, the only matching runners are *protected* (they take only protected-branch and tag jobs), or the host's `concurrent` cap is reached[^runner-config][^runner-advanced]. - **Duplicate work.** Branch and merge-request pipelines without a guard run twice per push[^workflow]; superseded pipelines run on unless jobs are `interruptible` and auto-cancel is set[^yaml]. - **Leaky trust boundaries.** A shell executor or privileged container on a shared runner lets any Developer reach the host[^runner-security]; fork merge requests can carry code that tries to steal secrets[^mr-pipelines]. - **Ungoverned deployments.** By default Developers can modify environments, and an older deploy job can finish after a newer one[^deploy-safety]. - **Platform drift.** A lagging self-managed instance leaves the security-fix window; catching up means *required upgrade stops* and *background migrations* before each hop[^maint][^upgrade-paths][^bbm]. **The cost of ignoring it** (illustrative): 40 engineers waiting for 6 pipelines a day, 10 avoidable minutes each, lose 40 × 6 × 10 = 2,400 minutes, 40 engineer-hours, every working day. ### What GitLabOps is not: boundaries and naming - **Not GitOps.** OpenGitOps defines GitOps by four principles: declarative, versioned and immutable, pulled automatically, continuously reconciled[^opengitops]. A CI job running `kubectl apply` is push-based; GitLab says "The GitOps workflow is recommended" and "You should not use a CI/CD workflow for production deployments."[^agent] GitOps *on* GitLab (Flux plus the agent) is covered here; the principles are on [GitOps](../gitops/). - **Not a product, role or certification** ([FAQ](#faq)). - **Not only administration.** On GitLab.com or Dedicated you still own pipelines, runners, secrets, environments and cost[^offerings]. **Two misconceptions, corrected by GitLab:** "Masking a CI/CD variable is not a guaranteed way to prevent malicious users from accessing variable values"[^variables]; and "The security risks are greater when using instance runners as they are available by default to all groups and projects"[^runner-config]. ## Where it comes from and where it sits ### Origins and evolution The word has no traceable origin. GitLab's operational surface has a history that older tutorials still miss: - **16.0:** create a runner in the UI or API first, then register it with its `glrt-` token[^new-workflow]. - **16.2:** the agent's built-in GitOps deprecated in favour of Flux[^agent-deprecated]. - **Runner 17.1:** Docker Autoscaler generally available, replacing Docker Machine[^docker-autoscaler]. - **17.5:** Docker Machine "scheduled for removal in GitLab 20.0 (May 2027)"[^docker-machine]; required stops fixed at x.2, x.5, x.8 and x.11[^upgrade-paths]. - **18.0:** registration tokens were not removed after all[^regtoken-issue], but stay "not recommended"[^new-workflow]. - **19.0:** job tokens default to JWT format, the legacy format usable until 20.0[^job-token]. - **Current docs:** the kaniko page is marked removed: "kaniko is no longer a maintained project"[^kaniko]. ### Glossary and mental model **Mental model: four loops**, each failing in its own way. When someone says "the pipeline is slow", first ask which loop is slow. ::: figure Four loops of GitLabOps and how each one fails ```mermaid %%{init: {"flowchart": {"wrappingWidth": 135, "nodeSpacing": 14, "rankSpacing": 26, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 subgraph L1["1 Configuration"] direction LR C["YAML and components merge into one pipeline graph"]:::core FC["Fails as: duplicate pipelines, stage barriers, invalid YAML"]:::risk C --- FC end subgraph L2["2 Scheduling"] direction LR S["A runner with matching tags, protection and free capacity takes each job"]:::core FS["Fails as: jobs stuck pending, long queue time"]:::risk S --- FS end subgraph L3["3 Delivery"] direction LR D["Deploy jobs or Flux change environments"]:::core FD["Fails as: unapproved, outdated or unsigned deployments"]:::risk D --- FD end subgraph L4["4 Platform, beneath loops 1 to 3"] direction LR P["GitLab version, capacity, backups"]:::neutral FP["Fails as: API errors, slow Gitaly, missed security fixes"]:::risk P --- FP end L1 --> L2 --> L3 L3 ~~~ L4 ``` Text equivalent: Every change passes three loops in order. Configuration merges YAML and components into one pipeline graph and fails as duplicate pipelines, stage barriers or invalid YAML (the CI Lint API catches the last[^lint-api]). Scheduling hands each job to a runner with matching tags, protection and capacity, and fails as pending jobs. Delivery changes environments and fails as unapproved, outdated or unsigned deployments. The platform loop underpins all three and fails as API errors, slow Gitaly or missed security fixes. ::: - **Stage, `needs`**: by default a job waits for every job in earlier stages; `needs` starts it as soon as named jobs finish[^yaml][^efficiency]. - **`rules`, `workflow`**: decide whether a job, or the whole pipeline, is created; `workflow` prevents duplicates[^workflow]. - **Merge-request pipeline**: runs on every change to a merge request's source branch and tests that branch alone[^mr-pipelines]; merged results pipelines and merge trains (Premium, Ultimate) test the result of merging it into the target[^merged-results][^merge-trains]. - **CI/CD component**: a versioned, reusable pipeline unit with typed inputs[^components]. - **Runner, executor**: the process that requests and runs jobs; the executor (Docker, Docker Autoscaler, Instance and Kubernetes are primary) decides isolation[^executors]. - **Runner scope, protected runner**: instance, group or project scope decides who can use, and abuse, a runner[^runner-scope]; a protected runner takes only protected-ref jobs[^runner-config]. - **Protected environment, deployment tier**: restricts who deploys and can require approvals (Premium, Ultimate); the tier (production, staging, testing, development, other) drives protection and DORA data[^environments][^protected-env]. - **Job token, ID token**: `CI_JOB_TOKEN` is valid only while the job runs, with the triggering user's access[^job-token]; an OIDC ID token (`id_tokens`) is exchanged for short-lived cloud credentials[^id-tokens]. - **Agent for Kubernetes**: `agentk` connects out from the cluster to GitLab's agent server[^agent]. - **Required stop, background migration**: a version you cannot skip; database work that must finish before the next hop[^bbm]. ### How GitLabOps relates to other Ops GitLabOps is a platform-specific layer beneath vendor-neutral disciplines: they supply the principles and policies, it applies them on GitLab. The boundaries are **editorial distinctions** chosen by this Atlas; no standard defines them. | Neighbour | Choose that page when… | |---|---| | [GitOps](../gitops/) | You design reconciliation itself, independent of the Git host | | [DevOps](../devops/) | You need culture, value streams and the DORA measurement model | | [DevSecOps](../devsecops/) | You set security policy and decide which findings block a merge | | [ReleaseOps](../releaseops/) | You decide *how* to release (canaries, flags, rollback policy) | | [PlatformOps](../platformops/) | GitLab is one component of an internal platform | | [TestOps](../testops/) | The problem is test signal and flaky tests, not pipeline mechanics | | [KubernetesOps](../kubernetesops/) | The problem is inside the cluster | ## Principles and operating model ### Charter and principles No official manifesto exists; this is **this Atlas's proposal**. ::: decision Proposed charter (Atlas proposal, not an industry standard) **Mission.** Trustworthy feedback on every change within an agreed time; attributable, authorised, reversible production changes; a platform inside its support window. **Goals to calibrate:** p95 merge-request feedback under 15 minutes; p95 queue time under 2 minutes; default-branch success above 90 %; at most two minor releases behind; no long-lived cloud keys in variables. **Non-goals:** release strategy, security policy, application incidents. ::: 1. **Pipelines are a product; developers are the users.** Publish service levels for feedback time and reliability. 2. **Reuse through pinned components**, "a specific commit SHA (preferred) or release version tag"[^components]. 3. **Isolation follows trust.** Privileged work only on isolated, ephemeral runners limited to protected branches[^runner-security]. 4. **Identity beats secrets.** Allowlisted job tokens; ID tokens bound to project, ref and environment[^job-token][^id-tokens]. 5. **Production is pulled or gated, and the pull source is guarded.** Write access to the promoted artifact, or to its signing identity, *is* deploy access[^agent][^protected-env]. 6. **Stay inside the security-fix window**[^maint]. 7. **Measure before tuning.** The [assurance indicators](#assurance-of-the-operating-model) turn these into testable invariants. The trade-offs remain: speed versus isolation, reuse versus team autonomy, GitLab-native features versus portability. ### Fundamentals and building blocks Five building blocks: - **Control plane**: Puma (web and API), Sidekiq (background jobs), Gitaly (Git repository access), PostgreSQL, Redis, object storage, the registry and the agent server[^arch]. GitLab runs it on GitLab.com and Dedicated[^offerings]; you run it on Self-Managed. - **Configuration**: `.gitlab-ci.yml`, components, variables and project settings such as the 60-minute default job timeout[^settings]. - **Execution**: runner managers, executors and caches[^runners]. - **Delivery**: environments, deployments, the agent and Flux. - **Assurance**: test and scan reports, metrics and audit events. Govern their retention too: job logs, artifacts (`expire_in`[^artifacts]) and, on Self-Managed, backups with `gitlab-secrets.json` and `gitlab.rb`[^backup]. ### Roles, ownership and RACI | Activity | Developer | CI/CD engineer | Runner operator | GitLab admin | Security | |---|---|---|---|---|---| | A project's pipeline | R/A | C | I | — | C | | Shared components | C | R/A | C | — | C | | Runners: create, size, retire | I | C | R/A | C | C | | Protected environments, approvers | C | C | I | — | C (A: environment owner) | | Scanners and policies | I | R | — | C | A | | Upgrades and restore tests | I | C | C | R/A | I | | "Jobs pending for everyone" | I | C | R/A | R (control plane) | — | R = responsible, A = accountable, C = consulted, I = informed. A small team that runs fleets and components as an internal product while teams own their pipelines usually works ([PlatformOps](../platformops/)); a central team writing everyone's YAML becomes a queue. ### Lifecycle: Day 0, Day 1, Day 2 | Phase | Work | Exit criteria | |---|---|---| | Day 0 | Offering and tier (merge trains, protected environments and DORA dashboards are paid[^merge-trains][^protected-env][^dora-gitlab]); sizing by requests per second[^refarch]; runner topology; identity policy | A decision record per choice | | Day 1 | Install; create and register runners[^new-workflow]; publish components; migrate pilots | Pilots green two weeks; baselines taken | | Day 2 | The cadence below: patches, minor upgrades, health, cost and clean-up reviews | In the security-fix window; SLOs met | ### Operating model and governance | Cadence | Activity | |---|---| | Weekly, 30 min | Pipeline health review (percentiles, top failing jobs, retries, cost) → a few owned tickets | | Twice monthly / monthly | Self-managed patches[^maint] / release notes and deprecations triage | | Every 1–2 months | Minor upgrade through any required stop, rehearsed on a clone[^upgrade-paths] | | Quarterly | Timed restore test; fleet capacity and isolation review | | Before May | Major-version readiness (20.0 is scheduled for 2027-05-20[^maint]) | Governance escalates from conventions to configuration teams cannot edit (see the [reuse comparison](#deep-comparisons)). A change to a shared component, runner image or fleet is a production change: review, canary group, rollback. ## Architecture and workflow ### Reference architecture A logical reference for containers deployed to Kubernetes with your own runner fleets. The fleet split is this Atlas's recommendation, derived from GitLab's runner security guidance[^runner-security]. ::: figure Runner fleets and the control plane, separated by trust ```mermaid %%{init: {"flowchart": {"wrappingWidth": 115, "nodeSpacing": 18, "rankSpacing": 36, "padding": 8}}}%% flowchart LR classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 FA["Fleet A, general and untrusted: autoscaler manager, one-job VMs"]:::core FB["Fleet B, protected refs only: protected runner, release pods"]:::risk CACHE[("Runner cache bucket")]:::neutral subgraph CP["GitLab control plane"] API["API and Sidekiq; artifacts in object storage"]:::core REG["Container registry"]:::core end CL["Cloud APIs via OIDC"]:::risk FA -->|"cache"| CACHE FA -->|"jobs, artifacts"| API FB -->|"jobs"| API FB -->|"push"| REG FB -->|"ID token"| CL ``` Text equivalent: Fleet A (general, untrusted jobs; one-job VMs from an autoscaler) writes to the runner cache bucket and exchanges jobs and artifacts with the GitLab API. Fleet B (protected refs only; release pods) requests jobs, pushes to the registry and exchanges OIDC ID tokens with cloud APIs. Red-tinted elements hold production-capable credentials. ::: | Component | Responsibility and control | |---|---| | Fleet A | Lint, test, scan, fork merge requests. Unprivileged; one job per VM (`max_use_count = 1`)[^docker-autoscaler]; no production variables | | Fleet B | Protected-ref builds; publish, sign, promote. Protected runner[^user-runners-api]; restricted egress; its own cache (protected branches already use separate caches by default[^caching]) | | Registry | Protected tag rule for `production`[^protected-tags]; clean-up policies never delete protected tags[^registry-cleanup] | | Agent and Flux | Pull-based reconciliation with signature checks[^flux-ocirepo]; the agent manages cluster access and shows state, and reconciles immediately only for Git sources in the agent configuration project and public projects[^agent-flux] | | Cloud trust | OIDC trust policies on `aud` and `sub`; on Self-Managed and Dedicated, AWS can check nothing else (promotion path, step 3)[^aws-oidc] | Runners and the agent connect out to GitLab, except receptive agents (Ultimate, Self-Managed, from 17.4)[^agent]. ### The promotion path: who can really deploy? Flux deploys whatever the `production` tag points to, so moving that tag is a production credential. Every job gets a registry user "with read-write access to the container registry"[^registry-auth], and "by default, users with the Developer, Maintainer, or Owner role can push and delete image tags"[^protected-tags]. Unguarded, a feature-branch job or a laptop could move `production` past every approval. ::: figure Promotion path from a protected build to production, with the controls that block a bypass ```mermaid %%{init: {"flowchart": {"wrappingWidth": 140, "nodeSpacing": 16, "rankSpacing": 30, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 PUB["Fleet B publish job: manifest artifact :sha, digest recorded"]:::core APR{{"Protected environment: an approver allows the deploy"}}:::decision KMS["KMS signing key; its role accepts only the main, protected-production subject, requested by the deploy job alone"]:::risk DEP["Fleet B deploy job: sign the digest, move the tag"]:::risk ATK["A Developer's job or laptop"]:::neutral TAG["Registry tag production; push limited by a protected tag rule"]:::core FX{{"Flux: is the production digest validly signed?"}}:::decision OK["Reconcile; agentk shows state in GitLab"]:::core NO["Refuse: SourceVerified False, alert"]:::risk PUB --> APR --> DEP KMS -.->|"sign"| DEP DEP --> TAG ATK -.->|"blocked by the rule"| TAG TAG --> FX FX -->|"yes"| OK FX -->|"no"| NO ``` Text equivalent: The publish job records the artifact digest; an approver allows the deployment; the deploy job signs the digest with a KMS key whose role accepts only the subject that this job alone requests, then moves the protected tag. A Developer's job or laptop is blocked by the tag rule. Flux reconciles only a validly signed artifact; otherwise it refuses and alerts. ::: 1. **Protected container tag** for `^production$`, push restricted to Maintainer (Free, generally available since 17.11; on Self-Managed it needs the registry metadata database)[^protected-tags][^registry-db]. A CI push counts as the job's user: the job token has "the same access level as the user that triggered the pipeline", or ran the manual job[^job-token], and in the 19.5 source the registry derives the denied tag patterns from that user's project role[^registry-src]. So the deploy job must be run by a Maintainer or Owner, and a Maintainer's feature-branch job passes the rule too: only step 2 stops an unsigned tag move. Rule changes apply once registry tokens expire (5 minutes by default, 15 minutes on GitLab.com)[^protected-tags]; test after that. 2. **Signature verification.** Flux's `OCIRepository` refuses anything not signed with the cosign key (`spec.verify`)[^flux-ocirepo]; the key sits in a cloud KMS[^cosign-kms] behind a role that accepts only this project's ID tokens[^aws-oidc]. On GitLab.com, keyless Sigstore signing is an alternative[^signing]. 3. **Bind the signing role to the production environment, not just `main`.** The default subject names only project, ref type and ref, so every `main` job, Fleet A's included, gets a token the role accepts. AWS cannot check `runner_id` or environment claims (on Self-Managed and Dedicated only `sub` and `aud`)[^aws-oidc], so add `environment_protected` and `deployment_tier` to the project's `ci_id_token_sub_claim_components` (18.7 and later) and require the deploy job's full subject[^projects-api][^id-tokens]. Confirm the string in a test project (the docs do not print it); the project's other trust policies see it too. 4. **Give that subject to the deploy job only.** Environment fields prove that a job *named* the protected production environment, not that anyone approved it. Approvals block "jobs deploying to the environment"[^protected-env], but a job with `environment:action` `prepare`, `verify` or `access` "does not trigger deployments"[^yaml], and `environment_action` cannot be a subject component[^projects-api]. GitLab's own change says "approval is not required when the job only accesses the environment", while an open issue says such jobs still need deployer access[^nondeploy-jobs]. So a post-deploy smoke test on `main` that names production and requests an ID token could sign without approval. Allow `id_tokens` only on the deploy job, enforce it through code-owner review of CI files[^codeowners], and test that a `verify` job on `main` cannot assume the role. 5. **Optionally, a separate CD project** whose members are only release managers[^deploy-safety]. **Residual risks.** Anyone the tag rule allows can re-point `production` to an *older* signed release; and a reviewed CI change can still add a job that signs. ### End-to-end workflow: merge request to production ::: figure Merge request to production, with failure and rollback paths ```mermaid --- config: sequence: actorMargin: 44 width: 96 diagramMarginX: 6 wrap: true boxMargin: 6 messageMargin: 30 --- sequenceDiagram participant Dev as Developer participant GL as GitLab participant RN as Runners participant Ap as Approver participant FX as Flux Dev->>GL: 1. Open MR RN->>GL: 2. Fleet A tests opt Note over Dev,GL: Pipeline fails GL-->>Dev: 3. Failed job Dev->>GL: 4. Push fix end Dev->>GL: 5. Merge RN->>GL: 6. Fleet B builds Ap->>GL: 7. Approve RN->>GL: 8. Sign, move tag FX->>GL: 9. Verify, pull opt Note over Ap,FX: Health checks fail Ap->>GL: 10. Rollback job FX->>GL: 11. Pull previous end ``` Text equivalent: (1–2) A merge request gets one pipeline, run on Fleet A. (3–4) If it fails, the developer pushes a fix, which cancels the superseded jobs. (5–6) Once green, it merges and protected Fleet B builds and records the digest. (7) An approver other than the triggerer approves. (8) The deploy job, serialised by a resource group, signs and moves the tag. (9) Flux verifies and reconciles. (10–11) If health checks fail, a rollback job re-tags the previous signed release. ::: By default "the user who triggers a deployment pipeline can't also approve the deployment job"; leave "Allow pipeline triggerer to approve deployment" off where separation of duties matters[^protected-env]. Each decision on this path has an owner and leaves a record: | Decision | Owner | Where it is recorded | |---|---|---| | Which pipeline runs | CI/CD engineer | `workflow:rules` in `.gitlab-ci.yml` | | Which runner may take a job | Runner operator | Tags and protection in the runner settings | | May this deployment proceed? | Environment owner | Approval and deployment record | | Roll back or roll forward | Service owner | Rollback job; policy on [ReleaseOps](../releaseops/) | GitLab's DORA dashboards count only successful deployments to **production**-tier environments, so tier them correctly[^dora-gitlab]. ## Practices, frameworks and tools ### Core practices and playbooks Each playbook ends with an acceptance test and a rollback. Also review clean-up policies and artifact expiry monthly[^registry-cleanup]. #### Playbook 1: a duplicate-free pipeline skeleton 1. `workflow:rules` that run merge-request pipelines, suppress branch pipelines while a merge request is open, and keep default-branch and tag pipelines[^workflow]. Put them in `.gitlab-ci.yml` itself: rules inside an included component do not enable merge-request pipelines[^mr-pipelines]. 2. `workflow:auto_cancel:on_new_commit: interruptible`; tests `interruptible: true`, deploys `false`[^yaml]. 3. `needs` instead of stage barriers; lint fast and first[^efficiency]. 4. `rules:changes` with `compare_to` on expensive jobs, because `rules:changes` is true for new branches and for pipelines without a push event[^yaml]. 5. With GitLab's security templates, `AST_ENABLE_MR_PIPELINES: "true"` (18.0 and later): "By default, the application security jobs are configured to run for branch pipelines only", so step 1 would silently drop them from merge requests[^ast-mr]. 6. A Lint API `dry_run` before merging[^lint-api]. **Acceptance:** one pipeline per push, scans included; a second push cancels the first's interruptible jobs. **Rollback:** revert; each pipeline uses its own commit's configuration. #### Playbook 2: runner fleets by trust 1. Inventory runners (scope, executor, tags, protection, privileged flag, last contact); retire unknown ones. 2. Define fleets by trust: general (unprivileged, ephemeral), protected (release credentials), special hardware[^fleet]. 3. Create runners as code: `POST /user/runners` with `tag_list`, `run_untagged=false` and, for the protected fleet, `access_level=ref_protected`; the token is shown once, so store it in a secret manager[^user-runners-api]. 4. Autoscale with the Docker Autoscaler or the Kubernetes executor, with at least two managers[^docker-autoscaler][^helm-runner][^fleet]. 5. Scrape `/metrics` from a private network only, because it has no authorisation[^runner-monitoring]; keep runner and GitLab versions aligned[^runner-version]. **Acceptance:** every fleet has an owner, capacity target and queue-time SLO; no privileged runner takes unprotected jobs. **Rollback:** pause the new runner, unpause the old; tags switch traffic per project. #### Playbook 3: secrets and identity hygiene 1. Classify variables: configuration, low-risk tokens (masked and hidden), production secrets (move out). 2. Cloud access through `id_tokens`, a service-specific `aud`, and trust policies on `sub` plus `project_id` (a separate key where supported, otherwise the first `sub` component, 19.1 and later), never user or email claims[^id-tokens][^projects-api]. 3. Keep the job token allowlist: "By default, the allowlist of any project only includes itself"[^job-token]. 4. Production variables **protected** and scoped to the production environment[^variables]. 5. With Ultimate, secret push protection, knowing it scans diffs only, skips files over 1 MiB and accepts the push on timeout[^push-protection]; pipeline secret detection is in every tier[^secret-detection]. **Acceptance:** no long-lived cloud keys in variables. **Rollback:** keep old credentials disabled, not deleted, for a release cycle. #### Playbook 4: deployment guardrails 1. `environment:name` and an explicit `deployment_tier` on every deploy job[^environments]. 2. Protected production environments with approvals (Premium, Ultimate); group-level protection by tier covers new projects[^protected-env]. 3. `resource_group: production`[^resource-groups] and "prevent outdated deployment jobs"[^deploy-safety]. 4. Flux with the agent for Kubernetes production; CI access through `ci_access` only to non-production environments, impersonating the job[^agent-ci]. 5. Guard the [promotion path](#the-promotion-path-who-can-really-deploy) (tag rule, signatures, deploy-job-only signing subject). 6. Decide rollback: "Allow job retries for rollback deployments" is on by default; GitLab advises disabling it in sensitive projects, where "you must run a new pipeline with a previous commit"[^deploy-safety]. **Acceptance:** a Developer's tag move is blocked, Flux refuses an unsigned artifact, and a `verify` job on `main` cannot assume the signing role. **Rollback:** only through review; deleting an approval rule can lose its history[^protected-env]. #### Playbook 5: self-managed upgrades 1. A minor upgrade every one to two months through any required stop[^upgrade-paths], plus the patch releases. Security fixes cover the current release and the two before it, so an instance installed on release N the day it ships is covered until N+3 ships, about three months: a quarterly rhythm leaves no slack[^maint]. 2. Before each hop: rehearse on a clone, back up including secrets and configuration, confirm background migrations finished ([runbook](#runbook-a-stuck-upgrade-hop))[^plan-upgrade][^backup]. 3. After: `gitlab:check`, `gitlab:doctor:secrets`, sign-in, clone and push, runner pickup, registry push and pull; then upgrade runners and agents[^runner-version]. **Acceptance:** checks pass; current or previous release. **Rollback:** a restore needs a backup of *exactly* the same version and edition[^backup], so back up and snapshot immediately before each hop. ### Methods, standards and frameworks GitLabOps has no standard of its own. It borrows OpenGitOps as a test of whether a setup is GitOps[^opengitops]; DORA's metrics (five in the guide updated 2026-01-05; GitLab's dashboards implement the earlier four)[^dora][^dora-gitlab]; the OWASP Top 10 CI/CD Security Risks as a threat checklist[^owasp-cicd]; and NIST SP 800-204D where an organisation or contract adopts it[^nist-204d]. Vendor rules such as required stops are mandatory; reference architectures are advice[^refarch]. ### Tooling landscape | Job | Options (October 2026) | Limit | |---|---|---| | Validate | CI Lint API `dry_run`[^lint-api] | Simulates creation, not runtime | | Reuse | Components, CI/CD Catalog[^components]; templates[^sast] | Same-name jobs merge silently | | Build images | Rootless BuildKit, Buildah, Docker-in-Docker[^buildkit][^docker-build]; kaniko removed[^kaniko] | Rootless needs user namespaces | | Observe | Runner metrics[^runner-monitoring]; CI/CD analytics[^ci-analytics] | Some panels need ClickHouse | | Executor[^executors] | Status | Good for | Watch out for | |---|---|---|---| | Docker | Primary | Builds on static hosts | The mounted Docker socket shares the host daemon[^runner-docker-install] | | Docker Autoscaler | Primary | Elastic one-job VMs | Cloud quotas; VM start-up | | Instance | Primary; clean only conditionally | Full OS or devices | Making hosts disposable is your job | | Kubernetes | Primary | A pod per job | Requests, limits, privileged pods | | Shell | Maintenance mode | Trusted single-purpose hosts | "Use it only for running trusted builds"[^runner-security] | | Docker Machine | Deprecated; removal in 20.0[^docker-machine] | Nothing new | Migrate | ### Automation and AI Automate the machinery, not the trust decisions: runner creation and rotation through the API, Lint dry runs on shared-component changes, reports of stale runners and unpinned components, upgrade gates ([runbook](#runbook-a-stuck-upgrade-hop)). GitLab Duo's Root Cause Analysis reads the last 100,000 characters of a failed job's log and proposes a fix (Duo add-on; not for trigger jobs or downstream pipelines)[^duo-rca]; approve the log sharing first, lint any suggested YAML, and let no assistant change protected pipelines, runners or environment protection without human approval[^owasp-cicd]. ## GitLabOps in practice ### Reference implementation A **representative design**, not a recipe: GitLab 19.x; a containerised Node.js service; Kubernetes production with Flux; AWS KMS for the signing key; Premium or Ultimate for approvals (on Free the signing role can be bound only to `main`). All hosts, IDs and tokens are placeholders. _Example status for this section: YAML (Atlas validator), TOML (smol-toml 1.3.1) and bash (`bash -n`) syntax-checked; nothing executed against a live GitLab instance, registry, KMS key or cluster._ #### 1. A reusable build component `templates/build-image/template.yml` in a project such as `platform/ci-components`, following GitLab's rootless BuildKit job[^buildkit]: no privileged container. Rootless BuildKit still needs "a runner that permits the system calls BuildKit uses to create user namespaces and mount points"; where it fails, GitLab suggests Buildah, so test it on each fleet. ```yaml spec: inputs: stage: default: build description: Stage in which the build job runs image_name: description: Registry path of the image, without a tag context: default: "." description: Build context directory runner_tag: default: protected-build description: Tag of the runner fleet allowed to build release images --- build-image: stage: $[[ inputs.stage ]] image: name: moby/buildkit:rootless # pin by digest in production entrypoint: [""] tags: ["$[[ inputs.runner_tag ]]"] interruptible: false variables: BUILDKITD_FLAGS: --oci-worker-no-process-sandbox before_script: - mkdir -p ~/.docker - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > ~/.docker/config.json script: - | buildctl-daemonless.sh build \ --frontend dockerfile.v0 \ --local context=$[[ inputs.context ]] \ --local dockerfile=$[[ inputs.context ]] \ --output type=image,name=$[[ inputs.image_name ]]:$CI_COMMIT_SHA,push=true rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - if: $CI_COMMIT_TAG ``` #### 2. The project pipeline ```yaml # .gitlab-ci.yml (illustrative, GitLab 19.x) include: - component: $CI_SERVER_FQDN/platform/ci-components/build-image@1.4.2 inputs: stage: build image_name: $CI_REGISTRY_IMAGE - template: Jobs/SAST.gitlab-ci.yml - template: Jobs/Secret-Detection.gitlab-ci.yml workflow: auto_cancel: on_new_commit: interruptible rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - if: $CI_COMMIT_TAG stages: [lint, test, build, publish, deploy] default: interruptible: true tags: [linux-general] retry: max: 2 when: [runner_system_failure, runner_interrupted] # infrastructure faults only, never script failures variables: NODE_IMAGE: node:22-bookworm-slim # pin by digest in production MANIFEST_REPO: $CI_REGISTRY_IMAGE/manifests TOOLS_IMAGE: $CI_REGISTRY/platform/images/release-tools:3 # internally vetted: flux, cosign, jq, AWS CLI AST_ENABLE_MR_PIPELINES: "true" # run the security templates in MR pipelines too (18.0 and later) .npm-cache: &npm-cache key: files: [package-lock.json] paths: [.npm/] lint: stage: lint image: $NODE_IMAGE cache: <<: *npm-cache policy: pull script: - npm ci --cache .npm --prefer-offline - npm run lint unit-test: stage: test needs: [] # start immediately instead of waiting for the lint stage image: $NODE_IMAGE cache: <<: *npm-cache script: - npm ci --cache .npm --prefer-offline - npm test artifacts: when: always reports: junit: junit.xml expire_in: 7 days publish-manifests: stage: publish image: $TOOLS_IMAGE tags: [protected-build] interruptible: false needs: [build-image] rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH script: - > flux push artifact "oci://$MANIFEST_REPO:$CI_COMMIT_SHORT_SHA" --path=./deploy --source="$CI_PROJECT_URL" --revision="$CI_COMMIT_BRANCH@sha1:$CI_COMMIT_SHA" --creds="$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" --output json > push.json - echo "MANIFEST_DIGEST=$(jq -r .digest push.json)" > manifest.env artifacts: reports: dotenv: manifest.env # hands the digest to deploy-production deploy-production: stage: deploy image: $TOOLS_IMAGE tags: [protected-build] interruptible: false needs: [publish-manifests] resource_group: production environment: name: production deployment_tier: production url: https://www.example.com id_tokens: SIGNER_ID_TOKEN: aud: https://gitlab.example.com variables: # IAM trust policy: StringEquals on sub = this job's subject incl. environment fields (promotion path, steps 3-4). # Only this job declares id_tokens. SIGNER_ROLE_ARN: arn:aws:iam::ACCOUNT_ID:role/manifest-signer COSIGN_KEY: awskms:///alias/manifest-signer COSIGN_YES: "true" AWS_REGION: eu-west-1 rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH when: manual # run by a Maintainer: the production tag rule checks the job's user script: - > creds=$(aws sts assume-role-with-web-identity --role-arn "$SIGNER_ROLE_ARN" --role-session-name "deploy-$CI_PIPELINE_ID" --web-identity-token "$SIGNER_ID_TOKEN" --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text) - export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" $creds) - printf '%s' "$CI_REGISTRY_PASSWORD" | cosign login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin - cosign sign --key "$COSIGN_KEY" "$MANIFEST_REPO@$MANIFEST_DIGEST" # transparency-log upload: decide, see below - flux tag artifact "oci://$MANIFEST_REPO:$CI_COMMIT_SHORT_SHA" --tag production --creds="$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" - cosign verify --key "$COSIGN_KEY" "$MANIFEST_REPO:production" > /dev/null # fail if the tag is not on a signed digest ``` **Why each piece.** The `workflow`, `interruptible`, `needs` and scanner settings are [Playbook 1](#core-practices-and-playbooks). Retries cover infrastructure failures only, so flaky tests stay visible; the JUnit report feeds the merge-request summary without changing job status[^unit-tests]. The publish job hands the artifact digest on through a dotenv report[^flux-cli][^yaml]. The deploy job, the only one with `id_tokens`, exchanges its token for short-lived AWS credentials[^aws-oidc], signs the recorded digest and moves the tag under `resource_group`[^resource-groups]; if someone re-points `:` in between, `production` lands on an unsigned digest, the final `cosign verify` fails and Flux refuses it. **Transparency log: decide, do not inherit.** `cosign sign` records each signature in a transparency log unless told not to: cosign 2 in the public `rekor.sigstore.dev`, so a private registry gets a public entry per release; cosign 3 in the services of a TUF-provided signing configuration unless you pass your own, keeping `--tlog-upload=false` only as a deprecated flag ("prefer using a --signing-config file with no transparency log services")[^cosign-cli]. Air-gapped instances need their own log or none; without an entry, verify with `--insecure-ignore-tlog=true`[^cosign-cli]. Flux's public-key verification sets `IgnoreTlog = true` (source-controller v1.9.6, with a comment calling it an oversight; re-check after upgrades)[^flux-sc-cosign]. _Status: defaults read from the cosign 2.6.5 and 3.1.3 and Flux sources; not executed._ **Settings outside YAML:** protected `main` with code-owner approval for CI files[^codeowners]; protected `production` with approvers and the triggerer-approval setting off[^protected-env]; the `^production$` tag rule, so a Maintainer runs the deploy job[^protected-tags]; the ID-token subject components[^projects-api]; "prevent outdated deployment jobs" and a deliberate rollback-retry choice[^deploy-safety]. #### 3. Runner fleets Fleet A: ephemeral VMs with the Docker Autoscaler and the AWS fleeting plugin[^docker-autoscaler][^runner-advanced]. _Example status: TOML syntax-checked; not executed._ ```toml concurrent = 40 check_interval = 3 log_level = "info" listen_address = "10.0.12.5:9252" # private interface only: /metrics has no authentication [[runners]] name = "fleet-a-general" url = "https://gitlab.example.com" token = "glrt-REPLACE_WITH_TOKEN_FROM_SECRET_STORE" executor = "docker-autoscaler" limit = 40 output_limit = 8192 [runners.docker] image = "alpine:3.20" privileged = false pull_policy = "always" [runners.cache] Type = "s3" Shared = true [runners.cache.s3] ServerAddress = "s3.amazonaws.com" BucketName = "example-runner-cache" BucketLocation = "eu-west-1" AuthenticationType = "iam" [runners.autoscaler] plugin = "aws" capacity_per_instance = 1 max_use_count = 1 max_instances = 40 [runners.autoscaler.plugin_config] name = "fleet-a-general-asg" [runners.autoscaler.connector_config] username = "ec2-user" use_external_addr = false [[runners.autoscaler.policy]] idle_count = 0 idle_time = "0s" periods = ["* * * * *"] [[runners.autoscaler.policy]] idle_count = 6 idle_time = "20m0s" periods = ["* 7-19 * * mon-fri"] timezone = "Europe/Berlin" ``` `idle_count` counts idle *job capacity*, not machines, and the last matching policy wins[^runner-advanced]; `pull_policy = "always"` follows GitLab's advice for shared runners[^runner-security]. Fleet B runs protected builds in a dedicated namespace via the Helm chart[^helm-runner]; protection is not a `config.toml` setting but the runner's `access_level=ref_protected`[^user-runners-api]. ```yaml # values-fleet-b.yaml for the gitlab/gitlab-runner Helm chart (illustrative) gitlabUrl: https://gitlab.example.com runnerToken: "" rbac: create: true concurrent: 10 runners: config: | [[runners]] [runners.kubernetes] namespace = "ci-protected" image = "alpine:3.20" privileged = false cpu_request = "500m" memory_request = "1Gi" cpu_limit = "2" memory_limit = "4Gi" [runners.kubernetes.node_selector] "workload" = "ci-protected" ``` #### 4. Flux for production, the agent for staging Flux follows `production` and verifies its signature first[^flux-ocirepo]; the key owner publishes the public key to the cluster[^cosign-kms]. _Example status: `bash -n` syntax-checked; not executed._ ```bash cosign public-key --key awskms:///alias/manifest-signer > cosign.pub kubectl -n flux-system create secret generic manifest-signer-pub --from-file=cosign.pub=cosign.pub ``` ```yaml apiVersion: source.toolkit.fluxcd.io/v1 kind: OCIRepository metadata: name: web namespace: flux-system spec: interval: 5m url: oci://registry.gitlab.example.com/shop/web/manifests ref: tag: production secretRef: name: gitlab-registry-pull # read-only registry credential verify: provider: cosign secretRef: name: manifest-signer-pub # unsigned or wrongly signed artifacts are refused --- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: web namespace: flux-system spec: interval: 10m sourceRef: kind: OCIRepository name: web path: ./ prune: true wait: true timeout: 5m ``` Staging may take pushes through the agent, limited to those environments and impersonating the CI job; without `access_as`[^agent-ci], a job inherits the agent's full permissions: ```yaml # .gitlab/agents/staging-agent/config.yaml in the agent configuration project ci_access: projects: - id: shop/web environments: - staging - review/* access_as: ci_job: {} --- # RoleBinding in the staging namespace: CI jobs appear in the group gitlab:ci_job apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: gitlab-ci-jobs-deploy namespace: web-staging subjects: - kind: Group name: gitlab:ci_job apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: edit # broad but namespace-bound; narrow it per project in real use apiGroup: rbac.authorization.k8s.io ``` With pull delivery the deployment record marks the promotion, not the finished rollout; accurate DORA data may need the Deployments API, for example called from a Flux notification[^dora-gitlab]. ### Hands-on labs Three labs, none executed here (commands are syntax-checked against the cited docs); use a test project, never production. A Flux bootstrap lab belongs with the reconciliation principles on [GitOps](../gitops/). #### Lab 1 (beginner): what your rules do, with no job run **Objective.** See which jobs a pipeline would contain per ref, with the CI Lint API's dry run[^lint-api]. **Prerequisites.** A project you own (GitLab.com Free is enough), an API token, `curl`, `jq`. **Cost and risk.** None. Commit this to `main`, then create a branch `feature-x`: ```yaml workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never - if: $CI_COMMIT_BRANCH stages: [lint, test, deploy] lint: stage: lint script: echo "lint" unit-test: stage: test needs: [] script: echo "test" deploy-docs: stage: deploy script: echo "deploy" rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` Run the dry run for `main`, then for `feature-x`. _Example status: `bash -n` syntax-checked; not executed._ ```bash #!/usr/bin/env bash set -euo pipefail : "${GITLAB_URL:?}" "${GITLAB_TOKEN:?}" "${PROJECT_ID:?}" REF="${1:-main}" REQ=$(mktemp) jq -n --rawfile content .gitlab-ci.yml --arg ref "${REF}" \ '{content: $content, dry_run: true, include_jobs: true, ref: $ref}' > "${REQ}" curl --silent --fail --request POST \ --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \ --header "Content-Type: application/json" \ --data @"${REQ}" \ --url "${GITLAB_URL}/api/v4/projects/${PROJECT_ID}/ci/lint" \ | jq '{valid, errors, warnings, jobs: [.jobs[]?.name]}' rm -f "${REQ}" ``` **Expected output.** `"valid": true`; `lint`, `unit-test` and `deploy-docs` on `main`, no `deploy-docs` on `feature-x`. **Validation.** Delete `needs: []`: the job list does not change, because `needs` changes *when* jobs start, not *whether* they exist. **Troubleshooting.** `401`/`403`: token scope or role. **Teardown.** Revoke the token; delete the lab project. #### Lab 2 (intermediate): your own runner, queueing **Objective.** Register a project runner, expose its metrics, and watch queueing when capacity is below demand. **Prerequisites.** A disposable Linux VM with Docker; Maintainer on a project. **Cost and risk.** The VM; mounting the Docker socket "delegates full control over the Docker daemon" to the runner container[^runner-docker-install], so never use your workstation. Create a project runner (Settings > CI/CD > Runners) tagged `lab-docker`, "Run untagged" off, and copy its `glrt-` token[^new-workflow]. _Example status: `bash -n` syntax-checked; not executed._ ```bash #!/usr/bin/env bash set -euo pipefail : "${RUNNER_TOKEN:?set RUNNER_TOKEN to the glrt- token}" # Runner container; metrics published on the host loopback only. docker run -d --name gitlab-runner --restart always \ -p 127.0.0.1:9252:9252 \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock \ gitlab/gitlab-runner:latest # Register with the authentication token (no tags or locking flags: those were set when the runner was created). docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner:latest register \ --non-interactive --url "https://gitlab.com/" --token "${RUNNER_TOKEN}" \ --executor docker --docker-image alpine:3.20 --description lab-docker # Expose metrics inside the container, then restart to be sure the change is loaded. sudo sed -i '1i listen_address = "0.0.0.0:9252"' /srv/gitlab-runner/config/config.toml grep -E '^(concurrent|listen_address)' /srv/gitlab-runner/config/config.toml docker restart gitlab-runner curl -s http://127.0.0.1:9252/metrics | grep -E '^gitlab_runner_(concurrent|limit|version_info)' ``` Set `concurrent = 1` in `config.toml`, restart the container and push: ```yaml queue-demo: tags: [lab-docker] image: alpine:3.20 parallel: 3 script: - sleep 60 ``` **Expected output.** The three jobs run one after another; with `concurrent = 3` they overlap. **Validation.** Later jobs show longer queued times; `gitlab_runner_concurrent` reports 1, then 3. **Troubleshooting.** Stuck `pending`: a tag typo or "Run untagged"; `410 Gone - runner registration disallowed`: a legacy registration token[^new-workflow]. **Teardown.** Delete the runner in the UI (invalidating its token), `docker rm -f gitlab-runner`, remove `/srv/gitlab-runner` (it holds the token), destroy the VM. #### Lab 3 (advanced): a before-and-after experiment **Objective.** Measure merge-request feedback time, change one thing, and decide from data. **Prerequisites.** Python 3.8+, a read-only token, a week of merge-request pipelines. **Cost and risk.** Read-only API calls, one per pipeline, so keep the window small on busy projects. Pipeline `duration` "excludes … any pending (queue) time"[^pipelines], so the script reports wall-clock feedback (`finished_at − created_at`), busy time and the wait in between; the API page's own example pipeline hides about 3 seconds of such wait[^pipelines-api]. _Example status: syntax-checked; the filtering, pagination, timestamp and percentile logic was exercised offline with a stubbed `api()` and a hand-written sample that includes the API page's example pipeline; not run against a live API._ ```python #!/usr/bin/env python3 """Merge-request pipeline health from the GitLab Pipelines API (read-only). Usage: GITLAB_URL=https://gitlab.example.com GITLAB_TOKEN= \ python3 pipeline_health.py [days] [source] feedback_s = finished_at - created_at (wall clock, includes every wait) busy_s = duration (job running time; GitLab excludes queue time) """ import json import math import os import sys import urllib.parse import urllib.request from datetime import datetime, timedelta, timezone def api(path, params): base = os.environ["GITLAB_URL"].rstrip("/") url = f"{base}/api/v4{path}?{urllib.parse.urlencode(params)}" req = urllib.request.Request(url, headers={"PRIVATE-TOKEN": os.environ["GITLAB_TOKEN"]}) with urllib.request.urlopen(req, timeout=30) as resp: return json.load(resp) def ts(value): """Parse GitLab timestamps such as 2022-09-21T01:05:07.200Z.""" return datetime.fromisoformat(value.replace("Z", "+00:00")) def percentile(values, pct): """Nearest-rank percentile: smallest value with at least pct % of the values at or below it.""" if not values: return None ordered = sorted(values) rank = max(1, math.ceil(pct / 100 * len(ordered))) return ordered[rank - 1] def summarise(details): finished = [d for d in details if d.get("status") in ("success", "failed")] result = {"considered": len(finished)} if finished: ok = sum(1 for d in finished if d["status"] == "success") result["success_rate"] = round(ok / len(finished), 3) series = {"feedback_s": [], "busy_s": [], "queued_s": [], "in_pipeline_wait_s": []} for d in finished: if not (d.get("created_at") and d.get("finished_at")): continue wall = (ts(d["finished_at"]) - ts(d["created_at"])).total_seconds() series["feedback_s"].append(wall) busy, queued = d.get("duration"), d.get("queued_duration") if busy is not None: series["busy_s"].append(busy) if queued is not None: series["queued_s"].append(queued) if busy is not None and queued is not None: series["in_pipeline_wait_s"].append(max(0.0, wall - busy - queued)) for name, values in series.items(): if values: result[name] = {"p50": percentile(values, 50), "p95": percentile(values, 95), "max": max(values)} return result def main(): project = urllib.parse.quote(sys.argv[1], safe="") days = int(sys.argv[2]) if len(sys.argv) > 2 else 7 source = sys.argv[3] if len(sys.argv) > 3 else "merge_request_event" since = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat() listed, page = [], 1 while True: batch = api(f"/projects/{project}/pipelines", {"updated_after": since, "source": source, "per_page": 100, "page": page}) if not batch: break listed.extend(batch) page += 1 # Filter again client-side, in case the server ignores an unknown source value. wanted = [p for p in listed if p.get("source") == source and p.get("status") in ("success", "failed")] details = [api(f"/projects/{project}/pipelines/{p['id']}", {}) for p in wanted] print(json.dumps(summarise(details), indent=2)) if __name__ == "__main__": main() ``` Take a 7-day baseline, make **one** change, measure 7 days later. **Expected output.** JSON with `considered`, `success_rate` and p50/p95/max per series. **Validation.** Spot-check pipelines in the UI; CI/CD analytics medians should read lower than `feedback_s`[^ci-analytics]. A large `in_pipeline_wait_s` points at the fleet. **Troubleshooting.** HTTP 404: use the numeric ID. **Teardown.** Revoke the token. ### Use cases and scenarios Illustrative composites, not reports of named organisations. | Scenario | Approach | Pitfall | Trade-off; success metric | |---|---|---|---| | Monorepo, 30 services, 45-minute pipelines | A child pipeline per service with `rules:changes` and `compare_to`; `libs/` changes trigger every dependant; at most two child levels[^downstream] | Without `compare_to`, every new branch runs every child[^yaml] | More configuration; p95 feedback < 15 min, no more escaped defects | | Regulated payments releases | Group-level protected `production` tier, approvals, triggerer barred[^protected-env]; a separate CD project of release managers[^deploy-safety]; the signing subject on its deploy job only; rollback retries off | A deploy job run by a release manager with only the Developer role fails at the Maintainer tag rule | Longer lead time; every production change, rollbacks included, has an approval record | | Open source with fork merge requests | Fork pipelines run in the fork; parent pipelines only after reading the diff[^mr-pipelines]; an unprivileged ephemeral fleet | A parent-project run gives the fork's CI configuration the parent's variables and runners[^mr-pipelines] | Slower first feedback; no unreviewed parent pipeline | | Self-managed upgrade debt | [Playbook 5](#core-practices-and-playbooks) hop by hop, then a one-to-two-month rhythm | Gates on versions before 19.5 cannot use `wait`[^bbm] | Several windows; back inside the security window | ## Measure, secure and pay ### Metrics and KPIs Measure the pipeline (what developers feel), the fleet (why) and the platform; DORA outcomes belong to [DevOps](../devops/). | KPI | Definition and source | Pitfall | |---|---|---| | MR feedback time p50, p95 | `finished_at − created_at` per pipeline (Lab 3)[^pipelines-api] | `duration` excludes every queue wait[^pipelines] | | Job queue time p95 | `gitlab_runner_job_queue_duration_seconds` histogram[^fleet] | Averages hide the morning peak | | Success rate | success ÷ (success + failed), cancellations excluded | Retries manufacture "green" | | Fleet saturation | Running jobs ÷ `concurrent` or `limit`[^fleet] | 100 % means queueing, not efficiency | | Cost per successful pipeline | (runners + storage + compute minutes) ÷ successful pipelines[^compute] | Rewards skipping tests if used alone | | Version lag | Minor releases behind current[^maint] | "Patched" yet outside the window | **Worked example (illustrative).** A week of 1,200 merge-request pipelines: 1,000 succeeded, 150 failed, 50 auto-cancelled. Success rate = 1,000 ÷ 1,150 = **87.0 %** (83.3 % if cancellations counted as failures). On hosted `medium` runners (cost factor 2) at 30 job-minutes each: 1,200 × 30 × 2 = **72,000** compute minutes[^compute]; at price *P*, cost per successful pipeline = **72 *P***. A p95 queue of 4.5 minutes plus a p95 duration of 22 minutes is **not** a p95 feedback time of 26.5: percentiles do not add, and both miss waits between stages. Pair every speed target with a quality guard. Fleet signals in PromQL (metric names from GitLab's runner docs; the third query is the fleet guide's own): ```promql # p95 job queue time across a fleet, 15-minute window (histogram documented in the fleet guide) histogram_quantile(0.95, sum by (le) (rate(gitlab_runner_job_queue_duration_seconds_bucket[15m]))) # Saturation: running jobs as a share of configured concurrency sum(gitlab_runner_jobs) / sum(gitlab_runner_concurrent) # Autoscaler start-up: p95 machine creation time histogram_quantile(0.95, sum by (le) (rate(gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket[5m]))) ``` ### Quality, security, privacy and compliance Most GitLabOps security work is about **who can make which job run where, with which identity**. The pipeline-specific OWASP CI/CD risks[^owasp-cicd], mapped to GitLab controls (identity reviews, dependency pinning, third-party services and logging are general practice on [DevSecOps](../devsecops/)): | Risk | GitLab controls | |---|---| | SEC-4 poisoned pipeline execution | Protected variables only on protected refs[^variables]; read fork diffs first[^mr-pipelines]; code owners for CI files[^codeowners]; separate protected caches[^caching]; CI file in another project[^settings] | | SEC-1 insufficient flow control | Protected branches, environments[^protected-env] and container tags[^protected-tags]; the signing subject on the deploy job only | | SEC-5 pipeline-based access control | Protected runners[^runner-config]; `ci_access` with `access_as: ci_job`[^agent-ci]; job token allowlist[^job-token] | | SEC-6 credential hygiene | OIDC with tight `aud` and `sub`[^id-tokens]; push protection (Ultimate)[^push-protection] | | SEC-7 insecure system configuration | Privileged only on isolated, ephemeral, protected runners; shell only for trusted builds[^runner-security] | | SEC-9 artifact integrity | Sign in the deploy job, verify in Flux[^flux-ocirepo] | Runner authentication tokens are credentials: "users with access to the runner authentication token can clone a runner and submit false jobs"[^runner-config]. Keep personal data out of fixtures and set artifact expiry deliberately[^artifacts]. GitLab produces evidence (approvals, deployments, audit events, scans), not compliance ([ComplianceOps](../complianceops/)). ### Cost, capacity and sustainability **Cost drivers.** Hosted runners: duration ÷ 60 × cost factor, with Linux factors 1, 2, 3, 6 and 12 from small to 2xlarge, and sizes above `medium` on Premium or Ultimate only (October 2026)[^compute][^hosted-linux]. Own runners: VM hours including idle capacity, storage, egress, operators. Self-Managed adds the control plane and upgrade labour. **Capacity with Little's law** (L = λ × W[^little]) on job slots, where W includes VM start-up and teardown when `max_use_count = 1`. Illustrative: 300 jobs an hour at peak (λ = 5 a minute) × (6 minutes run + about 1 minute overhead) = **35 busy slots**, five more than run time suggests; add burst headroom (1.3 × ≈ 45) and calibrate against p95 queue time. High queue time with low saturation points at tags or protection instead. Idle capacity (`idle_count`) buys start-up latency with VM hours; time-box it to working hours[^runner-advanced]. **Efficiency levers:** remove redundant work and stage barriers (Playbook 1); right-size machines, since over-provisioned runners waste money and under-provisioned ones waste developer time[^efficiency]; lock-file cache keys in a shared bucket[^caching]; artifact expiry and registry clean-up. With the registry metadata database, online garbage collection removes unreferenced data after 24 hours; with legacy metadata, schedule offline collection. Never run third-party garbage collection with the metadata database: it "will delete data associated with tagged images"[^registry-db]. We found no reliable carbon figures for CI; instance-hours are a proxy ([GreenOps](../greenops/)). ### Assurance of the operating model Does the operating model still hold? Check monthly with read-only API jobs; healthy is **zero** unless stated: - unowned runners, or runners silent for 30 days; privileged or shell runners taking unprotected jobs; - production credentials (variables, runners) reachable from unprotected refs; - jobs other than the deploy job declaring `id_tokens` against the production environment; - unprotected production environments, production jobs without a resource group, unpinned components; - days since the last restore test (healthy: ≤ 90) and minor releases behind (healthy: ≤ 2[^maint]). **Platform health and recovery (Self-Managed).** An outage stops every pipeline. Linux-package installations bundle Prometheus with host, PostgreSQL and Redis exporters "on by default"[^gitlab-prometheus]; alert on API latency, Sidekiq queues, Gitaly and PostgreSQL saturation and storage. Set recovery objectives, then choose `gitlab-backup` with separately stored secrets, per-component procedures for large instances[^backup], or Geo (Premium, Ultimate), "a warm-standby secondary site in a different region" that still needs backups because it replicates corruption[^geo]. Only a timed restore test proves it. ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Fix | |---|---|---| | `pending`, "no matching runners" | Tag mismatch or protected runners only[^runner-config] | Fix tags; `default:tags` in components | | `pending` with matching runners | `concurrent`, `limit`, `max_instances` or quota reached[^runner-monitoring] | Raise capacity or cut demand | | Two pipelines per push | No `workflow:rules` guard[^workflow] | Playbook 1 | | No scans in MR pipelines | Templates run "for branch pipelines only" | `AST_ENABLE_MR_PIPELINES`[^ast-mr] | | `410 Gone - runner registration disallowed` | Legacy registration token | Register with the runner's own token[^new-workflow] | | "Outdated deployment job" | A newer deployment already ran[^deploy-safety] | New pipeline for the earlier commit, unless rollback retries are allowed | | Flux stuck, `SourceVerified` False | `production` on an unsigned digest | Compare `flux get sources oci` with `MANIFEST_DIGEST`; re-run the deploy job; find who moved the tag[^flux-ocirepo] | | Merge train keeps restarting | A flaky job ejects its MR[^merge-trains] | Quarantine the test ([TestOps](../testops/)) | ### Runbook: jobs pending for everyone **Detection:** p95 queue time above 10 minutes for 15 minutes, or a flood of reports. ::: figure Diagnosing jobs stuck in pending ```mermaid %%{init: {"flowchart": {"wrappingWidth": 130, "nodeSpacing": 16, "rankSpacing": 28, "padding": 6}}}%% flowchart TB classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 A["Alert: p95 queue time high"]:::risk B{{"Everyone affected?"}}:::decision D{{"Managers online?"}}:::decision C["Check the project's tags, protection, runner enablement"]:::core E["Restart or fail over managers; check network, TLS"]:::core F{{"Fleet full?"}}:::decision G["Raise concurrent, limit, max instances or quota"]:::core J{{"API errors?"}}:::decision K["Check the control plane; tune runner request concurrency"]:::core A --> B B -->|"yes"| D B -->|"no"| C D -->|"yes"| F D -->|"no"| E F -->|"no"| J F -->|"yes"| G J -->|"yes"| K J -->|"no"| C ``` Text equivalent: If one project is affected, check its tags, protection and runner enablement. If everyone is, check that managers are online (else restart or fail over; check network and TLS), then whether the fleet is full (raise limits or quota), then API errors pointing at the control plane; otherwise return to per-project checks. ::: 1. Sample pending jobs for tags, projects and refs. 2. Managers alive? Last-contact times; `journalctl -u gitlab-runner --since "30 min ago"` for connection or TLS errors. 3. Full? `sum(gitlab_runner_jobs) / sum(gitlab_runner_concurrent)` near 1[^runner-monitoring][^fleet]. 4. Autoscaler? Machine states and creation time against `max_instances` and the cloud quota. 5. Control plane? Rising error and request-concurrency-exceeded counters point at the API or the default `request_concurrency = 1`[^runner-advanced]. ```text Metric names for steps 3-5 (GitLab Runner /metrics) gitlab_runner_jobs, gitlab_runner_concurrent, gitlab_runner_limit gitlab_runner_autoscaling_machine_states gitlab_runner_autoscaling_machine_creation_duration_seconds gitlab_runner_errors_total gitlab_runner_request_concurrency_exceeded_total ``` 6. Mitigate (pause schedules, move a team by tag, raise `max_instances`), then record the timeline and the alert that would have caught it. ### Runbook: a stuck upgrade hop ::: figure Self-managed upgrade hops through required stops, with gates between hops ```mermaid %%{init: {"flowchart": {"wrappingWidth": 210, "nodeSpacing": 30, "rankSpacing": 22, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 V0["Current: 18.3"]:::risk V1["18.5 stop"]:::core V2["18.8 stop"]:::core V3["18.11 stop"]:::core V4["19.2 stop"]:::core V5["Target: 19.5"]:::core G["Gate before each hop: backup with secrets, migrations finished, checks pass"]:::decision V0 --> V1 --> V2 --> V3 --> V4 --> V5 V1 -.- G V2 -.- G V3 -.- G V4 -.- G ``` Text equivalent: An illustrative path from 18.3 through the required stops 18.5, 18.8, 18.11 and 19.2 to 19.5; before every hop, a backup with secrets, finished background migrations and passing health checks. Confirm the path with GitLab's Upgrade Path tool. ::: **Symptom.** The upgrade stops because a background migration is still "active"[^bbm]. **Diagnosis.** Each gate runs on the version you are leaving, so on this path none can use `wait`, which arrives with 19.5. _Example status: `bash -n` syntax-checked; not executed. Commands and SQL quoted from GitLab's background-migrations page[^bbm]._ ```bash # 1. What is still queued, running or failed? sudo gitlab-rake gitlab:background_migrations:list # 18.9 and later sudo gitlab-rake gitlab:background_migrations:status # 18.8 and earlier # Any version: zero rows means all batched background migrations are complete. sudo gitlab-psql -c "SELECT job_class_name, table_name, column_name, job_arguments FROM batched_background_migrations WHERE status NOT IN(3, 6);" # 2. 19.5 and later only: block until nothing is paused, active or finalizing; non-zero exit if one failed. # The task has no internal timeout, so bound it. sudo timeout 6h gitlab-rake gitlab:background_migrations:wait # 3. Health checks before and after each hop. sudo gitlab-rake gitlab:check sudo gitlab-rake gitlab:doctor:secrets ``` Resolve a **failed** migration first (GitLab documents a manual `finalize`)[^bbm]. **Rollback:** restore the pre-hop backup to the same version and edition, with its secrets[^backup]. **Prevention:** gate hops in automation; rehearse on a clone[^plan-upgrade]. ### Anti-patterns and myths | Anti-pattern or myth | Why it fails | Instead | |---|---|---| | One shared pool with privileged Docker-in-Docker | Any Developer can reach the host[^runner-security] | Rootless builds; privileged only on protected, ephemeral runners | | `retry: 2` everywhere | Hides flaky tests, burns minutes | Retry infrastructure failures only | | `kubectl apply` to production with a stored kubeconfig | Long-lived credential, no drift correction[^agent] | Flux; CI promotes a signed artifact | | Upgrading straight to the newest version after a pause | Skipped stops and unfinished migrations fail[^upgrade-paths] | Hop by hop, every one to two months | | "The Kubernetes executor is always cheaper" | Pod requests and noisy neighbours add cost and risk | Dedicated CI node pools | ## Deciding and adopting ### When to use and when not to use **Invest** when GitLab serves more than a couple of teams, you run your own runners or instance, production needs approval evidence, CI cost is material, or developers routinely wait. **Keep it light** for one small team on GitLab.com with hosted runners: defaults, Playbook 1 and a monthly usage check. **Do not build a GitLab-specific practice** when GitLab only mirrors code built elsewhere, or several SCM platforms must stay portable: invest in [DevOps](../devops/), [GitOps](../gitops/) and [PlatformOps](../platformops/) and keep the GitLab layer thin. Someone must own fleet security, release notes, each May's major release and, on Self-Managed, restore tests; if nobody will, buy the more managed option. ### Decision framework ::: figure Choosing the hosting offering and runner strategy ```mermaid %%{init: {"flowchart": {"wrappingWidth": 130, "nodeSpacing": 16, "rankSpacing": 28, "padding": 6}}}%% flowchart TB classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 Q1{{"Must you run GitLab yourself, for example air-gapped?"}}:::decision Q2{{"Need single-tenant isolation without operating it?"}}:::decision COM["GitLab.com"]:::core DE["GitLab Dedicated"]:::core Q3{{"Jobs need a private network, special hardware or steady heavy load?"}}:::decision HR["Hosted runners: budget compute minutes"]:::neutral SM["Self-Managed: size by RPS; standalone to about 2,000 users, HA from 3,000"]:::core Q4{{"Well-run Kubernetes platform already available?"}}:::decision K8["Kubernetes executor on dedicated CI node pools"]:::core DA["Docker Autoscaler on ephemeral VMs"]:::core Q5{{"Any jobs privileged, deploying, or using production identity?"}}:::decision PF["Add a separate protected fleet"]:::core OK["Single unprivileged fleet is enough"]:::neutral Q1 -->|"no"| Q2 Q1 -->|"yes"| SM Q2 -->|"no"| COM Q2 -->|"yes"| DE COM --> Q3 DE --> Q3 Q3 -->|"no"| HR Q3 -->|"yes"| Q4 SM -->|"no hosted runners"| Q4 Q4 -->|"yes"| K8 Q4 -->|"no"| DA HR --> Q5 K8 --> Q5 DA --> Q5 Q5 -->|"yes"| PF Q5 -->|"no"| OK ``` Text equivalent: Must you run GitLab yourself? Then Self-Managed with your own runners (hosted runners exist only on GitLab.com and Dedicated). Otherwise Dedicated for single-tenant isolation, else GitLab.com with hosted runners, unless jobs need a private network, special hardware or heavy steady load. Own runners: Kubernetes executor on a well-run platform, else Docker Autoscaler. Privileged or production-identity jobs get a separate protected fleet. ::: The tree encodes GitLab's general sizing advice[^refarch] and runner security guidance[^runner-security]; the rest is this Atlas's judgement. **Worked comparison:** 80 developers on GitLab.com, private-network deployments, a platform team of two; scores are assumptions, except hosted-runner isolation: each job "runs in a newly provisioned VM, which is dedicated to the specific job"[^runners]. | Criterion (weight) | Hosted runners | Docker Autoscaler | Kubernetes executor | |---|---|---|---| | Isolation per job (0.30) | 5 | 5 | 3 | | Operating effort, higher = less (0.25) | 5 | 2 | 3 | | Cost at this volume (0.20) | 2 | 4 | 4 | | Private network access (0.15) | 1 | 5 | 5 | | Start-up latency (0.10) | 4 | 3 | 4 | | **Weighted total** | **3.70** | **3.85** | **3.60** | With no private-network deployments (network weight 0, effort 0.40) the totals become 4.30, 3.40 and 3.30: the answer flips, and the close base case could flip on your cost estimate alone. ### Maturity model **This Atlas's model**, not an official one. Find the level whose behaviour matches yours; move up when its exit criteria have held for a quarter. ::: figure GitLabOps maturity progression (Atlas model) ```mermaid %%{init: {"flowchart": {"wrappingWidth": 300, "rankSpacing": 18, "padding": 8}}}%% flowchart TB classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 L1["1 Ad hoc: copied pipelines, runners on any host, secrets in variables"]:::risk E1["Exit: runners owned, no duplicate pipelines"]:::decision L2["2 Repeatable: shared skeleton, protected default branches, runner inventory"]:::neutral E2["Exit: no long-lived cloud keys, no privileged runners on unprotected jobs"]:::decision L3["3 Managed: pinned components, fleets by trust, OIDC, protected environments and promotion tag"]:::core E3["Exit: dashboards live, SLOs defined, inside the security window"]:::decision L4["4 Measured: weekly review of feedback, queue, success and cost; upgrade calendar"]:::core E4["Exit: SLOs met for a quarter, restore tested within 90 days"]:::decision L5["5 Optimising: before/after data for every change, policies as code"]:::core L1 --> E1 --> L2 --> E2 --> L3 --> E3 --> L4 --> E4 --> L5 ``` Text equivalent: Five levels, each followed by an exit gate. 1 Ad hoc (copied pipelines, runners on any host, secrets in variables): exit when runners are owned and no pipeline runs twice. 2 Repeatable (shared skeleton, protected default branches, runner inventory): exit when no long-lived cloud keys remain and no privileged runner takes unprotected jobs. 3 Managed (pinned components, fleets by trust, OIDC, protected environments and promotion tag): exit when dashboards are live, SLOs defined and the instance inside the security window. 4 Measured (weekly review of feedback, queue, success and cost; upgrade calendar): exit when SLOs held for a quarter and a restore was tested within 90 days. 5 Optimising: before/after data for every change, policies as code. ::: ### Adoption roadmap: 30/60/90 days For level 1–2 with a small platform team. | Window | Work | Milestone | |---|---|---| | Days 1–30: see and stabilise | Runner inventory and owners; isolate privileged or shell runners serving untrusted jobs; Playbook 1 in the ten busiest projects; Lab 3 baseline | Baselines and inventory done | | Days 31–60: make the safe path easy | 3–5 pinned components; a protected fleet; OIDC for one cloud account (security signs off the trust policy); protected environments and promotion tag | Half the projects on the skeleton; first cloud keys gone | | Days 61–90: run it as a service | Queue-time dashboards and alerts; weekly review; upgrade calendar; registry clean-up | Queue-time SLO met for two weeks | ## Toolkit ### Checklists **New project** - [ ] Baseline component pinned; `workflow:rules` prevent duplicates; tests `interruptible`; scans in MR pipelines - [ ] Default branch protected; MR pipelines must pass; no secrets in `.gitlab-ci.yml`; Lint dry run passes on two refs **New runner fleet** - [ ] Owner and trust level; tags and access level set at creation; token in a secret manager - [ ] One clean environment per job; `privileged = false` unless protected and ephemeral; private metrics endpoint - [ ] Capacity sized with Little's law; time-boxed idle policy; runner version matches GitLab's **Production deployment readiness** - [ ] `deployment_tier: production`; protected environment with approvers; triggerer cannot approve; `resource_group` set - [ ] "Prevent outdated deployment jobs" on; rollback retries decided; rollback rehearsed - [ ] Tag rule tested (a Developer cannot move the tag; a Maintainer runs the deploy job); Flux refuses an unsigned artifact - [ ] Signing role bound to the production-environment subject; only the deploy job declares `id_tokens`; a `verify` job on `main` refused **Upgrade hop (Self-Managed)** - [ ] Path from the Upgrade Path tool; rehearsed and timed on a clone; Playbook 5's before and after steps done ### Templates **Runner request form** ```text Runner request - Requesting team / owner: - Purpose (build, test, deploy, special hardware): - Trust level: [ ] general [ ] protected refs only [ ] public/fork contributions - Scope: [ ] project [ ] group [ ] instance (admin approval) - Executor and image constraints: - Needs privileged mode? If yes, why, and why rootless builds are not enough: - Network access needed (registries, private endpoints, cloud APIs): - Expected peak jobs per hour and average job duration: - Credentials: OIDC audience and trust conditions (no long-lived keys): - Approval: runner operator ____ security (if protected/privileged) ____ ``` **Decision record for a pipeline or fleet change** ```text ADR-NNN: Status: proposed | accepted | superseded by ADR-MMM Date: YYYY-MM-DD Context: what problem, which data (baseline metrics), which GitLab version and tier Options considered: at least two, including "do nothing" Decision: the chosen option and the conditions under which it holds Consequences: cost, security, operating effort, lock-in, migration later Verification: which metric, which threshold, reviewed on which date Rollback: how to return to the previous state ``` **Monthly scorecard:** the charter's goals plus a retry rate ≤ 3 % and a falling cost per successful pipeline, each with an owner and last month's value (example targets, not benchmarks). ### Deep comparisons | Reuse mechanism, weakest to strongest | Versioning and inputs | Use when | |---|---|---| | Copy-paste YAML | No | Never for shared logic | | `include:` of files | By ref | Migration | | CI/CD components | Versions, typed inputs[^components] | Shared logic | | CI file in another project | By ref | Pipelines teams must not edit[^settings] | | Pipeline execution policies (Ultimate) | Policy project | Enforced jobs[^policies] | ### FAQ **Is "GitLabOps" a certification or job title?** No. Look for the skills or GitLab's own credentials, such as "GitLab Certified CI/CD Associate"[^gitlab-cert]; GitLab University's catalogue could not be read automatically, so check current names there. **Do I need Premium or Ultimate?** Not to start: merge-request pipelines, components, runners, environments and SAST are Free[^mr-pipelines][^components][^sast]. Paid tiers add merge trains, protected environments, policies, push protection and DORA dashboards[^merge-trains][^protected-env][^policies][^push-protection][^dora-gitlab]. **Hosted runners or our own?** Hosted runners need no fleet operations and isolate each job in a fresh VM, but bring compute-minute quotas and less network control[^runners][^compute]; see the [decision framework](#decision-framework). **How often must a self-managed instance be upgraded?** At least every two months, to the newest release ([Playbook 5](#core-practices-and-playbooks))[^maint]. Required stops limit how far one hop goes, not how often. **Can we compare GitLab's DORA dashboard with our own DORA data?** Only after aligning definitions: GitLab implements the earlier four metrics with its own data rules[^dora-gitlab]; the DORA guide now lists five[^dora]. ## Going further ### Advanced topics and emerging direction Worth mastering: merge trains sized to runner capacity[^merge-trains], dynamic child pipelines[^downstream] and policy projects[^policies]. Watch: the 20.0 removals in May 2027 ([origins](#origins-and-evolution)); the change that lets non-deploying jobs start without a manual action, behind a feature flag that is off by default in 19.5[^nondeploy-jobs]; and GitLab's push towards AI agents ("GitLab MCP server generally available" in 19.5)[^releases], which is vendor direction, not yet evidence. ### Learning path New to delivery? Start with [DevOps](../devops/); prerequisites are in [At a glance](#at-a-glance). | Stage | Focus | Practice | |---|---|---| | Day 1 | Stages, `rules`, `needs`, cache vs artifacts, MR pipelines[^yaml] | Lab 1 | | Week 1 | Runners, executors, protection, variables, environments | Lab 2; a skeleton component | | Month 1 | Autoscaling, OIDC, protected environments, Flux, metrics | Lab 3; the reference implementation in a sandbox | | Then | Fleet engineering, administration, CI security, GitOps on GitLab | Shadow an upgrade hop; threat-model pipelines with the OWASP list[^owasp-cicd] | Then go deeper on [GitOps](../gitops/), [DevSecOps](../devsecops/), [ReleaseOps](../releaseops/), [TestOps](../testops/) and [KubernetesOps](../kubernetesops/). ## Wrap-up ### Executive recap and cheat sheet **Decide:** the offering, runners by trust, signed pull for Kubernetes production, the tier whose controls you need. **Measure:** p95 feedback and queue time, success rate, cost per successful pipeline, version lag. **Watch:** privileged runners on shared fleets; a tag any Developer can move; a signing identity any `main` job can request; restores without the secrets file. ```text Lint dry run POST /api/v4/projects/:id/ci/lint {content, dry_run: true, include_jobs: true, ref} Create runner POST /api/v4/user/runners runner_type=group_type group_id=… tag_list=… access_level=ref_protected Register runner gitlab-runner register --non-interactive --url … --token glrt-… --executor docker Runner metrics listen_address = ":9252" → GET /metrics Migrations gate sudo gitlab-rake gitlab:background_migrations:wait (19.5+; earlier: :list or :status) MR scanners variables: AST_ENABLE_MR_PIPELINES: "true" Promotion guard protected container tag ^production$ + Flux OCIRepository spec.verify (cosign) Health checks sudo gitlab-rake gitlab:check ; sudo gitlab-rake gitlab:doctor:secrets Backup sudo gitlab-backup create (+ /etc/gitlab/gitlab-secrets.json and gitlab.rb separately) No duplicates workflow:rules … if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS → when: never One deploy at a time resource_group: production ``` ### A judgement call: one shared fleet or a separate build fleet? **Situation (illustrative).** A 60-developer organisation runs one autoscaled instance-runner fleet; a fifth of jobs build images with Docker-in-Docker, so the whole fleet is `privileged = true`. The platform team proposes a protected build fleet with rootless BuildKit (B); a senior engineer prefers the one privileged fleet (A). **Assumptions:** any Developer on any served project can start jobs; contractors contribute; VMs run one job each; a second fleet costs about a day a month. **Reasoning.** (1) *Worst failure:* privileged containers effectively disable container security and make breakout possible[^runner-security], so a compromised merge request in *any* project reaches the VM and its network (cache bucket, cloud metadata). (2) *Does ephemerality rescue A?* Partly; GitLab pairs ephemeral VMs with **dedicated runners restricted to protected branches**, and A has only the first half. (3) *Cost of B:* a second manager and a tag change; rootless builds may fail on some runners, where GitLab suggests Buildah[^buildkit]. (4) *What would change the answer:* if everyone who can trigger a job is equally trusted with production and the fleet reaches nothing sensitive, A's simplicity wins. **Decision: B.** A saves one visible fleet and accepts an invisible cost: every project's weakest contributor sets the security level of every build. **Uncertainty:** rootless failure rates here are unknown, so a two-week pilot on the five busiest image-building projects measures them first. ### Learning objectives and mastery check Objectives: explain the label and its GitOps boundary; design duplicate-free pipelines; size and segment fleets; close deployment bypasses; diagnose pending jobs and blocked upgrades; judge changes with percentiles. ::: answer 1. "We do GitLabOps, so we already do GitOps." What is wrong? GitOps needs pull-based, continuous reconciliation[^opengitops]; a CI job pushing manifests does neither, and GitLab advises against it for production[^agent]. GitOps *on* GitLab means Flux with the agent. ::: ::: answer 2. Jobs tagged `gpu` have pended for an hour; the GPU runner is online and idle. Three causes, quickest check first? The runner lacks one of the job's tags (it needs *all* of them); the runner is protected and the ref is not[^runner-config]; the runner is not enabled for the project or group[^runner-scope]. ::: ::: answer 3. Peak load is 240 jobs an hour, 5 minutes each, plus about 1 minute of VM overhead. How many slots? λ = 4 a minute, W = 6 minutes, so L = 24 busy slots[^little] (run time alone gives 20). Add headroom (≈ 31), then check quotas, `limit` and `concurrent`[^runner-advanced] against p95 queue time. ::: ::: answer 4. You are on 18.6 and must reach 19.5; the first hop fails on a background migration. What now? Inspect with `:status` (`:list` starts at 18.9) or the SQL check, poll until nothing is unfinished, and resolve failures first; `wait` only arrives in 19.5[^bbm]. Path: 18.8 → 18.11 → 19.2 → 19.5, each hop gated by a backup[^upgrade-paths]. ::: ::: answer 5. A contractor's fork merge request adds a job that prints environment variables. What protects you? The pipeline runs in the fork with the fork's variables, so the parent's secrets are absent; the maintainer must not run it in the parent project without reading the diff[^mr-pipelines]. ::: ::: answer 6. A lead says production is safe "because the deploy job needs an approval". Is it? No. Developers can move image tags by default[^protected-tags], so protect the tag and verify signatures in Flux[^flux-ocirepo]. A `verify` job on `main` naming production also gets the signing subject without approval[^id-tokens][^nondeploy-jobs], so only the deploy job may declare `id_tokens`. Test all three. ::: ### Conclusion Adopting GitLab gives you a delivery platform, not a delivery operation. The label has no standing; the craft does. **The hard parts are not in the YAML:** they are the queue before the runners, the trust boundary around privileged containers, the identity each job carries, the tag production pulls, and the upgrade path. All facts are dated October 2026 (GitLab 19.x); re-check them. ## Sources and further reading GitLab docs are the live 19.x pages, which carry no publication date. Ledger: `research/gitlabops-ledger.md`. [^gitlab-home]: [GitLab homepage](https://about.gitlab.com/) — GitLab Inc. Checked 2026-10-16. [^gitlab-topics-ops]: [How to Use GitLab for Ops](https://about.gitlab.com/topics/ops/) — GitLab Inc. Checked 2026-10-16. [^gitlab-cert]: [GitLab Certified CI/CD Associate](https://www.credly.com/org/gitlab/badge/gitlab-certified-ci-cd-associate) — GitLab, on Credly. Checked 2026-10-16. [^offerings]: [Choose a subscription](https://docs.gitlab.com/subscriptions/choosing_subscription/) — GitLab Docs. Checked 2026-10-16. [^maint]: [Release and maintenance policy](https://docs.gitlab.com/policy/maintenance/) — GitLab Docs. Checked 2026-10-16. [^releases]: [GitLab 19 release notes](https://docs.gitlab.com/releases/19/) — GitLab Docs. Checked 2026-10-16. [^upgrade-paths]: [Upgrade paths](https://docs.gitlab.com/update/upgrade_paths/) — GitLab Docs. Checked 2026-10-16. [^plan-upgrade]: [Before you upgrade](https://docs.gitlab.com/update/plan_your_upgrade/) — GitLab Docs. Checked 2026-10-16. [^bbm]: [Batched background migrations](https://docs.gitlab.com/update/background_migrations/) — GitLab Docs. Checked 2026-10-16. [^backup]: [Back up GitLab](https://docs.gitlab.com/administration/backup_restore/backup_gitlab/) and [overview](https://docs.gitlab.com/administration/backup_restore/) — GitLab Docs. Checked 2026-10-16. [^geo]: [Geo](https://docs.gitlab.com/administration/geo/) — GitLab Docs. Checked 2026-10-16. [^gitlab-prometheus]: [Monitoring GitLab with Prometheus](https://docs.gitlab.com/administration/monitoring/prometheus/) — GitLab Docs. Checked 2026-10-16. [^refarch]: [Reference architectures](https://docs.gitlab.com/administration/reference_architectures/) — GitLab Docs. Checked 2026-10-16. [^arch]: [Architecture overview](https://docs.gitlab.com/development/architecture/) — GitLab Docs. Checked 2026-10-16. [^yaml]: [CI/CD YAML syntax reference](https://docs.gitlab.com/ci/yaml/) — GitLab Docs. Checked 2026-10-16. [^workflow]: [`workflow` examples](https://docs.gitlab.com/ci/yaml/workflow/) — GitLab Docs. Checked 2026-10-16. [^pipelines]: [CI/CD pipelines](https://docs.gitlab.com/ci/pipelines/) — GitLab Docs. Checked 2026-10-16. [^mr-pipelines]: [Merge request pipelines](https://docs.gitlab.com/ci/pipelines/merge_request_pipelines/) — GitLab Docs. Checked 2026-10-16. [^merged-results]: [Merged results pipelines](https://docs.gitlab.com/ci/pipelines/merged_results_pipelines/) — GitLab Docs. Checked 2026-10-16. [^merge-trains]: [Merge trains](https://docs.gitlab.com/ci/pipelines/merge_trains/) — GitLab Docs. Checked 2026-10-16. [^codeowners]: [Code Owners](https://docs.gitlab.com/user/project/codeowners/) — GitLab Docs. Checked 2026-10-16. [^downstream]: [Downstream pipelines](https://docs.gitlab.com/ci/pipelines/downstream_pipelines/) — GitLab Docs. Checked 2026-10-16. [^components]: [CI/CD components](https://docs.gitlab.com/ci/components/) — GitLab Docs. Checked 2026-10-16. [^efficiency]: [Pipeline efficiency](https://docs.gitlab.com/ci/pipelines/pipeline_efficiency/) — GitLab Docs. Checked 2026-10-16. [^caching]: [Caching in GitLab CI/CD](https://docs.gitlab.com/ci/caching/) — GitLab Docs. Checked 2026-10-16. [^artifacts]: [Job artifacts](https://docs.gitlab.com/ci/jobs/job_artifacts/) — GitLab Docs. Checked 2026-10-16. [^settings]: [Customize pipeline configuration](https://docs.gitlab.com/ci/pipelines/settings/) — GitLab Docs. Checked 2026-10-16. [^unit-tests]: [Unit test reports](https://docs.gitlab.com/ci/testing/unit_test_reports/) — GitLab Docs. Checked 2026-10-16. [^lint-api]: [CI Lint API](https://docs.gitlab.com/api/lint/) — GitLab Docs. Checked 2026-10-16. [^pipelines-api]: [Pipelines API](https://docs.gitlab.com/api/pipelines/) — GitLab Docs. Checked 2026-10-16. [^runners]: [Runners](https://docs.gitlab.com/ci/runners/) and [GitLab-hosted runners](https://docs.gitlab.com/ci/runners/hosted_runners/) — GitLab Docs. Checked 2026-10-16. [^runner-scope]: [Manage runners](https://docs.gitlab.com/ci/runners/runners_scope/) — GitLab Docs. Checked 2026-10-16. [^runner-config]: [Configuring runners](https://docs.gitlab.com/ci/runners/configure_runners/) — GitLab Docs. Checked 2026-10-16. [^new-workflow]: [New runner registration workflow](https://docs.gitlab.com/ci/runners/new_creation_workflow/) and its [16.7 version](https://archives.docs.gitlab.com/16.7/ee/ci/runners/new_creation_workflow.html) — GitLab Docs. Checked 2026-10-16. [^regtoken-issue]: [Update guidance and content regarding the previously planned removal of the Runner Registration Token in GitLab 18.0](https://gitlab.com/gitlab-org/gitlab/-/issues/525239) — GitLab issue 525239, 2025. Checked 2026-10-16. [^user-runners-api]: [Users API: create a runner](https://docs.gitlab.com/api/users/) — GitLab Docs. Checked 2026-10-16. [^runner-docker-install]: [Run GitLab Runner in a container](https://docs.gitlab.com/runner/install/docker/) — GitLab Docs. Checked 2026-10-16. [^runner-version]: [GitLab Runner](https://docs.gitlab.com/runner/) — GitLab Docs. Checked 2026-10-16. [^executors]: [Executors](https://docs.gitlab.com/runner/executors/) — GitLab Docs. Checked 2026-10-16. [^docker-autoscaler]: [Docker Autoscaler executor](https://docs.gitlab.com/runner/executors/docker_autoscaler/) — GitLab Docs. Checked 2026-10-16. [^docker-machine]: [Docker Machine autoscaling](https://docs.gitlab.com/runner/configuration/autoscale/) — GitLab Docs. Checked 2026-10-16. [^helm-runner]: [GitLab Runner Helm chart](https://docs.gitlab.com/runner/install/kubernetes/), [its configuration](https://docs.gitlab.com/runner/install/kubernetes_helm_chart_configuration/), [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes/) — GitLab Docs. Checked 2026-10-16. [^runner-advanced]: [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration/) — GitLab Docs. Checked 2026-10-16. [^runner-security]: [Security for self-managed runners](https://docs.gitlab.com/runner/security/) — GitLab Docs. Checked 2026-10-16. [^runner-monitoring]: [Monitor runner usage](https://docs.gitlab.com/runner/monitoring/) — GitLab Docs. Checked 2026-10-16. [^fleet]: [Plan and operate a runner fleet](https://docs.gitlab.com/runner/fleet_scaling/) — GitLab Docs. Checked 2026-10-16. [^ci-analytics]: [CI/CD analytics](https://docs.gitlab.com/user/analytics/ci_cd_analytics/) — GitLab Docs. Checked 2026-10-16. [^compute]: [Compute minutes](https://docs.gitlab.com/ci/pipelines/compute_minutes/) — GitLab Docs. Checked 2026-10-16. [^hosted-linux]: [Hosted runners on Linux](https://docs.gitlab.com/ci/runners/hosted_runners/linux/) — GitLab Docs. Checked 2026-10-16. [^variables]: [CI/CD variables](https://docs.gitlab.com/ci/variables/) — GitLab Docs. Checked 2026-10-16. [^job-token]: [CI/CD job token](https://docs.gitlab.com/ci/jobs/ci_job_token/) — GitLab Docs. Checked 2026-10-16. [^id-tokens]: [OIDC authentication using ID tokens](https://docs.gitlab.com/ci/secrets/id_token_authentication/) — GitLab Docs. Checked 2026-10-16. [^aws-oidc]: [Configure OpenID Connect in AWS](https://docs.gitlab.com/ci/cloud_services/aws/) — GitLab Docs. Checked 2026-10-16. [^projects-api]: [Projects API](https://docs.gitlab.com/api/projects/) — GitLab Docs (`ci_id_token_sub_claim_components`). Checked 2026-10-16. [^nondeploy-jobs]: [Merge request 161860](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/161860) (merged 2024-08-12), its [feature-flag definition at v19.5.0-ee](https://gitlab.com/gitlab-org/gitlab/-/blob/v19.5.0-ee/ee/config/feature_flags/beta/prevent_blocking_non_deployment_jobs.yml) and [issue 437142](https://gitlab.com/gitlab-org/gitlab/-/issues/437142) — GitLab. Checked 2026-10-16. [^sast]: [SAST](https://docs.gitlab.com/user/application_security/sast/) — GitLab Docs. Checked 2026-10-16. [^ast-mr]: [Security configuration](https://docs.gitlab.com/user/application_security/detect/security_configuration/) — GitLab Docs. Checked 2026-10-16. [^secret-detection]: [Pipeline secret detection](https://docs.gitlab.com/user/application_security/secret_detection/pipeline/) — GitLab Docs. Checked 2026-10-16. [^push-protection]: [Secret push protection](https://docs.gitlab.com/user/application_security/secret_detection/secret_push_protection/) — GitLab Docs. Checked 2026-10-16. [^policies]: [Security policies](https://docs.gitlab.com/user/application_security/policies/) — GitLab Docs. Checked 2026-10-16. [^docker-build]: [Use Docker to build Docker images](https://docs.gitlab.com/ci/docker/using_docker_build/) — GitLab Docs. Checked 2026-10-16. [^kaniko]: [Use kaniko (removed)](https://docs.gitlab.com/ci/docker/using_kaniko/) — GitLab Docs. Checked 2026-10-16. [^buildkit]: [Build Docker images with BuildKit](https://docs.gitlab.com/ci/docker/using_buildkit/) — GitLab Docs. Checked 2026-10-16. [^registry-cleanup]: [Reduce container registry storage](https://docs.gitlab.com/user/packages/container_registry/reduce_container_registry_storage/) — GitLab Docs. Checked 2026-10-16. [^registry-db]: [Container registry metadata database](https://docs.gitlab.com/administration/packages/container_registry_metadata_database/) — GitLab Docs. Checked 2026-10-16. [^protected-tags]: [Protected container tags](https://docs.gitlab.com/user/packages/container_registry/protected_container_tags/) — GitLab Docs. Checked 2026-10-16. [^registry-auth]: [Authenticate with the container registry](https://docs.gitlab.com/user/packages/container_registry/authenticate_with_container_registry/) — GitLab Docs. Checked 2026-10-16. [^registry-src]: [container_registry_authentication_service.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/v19.5.0-ee/app/services/auth/container_registry_authentication_service.rb) and [auth.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/v19.5.0-ee/lib/gitlab/auth.rb) at tag v19.5.0-ee — GitLab source, 2026-10-15. Checked 2026-10-16. [^environments]: [Environments](https://docs.gitlab.com/ci/environments/) — GitLab Docs. Checked 2026-10-16. [^protected-env]: [Protected environments](https://docs.gitlab.com/ci/environments/protected_environments/) and [Deployment approvals](https://docs.gitlab.com/ci/environments/deployment_approvals/) — GitLab Docs. Checked 2026-10-16. [^deploy-safety]: [Deployment safety](https://docs.gitlab.com/ci/environments/deployment_safety/) — GitLab Docs. Checked 2026-10-16. [^resource-groups]: [Resource groups](https://docs.gitlab.com/ci/resource_groups/) — GitLab Docs. Checked 2026-10-16. [^agent]: [Agent for Kubernetes](https://docs.gitlab.com/user/clusters/agent/) and [installation](https://docs.gitlab.com/user/clusters/agent/install/) — GitLab Docs. Checked 2026-10-16. [^agent-ci]: [Using GitLab CI/CD with a Kubernetes cluster](https://docs.gitlab.com/user/clusters/agent/ci_cd_workflow/) — GitLab Docs. Checked 2026-10-16. [^agent-flux]: [GitOps with Flux and the agent](https://docs.gitlab.com/user/clusters/agent/gitops/) — GitLab Docs. Checked 2026-10-16. [^agent-deprecated]: [Agent GitOps (deprecated), 16.2](https://archives.docs.gitlab.com/16.2/ee/user/clusters/agent/gitops/agent.html) — GitLab Docs archive, 2023. Checked 2026-10-16. [^flux-cli]: [flux push artifact](https://fluxcd.io/flux/cmd/flux_push_artifact/), [flux tag artifact](https://fluxcd.io/flux/cmd/flux_tag_artifact/), [OCI artifacts cheatsheet](https://fluxcd.io/flux/cheatsheets/oci-artifacts/) — Flux (CNCF). Checked 2026-10-16. [^flux-ocirepo]: [OCI Repositories](https://fluxcd.io/flux/components/source/ocirepositories/) — Flux (CNCF). Checked 2026-10-16. [^flux-sc-cosign]: [source-controller v1.9.6 `cosign.go`](https://github.com/fluxcd/source-controller/blob/v1.9.6/internal/oci/cosign/cosign.go) — Flux, shipped with Flux 2.9.6 on 2026-10-01. Checked 2026-10-16. [^cosign-kms]: [Cosign key management](https://docs.sigstore.dev/cosign/key_management/overview/), [verifying signatures](https://docs.sigstore.dev/cosign/verifying/verify/), [cosign login](https://github.com/sigstore/cosign/blob/main/doc/cosign_login.md) — Sigstore. Checked 2026-10-16. [^cosign-cli]: [cosign sign](https://github.com/sigstore/cosign/blob/main/doc/cosign_sign.md) and [cosign verify](https://github.com/sigstore/cosign/blob/main/doc/cosign_verify.md); flag sources of 3.1.3 ([sign](https://github.com/sigstore/cosign/blob/v3.1.3/cmd/cosign/cli/options/sign.go), [rekor](https://github.com/sigstore/cosign/blob/v3.1.3/cmd/cosign/cli/options/rekor.go)) and 2.6.5 ([sign](https://github.com/sigstore/cosign/blob/v2.6.5/cmd/cosign/cli/options/sign.go), [rekor](https://github.com/sigstore/cosign/blob/v2.6.5/cmd/cosign/cli/options/rekor.go)) — Sigstore, released 2026-08-06. Checked 2026-10-16. [^signing]: [Keyless signing with Sigstore](https://docs.gitlab.com/ci/yaml/signing_examples/) — GitLab Docs. Checked 2026-10-16. [^opengitops]: [OpenGitOps principles v1.0.0](https://opengitops.dev/) — CNCF GitOps Working Group. Checked 2026-10-16. [^dora-gitlab]: [DORA metrics in GitLab](https://docs.gitlab.com/user/analytics/dora_metrics/) — GitLab Docs. Checked 2026-10-16. [^dora]: [DORA metrics guide](https://dora.dev/guides/dora-metrics/) — DORA, updated 2026-01-05. Checked 2026-10-16. [^duo-rca]: [Root Cause Analysis](https://docs.gitlab.com/user/gitlab_duo_chat/examples/) — GitLab Docs. Checked 2026-10-16. [^owasp-cicd]: [OWASP Top 10 CI/CD Security Risks](https://owasp.org/www-project-top-10-ci-cd-security-risks/) — OWASP Foundation. Checked 2026-10-16. [^nist-204d]: [NIST SP 800-204D](https://csrc.nist.gov/pubs/sp/800/204/d/final) — NIST, February 2024. Checked 2026-10-16. [^little]: [Little's law](https://en.wikipedia.org/wiki/Little%27s_law) — Wikipedia (J. D. C. Little, 1961). Checked 2026-10-16. **Changelog.** 2026-10-16: draft (GitLab 19.x). r1: signed, protected promotion; MR scanners; migration gates. r2: environment-bound signing role; transparency log. r3: deploy-job-only signing identity; Blueprint v2; 17,253 → 8,997 words. r4 (`reviews/gitlabops-score-r3.md`): restored the rootless-BuildKit caveat; CI tag pushes judged by the job's user (19.5 source); four-loops figure; standalone maturity figure; use-case pitfalls; 8,997 → 8,993 words. r4 follow-up (orchestrator): registration-token quote attributed to the workflow doc; GitLab.com's 15-minute rule propagation.