GitLabOps

GitLabOps is the work of running software delivery well on GitLab: designing .gitlab-ci.yml pipelines, operating the runners that execute jobs, governing environments and deployments, wiring in GitLab’s scanners and registries, and, on GitLab Self-Managed, keeping the instance healthy, backed up and upgraded.

Every push or merge request becomes a pipeline of jobs; a runner with matching tags runs each job in an executor (container, VM or Kubernetes pod), and deploy jobs record deployments to environments. GitLabOps keeps this loop fast, cheap, safe and dull.

Where to start. Beginners: the glossary, Lab 1. Engineers: the playbooks, reference implementation. Leaders: the decision framework, judgement call. Last verified 2026-10-16 against GitLab 19.x, when 19.5 was current and 19.4 and 19.3 received security fixes only^[3].

Starting is easy; trouble arrives with scale and time:

• Waiting, not building. Jobs stay pending because no runner has the tags, the only matching runners are protected (they take only protected-branch and tag jobs), or the host’s concurrent cap is reached^[4][5].
• Duplicate work. Branch and merge-request pipelines without a guard run twice per push^[6]; superseded pipelines run on unless jobs are interruptible and auto-cancel is set^[7].
• Leaky trust boundaries. A shell executor or privileged container on a shared runner lets any Developer reach the host^[8]; fork merge requests can carry code that tries to steal secrets^[9].
• Ungoverned deployments. By default Developers can modify environments, and an older deploy job can finish after a newer one^[10].
• Platform drift. A lagging self-managed instance leaves the security-fix window; catching up means required upgrade stops and background migrations before each hop^[3][11][12].

The cost of ignoring it (illustrative): 40 engineers waiting for 6 pipelines a day, 10 avoidable minutes each, lose 40 × 6 × 10 = 2,400 minutes, 40 engineer-hours, every working day.

• Not GitOps. OpenGitOps defines GitOps by four principles: declarative, versioned and immutable, pulled automatically, continuously reconciled^[13]. A CI job running kubectl apply is push-based; GitLab says “The GitOps workflow is recommended” and “You should not use a CI/CD workflow for production deployments.”^[14] GitOps on GitLab (Flux plus the agent) is covered here; the principles are on GitOps.
• Not a product, role or certification (FAQ).
• Not only administration. On GitLab.com or Dedicated you still own pipelines, runners, secrets, environments and cost^[15].

Two misconceptions, corrected by GitLab: “Masking a CI/CD variable is not a guaranteed way to prevent malicious users from accessing variable values”^[16]; and “The security risks are greater when using instance runners as they are available by default to all groups and projects”^[4].

The word has no traceable origin. GitLab’s operational surface has a history that older tutorials still miss:

• 16.0: create a runner in the UI or API first, then register it with its glrt- token^[17].
• 16.2: the agent’s built-in GitOps deprecated in favour of Flux^[18].
• Runner 17.1: Docker Autoscaler generally available, replacing Docker Machine^[19].
• 17.5: Docker Machine “scheduled for removal in GitLab 20.0 (May 2027)”^[20]; required stops fixed at x.2, x.5, x.8 and x.11^[11].
• 18.0: registration tokens were not removed after all^[21], but stay “not recommended”^[17].
• 19.0: job tokens default to JWT format, the legacy format usable until 20.0^[22].
• Current docs: the kaniko page is marked removed: “kaniko is no longer a maintained project”^[23].

Mental model: four loops, each failing in its own way. When someone says “the pipeline is slow”, first ask which loop is slow.

%%{init: {"flowchart": {"wrappingWidth": 135, "nodeSpacing": 14, "rankSpacing": 26, "padding": 8}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  subgraph L1["1 Configuration"]
    direction LR
    C["YAML and components merge into one pipeline graph"]:::core
    FC["Fails as: duplicate pipelines, stage barriers, invalid YAML"]:::risk
    C --- FC
  end
  subgraph L2["2 Scheduling"]
    direction LR
    S["A runner with matching tags, protection and free capacity takes each job"]:::core
    FS["Fails as: jobs stuck pending, long queue time"]:::risk
    S --- FS
  end
  subgraph L3["3 Delivery"]
    direction LR
    D["Deploy jobs or Flux change environments"]:::core
    FD["Fails as: unapproved, outdated or unsigned deployments"]:::risk
    D --- FD
  end
  subgraph L4["4 Platform, beneath loops 1 to 3"]
    direction LR
    P["GitLab version, capacity, backups"]:::neutral
    FP["Fails as: API errors, slow Gitaly, missed security fixes"]:::risk
    P --- FP
  end
  L1 --> L2 --> L3
  L3 ~~~ L4

• Stage, needs: by default a job waits for every job in earlier stages; needs starts it as soon as named jobs finish^[7][25].
• rules, workflow: decide whether a job, or the whole pipeline, is created; workflow prevents duplicates^[6].
• Merge-request pipeline: runs on every change to a merge request’s source branch and tests that branch alone^[9]; merged results pipelines and merge trains (Premium, Ultimate) test the result of merging it into the target^[26][27].
• CI/CD component: a versioned, reusable pipeline unit with typed inputs^[28].
• Runner, executor: the process that requests and runs jobs; the executor (Docker, Docker Autoscaler, Instance and Kubernetes are primary) decides isolation^[29].
• Runner scope, protected runner: instance, group or project scope decides who can use, and abuse, a runner^[30]; a protected runner takes only protected-ref jobs^[4].
• Protected environment, deployment tier: restricts who deploys and can require approvals (Premium, Ultimate); the tier (production, staging, testing, development, other) drives protection and DORA data^[31][32].
• Job token, ID token: CI_JOB_TOKEN is valid only while the job runs, with the triggering user’s access^[22]; an OIDC ID token (id_tokens) is exchanged for short-lived cloud credentials^[33].
• Agent for Kubernetes: agentk connects out from the cluster to GitLab’s agent server^[14].
• Required stop, background migration: a version you cannot skip; database work that must finish before the next hop^[12].

GitLabOps is a platform-specific layer beneath vendor-neutral disciplines: they supply the principles and policies, it applies them on GitLab. The boundaries are editorial distinctions chosen by this Atlas; no standard defines them.

No official manifesto exists; this is this Atlas’s proposal.

1. Pipelines are a product; developers are the users. Publish service levels for feedback time and reliability.
2. Reuse through pinned components, “a specific commit SHA (preferred) or release version tag”^[28].
3. Isolation follows trust. Privileged work only on isolated, ephemeral runners limited to protected branches^[8].
4. Identity beats secrets. Allowlisted job tokens; ID tokens bound to project, ref and environment^[22][33].
5. Production is pulled or gated, and the pull source is guarded. Write access to the promoted artifact, or to its signing identity, is deploy access^[14][32].
6. Stay inside the security-fix window^[3].
7. Measure before tuning.

The assurance indicators turn these into testable invariants. The trade-offs remain: speed versus isolation, reuse versus team autonomy, GitLab-native features versus portability.

Five building blocks:

• Control plane: Puma (web and API), Sidekiq (background jobs), Gitaly (Git repository access), PostgreSQL, Redis, object storage, the registry and the agent server^[34]. GitLab runs it on GitLab.com and Dedicated^[15]; you run it on Self-Managed.
• Configuration: .gitlab-ci.yml, components, variables and project settings such as the 60-minute default job timeout^[35].
• Execution: runner managers, executors and caches^[36].
• Delivery: environments, deployments, the agent and Flux.
• Assurance: test and scan reports, metrics and audit events. Govern their retention too: job logs, artifacts (expire_in^[37]) and, on Self-Managed, backups with gitlab-secrets.json and gitlab.rb^[38].

R = responsible, A = accountable, C = consulted, I = informed. A small team that runs fleets and components as an internal product while teams own their pipelines usually works (PlatformOps); a central team writing everyone’s YAML becomes a queue.

Governance escalates from conventions to configuration teams cannot edit (see the reuse comparison). A change to a shared component, runner image or fleet is a production change: review, canary group, rollback.

A logical reference for containers deployed to Kubernetes with your own runner fleets. The fleet split is this Atlas’s recommendation, derived from GitLab’s runner security guidance^[8].

%%{init: {"flowchart": {"wrappingWidth": 115, "nodeSpacing": 18, "rankSpacing": 36, "padding": 8}}}%%
flowchart LR
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  FA["Fleet A, general and untrusted: autoscaler manager, one-job VMs"]:::core
  FB["Fleet B, protected refs only: protected runner, release pods"]:::risk
  CACHE[("Runner cache bucket")]:::neutral
  subgraph CP["GitLab control plane"]
    API["API and Sidekiq; artifacts in object storage"]:::core
    REG["Container registry"]:::core
  end
  CL["Cloud APIs via OIDC"]:::risk
  FA -->|"cache"| CACHE
  FA -->|"jobs, artifacts"| API
  FB -->|"jobs"| API
  FB -->|"push"| REG
  FB -->|"ID token"| CL

Runners and the agent connect out to GitLab, except receptive agents (Ultimate, Self-Managed, from 17.4)^[14].

Flux deploys whatever the production tag points to, so moving that tag is a production credential. Every job gets a registry user “with read-write access to the container registry”^[48], and “by default, users with the Developer, Maintainer, or Owner role can push and delete image tags”^[43]. Unguarded, a feature-branch job or a laptop could move production past every approval.

%%{init: {"flowchart": {"wrappingWidth": 140, "nodeSpacing": 16, "rankSpacing": 30, "padding": 8}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  PUB["Fleet B publish job: manifest artifact :sha, digest recorded"]:::core
  APR{{"Protected environment: an approver allows the deploy"}}:::decision
  KMS["KMS signing key; its role accepts only the main, protected-production subject, requested by the deploy job alone"]:::risk
  DEP["Fleet B deploy job: sign the digest, move the tag"]:::risk
  ATK["A Developer's job or laptop"]:::neutral
  TAG["Registry tag production; push limited by a protected tag rule"]:::core
  FX{{"Flux: is the production digest validly signed?"}}:::decision
  OK["Reconcile; agentk shows state in GitLab"]:::core
  NO["Refuse: SourceVerified False, alert"]:::risk
  PUB --> APR --> DEP
  KMS -.->|"sign"| DEP
  DEP --> TAG
  ATK -.->|"blocked by the rule"| TAG
  TAG --> FX
  FX -->|"yes"| OK
  FX -->|"no"| NO

1. Protected container tag for ^production$, push restricted to Maintainer (Free, generally available since 17.11; on Self-Managed it needs the registry metadata database)^[43][49]. A CI push counts as the job’s user: the job token has “the same access level as the user that triggered the pipeline”, or ran the manual job^[22], and in the 19.5 source the registry derives the denied tag patterns from that user’s project role^[50]. So the deploy job must be run by a Maintainer or Owner, and a Maintainer’s feature-branch job passes the rule too: only step 2 stops an unsigned tag move. Rule changes apply once registry tokens expire (5 minutes by default, 15 minutes on GitLab.com)^[43]; test after that.
2. Signature verification. Flux’s OCIRepository refuses anything not signed with the cosign key (spec.verify)^[45]; the key sits in a cloud KMS^[51] behind a role that accepts only this project’s ID tokens^[47]. On GitLab.com, keyless Sigstore signing is an alternative^[52].
3. Bind the signing role to the production environment, not just main. The default subject names only project, ref type and ref, so every main job, Fleet A’s included, gets a token the role accepts. AWS cannot check runner_id or environment claims (on Self-Managed and Dedicated only sub and aud)^[47], so add environment_protected and deployment_tier to the project’s ci_id_token_sub_claim_components (18.7 and later) and require the deploy job’s full subject^[53][33]. Confirm the string in a test project (the docs do not print it); the project’s other trust policies see it too.
4. Give that subject to the deploy job only. Environment fields prove that a job named the protected production environment, not that anyone approved it. Approvals block “jobs deploying to the environment”^[32], but a job with environment:action prepare, verify or access “does not trigger deployments”^[7], and environment_action cannot be a subject component^[53]. GitLab’s own change says “approval is not required when the job only accesses the environment”, while an open issue says such jobs still need deployer access^[54]. So a post-deploy smoke test on main that names production and requests an ID token could sign without approval. Allow id_tokens only on the deploy job, enforce it through code-owner review of CI files^[55], and test that a verify job on main cannot assume the role.
5. Optionally, a separate CD project whose members are only release managers^[10].

Residual risks. Anyone the tag rule allows can re-point production to an older signed release; and a reviewed CI change can still add a job that signs.

---
config:
  sequence:
    actorMargin: 44
    width: 96
    diagramMarginX: 6
    wrap: true
    boxMargin: 6
    messageMargin: 30
---
sequenceDiagram
  participant Dev as Developer
  participant GL as GitLab
  participant RN as Runners
  participant Ap as Approver
  participant FX as Flux
  Dev->>GL: 1. Open MR
  RN->>GL: 2. Fleet A tests
  opt
    Note over Dev,GL: Pipeline fails
    GL-->>Dev: 3. Failed job
    Dev->>GL: 4. Push fix
  end
  Dev->>GL: 5. Merge
  RN->>GL: 6. Fleet B builds
  Ap->>GL: 7. Approve
  RN->>GL: 8. Sign, move tag
  FX->>GL: 9. Verify, pull
  opt
    Note over Ap,FX: Health checks fail
    Ap->>GL: 10. Rollback job
    FX->>GL: 11. Pull previous
  end

By default “the user who triggers a deployment pipeline can’t also approve the deployment job”; leave “Allow pipeline triggerer to approve deployment” off where separation of duties matters^[32]. Each decision on this path has an owner and leaves a record:

GitLab’s DORA dashboards count only successful deployments to production-tier environments, so tier them correctly^[39].

Each playbook ends with an acceptance test and a rollback. Also review clean-up policies and artifact expiry monthly^[44].

1. workflow:rules that run merge-request pipelines, suppress branch pipelines while a merge request is open, and keep default-branch and tag pipelines^[6]. Put them in .gitlab-ci.yml itself: rules inside an included component do not enable merge-request pipelines^[9].
2. workflow:auto_cancel:on_new_commit: interruptible; tests interruptible: true, deploys false^[7].
3. needs instead of stage barriers; lint fast and first^[25].
4. rules:changes with compare_to on expensive jobs, because rules:changes is true for new branches and for pipelines without a push event^[7].
5. With GitLab’s security templates, AST_ENABLE_MR_PIPELINES: "true" (18.0 and later): “By default, the application security jobs are configured to run for branch pipelines only”, so step 1 would silently drop them from merge requests^[56].
6. A Lint API dry_run before merging^[24].

Acceptance: one pipeline per push, scans included; a second push cancels the first’s interruptible jobs. Rollback: revert; each pipeline uses its own commit’s configuration.

1. Inventory runners (scope, executor, tags, protection, privileged flag, last contact); retire unknown ones.
2. Define fleets by trust: general (unprivileged, ephemeral), protected (release credentials), special hardware^[57].
3. Create runners as code: POST /user/runners with tag_list, run_untagged=false and, for the protected fleet, access_level=ref_protected; the token is shown once, so store it in a secret manager^[41].
4. Autoscale with the Docker Autoscaler or the Kubernetes executor, with at least two managers^[19][58][57].
5. Scrape /metrics from a private network only, because it has no authorisation^[59]; keep runner and GitLab versions aligned^[60].

Acceptance: every fleet has an owner, capacity target and queue-time SLO; no privileged runner takes unprotected jobs. Rollback: pause the new runner, unpause the old; tags switch traffic per project.

1. Classify variables: configuration, low-risk tokens (masked and hidden), production secrets (move out).
2. Cloud access through id_tokens, a service-specific aud, and trust policies on sub plus project_id (a separate key where supported, otherwise the first sub component, 19.1 and later), never user or email claims^[33][53].
3. Keep the job token allowlist: “By default, the allowlist of any project only includes itself”^[22].
4. Production variables protected and scoped to the production environment^[16].
5. With Ultimate, secret push protection, knowing it scans diffs only, skips files over 1 MiB and accepts the push on timeout^[61]; pipeline secret detection is in every tier^[62].

Acceptance: no long-lived cloud keys in variables. Rollback: keep old credentials disabled, not deleted, for a release cycle.

1. environment:name and an explicit deployment_tier on every deploy job^[31].
2. Protected production environments with approvals (Premium, Ultimate); group-level protection by tier covers new projects^[32].
3. resource_group: production^[63] and “prevent outdated deployment jobs”^[10].
4. Flux with the agent for Kubernetes production; CI access through ci_access only to non-production environments, impersonating the job^[64].
5. Guard the promotion path (tag rule, signatures, deploy-job-only signing subject).
6. Decide rollback: “Allow job retries for rollback deployments” is on by default; GitLab advises disabling it in sensitive projects, where “you must run a new pipeline with a previous commit”^[10].

Acceptance: a Developer’s tag move is blocked, Flux refuses an unsigned artifact, and a verify job on main cannot assume the signing role. Rollback: only through review; deleting an approval rule can lose its history^[32].

1. A minor upgrade every one to two months through any required stop^[11], plus the patch releases. Security fixes cover the current release and the two before it, so an instance installed on release N the day it ships is covered until N+3 ships, about three months: a quarterly rhythm leaves no slack^[3].
2. Before each hop: rehearse on a clone, back up including secrets and configuration, confirm background migrations finished (runbook)^[65][38].
3. After: gitlab:check, gitlab:doctor:secrets, sign-in, clone and push, runner pickup, registry push and pull; then upgrade runners and agents^[60].

Acceptance: checks pass; current or previous release. Rollback: a restore needs a backup of exactly the same version and edition^[38], so back up and snapshot immediately before each hop.

GitLabOps has no standard of its own. It borrows OpenGitOps as a test of whether a setup is GitOps^[13]; DORA’s metrics (five in the guide updated 2026-01-05; GitLab’s dashboards implement the earlier four)^[66][39]; the OWASP Top 10 CI/CD Security Risks as a threat checklist^[67]; and NIST SP 800-204D where an organisation or contract adopts it^[68]. Vendor rules such as required stops are mandatory; reference architectures are advice^[40].

Automate the machinery, not the trust decisions: runner creation and rotation through the API, Lint dry runs on shared-component changes, reports of stale runners and unpinned components, upgrade gates (runbook). GitLab Duo’s Root Cause Analysis reads the last 100,000 characters of a failed job’s log and proposes a fix (Duo add-on; not for trigger jobs or downstream pipelines)^[74]; approve the log sharing first, lint any suggested YAML, and let no assistant change protected pipelines, runners or environment protection without human approval^[67].

A representative design, not a recipe: GitLab 19.x; a containerised Node.js service; Kubernetes production with Flux; AWS KMS for the signing key; Premium or Ultimate for approvals (on Free the signing role can be bound only to main). All hosts, IDs and tokens are placeholders.

Example status for this section: YAML (Atlas validator), TOML (smol-toml 1.3.1) and bash (bash -n) syntax-checked; nothing executed against a live GitLab instance, registry, KMS key or cluster.

templates/build-image/template.yml in a project such as platform/ci-components, following GitLab’s rootless BuildKit job^[70]: no privileged container. Rootless BuildKit still needs “a runner that permits the system calls BuildKit uses to create user namespaces and mount points”; where it fails, GitLab suggests Buildah, so test it on each fleet.

spec:
  inputs:
    stage:
      default: build
      description: Stage in which the build job runs
    image_name:
      description: Registry path of the image, without a tag
    context:
      default: "."
      description: Build context directory
    runner_tag:
      default: protected-build
      description: Tag of the runner fleet allowed to build release images
---
build-image:
  stage: $[[ inputs.stage ]]
  image:
    name: moby/buildkit:rootless   # pin by digest in production
    entrypoint: [""]
  tags: ["$[[ inputs.runner_tag ]]"]
  interruptible: false
  variables:
    BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
  before_script:
    - mkdir -p ~/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > ~/.docker/config.json
  script:
    - |
      buildctl-daemonless.sh build \
        --frontend dockerfile.v0 \
        --local context=$[[ inputs.context ]] \
        --local dockerfile=$[[ inputs.context ]] \
        --output type=image,name=$[[ inputs.image_name ]]:$CI_COMMIT_SHA,push=true
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_TAG

# .gitlab-ci.yml (illustrative, GitLab 19.x)
include:
  - component: $CI_SERVER_FQDN/platform/ci-components/build-image@1.4.2
    inputs:
      stage: build
      image_name: $CI_REGISTRY_IMAGE
  - template: Jobs/SAST.gitlab-ci.yml
  - template: Jobs/Secret-Detection.gitlab-ci.yml

workflow:
  auto_cancel:
    on_new_commit: interruptible
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_TAG

stages: [lint, test, build, publish, deploy]

default:
  interruptible: true
  tags: [linux-general]
  retry:
    max: 2
    when: [runner_system_failure, runner_interrupted]   # infrastructure faults only, never script failures

variables:
  NODE_IMAGE: node:22-bookworm-slim   # pin by digest in production
  MANIFEST_REPO: $CI_REGISTRY_IMAGE/manifests
  TOOLS_IMAGE: $CI_REGISTRY/platform/images/release-tools:3   # internally vetted: flux, cosign, jq, AWS CLI
  AST_ENABLE_MR_PIPELINES: "true"   # run the security templates in MR pipelines too (18.0 and later)

.npm-cache: &npm-cache
  key:
    files: [package-lock.json]
  paths: [.npm/]

lint:
  stage: lint
  image: $NODE_IMAGE
  cache:
    <<: *npm-cache
    policy: pull
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run lint

unit-test:
  stage: test
  needs: []          # start immediately instead of waiting for the lint stage
  image: $NODE_IMAGE
  cache:
    <<: *npm-cache
  script:
    - npm ci --cache .npm --prefer-offline
    - npm test
  artifacts:
    when: always
    reports:
      junit: junit.xml
    expire_in: 7 days

publish-manifests:
  stage: publish
  image: $TOOLS_IMAGE
  tags: [protected-build]
  interruptible: false
  needs: [build-image]
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - >
      flux push artifact "oci://$MANIFEST_REPO:$CI_COMMIT_SHORT_SHA" --path=./deploy
      --source="$CI_PROJECT_URL" --revision="$CI_COMMIT_BRANCH@sha1:$CI_COMMIT_SHA"
      --creds="$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" --output json > push.json
    - echo "MANIFEST_DIGEST=$(jq -r .digest push.json)" > manifest.env
  artifacts:
    reports:
      dotenv: manifest.env   # hands the digest to deploy-production

deploy-production:
  stage: deploy
  image: $TOOLS_IMAGE
  tags: [protected-build]
  interruptible: false
  needs: [publish-manifests]
  resource_group: production
  environment:
    name: production
    deployment_tier: production
    url: https://www.example.com
  id_tokens:
    SIGNER_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    # IAM trust policy: StringEquals on sub = this job's subject incl. environment fields (promotion path, steps 3-4).
    # Only this job declares id_tokens.
    SIGNER_ROLE_ARN: arn:aws:iam::ACCOUNT_ID:role/manifest-signer
    COSIGN_KEY: awskms:///alias/manifest-signer
    COSIGN_YES: "true"
    AWS_REGION: eu-west-1
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual   # run by a Maintainer: the production tag rule checks the job's user
  script:
    - >
      creds=$(aws sts assume-role-with-web-identity --role-arn "$SIGNER_ROLE_ARN"
      --role-session-name "deploy-$CI_PIPELINE_ID" --web-identity-token "$SIGNER_ID_TOKEN"
      --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text)
    - export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" $creds)
    - printf '%s' "$CI_REGISTRY_PASSWORD" | cosign login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    - cosign sign --key "$COSIGN_KEY" "$MANIFEST_REPO@$MANIFEST_DIGEST"   # transparency-log upload: decide, see below
    - flux tag artifact "oci://$MANIFEST_REPO:$CI_COMMIT_SHORT_SHA" --tag production --creds="$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD"
    - cosign verify --key "$COSIGN_KEY" "$MANIFEST_REPO:production" > /dev/null   # fail if the tag is not on a signed digest

Why each piece. The workflow, interruptible, needs and scanner settings are Playbook 1. Retries cover infrastructure failures only, so flaky tests stay visible; the JUnit report feeds the merge-request summary without changing job status^[75]. The publish job hands the artifact digest on through a dotenv report^[76][7]. The deploy job, the only one with id_tokens, exchanges its token for short-lived AWS credentials^[47], signs the recorded digest and moves the tag under resource_group^[63]; if someone re-points :<sha> in between, production lands on an unsigned digest, the final cosign verify fails and Flux refuses it.

Transparency log: decide, do not inherit. cosign sign records each signature in a transparency log unless told not to: cosign 2 in the public rekor.sigstore.dev, so a private registry gets a public entry per release; cosign 3 in the services of a TUF-provided signing configuration unless you pass your own, keeping --tlog-upload=false only as a deprecated flag (“prefer using a –signing-config file with no transparency log services”)^[77]. Air-gapped instances need their own log or none; without an entry, verify with --insecure-ignore-tlog=true^[77]. Flux’s public-key verification sets IgnoreTlog = true (source-controller v1.9.6, with a comment calling it an oversight; re-check after upgrades)^[78]. Status: defaults read from the cosign 2.6.5 and 3.1.3 and Flux sources; not executed.

Settings outside YAML: protected main with code-owner approval for CI files^[55]; protected production with approvers and the triggerer-approval setting off^[32]; the ^production$ tag rule, so a Maintainer runs the deploy job^[43]; the ID-token subject components^[53]; “prevent outdated deployment jobs” and a deliberate rollback-retry choice^[10].

Fleet A: ephemeral VMs with the Docker Autoscaler and the AWS fleeting plugin^[19][5]. Example status: TOML syntax-checked; not executed.

concurrent = 40
check_interval = 3
log_level = "info"
listen_address = "10.0.12.5:9252"   # private interface only: /metrics has no authentication

[[runners]]
  name = "fleet-a-general"
  url = "https://gitlab.example.com"
  token = "glrt-REPLACE_WITH_TOKEN_FROM_SECRET_STORE"
  executor = "docker-autoscaler"
  limit = 40
  output_limit = 8192
  [runners.docker]
    image = "alpine:3.20"
    privileged = false
    pull_policy = "always"
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = "example-runner-cache"
      BucketLocation = "eu-west-1"
      AuthenticationType = "iam"
  [runners.autoscaler]
    plugin = "aws"
    capacity_per_instance = 1
    max_use_count = 1
    max_instances = 40
    [runners.autoscaler.plugin_config]
      name = "fleet-a-general-asg"
    [runners.autoscaler.connector_config]
      username = "ec2-user"
      use_external_addr = false
    [[runners.autoscaler.policy]]
      idle_count = 0
      idle_time = "0s"
      periods = ["* * * * *"]
    [[runners.autoscaler.policy]]
      idle_count = 6
      idle_time = "20m0s"
      periods = ["* 7-19 * * mon-fri"]
      timezone = "Europe/Berlin"

idle_count counts idle job capacity, not machines, and the last matching policy wins^[5]; pull_policy = "always" follows GitLab’s advice for shared runners^[8]. Fleet B runs protected builds in a dedicated namespace via the Helm chart^[58]; protection is not a config.toml setting but the runner’s access_level=ref_protected^[41].

# values-fleet-b.yaml for the gitlab/gitlab-runner Helm chart (illustrative)
gitlabUrl: https://gitlab.example.com
runnerToken: "<set-at-install-from-your-secret-store>"
rbac:
  create: true
concurrent: 10
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "ci-protected"
        image = "alpine:3.20"
        privileged = false
        cpu_request = "500m"
        memory_request = "1Gi"
        cpu_limit = "2"
        memory_limit = "4Gi"
        [runners.kubernetes.node_selector]
          "workload" = "ci-protected"

Flux follows production and verifies its signature first^[45]; the key owner publishes the public key to the cluster^[51]. Example status: bash -n syntax-checked; not executed.

cosign public-key --key awskms:///alias/manifest-signer > cosign.pub
kubectl -n flux-system create secret generic manifest-signer-pub --from-file=cosign.pub=cosign.pub

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: web
  namespace: flux-system
spec:
  interval: 5m
  url: oci://registry.gitlab.example.com/shop/web/manifests
  ref:
    tag: production
  secretRef:
    name: gitlab-registry-pull     # read-only registry credential
  verify:
    provider: cosign
    secretRef:
      name: manifest-signer-pub    # unsigned or wrongly signed artifacts are refused
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: web
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: OCIRepository
    name: web
  path: ./
  prune: true
  wait: true
  timeout: 5m

Staging may take pushes through the agent, limited to those environments and impersonating the CI job; without access_as^[64], a job inherits the agent’s full permissions:

# .gitlab/agents/staging-agent/config.yaml in the agent configuration project
ci_access:
  projects:
    - id: shop/web
      environments:
        - staging
        - review/*
      access_as:
        ci_job: {}
---
# RoleBinding in the staging namespace: CI jobs appear in the group gitlab:ci_job
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: gitlab-ci-jobs-deploy
  namespace: web-staging
subjects:
  - kind: Group
    name: gitlab:ci_job
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit                      # broad but namespace-bound; narrow it per project in real use
  apiGroup: rbac.authorization.k8s.io

With pull delivery the deployment record marks the promotion, not the finished rollout; accurate DORA data may need the Deployments API, for example called from a Flux notification^[39].

Three labs, none executed here (commands are syntax-checked against the cited docs); use a test project, never production. A Flux bootstrap lab belongs with the reconciliation principles on GitOps.

Objective. See which jobs a pipeline would contain per ref, with the CI Lint API’s dry run^[24]. Prerequisites. A project you own (GitLab.com Free is enough), an API token, curl, jq. Cost and risk. None. Commit this to main, then create a branch feature-x:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

stages: [lint, test, deploy]

lint:
  stage: lint
  script: echo "lint"

unit-test:
  stage: test
  needs: []
  script: echo "test"

deploy-docs:
  stage: deploy
  script: echo "deploy"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Run the dry run for main, then for feature-x. Example status: bash -n syntax-checked; not executed.

#!/usr/bin/env bash
set -euo pipefail
: "${GITLAB_URL:?}" "${GITLAB_TOKEN:?}" "${PROJECT_ID:?}"
REF="${1:-main}"
REQ=$(mktemp)
jq -n --rawfile content .gitlab-ci.yml --arg ref "${REF}" \
  '{content: $content, dry_run: true, include_jobs: true, ref: $ref}' > "${REQ}"
curl --silent --fail --request POST \
  --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
  --header "Content-Type: application/json" \
  --data @"${REQ}" \
  --url "${GITLAB_URL}/api/v4/projects/${PROJECT_ID}/ci/lint" \
  | jq '{valid, errors, warnings, jobs: [.jobs[]?.name]}'
rm -f "${REQ}"

Expected output. "valid": true; lint, unit-test and deploy-docs on main, no deploy-docs on feature-x. Validation. Delete needs: []: the job list does not change, because needs changes when jobs start, not whether they exist. Troubleshooting. 401/403: token scope or role. Teardown. Revoke the token; delete the lab project.

Objective. Register a project runner, expose its metrics, and watch queueing when capacity is below demand. Prerequisites. A disposable Linux VM with Docker; Maintainer on a project. Cost and risk. The VM; mounting the Docker socket “delegates full control over the Docker daemon” to the runner container^[73], so never use your workstation. Create a project runner (Settings > CI/CD > Runners) tagged lab-docker, “Run untagged” off, and copy its glrt- token^[17]. Example status: bash -n syntax-checked; not executed.

#!/usr/bin/env bash
set -euo pipefail
: "${RUNNER_TOKEN:?set RUNNER_TOKEN to the glrt- token}"
# Runner container; metrics published on the host loopback only.
docker run -d --name gitlab-runner --restart always \
  -p 127.0.0.1:9252:9252 \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest
# Register with the authentication token (no tags or locking flags: those were set when the runner was created).
docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner:latest register \
  --non-interactive --url "https://gitlab.com/" --token "${RUNNER_TOKEN}" \
  --executor docker --docker-image alpine:3.20 --description lab-docker
# Expose metrics inside the container, then restart to be sure the change is loaded.
sudo sed -i '1i listen_address = "0.0.0.0:9252"' /srv/gitlab-runner/config/config.toml
grep -E '^(concurrent|listen_address)' /srv/gitlab-runner/config/config.toml
docker restart gitlab-runner
curl -s http://127.0.0.1:9252/metrics | grep -E '^gitlab_runner_(concurrent|limit|version_info)'

Set concurrent = 1 in config.toml, restart the container and push:

queue-demo:
  tags: [lab-docker]
  image: alpine:3.20
  parallel: 3
  script:
    - sleep 60

Expected output. The three jobs run one after another; with concurrent = 3 they overlap. Validation. Later jobs show longer queued times; gitlab_runner_concurrent reports 1, then 3. Troubleshooting. Stuck pending: a tag typo or “Run untagged”; 410 Gone - runner registration disallowed: a legacy registration token^[17]. Teardown. Delete the runner in the UI (invalidating its token), docker rm -f gitlab-runner, remove /srv/gitlab-runner (it holds the token), destroy the VM.

Objective. Measure merge-request feedback time, change one thing, and decide from data. Prerequisites. Python 3.8+, a read-only token, a week of merge-request pipelines. Cost and risk. Read-only API calls, one per pipeline, so keep the window small on busy projects. Pipeline duration “excludes … any pending (queue) time”^[79], so the script reports wall-clock feedback (finished_at − created_at), busy time and the wait in between; the API page’s own example pipeline hides about 3 seconds of such wait^[80].

Example status: syntax-checked; the filtering, pagination, timestamp and percentile logic was exercised offline with a stubbed api() and a hand-written sample that includes the API page’s example pipeline; not run against a live API.

#!/usr/bin/env python3
"""Merge-request pipeline health from the GitLab Pipelines API (read-only).

Usage: GITLAB_URL=https://gitlab.example.com GITLAB_TOKEN=<read-only token> \
       python3 pipeline_health.py <project id or path> [days] [source]
feedback_s = finished_at - created_at (wall clock, includes every wait)
busy_s = duration (job running time; GitLab excludes queue time)
"""
import json
import math
import os
import sys
import urllib.parse
import urllib.request
from datetime import datetime, timedelta, timezone


def api(path, params):
    base = os.environ["GITLAB_URL"].rstrip("/")
    url = f"{base}/api/v4{path}?{urllib.parse.urlencode(params)}"
    req = urllib.request.Request(url, headers={"PRIVATE-TOKEN": os.environ["GITLAB_TOKEN"]})
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.load(resp)


def ts(value):
    """Parse GitLab timestamps such as 2022-09-21T01:05:07.200Z."""
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def percentile(values, pct):
    """Nearest-rank percentile: smallest value with at least pct % of the values at or below it."""
    if not values:
        return None
    ordered = sorted(values)
    rank = max(1, math.ceil(pct / 100 * len(ordered)))
    return ordered[rank - 1]


def summarise(details):
    finished = [d for d in details if d.get("status") in ("success", "failed")]
    result = {"considered": len(finished)}
    if finished:
        ok = sum(1 for d in finished if d["status"] == "success")
        result["success_rate"] = round(ok / len(finished), 3)
    series = {"feedback_s": [], "busy_s": [], "queued_s": [], "in_pipeline_wait_s": []}
    for d in finished:
        if not (d.get("created_at") and d.get("finished_at")):
            continue
        wall = (ts(d["finished_at"]) - ts(d["created_at"])).total_seconds()
        series["feedback_s"].append(wall)
        busy, queued = d.get("duration"), d.get("queued_duration")
        if busy is not None:
            series["busy_s"].append(busy)
        if queued is not None:
            series["queued_s"].append(queued)
        if busy is not None and queued is not None:
            series["in_pipeline_wait_s"].append(max(0.0, wall - busy - queued))
    for name, values in series.items():
        if values:
            result[name] = {"p50": percentile(values, 50), "p95": percentile(values, 95), "max": max(values)}
    return result


def main():
    project = urllib.parse.quote(sys.argv[1], safe="")
    days = int(sys.argv[2]) if len(sys.argv) > 2 else 7
    source = sys.argv[3] if len(sys.argv) > 3 else "merge_request_event"
    since = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
    listed, page = [], 1
    while True:
        batch = api(f"/projects/{project}/pipelines",
                    {"updated_after": since, "source": source, "per_page": 100, "page": page})
        if not batch:
            break
        listed.extend(batch)
        page += 1
    # Filter again client-side, in case the server ignores an unknown source value.
    wanted = [p for p in listed if p.get("source") == source and p.get("status") in ("success", "failed")]
    details = [api(f"/projects/{project}/pipelines/{p['id']}", {}) for p in wanted]
    print(json.dumps(summarise(details), indent=2))


if __name__ == "__main__":
    main()

Take a 7-day baseline, make one change, measure 7 days later. Expected output. JSON with considered, success_rate and p50/p95/max per series. Validation. Spot-check pipelines in the UI; CI/CD analytics medians should read lower than feedback_s^[72]. A large in_pipeline_wait_s points at the fleet. Troubleshooting. HTTP 404: use the numeric ID. Teardown. Revoke the token.

Illustrative composites, not reports of named organisations.

Measure the pipeline (what developers feel), the fleet (why) and the platform; DORA outcomes belong to DevOps.

Worked example (illustrative). A week of 1,200 merge-request pipelines: 1,000 succeeded, 150 failed, 50 auto-cancelled. Success rate = 1,000 ÷ 1,150 = 87.0 % (83.3 % if cancellations counted as failures). On hosted medium runners (cost factor 2) at 30 job-minutes each: 1,200 × 30 × 2 = 72,000 compute minutes^[82]; at price P, cost per successful pipeline = 72 P. A p95 queue of 4.5 minutes plus a p95 duration of 22 minutes is not a p95 feedback time of 26.5: percentiles do not add, and both miss waits between stages. Pair every speed target with a quality guard.

Fleet signals in PromQL (metric names from GitLab’s runner docs; the third query is the fleet guide’s own):

# p95 job queue time across a fleet, 15-minute window (histogram documented in the fleet guide)
histogram_quantile(0.95, sum by (le) (rate(gitlab_runner_job_queue_duration_seconds_bucket[15m])))

# Saturation: running jobs as a share of configured concurrency
sum(gitlab_runner_jobs) / sum(gitlab_runner_concurrent)

# Autoscaler start-up: p95 machine creation time
histogram_quantile(0.95, sum by (le) (rate(gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket[5m])))

Most GitLabOps security work is about who can make which job run where, with which identity. The pipeline-specific OWASP CI/CD risks^[67], mapped to GitLab controls (identity reviews, dependency pinning, third-party services and logging are general practice on DevSecOps):

Runner authentication tokens are credentials: “users with access to the runner authentication token can clone a runner and submit false jobs”^[4]. Keep personal data out of fixtures and set artifact expiry deliberately^[37]. GitLab produces evidence (approvals, deployments, audit events, scans), not compliance (ComplianceOps).

Cost drivers. Hosted runners: duration ÷ 60 × cost factor, with Linux factors 1, 2, 3, 6 and 12 from small to 2xlarge, and sizes above medium on Premium or Ultimate only (October 2026)^[82][83]. Own runners: VM hours including idle capacity, storage, egress, operators. Self-Managed adds the control plane and upgrade labour.

Capacity with Little’s law (L = λ × W^[84]) on job slots, where W includes VM start-up and teardown when max_use_count = 1. Illustrative: 300 jobs an hour at peak (λ = 5 a minute) × (6 minutes run + about 1 minute overhead) = 35 busy slots, five more than run time suggests; add burst headroom (1.3 × ≈ 45) and calibrate against p95 queue time. High queue time with low saturation points at tags or protection instead. Idle capacity (idle_count) buys start-up latency with VM hours; time-box it to working hours^[5].

Efficiency levers: remove redundant work and stage barriers (Playbook 1); right-size machines, since over-provisioned runners waste money and under-provisioned ones waste developer time^[25]; lock-file cache keys in a shared bucket^[42]; artifact expiry and registry clean-up. With the registry metadata database, online garbage collection removes unreferenced data after 24 hours; with legacy metadata, schedule offline collection. Never run third-party garbage collection with the metadata database: it “will delete data associated with tagged images”^[49]. We found no reliable carbon figures for CI; instance-hours are a proxy (GreenOps).

Does the operating model still hold? Check monthly with read-only API jobs; healthy is zero unless stated:

• unowned runners, or runners silent for 30 days; privileged or shell runners taking unprotected jobs;
• production credentials (variables, runners) reachable from unprotected refs;
• jobs other than the deploy job declaring id_tokens against the production environment;
• unprotected production environments, production jobs without a resource group, unpinned components;
• days since the last restore test (healthy: ≤ 90) and minor releases behind (healthy: ≤ 2^[3]).

Platform health and recovery (Self-Managed). An outage stops every pipeline. Linux-package installations bundle Prometheus with host, PostgreSQL and Redis exporters “on by default”^[85]; alert on API latency, Sidekiq queues, Gitaly and PostgreSQL saturation and storage. Set recovery objectives, then choose gitlab-backup with separately stored secrets, per-component procedures for large instances^[38], or Geo (Premium, Ultimate), “a warm-standby secondary site in a different region” that still needs backups because it replicates corruption^[86]. Only a timed restore test proves it.

Detection: p95 queue time above 10 minutes for 15 minutes, or a flood of reports.

%%{init: {"flowchart": {"wrappingWidth": 130, "nodeSpacing": 16, "rankSpacing": 28, "padding": 6}}}%%
flowchart TB
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  A["Alert: p95 queue time high"]:::risk
  B{{"Everyone affected?"}}:::decision
  D{{"Managers online?"}}:::decision
  C["Check the project's tags, protection, runner enablement"]:::core
  E["Restart or fail over managers; check network, TLS"]:::core
  F{{"Fleet full?"}}:::decision
  G["Raise concurrent, limit, max instances or quota"]:::core
  J{{"API errors?"}}:::decision
  K["Check the control plane; tune runner request concurrency"]:::core
  A --> B
  B -->|"yes"| D
  B -->|"no"| C
  D -->|"yes"| F
  D -->|"no"| E
  F -->|"no"| J
  F -->|"yes"| G
  J -->|"yes"| K
  J -->|"no"| C

1. Sample pending jobs for tags, projects and refs.
2. Managers alive? Last-contact times; journalctl -u gitlab-runner --since "30 min ago" for connection or TLS errors.
3. Full? sum(gitlab_runner_jobs) / sum(gitlab_runner_concurrent) near 1^[59][57].
4. Autoscaler? Machine states and creation time against max_instances and the cloud quota.
5. Control plane? Rising error and request-concurrency-exceeded counters point at the API or the default request_concurrency = 1^[5].

Metric names for steps 3-5 (GitLab Runner /metrics)
gitlab_runner_jobs, gitlab_runner_concurrent, gitlab_runner_limit
gitlab_runner_autoscaling_machine_states
gitlab_runner_autoscaling_machine_creation_duration_seconds
gitlab_runner_errors_total
gitlab_runner_request_concurrency_exceeded_total

6. Mitigate (pause schedules, move a team by tag, raise max_instances), then record the timeline and the alert that would have caught it.

%%{init: {"flowchart": {"wrappingWidth": 210, "nodeSpacing": 30, "rankSpacing": 22, "padding": 8}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  V0["Current: 18.3"]:::risk
  V1["18.5 stop"]:::core
  V2["18.8 stop"]:::core
  V3["18.11 stop"]:::core
  V4["19.2 stop"]:::core
  V5["Target: 19.5"]:::core
  G["Gate before each hop: backup with secrets, migrations finished, checks pass"]:::decision
  V0 --> V1 --> V2 --> V3 --> V4 --> V5
  V1 -.- G
  V2 -.- G
  V3 -.- G
  V4 -.- G

Symptom. The upgrade stops because a background migration is still “active”^[12]. Diagnosis. Each gate runs on the version you are leaving, so on this path none can use wait, which arrives with 19.5. Example status: bash -n syntax-checked; not executed. Commands and SQL quoted from GitLab’s background-migrations page^[12].

# 1. What is still queued, running or failed?
sudo gitlab-rake gitlab:background_migrations:list      # 18.9 and later
sudo gitlab-rake gitlab:background_migrations:status    # 18.8 and earlier
# Any version: zero rows means all batched background migrations are complete.
sudo gitlab-psql -c "SELECT job_class_name, table_name, column_name, job_arguments FROM batched_background_migrations WHERE status NOT IN(3, 6);"
# 2. 19.5 and later only: block until nothing is paused, active or finalizing; non-zero exit if one failed.
#    The task has no internal timeout, so bound it.
sudo timeout 6h gitlab-rake gitlab:background_migrations:wait
# 3. Health checks before and after each hop.
sudo gitlab-rake gitlab:check
sudo gitlab-rake gitlab:doctor:secrets

Resolve a failed migration first (GitLab documents a manual finalize)^[12]. Rollback: restore the pre-hop backup to the same version and edition, with its secrets^[38]. Prevention: gate hops in automation; rehearse on a clone^[65].

Invest when GitLab serves more than a couple of teams, you run your own runners or instance, production needs approval evidence, CI cost is material, or developers routinely wait. Keep it light for one small team on GitLab.com with hosted runners: defaults, Playbook 1 and a monthly usage check. Do not build a GitLab-specific practice when GitLab only mirrors code built elsewhere, or several SCM platforms must stay portable: invest in DevOps, GitOps and PlatformOps and keep the GitLab layer thin. Someone must own fleet security, release notes, each May’s major release and, on Self-Managed, restore tests; if nobody will, buy the more managed option.

%%{init: {"flowchart": {"wrappingWidth": 130, "nodeSpacing": 16, "rankSpacing": 28, "padding": 6}}}%%
flowchart TB
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  Q1{{"Must you run GitLab yourself, for example air-gapped?"}}:::decision
  Q2{{"Need single-tenant isolation without operating it?"}}:::decision
  COM["GitLab.com"]:::core
  DE["GitLab Dedicated"]:::core
  Q3{{"Jobs need a private network, special hardware or steady heavy load?"}}:::decision
  HR["Hosted runners: budget compute minutes"]:::neutral
  SM["Self-Managed: size by RPS; standalone to about 2,000 users, HA from 3,000"]:::core
  Q4{{"Well-run Kubernetes platform already available?"}}:::decision
  K8["Kubernetes executor on dedicated CI node pools"]:::core
  DA["Docker Autoscaler on ephemeral VMs"]:::core
  Q5{{"Any jobs privileged, deploying, or using production identity?"}}:::decision
  PF["Add a separate protected fleet"]:::core
  OK["Single unprivileged fleet is enough"]:::neutral
  Q1 -->|"no"| Q2
  Q1 -->|"yes"| SM
  Q2 -->|"no"| COM
  Q2 -->|"yes"| DE
  COM --> Q3
  DE --> Q3
  Q3 -->|"no"| HR
  Q3 -->|"yes"| Q4
  SM -->|"no hosted runners"| Q4
  Q4 -->|"yes"| K8
  Q4 -->|"no"| DA
  HR --> Q5
  K8 --> Q5
  DA --> Q5
  Q5 -->|"yes"| PF
  Q5 -->|"no"| OK

The tree encodes GitLab’s general sizing advice^[40] and runner security guidance^[8]; the rest is this Atlas’s judgement. Worked comparison: 80 developers on GitLab.com, private-network deployments, a platform team of two; scores are assumptions, except hosted-runner isolation: each job “runs in a newly provisioned VM, which is dedicated to the specific job”^[36].

With no private-network deployments (network weight 0, effort 0.40) the totals become 4.30, 3.40 and 3.30: the answer flips, and the close base case could flip on your cost estimate alone.

This Atlas’s model, not an official one. Find the level whose behaviour matches yours; move up when its exit criteria have held for a quarter.

%%{init: {"flowchart": {"wrappingWidth": 300, "rankSpacing": 18, "padding": 8}}}%%
flowchart TB
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  L1["1 Ad hoc: copied pipelines, runners on any host, secrets in variables"]:::risk
  E1["Exit: runners owned, no duplicate pipelines"]:::decision
  L2["2 Repeatable: shared skeleton, protected default branches, runner inventory"]:::neutral
  E2["Exit: no long-lived cloud keys, no privileged runners on unprotected jobs"]:::decision
  L3["3 Managed: pinned components, fleets by trust, OIDC, protected environments and promotion tag"]:::core
  E3["Exit: dashboards live, SLOs defined, inside the security window"]:::decision
  L4["4 Measured: weekly review of feedback, queue, success and cost; upgrade calendar"]:::core
  E4["Exit: SLOs met for a quarter, restore tested within 90 days"]:::decision
  L5["5 Optimising: before/after data for every change, policies as code"]:::core
  L1 --> E1 --> L2 --> E2 --> L3 --> E3 --> L4 --> E4 --> L5

For level 1–2 with a small platform team.

New project

• Baseline component pinned; workflow:rules prevent duplicates; tests interruptible; scans in MR pipelines
• Default branch protected; MR pipelines must pass; no secrets in .gitlab-ci.yml; Lint dry run passes on two refs

New runner fleet

• Owner and trust level; tags and access level set at creation; token in a secret manager
• One clean environment per job; privileged = false unless protected and ephemeral; private metrics endpoint
• Capacity sized with Little’s law; time-boxed idle policy; runner version matches GitLab’s

Production deployment readiness

• deployment_tier: production; protected environment with approvers; triggerer cannot approve; resource_group set
• “Prevent outdated deployment jobs” on; rollback retries decided; rollback rehearsed
• Tag rule tested (a Developer cannot move the tag; a Maintainer runs the deploy job); Flux refuses an unsigned artifact
• Signing role bound to the production-environment subject; only the deploy job declares id_tokens; a verify job on main refused

Upgrade hop (Self-Managed)

• Path from the Upgrade Path tool; rehearsed and timed on a clone; Playbook 5’s before and after steps done

Runner request form

Runner request
- Requesting team / owner:
- Purpose (build, test, deploy, special hardware):
- Trust level: [ ] general  [ ] protected refs only  [ ] public/fork contributions
- Scope: [ ] project  [ ] group  [ ] instance (admin approval)
- Executor and image constraints:
- Needs privileged mode? If yes, why, and why rootless builds are not enough:
- Network access needed (registries, private endpoints, cloud APIs):
- Expected peak jobs per hour and average job duration:
- Credentials: OIDC audience and trust conditions (no long-lived keys):
- Approval: runner operator ____  security (if protected/privileged) ____

Decision record for a pipeline or fleet change

ADR-NNN: <decision title>
Status: proposed | accepted | superseded by ADR-MMM
Date: YYYY-MM-DD
Context: what problem, which data (baseline metrics), which GitLab version and tier
Options considered: at least two, including "do nothing"
Decision: the chosen option and the conditions under which it holds
Consequences: cost, security, operating effort, lock-in, migration later
Verification: which metric, which threshold, reviewed on which date
Rollback: how to return to the previous state

Monthly scorecard: the charter’s goals plus a retry rate ≤ 3 % and a falling cost per successful pipeline, each with an owner and last month’s value (example targets, not benchmarks).

Is “GitLabOps” a certification or job title? No. Look for the skills or GitLab’s own credentials, such as “GitLab Certified CI/CD Associate”^[88]; GitLab University’s catalogue could not be read automatically, so check current names there.

Do I need Premium or Ultimate? Not to start: merge-request pipelines, components, runners, environments and SAST are Free^[9][28][69]. Paid tiers add merge trains, protected environments, policies, push protection and DORA dashboards^{[27][32][87][61][39]}.

Hosted runners or our own? Hosted runners need no fleet operations and isolate each job in a fresh VM, but bring compute-minute quotas and less network control^[36][82]; see the decision framework.

How often must a self-managed instance be upgraded? At least every two months, to the newest release (Playbook 5)^[3]. Required stops limit how far one hop goes, not how often.

Can we compare GitLab’s DORA dashboard with our own DORA data? Only after aligning definitions: GitLab implements the earlier four metrics with its own data rules^[39]; the DORA guide now lists five^[66].

Worth mastering: merge trains sized to runner capacity^[27], dynamic child pipelines^[81] and policy projects^[87]. Watch: the 20.0 removals in May 2027 (origins); the change that lets non-deploying jobs start without a manual action, behind a feature flag that is off by default in 19.5^[54]; and GitLab’s push towards AI agents (“GitLab MCP server generally available” in 19.5)^[89], which is vendor direction, not yet evidence.

New to delivery? Start with DevOps; prerequisites are in At a glance.

Then go deeper on GitOps, DevSecOps, ReleaseOps, TestOps and KubernetesOps.

Decide: the offering, runners by trust, signed pull for Kubernetes production, the tier whose controls you need. Measure: p95 feedback and queue time, success rate, cost per successful pipeline, version lag. Watch: privileged runners on shared fleets; a tag any Developer can move; a signing identity any main job can request; restores without the secrets file.

Lint dry run         POST /api/v4/projects/:id/ci/lint  {content, dry_run: true, include_jobs: true, ref}
Create runner        POST /api/v4/user/runners  runner_type=group_type group_id=… tag_list=… access_level=ref_protected
Register runner      gitlab-runner register --non-interactive --url … --token glrt-… --executor docker
Runner metrics       listen_address = "<private-ip>:9252"  →  GET /metrics
Migrations gate      sudo gitlab-rake gitlab:background_migrations:wait   (19.5+; earlier: :list or :status)
MR scanners          variables: AST_ENABLE_MR_PIPELINES: "true"
Promotion guard      protected container tag ^production$ + Flux OCIRepository spec.verify (cosign)
Health checks        sudo gitlab-rake gitlab:check ; sudo gitlab-rake gitlab:doctor:secrets
Backup               sudo gitlab-backup create   (+ /etc/gitlab/gitlab-secrets.json and gitlab.rb separately)
No duplicates        workflow:rules  … if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS → when: never
One deploy at a time resource_group: production

Situation (illustrative). A 60-developer organisation runs one autoscaled instance-runner fleet; a fifth of jobs build images with Docker-in-Docker, so the whole fleet is privileged = true. The platform team proposes a protected build fleet with rootless BuildKit (B); a senior engineer prefers the one privileged fleet (A). Assumptions: any Developer on any served project can start jobs; contractors contribute; VMs run one job each; a second fleet costs about a day a month.

Reasoning. (1) Worst failure: privileged containers effectively disable container security and make breakout possible^[8], so a compromised merge request in any project reaches the VM and its network (cache bucket, cloud metadata). (2) Does ephemerality rescue A? Partly; GitLab pairs ephemeral VMs with dedicated runners restricted to protected branches, and A has only the first half. (3) Cost of B: a second manager and a tag change; rootless builds may fail on some runners, where GitLab suggests Buildah^[70]. (4) What would change the answer: if everyone who can trigger a job is equally trusted with production and the fleet reaches nothing sensitive, A’s simplicity wins.

Decision: B. A saves one visible fleet and accepts an invisible cost: every project’s weakest contributor sets the security level of every build. Uncertainty: rootless failure rates here are unknown, so a two-week pilot on the five busiest image-building projects measures them first.

Objectives: explain the label and its GitOps boundary; design duplicate-free pipelines; size and segment fleets; close deployment bypasses; diagnose pending jobs and blocked upgrades; judge changes with percentiles.

Adopting GitLab gives you a delivery platform, not a delivery operation. The label has no standing; the craft does. The hard parts are not in the YAML: they are the queue before the runners, the trust boundary around privileged containers, the identity each job carries, the tag production pulls, and the upgrade path. All facts are dated October 2026 (GitLab 19.x); re-check them.

GitLab docs are the live 19.x pages, which carry no publication date. Ledger: research/gitlabops-ledger.md.

Changelog. 2026-10-16: draft (GitLab 19.x). r1: signed, protected promotion; MR scanners; migration gates. r2: environment-bound signing role; transparency log. r3: deploy-job-only signing identity; Blueprint v2; 17,253 → 8,997 words. r4 (reviews/gitlabops-score-r3.md): restored the rootless-BuildKit caveat; CI tag pushes judged by the job’s user (19.5 source); four-loops figure; standalone maturity figure; use-case pitfalls; 8,997 → 8,993 words. r4 follow-up (orchestrator): registration-token quote attributed to the workflow doc; GitLab.com’s 15-minute rule propagation.