GitOps

GitOps is a way of operating software systems in which you write down the state you want (declaratively), keep every version of that declaration in a store that cannot be silently rewritten (usually Git), and let a software agent fetch it and keep the running system matching it. People change production by changing the declaration through a reviewed pull request. They do not log in to production and change it by hand.

The mental picture is a thermostat. You set the temperature you want, and a control loop keeps measuring the room and switching the heating on or off until the two match. In GitOps the setting is a commit, the room is your cluster, and the thermostat is a reconciler such as Argo CD or Flux. When someone changes the room directly (a kubectl edit at 2 a.m.), the loop notices the difference, called drift, and either reverts it or raises the alarm.

Who this page is for. Beginners (the core idea and a test for spotting it), engineers (repositories, promotion, secrets, drift), architects (pull or push, hub or per-cluster agents) and leaders (governance, evidence, risk). Last verified: 2026-10-17.

Without GitOps, most teams deploy with a pipeline that ends in a command such as kubectl apply or helm upgrade, plus occasional manual fixes. Four problems follow.

1. Nobody can say what production should look like. The last pipeline run and every manual edit since then together define it. Weaveworks, which coined the term, put it this way in 2017: small ad-hoc changes accumulate until “you can no longer distinguish between how the system should be and how it is”^[10].
2. Drift is invisible. A hotfix applied by hand survives until the next deploy silently removes it, or the next deploy fails because of it.
3. CI holds the keys to production. A push pipeline needs credentials for every environment it deploys to. NIST’s DevSecOps guidance names this as the security downside of the push model: credentials can be exposed “outside of the deployment environment”^[5].
4. Rebuilding is archaeology. After losing a cluster you must work out which pipelines, in which order, with which parameters, recreate it.

GitOps answers each with a mechanism, and because the desired state is “sufficient to recreate the system”^[11], a new cluster can be pointed at the same declarations:

What the evidence does and does not show. The benefits above are mechanisms, not measured outcomes: no independent study was found (October 2026) measuring, say, change failure rate before and after adoption. CNCF’s 2025 survey reports that 58% of “cloud native innovators” use GitOps principles extensively, against 23% of “adopters”^[12], which is correlation with organisations mature in other ways, not proof that GitOps causes the maturity.

• Not “we keep YAML in Git”. Storing manifests in Git satisfies at most the first two principles. If a pipeline pushes them and nothing watches the cluster afterwards, there is no reconciliation.
• Not a replacement for CI. CI still builds, tests, scans and publishes artefacts. GitOps replaces the deploy step: the pipeline ends with a commit or a published artefact, not with a write to the cluster^[5].
• Not tied to Git. The principles call for a state store with immutable, versioned history, which “should provide access control and auditing”. Git is “the canonical example”, but “any other system that meets these criteria may be used” ^[11]. Flux calls registry-based delivery “Gitless GitOps”^[13].
• Not Kubernetes-only. Kubernetes made GitOps practical because its API is declarative and built from control loops ^[14], so most tooling lives there. The same pattern is used for cloud resources through Kubernetes-hosted controllers (Crossplane; tofu-controller, a community OpenTofu/Terraform controller for Flux maintained in the separate flux-iac organisation, formerly Weave TF-Controller^[15]).
• Not backup or data recovery. The OpenGitOps desired state “generally does not include persistent application data, eg. database contents”^[11]. See BackupOps and DisasterRecoveryOps.
• Not a release strategy. Canaries, blue-green and feature flags decide how much traffic sees a change; that belongs to ReleaseOps. GitOps decides how the declared state reaches the cluster.
• Not infrastructure as code. IaC is the declarative language and lifecycle (InfraOps); GitOps is one operating model for applying it continuously.

The shift between 2017 and 2021 matters. The 2017 post described “diff tools” that “detect any divergence and notify us via Slack alerts”, with separate sync tools for convergence^[10]. The 2021 principles require agents that pull on their own and “attempt to apply the desired state”^[1]. Many arguments about whether a setup “is GitOps” come down to which of the two definitions someone has in mind.

The mental model is the control loop that Kubernetes itself is built from: controllers “watch the state of your cluster, then make or request changes where needed”^[14]. A GitOps reconciler is one more controller whose desired state comes from outside the cluster.

flowchart TB
  PR["Reviewed pull request merged"]:::neutral --> DS["Desired state: versioned declarations in the state store"]:::core
  DS --> PULL["Agent pulls the declarations"]:::core
  PULL --> OBS["Agent observes the actual state"]:::core
  OBS --> CMP{"Actual matches desired?"}:::decision
  CMP -- "No" --> ACT["Act: apply, prune or roll back"]:::core
  CMP -- "Yes" --> REP["Report: status, events, commit status"]:::neutral
  ACT --> REP
  REP -- "next interval or webhook" --> PULL
  DRIFT["Change made outside the state store"]:::risk -. "creates drift" .-> OBS
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

flowchart LR
  DEV["DevOps: pipeline and culture"]:::neutral
  INF["InfraOps: IaC lifecycle"]:::neutral
  SEC["DevSecOps: signing and policy"]:::neutral
  PLAT["PlatformOps: golden paths"]:::neutral
  G["GitOps: reconcile declared state"]:::core
  K8S["KubernetesOps: the cluster"]:::neutral
  REL["ReleaseOps: canaries, exposure"]:::neutral
  GL["GitLabOps: one platform"]:::neutral
  API["API Ops: API configuration"]:::neutral
  DEV --> G
  INF --> G
  SEC --> G
  PLAT --> G
  G --> K8S
  G --> REL
  G --> GL
  G --> API
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

The OpenGitOps document says the desired state of a GitOps-managed system must be^[1]:

1. Declarative. “A system managed by GitOps must have its desired state expressed declaratively.”
2. Versioned and Immutable. “Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.”
3. Pulled Automatically. “Software agents automatically pull the desired state declarations from the source.”
4. Continuously Reconciled. “Software agents continuously observe actual system state and attempt to apply the desired state.”

The glossary carries much of the meaning:

• Why pull? Because the agents “must be able to access the desired state from the state store at any time, not only when there is an intentional change in the state store triggering a push event”, so reconciliation can be triggered “whenever there is a divergence”^[11].
• How continuous? “Continuous” means “reconciliation continues to happen, not that it must be instantaneous”^[11]. A three-minute polling interval qualifies.
• Immutable how? A branch is a mutable pointer to immutable commits. The principle is broken by references inside the declarations, such as an image tag like :latest, a remote Kustomize base on HEAD or an unpinned chart range: Argo CD warns that such manifests “change their meaning from one day to the next” without any commit in your repository^[30].
• Policies are part of the system. The glossary’s “software system” includes “Policies for controlling access and management of repositories, deployments, runtimes”^[11].

Ask four questions, one per principle:

1. Could you recreate the system’s configuration from the declarations alone, apart from persistent data?
2. Is every version immutable, retained, access-controlled and audited, including what the declarations reference (images, charts, bases)?
3. Does an agent fetch the declarations by itself, whenever it needs them, without a pipeline pushing them?
4. Does the agent keep comparing actual with desired state, and act or alert when they diverge, not only when a new commit arrives?

“Partial” is not an insult: manual sync is a sensible first step and a defensible permanent choice for some production systems.

The following charter is this Atlas’s proposal, not a published standard. Adapt the numbers to your risk.

• Mission. Every change to a managed system is declared, reviewed and versioned; the system converges to it automatically; every difference between declaration and reality is either corrected or recorded as an exception.
• Invariant 1 — one path to production. A production change has a merged commit or a time-boxed break-glass record that is closed by a commit.
• Invariant 2 — no mutable references in production. Images by digest, charts and remote bases by exact version or commit.
• Invariant 3 — least-privilege reconcilers. A reconciler can change only what its owning team may change.
• Invariant 4 — guarded deletion. Pruning is enabled, and the objects whose loss is catastrophic (namespaces, persistent volumes, CRDs) are protected from it.
• Invariant 5 — data is not in Git. Backups and restore tests exist for everything the declarations cannot recreate.
• Measurable goals. Reconciliation lag p90 within a target your users accept; zero unticketed out-of-band changes per month; a successful rebuild drill each quarter (see metrics).
• Non-goals. Deciding canary percentages, owning CI build quality, or replacing change management where regulation requires it.

Typical roles: the platform team runs the reconcilers and owns the repository contract; application teams own their manifests and overlays; code owners approve production paths; security owns signing, secret keys and policy; SRE/on-call handles drift alerts and break-glass; change or compliance consumes the evidence. Small organisations combine roles but keep approvals separate.

R = responsible, A = accountable, C = consulted, I = informed.

• Day 0 — design. Choose the state store and repository layout, the reconciler, hub or per-cluster topology, tenancy model, secrets pattern, drift policy and the break-glass procedure. Decide what stays out of GitOps (data, operator-generated objects).
• Day 1 — bootstrap and migrate. Install the reconciler so that it manages itself: Flux’s bootstrap commits its own manifests and creates the objects that reconcile them^[13]. In Argo CD a parent owns the Applications: an ApplicationSet (Argo CD’s recommendation) or an “app of apps”, which its documentation calls “an admin-only tool”, so only admins should push to the parent’s source^[34]. A parent with self-heal restores each child’s spec, including a syncPolicy changed during an incident (break-glass); one with the deletion finalizer deletes its children with it. Migrate existing workloads one namespace at a time, with automated sync off or prune disabled until the diffs are understood.
• Day 2 — operate. Upgrade the reconciler (it is now production-critical), rotate keys, review exceptions, tune intervals and webhooks, watch reconciliation lag and failures, and rehearse rebuilds.

• Approval is a pull-request review. Use branch protection on the paths reconcilers read: required reviews (with code-owner review for production paths), required status checks, optionally required signed commits, and no bypass for administrators ^[35]. Other Git hosts offer equivalents.
• Evidence is a chain, not a commit. Git shows what was approved. Argo CD’s documentation notes that this audit log “only applies to what happened in Git and does not necessarily correlate one-to-one with events that happen in a cluster”^[36]. Keep the reconciler’s events, which name the revision applied, alongside the pull request.
• Freeze windows are declarative too. Argo CD sync windows allow or deny syncs on a cron schedule; deny overrides allow ^[37].
• Break-glass is designed in advance: scope, time limit, record and the route back to Git (template).
• Exceptions are visible. Every ignoreDifferences rule or Flux ignore annotation is tolerated drift: record it (template) and review it.

The reference architecture below is vendor-neutral. Argo CD packs the source fetcher and renderer into its repo-server and the applier into its application controller^[32]; Flux splits them into source-, kustomize- and helm-controllers^[29].

flowchart TB
  subgraph OUTSIDE["Outside the cluster: change and build"]
    ENG["Engineer opens a pull request"]:::neutral
    CI["CI: build, test, scan, sign"]:::neutral
    REV{"Code owners review and merge"}:::decision
  end
  subgraph STORES["State stores"]
    CFG["Config repository with protected paths"]:::core
    REG["OCI registry: images and charts by digest"]:::core
    VAULT["External secret manager"]:::neutral
  end
  subgraph CLUSTER["Inside the cluster trust boundary"]
    SRC["Source fetcher: clone and verify"]:::core
    RND["Renderer: Kustomize or Helm"]:::core
    APL["Applier: diff, apply, prune as a scoped account"]:::core
    KAPI["Kubernetes API server"]:::neutral
    NODES["Nodes: kubelet runs Pods"]:::neutral
    SECOP["Secrets operator"]:::neutral
    NOTIF["Notifier: events, commit status, alerts"]:::neutral
  end
  ENG --> REV
  ENG --> CI
  CI -- "publishes digest" --> REG
  CI -- "check results" --> REV
  REV --> CFG
  CFG -. "pulled" .-> SRC
  SRC --> RND
  RND --> APL
  APL --> KAPI
  KAPI --> NODES
  REG -. "images pulled by kubelet" .-> NODES
  VAULT -. "pulled" .-> SECOP
  SECOP --> KAPI
  APL --> NOTIF
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

flowchart TB
  subgraph PULL["Pull model"]
    direction TB
    Q1["Merge to main"]:::neutral --> Q2["Config repo or registry"]:::core
    Q2 -- "read-only pull" --> Q3["Agent inside the cluster"]:::core
    Q3 -- "in-cluster identity" --> Q4["Kubernetes API: hand edits reverted"]:::core
  end
  subgraph PUSH["Push model"]
    direction TB
    P1["Merge to main"]:::neutral --> P2["CI runner outside the cluster"]:::risk
    P2 -- "uses a stored cluster credential" --> P3["Kubernetes API: hand edits persist unnoticed"]:::risk
  end
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

NIST SP 800-204C states the case for pull bluntly: “Since CD tools have the keys to production systems, push-based models are rendered insecure.” With pull, “the deployment environment credentials (e.g., cluster credentials) are not exposed outside of the production environment”^[5]. Microsoft’s Flux documentation makes the same point: “No access to clusters is needed by the source repo or by any other cluster”^[42].

Two counterweights keep this honest:

1. The agent is powerful. By default Argo CD’s controller uses a cluster-admin-level role^[36], and Flux’s controllers run under a service account that “has the cluster-admin role bound to it. This is convenient for a deployment in which all users are trusted”^[33]. Pull moves the credential inside the boundary; it does not make it small (see security).
2. The repository becomes the credential. Argo CD’s security guide warns that unauthorised write access to a trusted repository lets an attacker deploy malicious images or delete resources through pruning^[36]. Pull removes one place where production credentials sit; it does not protect what is merged.

A webhook does not make a setup push-based. Argo CD polls every two to three minutes by default (120 seconds plus up to 60 seconds of jitter^[43]) and accepts webhooks from Git hosts and registries to skip the wait^[44]; Flux’s Receiver does the same^[45]. The webhook only says “look now”; the agent still pulls and applies. The price is an inbound endpoint, usually an Ingress to the Argo CD API server or Flux’s webhook-receiver Service. Argo CD’s /api/webhook “currently lacks rate limiting protection”, so set a shared secret and a payload-size limit; a Flux Receiver checks a token. Expose only that path.

sequenceDiagram
  participant E as Engineer
  participant G as Git host and CI
  participant A as Agent
  participant K as Cluster
  E->>G: Open pull request
  G->>G: CI renders, diffs, checks policy
  E->>G: Approve and merge
  G-->>A: Webhook (optional)
  A->>G: Pull new revision
  A->>K: Dry-run apply and diff
  A->>K: Apply and prune
  K-->>A: Rollout status
  alt Healthy
    A-->>G: Commit status: success
  else Unhealthy or failed
    A-->>G: Commit status: failure, alert
    E->>G: Merge a revert
    A->>K: Reconcile previous revision
  end

Separate configuration from application source. Argo CD recommends a separate configuration repository: a replica change should not trigger a build, the audit history stays readable, access can differ, and CI pushing manifests to the source repository can cause build loops^[30]. Flux documents four layouts: a monorepo (apps/, infrastructure/, clusters/, environments as directories on one branch), a repository per environment when production access must be restricted (Git cannot restrict reads per file), a repository per team and a repository per app^[47].

Model environments as directories, not branches. Kustomize overlays exist to create “development, QA, staging and production environment variants” of a shared base^[26]. Kostis Kapelonis, writing on a GitOps vendor’s blog, calls branch-per- environment “an anti-pattern”: merges carry unintended changes and the wrong order of changes, cherry-picks do not scale, and hand edits in upstream branches are never back-ported^[48]. Directories make the difference between staging and production a reviewable file diff.

Promote by changing one value in the next environment. CI records an immutable digest. Promotion is a pull request that sets that digest in the next overlay. Automation can open the request: Flux image automation scans a registry, picks a tag by policy, updates manifests and commits, optionally to a separate branch so a human merges it^[49]. Kargo, an open-source project from Akuity, orchestrates multi-stage promotion on top of GitOps^[50].

flowchart TB
  B["CI builds the image and records its digest"]:::neutral --> D["Automation updates the dev overlay"]:::core
  D -- "healthy, tests pass" --> SP["Bot opens PR: staging overlay to the same digest"]:::core
  D -. "fails" .-> FIX["Fix forward or revert"]:::risk
  SP -- "checks and review pass" --> PP["PR: production overlay to the same digest"]:::core
  SP -. "fails" .-> FIX
  PP -- "code owners approve, window open" --> REC["Production reconciles; health checked"]:::core
  PP -. "no approval or window closed" .-> WAIT["Wait for the window or escalate"]:::neutral
  REC -. "unhealthy" .-> RV["Revert the production commit"]:::risk
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Review what will run, not only the template. A one-line change to Helm values can change hundreds of rendered lines. Either render in CI and post the diff on the pull request, or commit rendered manifests. Argo CD’s Source Hydrator (beta since 3.5.0) implements the rendered manifest pattern by pushing hydrated manifests to Git before syncing them^[22].

Never commit plain Kubernetes Secrets: their values are only base64-encoded, and even inside the cluster Secrets are, by default, “stored unencrypted in the API server’s underlying data store (etcd)”^[51].

Argo CD recommends populating secrets on the destination cluster (Sealed Secrets, External Secrets Operator, Secrets Store CSI Driver) so the reconciler never sees them^[40]. For SOPS, Flux recommends age over OpenPGP where possible and encrypting only data and stringData^[53]; who can read the decryption key is covered under the reference implementation.

“Correct all drift” sounds right and fails in practice, because other controllers legitimately write to the objects you declare.

Declarations arrive all at once; clusters need order. Custom resource definitions must exist before custom resources, and a controller must be ready before objects that depend on its webhooks. Flux expresses order between Kustomizations with dependsOn and gates on readiness with wait or healthChecks^[25]. Argo CD orders within one sync by phases (PreSync, Sync, PostSync) and by argocd.argoproj.io/sync-wave annotations, and prunes in reverse wave order^[63]. “Healthy” means Kubernetes reports the objects ready, not that the service is correct; add a smoke test (an Argo CD PostSync hook, or a Job checked by Flux health checks) when readiness probes are weak.

No law or certification standard requires GitOps. Regulated organisations can use its pull-request and reconciliation evidence for their own change-control requirements; mapping those belongs to ComplianceOps.

Pick by job first, then by product. Versions and features below are as of October 2026.

Neither column is “better”: a shared UI and a central fleet view favour Argo CD; per-cluster autonomy and Kubernetes-native RBAC favour Flux. CNCF’s 2025 survey of Argo CD users found 42% running more than 500 applications per instance and 25% connecting more than 20 clusters^[68]; as a survey of its own users, it says nothing about market share.

Automate the mechanical steps (pull-request checks, development image updates, promotion pull requests, statuses); keep human judgement for high-risk production promotion, pruning protected kinds and break-glass.

AI assistants can draft manifests, summarise rendered diffs or explain why an application is out of sync. An assistant or agent that can push to a watched path is a deployer: give it the same scoped, reviewable access as a CI bot, behind the same review and policy checks. No evidence was found (as of October 2026) that AI-specific tooling improves GitOps outcomes.

Assumptions. Two Kubernetes 1.36 clusters (staging and production), Flux 2.9 installed by bootstrap in each with Flux’s multi-tenancy lockdown, one configuration repository fleet-config on a Git host with branch protection, SOPS with one age key per tenant and environment, and a tenant team “shop” that may change only its own namespace (hostnames are placeholders). An Argo CD equivalent follows.

Example status: layout only (text).

fleet-config/
├── clusters/
│   ├── staging/
│   │   ├── flux-system/          # written by flux bootstrap, plus the lockdown patch
│   │   ├── infrastructure.yaml   # platform add-ons, reconciled as cluster admin
│   │   └── tenants.yaml          # tenant onboarding, depends on infrastructure
│   └── production/               # same files, production paths
├── infrastructure/
│   ├── base/                     # ingress, cert-manager, policy controller
│   ├── staging/
│   └── production/
├── tenants/
│   ├── staging/shop.yaml
│   └── production/shop.yaml      # namespace, service account, Flux objects for "shop"
├── apps/
│   ├── base/shop/                # Deployment, Service, ServiceAccount
│   ├── staging/shop/             # image digest, replicas, encrypted secrets
│   └── production/shop/
└── .sops.yaml                    # one age recipient per tenant and environment

Lock Flux down first. By default a Kustomization that names no service account is applied as cluster-admin, and Flux grants the built-in admin role rights over Flux objects^[25][33]. So a tenant holding admin in its namespace could commit its own Kustomization without serviceAccountName and have Flux apply it as cluster-admin. Flux’s documented lockdown closes this. Such objects fall back to their namespace’s default account, which has no permissions; cross-namespace source references and Kustomize remote bases are refused (principle 2); flux-system keeps the controller’s account^[33].

Example status: YAML syntax-checked; patches as in Flux’s multi-tenancy guide^[33], except that source-watcher is left out of the --default-service-account patch, because the guide says it accepts that flag only from Flux 2.10.0; not applied.

# clusters/production/flux-system/kustomization.yaml: bootstrap + lockdown (Flux 2.9)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-cross-namespace-refs=true
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller|notification-controller)"
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-cross-namespace-refs=true
    target:
      kind: Deployment
      name: "(image-reflector-controller|image-automation-controller|source-watcher)"
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-remote-bases=true
    target:
      kind: Deployment
      name: kustomize-controller
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --default-service-account=default
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller)"   # add source-watcher from Flux 2.10
  - patch: |
      - op: add
        path: /spec/serviceAccountName
        value: kustomize-controller
    target:
      kind: Kustomization
      name: flux-system

The platform’s own Kustomizations install add-ons and tenant namespaces, so they name the controller’s account explicitly:

Example status: YAML syntax-checked by the validator; fields follow the Flux v1 Kustomization API^[25]; not applied to a cluster.

# clusters/production/infrastructure.yaml and tenants.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  serviceAccountName: kustomize-controller   # platform scope: cluster-admin, on purpose
  interval: 30m          # drift check; new commits are picked up when the source changes
  retryInterval: 2m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/production
  prune: true
  wait: true             # health-check everything this applies
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenants
  namespace: flux-system
spec:
  serviceAccountName: kustomize-controller
  dependsOn:
    - name: infrastructure
  interval: 30m
  timeout: 3m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./tenants/production
  prune: true

The tenant file gives “shop” a namespace, a service account that Flux impersonates, and its own source. With the lockdown set, the Kustomization below can do only what shop-reconciler may do in shop^[33][25]. Because that account holds admin, commits under apps/production/shop/ can also create Flux objects in shop; under the lockdown those run only as accounts in shop too.

Example status: YAML syntax-checked; not applied. The trusted-signers, shop-config-auth and sops-age Secrets are created out of band during cluster bootstrap and are never committed.

# tenants/production/shop.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    kustomize.toolkit.fluxcd.io/prune: disabled   # never garbage-collect the namespace
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: shop-reconciler
  namespace: shop
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: shop-reconciler-admin
  namespace: shop
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin                # namespace-scoped through the RoleBinding
subjects:
  - kind: ServiceAccount
    name: shop-reconciler
    namespace: shop
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: shop-config
  namespace: shop
spec:
  interval: 1m
  url: ssh://git@git.example.com/platform/fleet-config.git
  ref:
    branch: main
  secretRef:
    name: shop-config-auth     # read-only deploy key
  verify:
    mode: HEAD                 # the HEAD commit must carry a trusted signature
    secretRef:
      name: trusted-signers    # .asc (PGP) or .sshpub (SSH) public keys
  ignore: |                    # trims the artefact; NOT access control
    /*
    !/apps/production/shop/
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: shop
  namespace: shop
spec:
  serviceAccountName: shop-reconciler
  interval: 10m
  retryInterval: 1m
  timeout: 3m
  sourceRef:
    kind: GitRepository
    name: shop-config
  path: ./apps/production/shop
  targetNamespace: shop
  prune: true
  wait: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Who can read these credentials. ignore only trims the artefact Flux builds. The deploy key in shop-config-auth reads all of fleet-config: in a monorepo every tenant can read every tenant’s configuration, because Git “is not designed to restrict access to certain files”^[47]. Both Secrets sit in namespace shop, and “anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace”^[51]. That covers the shop team, shop-reconciler (admin allows “read/write access to most resources in a namespace”^[69], Pods included), and so any commit merged under apps/production/shop/, plus cluster administrators. So the age key must decrypt nothing but shop’s production Secrets: one key per tenant and environment (see Lab 3), or a per-tenant cloud KMS key reached through Flux’s object-level workload identity (behind a feature gate in Flux 2.9^[25]). If tenants must not read each other’s configuration, use Flux’s repository-per-team layout with a deploy key per tenant repository^[47].

What a signature proves. When merges happen on the Git host, the HEAD of main is a commit the host created. GitHub.com signs web-interface commits with its own key (https://github.com/web-flow.gpg); GitHub Enterprise Server does so only if a site administrator has enabled web commit signing, with the key at https://HOSTNAME/web-flow.gpg. Rebase-and-merge adds commits “without commit signature verification”^[70]. So trusted-signers must include the host’s key where the host signs, or a release bot must sign merges or tags (mode: Tag or TagAndHEAD)^[38]. The signature then proves the change came through the host or the bot; who approved it comes from branch protection.

The same intent in Argo CD uses an AppProject to fence the tenant and an Application with automated sync. Cluster-scoped kinds are allow-listed in a project, so leaving the list out permits none^[27]. Without a deletion finalizer, kubectl delete of the Application does not cascade to the workloads, but argocd app delete cascades by default^[71].

Example status: YAML syntax-checked; fields follow the Argo CD 3.5 documentation^[27][24][37]; not applied.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: shop
  namespace: argocd
spec:
  description: Shop team applications
  sourceRepos:
    - https://git.example.com/platform/fleet-config.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: shop
  syncWindows:
    - kind: deny                 # weekend freeze; on-call may still sync manually
      schedule: "0 18 * * 5"
      timeZone: "Europe/Amsterdam"
      duration: 62h
      applications:
        - "*"
      manualSync: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop-production
  namespace: argocd
spec:
  project: shop
  source:
    repoURL: https://git.example.com/platform/fleet-config.git
    targetRevision: main
    path: apps/production/shop
  destination:
    server: https://kubernetes.default.svc
    namespace: shop
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 3m

Pull-request checks need no cluster credentials. Render every directory that has a kustomization.yaml, including tenants/ and clusters/; run the image guard from Lab 3 on application and infrastructure renders; and refuse cluster-scoped RBAC or CRDs in tenant paths. The lockdown would make such an apply fail; this check makes the pull request fail first.

Example status: bash -n syntax-checked; logic executed on 2026-10-16 in a throw-away repository with a stub kustomize that concatenates the resources each kustomization.yaml lists; not run with real kustomize.

#!/usr/bin/env bash
# ci/validate.sh: run on every pull request (needs kustomize and python3; no cluster)
set -euo pipefail
status=0
rendered="$(mktemp)"
trap 'rm -f "$rendered"' EXIT
mapfile -t dirs < <(find . -name kustomization.yaml -not -path './.git/*' \
  -exec dirname {} \; | sort)
for dir in "${dirs[@]}"; do
  echo "== ${dir}"
  if ! kustomize build "$dir" > "$rendered"; then
    echo "render failed: ${dir}"
    status=1
    continue
  fi
  case "$dir" in
    ./apps/*)
      python3 ci/check_images.py "$rendered" || status=1
      if grep -Eq '^kind: (ClusterRole(Binding)?|CustomResourceDefinition)$' "$rendered"
      then
        echo "cluster-scoped RBAC or CRD in a tenant path: ${dir}"
        status=1
      fi ;;
    ./infrastructure/*) python3 ci/check_images.py "$rendered" || status=1 ;;
  esac
done
exit "$status"

Labs 1–3 were not executed here (no container runtime or Kubernetes tools); their commands follow the cited documentation and pass syntax checks. The Python scripts in Labs 3 and 4 were executed (2026-10-16, Python 3.8.10); their outputs are real.

Objective. See the reconciliation loop: declare an application, change it by hand, watch it come back, then turn self-heal off and see drift persist. Prerequisites. Docker; kind v0.33.0^[72]; kubectl 1.35 or later; about 4 GB of free memory; internet access to GitHub and public registries. Cost and risk. Free and local. The cluster is disposable.

Example status: bash -n and YAML syntax-checked; not executed. Pinned to Argo CD v3.5.5, which is tested on Kubernetes 1.33–1.36 ^[67], hence the 1.36 node image.

1. Create the cluster and install Argo CD with server-side apply, which its CRDs need^[73]. Run these commands.

kind create cluster --name gitops-lab \
  --image kindest/node:v1.36.4@sha256:099e049362a1526b2db71494e1947aae99bd16290d7c895f2b7ea312e3cbfaed
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.5/manifests/install.yaml
kubectl -n argocd rollout status statefulset/argocd-application-controller --timeout=300s
kubectl -n argocd rollout status deployment/argocd-repo-server --timeout=300s

2. Save as guestbook-app.yaml and apply it with kubectl apply -f guestbook-app.yaml. The revision is a commit SHA, so the manifests cannot change under you; the image tag they name (gb-frontend:v5) still can, and the Lab 3 guard would flag it:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: 8088f4c0d970abb09e250248cc97e35623447cb5
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

3. Check status, create drift, and watch it disappear:

kubectl -n argocd get application guestbook \
  -o jsonpath='{.status.sync.status}{" "}{.status.health.status}{"\n"}'
kubectl -n guestbook scale deployment guestbook-ui --replicas=3
sleep 30
kubectl -n guestbook get deployment guestbook-ui -o jsonpath='{.spec.replicas}{"\n"}'
kubectl -n argocd get events --field-selector involvedObject.name=guestbook

4. Turn self-heal off, repeat the scale, and compare:

kubectl -n argocd patch application guestbook --type merge \
  -p '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":false}}}}'
kubectl -n guestbook scale deployment guestbook-ui --replicas=3
sleep 30
kubectl -n argocd get application guestbook -o jsonpath='{.status.sync.status}{"\n"}'

Expected output. After step 2, within a few minutes: Synced Healthy. In step 3 the replica count returns to 1 (self-heal retries after a 5-second default timeout^[24]) and the events list sync operations. In step 4 the count stays at 3 and the status is OutOfSync: Argo CD sees the drift but, with self-heal off, live changes “will not trigger automated sync”^[24]. Validation. You can explain why steps 3 and 4 differ using principle 4. Troubleshooting. CRD errors about annotation size mean --server-side was missing; Unknown sync status usually means the repo-server cannot reach GitHub; ImagePullBackOff means the node cannot reach the registry. Teardown. kind delete cluster --name gitops-lab.

Objective. Reconcile a pinned release with Flux, see a kubectl set image reverted, practise suspend and resume as a break-glass step, and change the desired state. Prerequisites. As Lab 1 (Flux 2.9 supports Kubernetes 1.34–1.36^[65]). Cost and risk. Free and local; this uses Flux’s “dev install”, which Flux documents for testing only^[74].

Example status: bash -n and YAML syntax-checked; not executed.

kind create cluster --name flux-lab \
  --image kindest/node:v1.36.4@sha256:099e049362a1526b2db71494e1947aae99bd16290d7c895f2b7ea312e3cbfaed
kubectl apply -f https://github.com/fluxcd/flux2/releases/download/v2.9.6/install.yaml
kubectl -n flux-system rollout status deployment/source-controller --timeout=180s
kubectl -n flux-system rollout status deployment/kustomize-controller --timeout=180s

Save as podinfo.yaml and apply it. The source is pinned to tag 6.15.0 of the podinfo demo application; its kustomize/ directory holds a Deployment, a Service and a HorizontalPodAutoscaler. Because the autoscaler owns replicas, this lab creates drift on the image instead.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo
  ref:
    tag: "6.15.0"
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 2m
  timeout: 2m
  sourceRef:
    kind: GitRepository
    name: podinfo
  path: ./kustomize
  targetNamespace: default
  prune: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: podinfo
      namespace: default

kubectl apply -f podinfo.yaml
kubectl -n flux-system wait kustomization/podinfo --for=condition=ready --timeout=3m
img() {
  kubectl -n default get deployment podinfo \
    -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
}
reconcile() {
  kubectl -n flux-system annotate --field-manager=flux-client-side-apply --overwrite \
    kustomization/podinfo reconcile.fluxcd.io/requestedAt="$(date +%s)"
}
set_suspend() {  # $1 = true or false
  kubectl -n flux-system patch kustomization podinfo \
    --field-manager=flux-client-side-apply --type merge -p "{\"spec\":{\"suspend\":$1}}"
}
OLD=ghcr.io/stefanprodan/podinfo:6.14.1
img                                                    # 1. desired image
kubectl -n default set image deployment/podinfo podinfod="$OLD"
reconcile; sleep 20; img                               # 2. drift reverted
set_suspend true
kubectl -n default set image deployment/podinfo podinfod="$OLD"
sleep 20; img                                          # 3. break-glass: fix survives
set_suspend false
reconcile; sleep 20; img                               # 4. resume: declared image
kubectl -n flux-system patch gitrepository podinfo --type merge \
  -p '{"spec":{"ref":{"tag":"6.14.1"}}}'
sleep 90; img                                          # 5. new desired state

Expected output. Lines 1, 2 and 4 print ghcr.io/stefanprodan/podinfo:6.15.0; line 3 prints …:6.14.1, because suspension pauses drift correction^[25]; line 5 prints …:6.14.1 again, this time as the declared state. Validation. kubectl -n flux-system get kustomization podinfo shows READY True and the applied revision. In a real setup step 5 is a merged pull request, because the GitRepository itself lives in Git after bootstrap^[13]; GitLabOps shows that loop through GitLab’s agent for Kubernetes. Troubleshooting. kubectl -n flux-system describe kustomization podinfo and kubectl -n flux-system get events show build, apply and health-check errors; a health-check timeout usually means the image could not be pulled. Teardown. kind delete cluster --name flux-lab.

Objective. Build a base-and-overlay layout, encrypt a Secret with SOPS and age, and block mutable images and plain Secrets before merge. Prerequisites. kustomize 5.8, sops 3.13, age 1.3 (current releases in October 2026), Python 3.8 or later. Cost and risk. Free and local, but the .agekey files are private keys: keep them out of Git. If one is ever committed, rotate it and every value it encrypted, because Git history keeps the old ciphertext.

Example status: shell steps bash -n syntax-checked and not executed (tools not installed here); check_images.py executed on 2026-10-16 on the sample described in step 3.

1. Create a base and a production overlay, a variant of the base^[26]: save each document below as the file its first comment names, then copy the two overlay files to apps/staging/shop/ with another count and digest. The base pins a digest too, because the reference implementation’s ci/validate.sh runs the guard on every render under apps/; Kustomize’s images field replaces it by image name, and replicas sets the count^[75].

Example status: YAML syntax-checked; not rendered (kustomize not installed here). The digests are placeholders.

# apps/base/shop/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
images:
  - name: registry.example.com/shop/web
    digest: sha256:1111111111111111111111111111111111111111111111111111111111111111
---
# apps/base/shop/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: registry.example.com/shop/web   # the kustomization pins the digest
          ports:
            - containerPort: 8080
---
# apps/base/shop/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080
---
# apps/production/shop/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop
resources:
  - ../../base/shop
  - secret-db.yaml
replicas:
  - name: web
    count: 3
images:
  - name: registry.example.com/shop/web
    digest: sha256:2222222222222222222222222222222222222222222222222222222222222222
---
# apps/production/shop/secret-db.yaml: plain until step 2 encrypts it
apiVersion: v1
kind: Secret
metadata:
  name: db
type: Opaque
stringData:
  password: lab-placeholder-not-a-real-secret

2. Create one key per environment for the shop tenant and a SOPS rule per path, then encrypt each secret-db.yaml so only the stringData values are encrypted^[53]. A file that matches no rule is refused, so nothing is encrypted to the wrong key:

for env in staging production; do
  age-keygen -o "shop-${env}.agekey"   # private key: outside Git, backed up offline
done
echo "*.agekey" >> .gitignore
pub_stg="$(age-keygen -y shop-staging.agekey)"
pub_prd="$(age-keygen -y shop-production.agekey)"
cat > .sops.yaml <<EOF
creation_rules:                             # one recipient per tenant and environment
  - path_regex: apps/staging/shop/secret.*\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: ${pub_stg}
  - path_regex: apps/production/shop/secret.*\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: ${pub_prd}
EOF
for env in staging production; do
  sops --encrypt --in-place "apps/${env}/shop/secret-db.yaml"
done
grep -c 'ENC\[' apps/production/shop/secret-db.yaml   # expect 1 or more
kustomize build apps/production/shop > rendered-production.yaml

3. Save the guard as ci/check_images.py and run it on your render: it should print 0 finding(s). The executed run below used a sample render that adds the mistakes the guard exists for: beside an encrypted Secret, a plain Secret, a Deployment with one pinned and one unpinned container, and a CronJob on :latest.

"""Pre-merge guard for a GitOps repository: refuse mutable images and unencrypted Secrets.

Usage: kustomize build overlays/production | python3 check_images.py
   or: python3 check_images.py rendered.yaml
Reads rendered Kubernetes YAML (documents separated by ---). Standard library only and
line-based on purpose: it inspects renderer output, not hand-written YAML, so it needs no
YAML parser. Exit code 1 on any finding.
"""
import re
import sys

IMAGE = re.compile(r"^\s*-?\s*image:\s*['\"]?([^'\"\s]+)")
KIND = re.compile(r"^kind:\s*(\S+)")
NAME = re.compile(r"^\s{2}name:\s*(\S+)")


def documents(text):
    """Split a multi-document YAML stream on lines that are exactly '---'."""
    doc = []
    for line in text.splitlines():
        if line.strip() == "---":
            if doc:
                yield doc
            doc = []
        else:
            doc.append(line)
    if doc:
        yield doc


def check(text):
    findings = []
    for lines in documents(text):
        kind = next((m.group(1) for m in map(KIND.match, lines) if m), "?")
        name = next((m.group(1) for m in map(NAME.match, lines) if m), "?")
        where = "%s/%s" % (kind, name)
        if kind == "Secret" and not any(l.startswith("sops:") for l in lines):
            findings.append("%s: Secret is not SOPS-encrypted (no top-level sops: block)"
                            % where)
        for line in lines:
            m = IMAGE.match(line)
            if not m:
                continue
            ref = m.group(1)
            if "@sha256:" in ref:
                continue
            tag = ref.rsplit(":", 1)[1] if ":" in ref.rsplit("/", 1)[-1] else "(none)"
            reason = ("mutable tag 'latest'" if tag == "latest"
                      else "tag %s without digest" % tag)
            findings.append("%s: image %s uses %s" % (where, ref, reason))
    return findings


def main():
    if len(sys.argv) > 1:
        text = open(sys.argv[1], encoding="utf-8").read()
    else:
        text = sys.stdin.read()
    findings = check(text)
    for f in findings:
        print("FAIL " + f)
    print("%d finding(s)" % len(findings))
    return 1 if findings else 0


if __name__ == "__main__":
    sys.exit(main())

Expected output (actual output of python3 check_images.py rendered-production.yaml on the sample; exit code 1):

FAIL Secret/legacy-api-key: Secret is not SOPS-encrypted (no top-level sops: block)
FAIL Deployment/checkout: image registry.example.com/infra/log-shipper:2.4.1 uses tag 2.4.1 without digest
FAIL CronJob/nightly-report: image registry.example.com/shop/report:latest uses mutable tag 'latest'
3 finding(s)

Validation. SOPS_AGE_KEY_FILE=shop-production.agekey sops --decrypt apps/production/shop/secret-db.yaml shows the plain values, and the staging key fails on the same file, so the keys are separated (SOPS reads age keys from SOPS_AGE_KEY_FILE^[76]). On the sample, pinning both images by digest and encrypting the second Secret brings the output to 0 finding(s), exit 0. Troubleshooting. sops reporting “no matching creation rules” means the file path does not match path_regex; a Secret with encrypted metadata will fail in Flux, which supports encrypting only data or stringData^[53]. Teardown. Delete the lab directory and the two practice .agekey files.

Objective. Compute the metrics of this page from one week of events for one cluster; in production, take the inputs from the data sources listed in the metrics table. Prerequisites. Python 3.8 or later. Cost and risk. None.

Example status: executed on 2026-10-16 with Python 3.8.10; JSON and Python syntax-checked by the validator. The data is invented for the exercise.

Save as gitops_events.json:

{
  "cluster": "prod-eu-1",
  "window": {"start": "2026-09-07T00:00:00Z", "end": "2026-09-14T00:00:00Z"},
  "managed_apps": 12,
  "changes": [
    {"pr": 101, "merged_at": "2026-09-07T09:02:00Z",
     "healthy_at": "2026-09-07T09:06:30Z", "outcome": "healthy"},
    {"pr": 102, "merged_at": "2026-09-07T13:40:00Z",
     "healthy_at": "2026-09-07T13:43:10Z", "outcome": "healthy"},
    {"pr": 103, "merged_at": "2026-09-08T10:15:00Z",
     "healthy_at": null, "outcome": "failed", "fixed_by_pr": 104},
    {"pr": 104, "merged_at": "2026-09-08T10:41:00Z",
     "healthy_at": "2026-09-08T10:45:00Z", "outcome": "healthy"},
    {"pr": 105, "merged_at": "2026-09-09T08:55:00Z",
     "healthy_at": "2026-09-09T09:03:40Z", "outcome": "healthy"},
    {"pr": 106, "merged_at": "2026-09-09T16:20:00Z",
     "healthy_at": "2026-09-09T16:22:50Z", "outcome": "healthy"},
    {"pr": 107, "merged_at": "2026-09-10T11:05:00Z",
     "healthy_at": "2026-09-10T11:31:00Z", "outcome": "healthy"},
    {"pr": 108, "merged_at": "2026-09-11T09:30:00Z",
     "healthy_at": "2026-09-11T09:33:20Z", "outcome": "healthy"},
    {"pr": 109, "merged_at": "2026-09-11T15:10:00Z",
     "healthy_at": null, "outcome": "failed", "fixed_by_pr": 110},
    {"pr": 110, "merged_at": "2026-09-11T15:52:00Z",
     "healthy_at": "2026-09-11T15:55:30Z", "outcome": "healthy"}
  ],
  "drift": [
    {"object": "Deployment/checkout", "cause": "human", "ticket": null,
     "detected_at": "2026-09-07T22:10:00Z", "resolved_at": "2026-09-07T22:10:20Z",
     "resolution": "reverted"},
    {"object": "ConfigMap/feature-flags", "cause": "controller", "ticket": null,
     "detected_at": "2026-09-08T03:00:00Z", "resolved_at": "2026-09-08T11:40:00Z",
     "resolution": "ignore-rule-added"},
    {"object": "Deployment/search", "cause": "human", "ticket": "INC-2291",
     "detected_at": "2026-09-09T19:45:00Z", "resolved_at": "2026-09-10T08:30:00Z",
     "resolution": "adopted-into-git"},
    {"object": "Service/payments", "cause": "human", "ticket": null,
     "detected_at": "2026-09-10T02:12:00Z", "resolved_at": "2026-09-10T02:12:30Z",
     "resolution": "reverted"},
    {"object": "HorizontalPodAutoscaler/web", "cause": "human", "ticket": null,
     "detected_at": "2026-09-12T14:00:00Z", "resolved_at": "2026-09-12T14:00:10Z",
     "resolution": "reverted"},
    {"object": "Deployment/search", "cause": "human", "ticket": null,
     "detected_at": "2026-09-13T21:05:00Z", "resolved_at": "2026-09-13T21:05:20Z",
     "resolution": "reverted"}
  ],
  "out_of_sync": [
    {"app": "web", "from": "2026-09-08T03:00:00Z", "to": "2026-09-08T11:40:00Z"},
    {"app": "search", "from": "2026-09-09T19:45:00Z", "to": "2026-09-10T08:30:00Z"},
    {"app": "checkout", "from": "2026-09-08T10:15:00Z", "to": "2026-09-08T10:45:00Z"},
    {"app": "web", "from": "2026-09-11T15:10:00Z", "to": "2026-09-11T15:55:30Z"}
  ]
}

Save as gitops_metrics.py and run python3 gitops_metrics.py gitops_events.json:

"""Compute GitOps operating metrics for ONE cluster from a weekly event log.

Usage: python3 gitops_metrics.py gitops_events.json
Metric definitions are this Atlas's (see the Metrics and KPIs section).
Standard library only; Python 3.8+.
"""
import json
import statistics
import sys
from datetime import datetime


def ts(value):
    """Parse an ISO-8601 UTC timestamp such as 2026-09-07T09:02:00Z."""
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def minutes(start, end):
    return (ts(end) - ts(start)).total_seconds() / 60


def p90(values):
    """Nearest-rank 90th percentile (small samples: report it, do not over-read it)."""
    ordered = sorted(values)
    rank = max(1, -(-9 * len(ordered) // 10))  # ceil(0.9 * n)
    return ordered[rank - 1]


def main(path):
    with open(path, encoding="utf-8") as fh:
        data = json.load(fh)
    start, end = data["window"]["start"], data["window"]["end"]
    window_min = minutes(start, end)

    changes = data["changes"]
    lags = [minutes(c["merged_at"], c["healthy_at"])
            for c in changes if c["outcome"] == "healthy"]
    failed = [c for c in changes if c["outcome"] == "failed"]
    print("Cluster %s, %s to %s" % (data["cluster"], start[:10], end[:10]))
    print("Changes merged: %d, reached healthy: %d, failed: %d"
          % (len(changes), len(lags), len(failed)))
    print("Reconciliation lag (merge -> healthy): "
          "median %.1f min, p90 %.1f min, max %.1f min"
          % (statistics.median(lags), p90(lags), max(lags)))
    print("Apply failure rate: %d/%d = %.0f%%"
          % (len(failed), len(changes), 100.0 * len(failed) / len(changes)))

    drift = data["drift"]
    human = [d for d in drift if d["cause"] == "human"]
    unticketed = [d for d in human if not d["ticket"]]
    resolve = [minutes(d["detected_at"], d["resolved_at"]) for d in drift]
    mix = {}
    for d in drift:
        mix[d["resolution"]] = mix.get(d["resolution"], 0) + 1
    print("Drift events: %d (human %d, controller %d); median time to resolve %.1f min"
          % (len(drift), len(human), len(drift) - len(human), statistics.median(resolve)))
    print("Resolution mix: "
          + ", ".join("%s %d" % (k, v) for k, v in sorted(mix.items())))
    print("Out-of-band human changes: %d, of which without a break-glass ticket: %d"
          % (len(human), len(unticketed)))

    oos = sum(minutes(o["from"], o["to"]) for o in data["out_of_sync"])
    share = 100.0 * oos / (data["managed_apps"] * window_min)
    print("Time out of sync: %.0f app-minutes of %.0f = %.2f%%"
          % (oos, data["managed_apps"] * window_min, share))


if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.exit("usage: python3 gitops_metrics.py gitops_events.json")
    main(sys.argv[1])

Expected output (actual output of the executed run):

Cluster prod-eu-1, 2026-09-07 to 2026-09-14
Changes merged: 10, reached healthy: 8, failed: 2
Reconciliation lag (merge -> healthy): median 3.8 min, p90 26.0 min, max 26.0 min
Apply failure rate: 2/10 = 20%
Drift events: 6 (human 5, controller 1); median time to resolve 0.4 min
Resolution mix: adopted-into-git 1, ignore-rule-added 1, reverted 4
Out-of-band human changes: 5, of which without a break-glass ticket: 4
Time out of sync: 1360 app-minutes of 120960 = 1.12%

Validation. Recompute one number by hand (the worked calculation in Metrics and KPIs does this). Questions. Why does the p90 equal the maximum? Why does a median time-to-resolve of 0.4 minutes hide the most important drift events? Troubleshooting. ValueError from fromisoformat means a timestamp is not ISO-8601; Python older than 3.8 is not supported. Teardown. None.

All five scenarios are illustrative: they combine documented tool behaviour into realistic situations, not reports from named organisations.

GitOps changes how changes reach production, so measure the mechanism here and leave delivery outcomes to DORA’s metrics on DevOps. The definitions below are this Atlas’s; no standard defines GitOps metrics.

Take the Lab 4 week. Eight of ten merged changes became healthy. Their lags, in minutes, sorted: 2.8, 3.2, 3.3, 3.5, 4.0, 4.5, 8.7, 26.0.

• Median lag = (3.5 + 4.0) ÷ 2 = 3.75 min (printed as 3.8).
• p90 by nearest rank = value number ⌈0.9 × 8⌉ = 8 = 26.0 min. With eight samples the p90 is the maximum; report it, but do not set targets on a week of data.
• Apply failure rate = 2 ÷ 10 = 20%. Both failures were fixed by follow-up pull requests merged 26 and 42 minutes later.
• Time out of sync = 520 + 765 + 30 + 45.5 = 1,360.5 app-minutes, over 12 applications × 10,080 minutes = 120,960 app-minutes, which is 1.12%.
• The story behind the numbers. Two drift events account for (520 + 765) ÷ 1,360.5 = 94% of the out-of-sync time: a controller rewriting feature-flags, tolerated only after an ignore rule was reviewed and merged 8 hours 40 minutes later, and the hotfix on search, adopted into Git the next morning. Yet the median time to resolve drift is 0.4 minutes, because self-heal reverted the four other events within seconds. The median says “drift is a non-issue”; the distribution says “two exceptions ran for hours outside Git”. Review the tail, and review the four unticketed human changes.

Targets. No published benchmarks exist for these measures. Measure four weeks, then set targets from your own baseline and from what users of the platform accept (for example, “p90 lag under 10 minutes for application changes”).

Vanity metrics. “Percentage of services on GitOps”, “syncs per day” and “number of Applications” grow with effort, not with control.

Example status: PromQL written for this page and not executed; metric and label names are from the Argo CD and Flux documentation ^[78][80]. The Flux resource metric requires the kube-state-metrics configuration from Flux’s monitoring example.

# Argo CD: applications currently out of sync, per project
count by (project) (argocd_app_info{sync_status="OutOfSync"})

# Argo CD: share of sync operations that ended in Error or Failed over 7 days
sum(increase(argocd_app_sync_total{phase=~"Error|Failed"}[7d]))
  / sum(increase(argocd_app_sync_total[7d]))

# Flux: Kustomizations that are not ready, per namespace
count by (exported_namespace) (
  gotk_resource_info{customresource_kind="Kustomization", ready="False"})

# Flux: p90 reconcile duration by kind over the last hour
histogram_quantile(0.9,
  sum by (le, kind) (rate(gotk_reconcile_duration_seconds_bucket[1h])))

The most useful security question in GitOps is not “is the cluster locked down?” but “who can change what the reconciler reads?”. Each row below is a path to production.

Shrink the reconciler. In Flux, impersonate tenant service accounts under the lockdown shown in the reference implementation^[33]. In Argo CD, use dedicated AppProjects (the default project permits any source, destination and kind^[27]), limit the controller’s write rights by namespace and kind^[36], and evaluate the beta impersonation feature, whose documentation warns that without it a compromised Argo CD gives attackers cluster-admin access^[23].

Keep the renderer narrow. Disable unused config-management tools^[36] and inject no secrets at render time (secrets).

Evidence and retention. Keep the evidence chain for your audit period, with audit-log entries for any out-of-band action^[79]; Argo CD’s guide recommends persisting its Kubernetes Events with an exporter if you need them for longer^[36]. Retention periods, and whether pull-request approval satisfies segregation-of-duties rules, depend on your regulator; map them with ComplianceOps.

Supply-chain controls that act before a change reaches the state store (scanning, signing images, admission policies) belong to DevSecOps; cluster hardening belongs to KubernetesSecOps.

• Reconciler capacity. Argo CD’s repo-server forks Helm or Kustomize per render (--parallelismlimit avoids out-of-memory kills), the application controller can be sharded, and the HA manifests need at least three nodes^[81]. Flux’s cost grows with the number of objects and each interval^[25]; shorter intervals buy faster drift correction with more API and Git traffic.
• Git host and registry load. Every source is polled on its interval; with hundreds of applications, prefer webhooks for speed and longer intervals for drift checks (Flux’s recommended production example uses one minute for the source and an hour for drift detection)^[25].
• Monitoring cost. Exposing Application labels as metrics multiplies series; Argo CD warns about cardinality^[78].
• People cost. Review time is the real price of “every change is a pull request”; automate low-risk promotions.
• Sustainability. This page found no evidence that GitOps changes energy use materially. Carbon and cost optimisation of workloads belong to GreenOps and FinOps.

A GitOps setup can look healthy while quietly failing. Five checks keep it honest:

1. Is the loop running? Alert when a source or Kustomization is not ready, or when an application has not reconciled within several intervals; a dead reconciler leaves clusters stale, not down.
2. Is Git the only path? Review out-of-band human writes from audit logs weekly; every one needs a break-glass record.
3. Are exceptions shrinking? Review every ignore rule and every prune: disabled annotation quarterly; remove the stale ones.
4. Can we rebuild? Run a quarterly drill: new cluster, bootstrap, restore data, measure rebuild time.
5. Does the evidence chain hold? Sample ten production changes a quarter and trace each from pull request to approval to applied revision to healthy status.

flowchart TB
  A["Alert: applications missing or namespace terminating"]:::risk --> B["Step 1: freeze reconciliation for the affected scope"]:::core
  B --> C{"Step 2: data-bearing objects deleted?"}:::decision
  C -- "Yes, in parallel" --> D["Step 4: restore data from backup"]:::risk
  C -- "Yes or no" --> E["Step 3: find the commit and revert it by pull request"]:::core
  E --> G["Step 5: resume reconciliation and watch health"]:::core
  G --> H{"All healthy and data verified?"}:::decision
  D --> H
  H -- "Yes" --> I["Step 6: prune protection, CI deletion check"]:::neutral
  H -- "No" --> R["Keep the freeze; repeat steps 3 and 5"]:::risk
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

1. Freeze (minutes 0–5). Stop the reconciler from acting on the bad revision. A Flux suspension pauses new revisions and drift correction; it holds because suspend is not declared in Git (a declared suspend: false “does not allow for ‘hot patching’” ^[25]). Turning off Argo CD’s automated sync holds only if nothing restores it: a parent application with self-heal, or an ApplicationSet, puts the child’s syncPolicy back, and self-heal then reverts your live fix. Prepare the exemption in advance: in the parent, ignoreDifferences on /spec/syncPolicy/automated with RespectIgnoreDifferences=true^[34]; in an ApplicationSet, ignoreApplicationDifferences on /spec/syncPolicy, documented for “temporarily toggling auto-sync” ^[77]. A deny sync window blocks all syncs without editing the Application^[37], but its AppProject usually lives in Git too.

Example status: bash -n syntax-checked; not executed. Flux names match the reference implementation; the Argo CD application name is illustrative and assumes the exemption above.

# Flux: pause the platform's tenants Kustomization
kubectl -n flux-system patch kustomization tenants \
  --field-manager=flux-client-side-apply --type merge -p '{"spec":{"suspend":true}}'
# Argo CD equivalent: stop automated sync for the application that manages tenants
argocd app set tenants-production --sync-policy none

2. Assess (minutes 5–15). List what was deleted and whether it held data. Reconciler events and Argo CD’s sync history name the revision and the pruned objects; Git shows the change.

argocd app history tenants-production                        # Argo CD: revisions synced
# Flux: what the tenants Kustomization did
kubectl -n flux-system get events --field-selector involvedObject.name=tenants
git log --oneline -5 -- tenants/production
SUSPECT_SHA=abc1234   # placeholder: the commit that dropped payments.yaml
git show --stat "$SUSPECT_SHA" -- tenants/production
kubectl get namespace payments -o jsonpath='{.status.phase}{"\n"}'

3. Restore configuration (minutes 15–30). Revert the offending commit through a pull request (an emergency approval path is fine; skipping Git is not). If the namespace is still Terminating, wait for deletion to finish before resuming, because a terminating namespace “cannot be used for new objects”^[82].

git revert --no-edit "$SUSPECT_SHA"
git push origin HEAD:revert-tenants-prune   # then merge it via the emergency path

4. Restore data (in parallel). Configuration comes back from Git; data does not^[11]. Restore the ledger volume from the latest backup and reconcile any transactions since, following DisasterRecoveryOps.

5. Resume and verify. Resume reconciliation (suspend: false, or re-enable automated sync), then confirm every application is Synced and healthy and the restored data passes application-level checks.

6. Prevent. Mark namespaces and claims as not prunable (Argo CD Prune=false or Prune=confirm^[58]; Flux kustomize.toolkit.fluxcd.io/prune: disabled^[25]); use a Retain reclaim policy for critical data classes^[83]; and add a CI check that renders every directory with a kustomization.yaml (including tenants/, where this incident happened) and fails when a pull request removes a protected kind:

Example status: bash -n syntax-checked; logic executed on 2026-10-16 in a throw-away repository shaped like the reference layout, with a stub kustomize that concatenates the resources each kustomization.yaml lists; not run with real kustomize.

#!/usr/bin/env bash
# ci/deletion-check.sh: fail if a PR removes Namespaces, PVCs or CRDs from any render.
# Usage: ci/deletion-check.sh origin/main   (run from the repo root; needs git, kustomize)
set -euo pipefail
base_ref="${1:-origin/main}"
work="$(mktemp -d)"
trap 'git worktree remove --force "$work/base" >/dev/null 2>&1 || true
      rm -rf "$work"' EXIT
git worktree add --quiet --detach "$work/base" "$base_ref"

dirs() {  # every directory with a kustomization.yaml, tenants/ and clusters/ included
  find . -name kustomization.yaml -not -path './.git/*' -exec dirname {} \;
}
render() {  # $1 = directory, $2 = output; empty if the directory is absent on that side
  : > "$2"
  [ -f "$1/kustomization.yaml" ] || return 0
  kustomize build "$1" \
    | awk '/^kind:/ {k=$2; next} k && /^  name:/ {print k "/" $2; k=""}' | sort -u > "$2"
}

all="$( { (cd "$work/base" && dirs); dirs; } | sort -u)"
status=0
for dir in $all; do
  if ! render "$work/base/$dir" "$work/before" || ! render "$dir" "$work/after"; then
    echo "render failed: ${dir}"   # a broken render must not pass as "nothing removed"
    status=1
    continue
  fi
  removed="$(comm -23 "$work/before" "$work/after" \
    | grep -E '^(Namespace|PersistentVolumeClaim|CustomResourceDefinition)/' || true)"
  if [ -n "$removed" ]; then
    printf 'Protected objects removed from %s:\n%s\n' "$dir" "$removed"
    status=1
  fi
done
exit "$status"

On the test repository, the incident’s change (the payments.yaml line dropped from tenants/production/kustomization.yaml) printed Protected objects removed from ./tenants/production: and Namespace/payments, and exited 1. Removing apps/production/shop/ reported PersistentVolumeClaim/orders-db; a broken render on either side printed render failed and exited 1; no change exited 0. Because a broken base render fails too, a pull request that repairs a broken main needs a code owner’s override. The awk step assumes renderer output with kind before metadata, which kustomize produces; it is not a YAML parser.

Each has a home section above; this list is the short form.

• “We use Git, so we do GitOps.” Without a pulling, reconciling agent it is versioned push deployment (four-question test).
• Branch per environment. Merges drag unrelated changes along^[48]; use directories (layout).
• Mutable references in production. Production changes without a commit^[30]; pin digests and versions, and check in CI.
• Self-heal without break-glass. Responders fight the reconciler, or turn GitOps off for good (judgement call).
• Myth: pull is automatically secure. The repository becomes the credential and the reconciler is highly privileged (push versus pull).

Good fit. The target exposes a declarative API with a reconciler you can run (Kubernetes above all; cloud resources through Kubernetes-hosted controllers). Several people or teams change the same environments. You need evidence of who approved what. Drift has hurt you. You run more than a handful of clusters or environments that should stay alike.

Poor fit, or only partly.

• Imperative or one-off work (schema migrations, data fixes, certificate ceremonies): GitOps assumes repeatable convergence; design these as runbooks, triggered from declared Jobs if you must.
• Objects created by other controllers: operators, autoscalers and admission webhooks own parts of the state. Do not declare what they manage.
• Targets with no agent: SaaS settings or appliances without an API a reconciler can drive. Push from CI with scoped credentials and an audit log, and say honestly that it is push.
• A single small service on one host: the reconciler and review discipline may cost more than they save.
• Teams without review capacity: pull requests that nobody reads add delay without adding control.

Hidden responsibilities. Running and upgrading reconcilers, key management, the exception list, break-glass drills, and on-call for the delivery system itself.

flowchart TB
  S["A component you want to manage"]:::neutral --> Q1{"Configuration exposed through a declarative API?"}:::decision
  Q1 -- "No" --> N1["Do not force it: scripts or push, with an audit log"]:::risk
  Q1 -- "Yes" --> Q2{"Can an agent run near it and reach the store?"}:::decision
  Q2 -- "No" --> N2["Push from CI with scoped credentials, or ship artefacts to a local agent"]:::neutral
  Q2 -- "Yes" --> Q3{"Configuration, not data or one-off actions?"}:::decision
  Q3 -- "No" --> N3["Keep data and migrations out: runbooks and backups"]:::risk
  Q3 -- "Yes" --> Q4{"Can the team review changes and own break-glass?"}:::decision
  Q4 -- "No" --> N4["Start with manual sync and drift alerts"]:::neutral
  Q4 -- "Yes" --> Y["Adopt: automated sync, self-heal and guarded prune"]:::core
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Weighted comparison (illustrative). Scores 1 (poor) to 5 (good) are this Atlas’s judgement for the stated assumptions, not measurements; change the weights to fit your situation.

Assumptions: a multi-team Kubernetes platform with ten clusters, regulated change evidence, and a 24/7 on-call rotation.

For a four-person start-up with one cluster, weights might be 10, 15, 5, 20, 25 and 25. The scores become 3.55, 3.60 and 3.50: a near tie. At that size, choose by what you expect to grow into.

This Atlas’s model (no official GitOps maturity model exists). Move up only when the exit criteria hold for a full quarter.

Quick win: drift alerts alone (manual sync) show hand changes nobody knew about. Dependencies: branch protection, an artefact registry with digests, and an owner for the reconciler on-call.

Quick start

• Write down what is in scope (clusters, namespaces, kinds) and what stays out (data, operator-owned objects)
• Choose a state store and layout; record it in an ADR
• Install the reconciler so it manages itself (bootstrap)
• Render every overlay in CI and block mutable images and plain Secrets
• Turn on drift alerts before automatic correction

Application onboarding preflight

• Images referenced by digest; charts and remote bases pinned
• No replicas for autoscaled workloads; known mutated fields listed as narrow ignore rules
• Secrets follow the chosen pattern; no plain Secret manifests
• Health checks or readiness probes reflect real readiness
• Namespaces and claims carry prune protection
• Owner, code owners and on-call contact recorded

Reconciler production readiness

• Version pinned and within the supported Kubernetes range^[67][65]
• Tenants fenced: AppProjects, not the default project^[27], or the Flux lockdown flags, so no tenant Kustomization or HelmRelease runs as cluster-admin^[33]
• Webhooks configured; intervals chosen deliberately
• Metrics scraped; alerts for not-ready sources and stale reconciliation
• Credential expiry dates tracked; decryption keys backed up outside the cluster
• Upgrade procedure tested in staging

Security review

• Branch protection on watched paths, with no administrator bypass^[35]
• Commit or source verification on production sources^[38][66]
• CI cannot write to production paths without review; bots scoped to one repository and path
• Remote bases refused (Flux --no-remote-bases) or pinned by commit^[33][36]
• Unused config-management tools disabled on the renderer^[36]
• No render-time secret injection^[40]

Incident and break-glass

• Who may suspend reconciliation, for which scope, for how long
• Suspension cannot be undone by Git or a parent (preconditions in runbook step 1)
• Suspension creates a ticket and an alert if it exceeds the time limit
• The fix reaches Git before reconciliation resumes
• Revert-by-pull-request rehearsed; emergency approval path known

Monthly review

• Reconciliation lag, apply failures, drift and out-of-band changes reviewed
• Exception list (ignore rules, prune-disabled objects) reviewed and pruned
• Reconciler and controller versions checked against support windows

1. Architecture decision record: repository and promotion strategy

ADR-NNN: GitOps repository and promotion strategy
Status: proposed | accepted | superseded by ADR-NNN        Date: YYYY-MM-DD
Context: number of clusters, teams, environments; regulatory evidence needs;
  Git host; registry.
Decision:
  State store: Git repository <name> (or OCI artefacts in <registry>)
  Layout: monorepo with apps/, infrastructure/, clusters/ | repo per environment
          | repo per team
  Environments: directories with Kustomize overlays (no branch per environment)
  Promotion: PR that sets the image digest in the next overlay;
             automation allowed up to <environment>
  Reconciler: <Argo CD | Flux> <version>; topology: hub | per cluster | managed
  Tenancy: <AppProjects | impersonated service accounts>; platform paths owned by <team>
  Secrets: <SOPS+age | External Secrets Operator | Sealed Secrets>
  Drift policy: revert by default; tolerated fields listed in <file>
Consequences: what gets easier, what gets harder, what we must operate.
Review date: YYYY-MM-DD

2. Break-glass record

Break-glass record BG-YYYY-NNN
Incident: INC-NNNN            Opened by: <name>            Approved by: <on-call lead>
Scope suspended: <cluster>/<namespace>/<Kustomization or Application>  (smallest possible)
Suspended at: YYYY-MM-DD HH:MM UTC        Time limit: <hours>   Alert if exceeded: yes
Manual changes made (exact commands or diff):
Pull request that brings the change into Git: <link>          Merged at:
Reconciliation resumed at:                 Post-resume state: Synced/Healthy (yes/no)
Follow-up actions:

3. Application onboarding request

Application: <name>        Team: <team>        Code owners: <group>
Environments and clusters: dev / staging / production on <clusters>
Source of manifests: <path in config repo> | <app repo path or chart + version>
Images: registry path; digest pinning confirmed (yes/no)
Secrets: pattern used; keys or secret-manager paths
Autoscaled or operator-managed fields: <list>
Data-bearing objects: <PVCs, databases>; backup owner: <team>
Health: readiness probes; extra smoke test (yes/no)
Promotion: automated up to <environment>; production approval by <group>
Risk class: low | medium | high        On-call contact: <rota>

4. Drift exception record

Exception DX-NNN    Object: <kind>/<namespace>/<name>    Field(s): <JSON pointer>
Why it differs: <controller, webhook, autoscaler, vendor operator>
Mechanism: ignoreDifferences | ssa annotation | omitted from Git | prune disabled
Owner: <team>    Created: YYYY-MM-DD    Review by: YYYY-MM-DD
Removal condition: <what must change for this exception to go away>

GitOps against push CD is covered in push versus pull and the weighted comparison. Three other approaches compete with it:

Do I need Kubernetes to do GitOps? No, but it helps. The principles are tool-neutral^[1]. Outside Kubernetes you need a reconciler for the target: Crossplane or the OpenTofu/Terraform controller for cloud resources, or pull agents such as ansible-pull for hosts, with less mature drift reporting^[21][15][31].

How do I roll back? Revert the commit and let the reconciler apply it. Argo CD’s own rollback cannot be used while automated sync is enabled^[24]. Flux can roll back a failed Helm upgrade automatically when remediation is configured^[59]; when it does, Git still declares the failed version, so follow with a revert so that Git tells the truth.

Can CI still deploy to development? Yes: push or automated image updates in development, pull with review for production. Be explicit about which environments meet which principles.

What about database migrations? Treat them as application concerns: backward-compatible migrations run by the application or by a declared Job with ordering (sync waves or dependsOn)^[63][25]. Never rely on GitOps to undo a migration; reverting a manifest does not revert data.

Does GitOps satisfy a change advisory board? It provides evidence (approvals, applied revisions), but deciding which changes need which approval is change management (ITOps): automate low-risk standard changes and keep human approval where policy demands it.

Status as of October 2026; each topic is introduced earlier on the page.

• Prerequisites. Git and pull requests; Kubernetes objects and kubectl; Kustomize or Helm basics; CI concepts from DevOps. KubernetesOps helps.
• One day. Read the four principles and the glossary^[1][11]; do Lab 1; run the four-question test on your current setup.
• One week. Labs 2 and 3; design a repository layout for two environments; write the drift policy and break-glass procedure for one service.
• Thirty days. Run the first 30 days of the roadmap on a real non-critical service; compute the metrics with Lab 4’s method from your own events.
• Specialisations. Multi-tenancy and reconciler security (DevSecOps, KubernetesSecOps); fleet management (PlatformOps); progressive delivery (ReleaseOps); cloud resources (InfraOps); GitLab wiring (GitLabOps).
• Portfolio projects. A public repository with overlays, SOPS-encrypted secrets, a CI guard and a documented drift policy; a written rebuild drill with measured times.

Option A — keep self-heal on and fix only through an emergency pull request. The invariant holds, the evidence chain stays intact, and nobody has to remember to undo anything. It fails when the path to Git is the bottleneck (approver asleep, Git host degraded) or when the reconciler is part of the problem, for example by re-applying a bad revision.

Option B — suspend reconciliation for the one affected application, patch live, then land the fix in Git before resuming. Service recovers faster. The risk is the forgotten suspension: drift becomes permanent, the next merge silently overwrites the fix, or the application stays outside GitOps for weeks.

Assumptions. The emergency path usually takes about ten minutes (illustrative; measure your own); suspension is possible per application or per object (Flux supports both a suspended Kustomization and a per-object kustomize.toolkit.fluxcd.io/reconcile: disabled annotation^[25]; Argo CD can switch off automation for one application or block syncs with a sync window ^[24][37]); nothing undoes the suspension (the preconditions in runbook step 1); suspension actions are audited.

Why the attractive choice can be wrong. “Never bypass GitOps” sounds principled, but it turns a reliability control into an outage multiplier when approvals or the Git host are slow. “Just patch it” sounds pragmatic, but as a habit it stops Git describing production.

Decision. Choose B, designed in advance: the smallest possible scope, a time limit with an alert (for example four hours), a break-glass record, a fix pull request opened during the incident, and resumption only after it merges. Count the action as an out-of-band change with a ticket in your metrics. Prefer A whenever the emergency path is reliably faster than your recovery target and the Git host is healthy.

Uncertainty. No published data compares recovery times for the two approaches; the answer depends on your approval latency. Measure both in game days and revisit the rule.

After this page you should be able to:

• Explain the four OpenGitOps principles and why “pull” enables continuous reconciliation.
• Classify a delivery setup with the four-question test.
• Design a repository layout, promotion flow, secrets pattern and drift policy for several teams.
• Operate a reconciler safely: least privilege, prune guards, break-glass, metrics.
• Evaluate when GitOps is the wrong tool, and defend the choice to a leader or auditor.

GitOps is a small idea with large consequences: write down what production should be, keep every version, and let a control loop close the gap. Its definition is now stable and vendor-neutral, two mature reconcilers implement it, and both standards guidance and cloud providers treat the pull model as normal.

The idea is also easy to misread. Pull removes cluster credentials from CI, but it turns the repository into the credential and puts a cluster-admin reconciler inside the cluster unless you narrow it. Self-heal stops quiet drift, but it will fight your responders unless you design break-glass in advance. Git recreates configuration, not data. Most GitOps failures are failures of these edges: mutable references, unguarded pruning, forgotten suspensions, stale reconcilers, and guarantees taken without their preconditions.

Adopt it where the target is declarative and many hands change it; start with drift alerts and manual sync; earn automation by measuring lag, failures and out-of-band changes; and say plainly when a system is better served by an honest push pipeline. GitOps is not “using Git”. It is choosing to make the declared state the only path to production, and then doing the work to keep that true.

All sources were opened on 2026-10-16 (those changed in the post-score polish again on 2026-10-17); retrieval notes, quotation sources and the gaps that could not be checked are in the evidence ledger (research/gitops-ledger.md). Version-specific facts are as of October 2026.

Further reading

• opengitops.dev — the principles’ home page and community links.
• flux2-kustomize-helm-example and flux2-multi-tenancy — reference repositories linked from Flux’s repository-structure guide.
• argocd-example-apps — the sample applications used in Lab 1.

Changelog

• 2026-10-16 — First published draft (managing author): research, ledger, 13 parts, 8 diagrams, 4 labs (2 scripts executed). Awaiting independent fact-check.
• 2026-10-16 — Revision after fact-check r1 (reviews/gitops-r1-response.md): Flux lockdown and credential-reach notes; per-tenant, per-environment SOPS keys; CI scripts render tenants/ and clusters/ (re-tested with a stub); ApplicationSet and break-glass preconditions; signing, webhook, attribution and Lab 4 fixes; repetition cut. Argo CD 3.6 was at rc3 on 2026-10-15: refresh the 3.5 lines when it ships.
• 2026-10-17, post-score polish (reviews/gitops-polish.md): GitHub Enterprise Server signing condition; polling range; multi-source self-heal caveat; self-contained Lab 3; runbook figure with step numbers and parallel restore; shorter decision-tree labels; restatements cut. Argo CD 3.6 still rc3 on 2026-10-17.