---
title: "GitOps"
slug: gitops
category: delivery
term_status: established
evidence_status: verified
summary: "GitOps is a way of operating software systems in which the desired state is declared in a versioned, immutable store such as Git, and software agents continuously pull it and reconcile the running system to match."
audience: [Beginners, Platform engineers, Kubernetes operators, Release engineers, Security reviewers, Engineering leaders]
last_verified: 2026-10-17
key_takeaways:
  - "GitOps is defined by four OpenGitOps principles: declarative, versioned and immutable, pulled automatically, continuously reconciled. A CI job that runs kubectl apply meets two of them at most."
  - "Git is the usual state store, not a requirement: an OCI registry or bucket with immutable versions, access control and audit also qualifies."
  - "Pull keeps deploy credentials inside the cluster, but Argo CD and Flux reconcile as cluster-admin by default. Narrow them with projects and impersonated service accounts."
  - "Whoever can change what the reconciler reads can deploy. Protect branches, tags, remote bases and chart sources as production credentials."
  - "Set a drift policy per resource (revert, tolerate or alert) and keep a break-glass path, or self-heal will fight your incident responders."
  - "GitOps recreates configuration, not data. A rebuild from Git still needs backups for databases and volumes."
related: [gitlabops, devops, releaseops, kubernetesops, platformops, infraops, devsecops, api-ops]
---

## What GitOps is

### GitOps in 30 seconds

**GitOps** is a way of operating software systems in which you write down the state you want (declaratively), keep every
version of that declaration in a store that cannot be silently rewritten (usually Git), and let a software agent fetch it and keep
the running system matching it. People change production by changing the declaration through a reviewed pull request. They do not
log in to production and change it by hand.

The mental picture is a thermostat. You set the temperature you want, and a control loop keeps measuring the room and switching the
heating on or off until the two match. In GitOps the setting is a commit, the room is your cluster, and the thermostat is a
reconciler such as Argo CD or Flux. When someone changes the room directly (a `kubectl edit` at 2 a.m.), the loop notices the
difference, called **drift**, and either reverts it or raises the alarm.

**Who this page is for.** Beginners (the core idea and a test for spotting it), engineers (repositories, promotion, secrets, drift),
architects (pull or push, hub or per-cluster agents) and leaders (governance, evidence, risk). Last verified: 2026-10-17.

::: note Term status: established, with loose edges
GitOps has a vendor-neutral definition: the four **OpenGitOps principles**, version 1.0.0, published in October 2021 by the
OpenGitOps project, a CNCF Sandbox project whose mission is "to define a vendor-neutral, principle-led meaning of
GitOps"[^ogo-principles][^ogo-project]. Two independent implementations, Flux and Argo, reached CNCF graduation in 2022
[^cncf-flux-page][^cncf-argo-page], and NIST guidance describes the GitOps workflow model[^nist-204c]: the practice is
**established**. Two caveats. First, governance moved: the GitOps Working Group, chartered under the CNCF App Delivery TAG to
produce a vendor-neutral definition[^wg-charter], is now listed as "Merged into OpenGitOps meetings"[^tag-readme], and CNCF's TAG page
(checked October 2026) lists TAG App Delivery among the legacy TAGs spun down during 2025, mapped to TAG Developer Experience
[^cncf-tags]. Second, marketing uses "GitOps" for almost any Git-triggered pipeline; this page uses the principles as the test.
:::

::: note Overlap with GitLabOps (editorial distinction)
[GitLabOps](../gitlabops/) uses the same vocabulary. **Editorial distinction:** this page teaches GitOps independently of the Git
host and the reconciler; GitLabOps covers operating one product, including GitLab's own recommendation to use Flux with its agent
for Kubernetes[^gitlab-gitops]. The split is this Atlas's choice, not an industry boundary.
:::

### At a glance

| | |
|---|---|
| **Goal** | Production changes only through reviewed, versioned declarations, with drift that is detected and either corrected or justified |
| **Main users** | Platform teams (run the reconcilers), application teams (own their manifests), release engineers, security and audit |
| **Scope** | The four principles, pull versus push, reconciliation and drift, repository layout, promotion, secrets, reconciler security, GitOps beyond Kubernetes |
| **Prerequisites** | Git and pull-request review; declarative configuration (Kubernetes manifests, Kustomize or Helm); CI basics from [DevOps](../devops/) |
| **Key outputs** | A configuration repository per scope, a reconciler per cluster or fleet, a promotion procedure, a drift policy, a break-glass procedure |
| **Term status** | Established (OpenGitOps v1.0.0; two CNCF graduated reconcilers); the label is used loosely by vendors |
| **Linked, not taught here** | Canaries and feature flags ([ReleaseOps](../releaseops/)), IaC fundamentals ([InfraOps](../infraops/)), cluster operations ([KubernetesOps](../kubernetesops/)), GitLab specifics ([GitLabOps](../gitlabops/)), API configuration pipelines ([API Ops](../api-ops/)) |

### The problem GitOps solves

Without GitOps, most teams deploy with a pipeline that ends in a command such as `kubectl apply` or `helm upgrade`, plus occasional
manual fixes. Four problems follow.

1. **Nobody can say what production should look like.** The last pipeline run and every manual edit since then together define it.
   Weaveworks, which coined the term, put it this way in 2017: small ad-hoc changes accumulate until "you can no longer distinguish
   between how the system should be and how it is"[^weave-2017].
2. **Drift is invisible.** A hotfix applied by hand survives until the next deploy silently removes it, or the next deploy fails
   because of it.
3. **CI holds the keys to production.** A push pipeline needs credentials for every environment it deploys to. NIST's DevSecOps
   guidance names this as the security downside of the push model: credentials can be exposed "outside of the deployment
   environment"[^nist-204c].
4. **Rebuilding is archaeology.** After losing a cluster you must work out which pipelines, in which order, with which parameters,
   recreate it.

GitOps answers each with a mechanism, and because the desired state is "sufficient to recreate the system"[^ogo-glossary], a new
cluster can be pointed at the same declarations:

| Before | After |
|---|---|
| "What is running?" means querying the cluster and pipeline logs | The commit the reconciler reports as applied, plus a list of tolerated exceptions |
| Hotfixes by hand, forgotten until the next deploy | Hand edits are reverted or alerted on within minutes, so fixes go through a pull request |
| CI has write credentials for every cluster | CI writes to Git or a registry; the in-cluster agent applies |
| Rollback means re-running an old pipeline | Rollback means reverting a commit |
| Rebuild means reconstructing pipeline history | Rebuild means bootstrapping a reconciler at the same path and restoring data from backups |

**What the evidence does and does not show.** The benefits above are mechanisms, not measured outcomes: no independent study was
found (October 2026) measuring, say, change failure rate before and after adoption. CNCF's 2025 survey reports that 58% of "cloud
native innovators" use GitOps principles extensively, against 23% of "adopters"[^cncf-annual-2026], which is correlation with
organisations mature in other ways, not proof that GitOps causes the maturity.

### What GitOps is not

- **Not "we keep YAML in Git".** Storing manifests in Git satisfies at most the first two principles. If a pipeline pushes them and
  nothing watches the cluster afterwards, there is no reconciliation.
- **Not a replacement for CI.** CI still builds, tests, scans and publishes artefacts. GitOps replaces the *deploy* step: the
  pipeline ends with a commit or a published artefact, not with a write to the cluster[^nist-204c].
- **Not tied to Git.** The principles call for a *state store* with immutable, versioned history, which "should provide access
  control and auditing". Git is "the canonical example", but "any other system that meets these criteria may be used"
  [^ogo-glossary]. Flux calls registry-based delivery "Gitless GitOps"[^flux-concepts].
- **Not Kubernetes-only.** Kubernetes made GitOps practical because its API is declarative and built from control loops
  [^k8s-controller], so most tooling lives there. The same pattern is used for cloud resources through Kubernetes-hosted controllers
  ([Crossplane](#tooling-landscape); tofu-controller, a community OpenTofu/Terraform controller for Flux maintained in the separate
  flux-iac organisation, formerly Weave TF-Controller[^tofu-controller]).
- **Not backup or data recovery.** The OpenGitOps desired state "generally does not include persistent application data, eg. database
  contents"[^ogo-glossary]. See [BackupOps](../backupops/) and [DisasterRecoveryOps](../disasterrecoveryops/).
- **Not a release strategy.** Canaries, blue-green and feature flags decide *how much* traffic sees a change; that belongs to
  [ReleaseOps](../releaseops/). GitOps decides *how* the declared state reaches the cluster.
- **Not infrastructure as code.** IaC is the declarative language and lifecycle ([InfraOps](../infraops/)); GitOps is one operating
  model for applying it continuously.

## Origins, vocabulary and neighbours

### Origins and evolution

| When | Milestone | Why it matters |
|---|---|---|
| 2017-08-07 | Alexis Richardson (Weaveworks) publishes "GitOps - Operations by Pull Request": "“GitOps” is our name for how we use developer tooling to drive operations."[^weave-2017] | Names the practice; see the 2017-versus-2021 note below |
| 2017 | Argo created at Applatix, later acquired by Intuit[^cncf-argo-grad] | The origin of Argo CD |
| 2019-07-15 | Flux, the open-source project behind Weaveworks' tooling[^weave-2017], accepted by CNCF[^cncf-flux-page] | A vendor-neutral home for Weaveworks' GitOps operator |
| 2021 | OpenGitOps becomes a CNCF Sandbox project (2021-01-26)[^cncf-ogo] and releases Principles and Glossary v1.0.0 (2021-10-08)[^ogo-principles] | A vendor-neutral definition; "pulled" and "continuously reconciled" become part of it |
| March 2022 | NIST SP 800-204C describes the pull-based GitOps workflow model[^nist-204c] | A standards body recommends pull for credential reasons |
| 2022-11-30 / 2022-12-06 | Flux and Argo graduate in CNCF[^cncf-flux-page][^cncf-argo-page] | Both reconcilers reach the foundation's top maturity level |
| February 2024 | Weaveworks shuts down; Flux continues under CNCF with maintainers employed elsewhere[^sdtimes-weave][^gitlab-flux-2024] | A foundation project can outlive its sponsor; vendor health still matters |
| 2025 | CNCF restructures its TAGs; TAG App Delivery is spun down[^cncf-tags] | The definition now lives with OpenGitOps |
| 2025-11 | AWS launches a managed Argo CD capability for EKS[^eks-capabilities]; Crossplane graduates in CNCF[^crossplane] | Reconcilers become managed services; declarative cloud control planes mature |
| 2026 | Argo CD 3.5 marks its rendered-manifest and least-privilege sync features as beta[^argo-hydrator][^argo-impersonation] | The frontier moves from "can we reconcile?" to "can we review and constrain what is reconciled?" |

The shift between 2017 and 2021 matters. The 2017 post described "diff tools" that "detect any divergence and notify us via Slack
alerts", with separate sync tools for convergence[^weave-2017]. The 2021 principles require agents that pull on their own and
"attempt to apply the desired state"[^ogo-principles]. Many arguments about whether a setup "is GitOps" come down to which of the
two definitions someone has in mind.

### Glossary and mental model

The mental model is the control loop that Kubernetes itself is built from: controllers "watch the state of your cluster, then make
or request changes where needed"[^k8s-controller]. A GitOps reconciler is one more controller whose desired state comes from outside
the cluster.

::: figure The GitOps reconciliation loop
```mermaid
flowchart TB
  PR["Reviewed pull request merged"]:::neutral --> DS["Desired state: versioned declarations in the state store"]:::core
  DS --> PULL["Agent pulls the declarations"]:::core
  PULL --> OBS["Agent observes the actual state"]:::core
  OBS --> CMP{"Actual matches desired?"}:::decision
  CMP -- "No" --> ACT["Act: apply, prune or roll back"]:::core
  CMP -- "Yes" --> REP["Report: status, events, commit status"]:::neutral
  ACT --> REP
  REP -- "next interval or webhook" --> PULL
  DRIFT["Change made outside the state store"]:::risk -. "creates drift" .-> OBS
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: A merged pull request updates the desired state in the state store. The agent pulls the declarations, observes the
actual state and compares the two. If they differ, it acts by applying, pruning or rolling back, then reports; if they match, it
reports. The loop repeats on the next interval or when a webhook asks for a refresh. A change made outside the state store creates
drift, which the next observation detects.
:::

| Term | Meaning | Example |
|---|---|---|
| Desired state | "The aggregate of all configuration data that is sufficient to recreate the system"[^ogo-glossary] | Manifests for every Deployment, Service, policy and add-on in a cluster |
| Actual (live) state | What the API server currently holds and runs | The Deployment as `kubectl get -o yaml` shows it |
| State store | Keeps immutable versions of the desired state, and "should provide access control and auditing"[^ogo-glossary] | A Git repository with protected branches; an OCI registry |
| Reconciler (agent) | Controller that pulls desired state and drives actual state towards it | Argo CD application controller; Flux kustomize-controller |
| Sync / reconcile | One apply pass from desired to actual state | Argo CD "sync"; a Flux Kustomization run |
| Drift | Actual state "has moved or is in the process of moving away from the desired state"[^ogo-glossary] | A replica count changed by hand |
| Self-heal | Re-applying desired state when the live state drifts, not only on new commits | Argo CD `selfHeal: true`[^argo-autosync] |
| Prune | Deleting live objects that are no longer declared | Argo CD `prune: true`; Flux `.spec.prune`[^flux-kustomization] |
| Health | Whether applied objects became ready, as distinct from "applied" | Deployment rollout complete; Flux `wait`/`healthChecks` |
| Base and overlay | A shared configuration and its per-environment variants[^kustomize-glossary] | `apps/base` plus `apps/production` |
| Kustomization (Flux) vs `kustomization.yaml` | Two things, one word: the Flux object (`kustomize.toolkit.fluxcd.io`) says which path to apply, how often and as whom[^flux-kustomization]; Kustomize's file lists one directory's resources and patches[^kustomize-glossary] | The Flux `tenants` Kustomization applies `tenants/production/`, whose `kustomization.yaml` lists `shop.yaml` |
| Reconciler objects | Argo CD: Application (source to destination), AppProject (the tenant fence), ApplicationSet (generates Applications)[^argo-projects][^argo-appset]. Flux: GitRepository or OCIRepository (sources), Kustomization and HelmRelease (what to apply)[^flux-components] | An ApplicationSet that creates one Application per cluster |
| Promotion | Moving a verified version to the next environment by changing that environment's declaration | A pull request that sets the production image digest |
| Rendered manifests | The final YAML after Helm or Kustomize, committed for review | Argo CD Source Hydrator (beta)[^argo-hydrator] |
| Break-glass | A recorded, time-boxed exception that suspends reconciliation for an emergency change | Suspend one Kustomization, fix, merge the fix, resume |

### How GitOps relates to other Ops

::: figure Where GitOps sits among neighbouring Atlas pages
```mermaid
flowchart LR
  DEV["DevOps: pipeline and culture"]:::neutral
  INF["InfraOps: IaC lifecycle"]:::neutral
  SEC["DevSecOps: signing and policy"]:::neutral
  PLAT["PlatformOps: golden paths"]:::neutral
  G["GitOps: reconcile declared state"]:::core
  K8S["KubernetesOps: the cluster"]:::neutral
  REL["ReleaseOps: canaries, exposure"]:::neutral
  GL["GitLabOps: one platform"]:::neutral
  API["API Ops: API configuration"]:::neutral
  DEV --> G
  INF --> G
  SEC --> G
  PLAT --> G
  G --> K8S
  G --> REL
  G --> GL
  G --> API
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: GitOps sits in the middle. DevOps, InfraOps, DevSecOps and PlatformOps feed it; KubernetesOps, ReleaseOps,
GitLabOps and API Ops use or specialise it. The table below gives each relationship.
:::

| Neighbour | Shared ground | This page's focus | Go there for |
|---|---|---|---|
| [DevOps](../devops/) | Pipelines, trunk-based work, delivery metrics | What happens after CI: pull and reconcile | CI/CD design, DORA's five metrics |
| [InfraOps](../infraops/) | Declarative definitions, drift | Continuous reconciliation by an agent | Writing IaC, state files, modules |
| [KubernetesOps](../kubernetesops/) | The Kubernetes API, controllers | Delivering declared state to clusters | Cluster upgrades, capacity, troubleshooting |
| [ReleaseOps](../releaseops/) | Promotion between environments | The mechanism that moves declarations | Canary, blue-green, feature flags |
| [PlatformOps](../platformops/) | Self-service deployment | The reconciler and repository contract | Running the platform as a product |
| [DevSecOps](../devsecops/) | Supply-chain controls | Who can change what the reconciler reads | Scanning and gate policy |
| [GitLabOps](../gitlabops/) | Flux, merge requests, environments | Host-neutral principles | GitLab agent and Flux wiring |
| [API Ops](../api-ops/) | Declarative config from Git | Kubernetes-centred reconcilers | API gateway and contract pipelines |

## Principles and operating model

### The four principles, read carefully

The OpenGitOps document says the desired state of a GitOps-managed system must be[^ogo-principles]:

1. **Declarative.** "A system managed by GitOps must have its desired state expressed declaratively."
2. **Versioned and Immutable.** "Desired state is stored in a way that enforces immutability, versioning and retains a complete
   version history."
3. **Pulled Automatically.** "Software agents automatically pull the desired state declarations from the source."
4. **Continuously Reconciled.** "Software agents continuously observe actual system state and attempt to apply the desired state."

The glossary carries much of the meaning:

- *Why pull?* Because the agents "must be able to access the desired state from the state store at *any* time, not only when there
  is an intentional change in the state store triggering a push event", so reconciliation can be triggered "whenever there is a
  divergence"[^ogo-glossary].
- *How continuous?* "Continuous" means "reconciliation continues to happen, not that it must be instantaneous"[^ogo-glossary]. A
  three-minute polling interval qualifies.
- *Immutable how?* A branch is a mutable pointer to immutable commits. The principle is broken by references *inside* the
  declarations, such as an image tag like `:latest`, a remote Kustomize base on `HEAD` or an unpinned chart range: Argo CD warns that
  such manifests "change their meaning from one day to the next" without any commit in your repository[^argo-best].
- *Policies are part of the system.* The glossary's "software system" includes "Policies for controlling access and management of
  repositories, deployments, runtimes"[^ogo-glossary].

#### The "Is it GitOps?" test (this Atlas's checklist)

Ask four questions, one per principle:

1. Could you recreate the system's configuration from the declarations alone, apart from persistent data?
2. Is every version immutable, retained, access-controlled and audited, including what the declarations reference (images, charts,
   bases)?
3. Does an agent fetch the declarations by itself, whenever it needs them, without a pipeline pushing them?
4. Does the agent keep comparing actual with desired state, and act or alert when they diverge, not only when a new commit arrives?

| Setup | 1 | 2 | 3 | 4 | Verdict |
|---|---|---|---|---|---|
| CI job runs `kubectl apply -f k8s/` after merge | Yes | Partly (tags often mutable) | No | No | Declarative CD with push, not GitOps |
| Argo CD, manual sync only | Yes | Depends on references | Yes | Observes and reports drift; applies only when a person syncs | Partial: detection without automatic correction |
| Argo CD automated sync, `selfHeal` off | Yes | Depends | Yes | Applies new commits; live edits "will not trigger automated sync"[^argo-autosync] | Partial: reconciles commits, tolerates live drift |
| Argo CD automated sync with `selfHeal`, or a Flux Kustomization | Yes | Depends | Yes | Yes; Flux corrects drift every `.spec.interval`[^flux-kustomization] | GitOps |
| CI job runs `terraform apply` after merge | Yes | Yes if modules and providers are pinned | No | No | IaC with pull-request automation |
| OpenTofu/Terraform controller for Flux | Yes | Yes | Yes | Yes, or drift detection only, by configuration[^tofu-controller] | GitOps for cloud resources |
| `ansible-pull` from cron on each host | Mostly (playbooks can be imperative) | Yes | Yes: it "inverts the default *push* architecture"[^ansible-pull] | Re-applies on schedule; no drift report unless you build one | Close, depending on playbook design |

"Partial" is not an insult: manual sync is a sensible first step and a defensible permanent choice for some production systems.

### This Atlas's proposed charter and invariants

The following charter is **this Atlas's proposal**, not a published standard. Adapt the numbers to your risk.

- **Mission.** Every change to a managed system is declared, reviewed and versioned; the system converges to it automatically; every
  difference between declaration and reality is either corrected or recorded as an exception.
- **Invariant 1 — one path to production.** A production change has a merged commit or a time-boxed break-glass record that is
  closed by a commit.
- **Invariant 2 — no mutable references in production.** Images by digest, charts and remote bases by exact version or commit.
- **Invariant 3 — least-privilege reconcilers.** A reconciler can change only what its owning team may change.
- **Invariant 4 — guarded deletion.** Pruning is enabled, and the objects whose loss is catastrophic (namespaces, persistent
  volumes, CRDs) are protected from it.
- **Invariant 5 — data is not in Git.** Backups and restore tests exist for everything the declarations cannot recreate.
- **Measurable goals.** Reconciliation lag p90 within a target your users accept; zero unticketed out-of-band changes per month;
  a successful rebuild drill each quarter (see [metrics](#metrics-and-kpis)).
- **Non-goals.** Deciding canary percentages, owning CI build quality, or replacing change management where regulation requires it.

### Fundamentals and building blocks

| Building block | Job | Examples (as of October 2026) | What breaks without it |
|---|---|---|---|
| State store | Immutable, versioned, access-controlled declarations | Git with protected branches; OCI registry | No reliable record of intent |
| Artefact registry | Stores images and charts by digest | Any OCI registry | "Version" references that change meaning |
| Renderer | Turns Kustomize or Helm into plain manifests | Argo CD repo-server; Flux kustomize- and helm-controllers | Reviewers approve templates, not what will run |
| Reconciler | Pulls, diffs, applies, prunes, reports health | Argo CD application controller[^argo-arch]; Flux controllers[^flux-components] | No loop: back to push |
| Tenancy and policy | Limits what each team's declarations may touch | Argo CD AppProjects[^argo-projects]; Flux `serviceAccountName` with the lockdown flags[^flux-multitenancy] | Any repository writer can change anything |
| Secrets mechanism | Keeps secret values out of plain Git | SOPS, Sealed Secrets, External Secrets Operator | Credentials in Git history forever |
| Feedback | Tells humans what was applied and what failed | Commit statuses, events, chat alerts, metrics | Silent failure; stale clusters |
| Promotion mechanism | Moves verified versions between environments | Pull requests per environment; image automation; Kargo | Ad-hoc copying between folders |

### Roles, ownership and RACI

Typical roles: the **platform team** runs the reconcilers and owns the repository contract; **application teams** own their manifests
and overlays; **code owners** approve production paths; **security** owns signing, secret keys and policy; **SRE/on-call** handles drift
alerts and break-glass; **change or compliance** consumes the evidence. Small organisations combine roles but keep approvals separate.

R = responsible, A = accountable, C = consulted, I = informed.

| Activity | Platform team | App team | Code owners | Security | On-call / SRE |
|---|---|---|---|---|---|
| Install and upgrade reconcilers | A/R | I | I | C | C |
| Repository layout and tenancy rules | A/R | C | C | C | I |
| Onboard an application | A | R | C | C | I |
| Change an application's staging overlay | I | A/R | I | I | I |
| Promote to production | I | R | A | I | C |
| Rotate secret-encryption keys | R | I | I | A | I |
| Triage a drift alert | C | R | I | I | A |
| Break-glass change | I | C | I | I | A/R |
| Quarterly rebuild drill | A/R | C | I | I | R |

### Lifecycle: Day 0, Day 1, Day 2

- **Day 0 — design.** Choose the state store and repository layout, the reconciler, hub or per-cluster topology, tenancy model,
  secrets pattern, drift policy and the break-glass procedure. Decide what stays out of GitOps (data, operator-generated objects).
- **Day 1 — bootstrap and migrate.** Install the reconciler so that it manages itself: Flux's bootstrap commits its own manifests and
  creates the objects that reconcile them[^flux-concepts]. In Argo CD a parent owns the Applications: an ApplicationSet (Argo CD's
  recommendation) or an "app of apps", which its documentation calls "an admin-only tool", so only admins should push to the parent's
  source[^argo-bootstrap]. A parent with self-heal restores each child's spec, including a `syncPolicy` changed during an incident
  ([break-glass](#worked-runbook-a-refactor-that-prunes-production)); one with the deletion finalizer deletes its children with it.
  Migrate existing workloads one namespace at a time, with automated sync off or prune disabled until the diffs are understood.
- **Day 2 — operate.** Upgrade the reconciler (it is now production-critical), rotate keys, review exceptions, tune intervals and
  webhooks, watch reconciliation lag and failures, and rehearse rebuilds.

### Operating model and governance

- **Approval is a pull-request review.** Use branch protection on the paths reconcilers read: required reviews (with code-owner review
  for production paths), required status checks, optionally required signed commits, and no bypass for administrators
  [^gh-protected]. Other Git hosts offer equivalents.
- **Evidence is a chain, not a commit.** Git shows what was *approved*. Argo CD's documentation notes that this audit log "only applies
  to what happened in Git and does not necessarily correlate one-to-one with events that happen in a cluster"[^argo-security].
  Keep the reconciler's events, which name the revision applied, alongside the pull request.
- **Freeze windows are declarative too.** Argo CD sync windows allow or deny syncs on a cron schedule; deny overrides allow
  [^argo-syncwindows].
- **Break-glass is designed in advance:** scope, time limit, record and the route back to Git ([template](#templates)).
- **Exceptions are visible.** Every `ignoreDifferences` rule or Flux ignore annotation is tolerated drift: record it
  ([template](#templates)) and review it.

## Architecture and workflow

### Reference architecture

The reference architecture below is vendor-neutral. Argo CD packs the source fetcher and renderer into its repo-server and the
applier into its application controller[^argo-arch]; Flux splits them into source-, kustomize- and helm-controllers[^flux-components].

::: figure Reference architecture with trust boundaries
```mermaid
flowchart TB
  subgraph OUTSIDE["Outside the cluster: change and build"]
    ENG["Engineer opens a pull request"]:::neutral
    CI["CI: build, test, scan, sign"]:::neutral
    REV{"Code owners review and merge"}:::decision
  end
  subgraph STORES["State stores"]
    CFG["Config repository with protected paths"]:::core
    REG["OCI registry: images and charts by digest"]:::core
    VAULT["External secret manager"]:::neutral
  end
  subgraph CLUSTER["Inside the cluster trust boundary"]
    SRC["Source fetcher: clone and verify"]:::core
    RND["Renderer: Kustomize or Helm"]:::core
    APL["Applier: diff, apply, prune as a scoped account"]:::core
    KAPI["Kubernetes API server"]:::neutral
    NODES["Nodes: kubelet runs Pods"]:::neutral
    SECOP["Secrets operator"]:::neutral
    NOTIF["Notifier: events, commit status, alerts"]:::neutral
  end
  ENG --> REV
  ENG --> CI
  CI -- "publishes digest" --> REG
  CI -- "check results" --> REV
  REV --> CFG
  CFG -. "pulled" .-> SRC
  SRC --> RND
  RND --> APL
  APL --> KAPI
  KAPI --> NODES
  REG -. "images pulled by kubelet" .-> NODES
  VAULT -. "pulled" .-> SECOP
  SECOP --> KAPI
  APL --> NOTIF
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: Three zones. Outside the cluster, an engineer opens a pull request; CI builds, tests, scans and signs, publishes
images by digest to the registry and reports check results; code owners review and merge. The state stores are the configuration
repository with protected paths, the OCI registry and an external secret manager. Inside the cluster, a source fetcher pulls and
verifies the configuration, a renderer runs Kustomize or Helm, and an applier diffs, applies and prunes through the API server as a
scoped account; nodes run the Pods, and their kubelet pulls images from the registry; a secrets operator pulls values from the secret
manager; a notifier reports events, commit statuses and alerts. Every arrow into the cluster is a pull; an optional webhook (not
drawn) is the one inbound call.
:::

| Component | Trust notes |
|---|---|
| Config repository | Write access to a watched path is deploy access[^argo-security] |
| CI | Registry push and, at most, the right to open pull requests or commit to non-production paths; no cluster credentials |
| Registry | Reference by digest; restrict who can push or move tags |
| Source fetcher | Read-only credentials; Flux can verify Git commit signatures[^flux-gitrepo] |
| Renderer | Argo CD runs `helm template` and `kustomize build`; its security guide warns that malicious charts or kustomizations can read files outside their tree, so treat repository content as untrusted input[^argo-security][^argo-helm] |
| Applier | Cluster-admin by default; narrow it (see [push versus pull](#push-versus-pull-where-the-credentials-live)) |
| Secrets operator | Keeps values out of Git and out of the reconciler[^argo-secrets] |
| Notifier | Flux posts commit statuses to GitHub, GitLab, Bitbucket, Azure DevOps and Gitea[^flux-providers] |

### Push versus pull: where the credentials live

::: figure Push and pull deployment: who holds the cluster credential
```mermaid
flowchart TB
  subgraph PULL["Pull model"]
    direction TB
    Q1["Merge to main"]:::neutral --> Q2["Config repo or registry"]:::core
    Q2 -- "read-only pull" --> Q3["Agent inside the cluster"]:::core
    Q3 -- "in-cluster identity" --> Q4["Kubernetes API: hand edits reverted"]:::core
  end
  subgraph PUSH["Push model"]
    direction TB
    P1["Merge to main"]:::neutral --> P2["CI runner outside the cluster"]:::risk
    P2 -- "uses a stored cluster credential" --> P3["Kubernetes API: hand edits persist unnoticed"]:::risk
  end
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: Two columns, push on the left and pull on the right. In the push model a merge to main triggers a CI runner outside the cluster, which uses a stored cluster
credential to write to the Kubernetes API; hand edits made directly in the cluster persist unnoticed. In the pull model a merge updates
the configuration repository or registry; an agent inside the cluster pulls it with read-only access and applies it with its own
in-cluster identity; hand edits are detected and reverted.
:::

NIST SP 800-204C states the case for pull bluntly: "Since CD tools have the keys to production systems, push-based models are rendered
insecure." With pull, "the deployment environment credentials (e.g., cluster credentials) are not exposed outside of the production
environment"[^nist-204c]. Microsoft's Flux documentation makes the same point: "No access to clusters is needed by the source repo or
by any other cluster"[^aks-flux].

Two counterweights keep this honest:

1. **The agent is powerful.** By default Argo CD's controller uses a cluster-admin-level role[^argo-security], and Flux's controllers
   run under a service account that "has the `cluster-admin` role bound to it. This is convenient for a deployment in which all users
   are trusted"[^flux-multitenancy]. Pull moves the credential inside the boundary; it does not make it small (see
   [security](#security-and-compliance-who-can-really-deploy)).
2. **The repository becomes the credential.** Argo CD's security guide warns that unauthorised write access to a trusted repository
   lets an attacker deploy malicious images or delete resources through pruning[^argo-security]. Pull removes one place where
   production credentials sit; it does not protect what is merged.

A **webhook does not make a setup push-based.** Argo CD polls every two to three minutes by default (120 seconds plus up to 60 seconds
of jitter[^argo-faq]) and accepts webhooks from Git hosts and registries to skip the wait[^argo-webhook]; Flux's `Receiver` does the
same[^flux-receivers]. The webhook only says "look now"; the agent still pulls and applies. The price is an inbound endpoint, usually
an Ingress to the Argo CD API server or Flux's `webhook-receiver` Service. Argo CD's `/api/webhook` "currently lacks rate limiting
protection", so set a shared secret and a payload-size limit; a Flux `Receiver` checks a token. Expose only that path.

### Hub-and-spoke or an agent per cluster

| Topology | How it works | Strengths | Costs and risks |
|---|---|---|---|
| Central instance (hub) pushing to remote clusters | One Argo CD reconciles many clusters; it stores each cluster's credentials as Secrets in its own namespace[^argo-security] | One view and one RBAC model for the fleet; ApplicationSet templates many Applications[^argo-appset] | The hub holds credentials for every cluster and needs network reach to each API server; a hub outage pauses delivery everywhere |
| Agent per cluster | Flux (or Argo CD) installed in each cluster, each pulling its own path | No inbound access to clusters unless webhooks are enabled; "each cluster's control plane is logically separated"[^aks-flux] | N installations to upgrade and monitor; fleet view needs aggregation |
| Hub with pulling agents | Lightweight agents connect out to a central control plane; "Control plane never needs direct cluster access"[^argocd-agent] | Central view without central credentials; suits edge and unreliable networks | argocd-agent is pre-1.0 (v0.10.0, August 2026); check maturity before production |
| Managed reconciler | The EKS Capability for Argo CD runs "in AWS-owned infrastructure separate from your clusters"[^eks-capabilities]; the AKS/Arc Flux extension runs in each cluster, latest three versions supported[^aks-flux] | Fewer upgrades to engineer | Provider feature set and cadence; lock-in |

### End-to-end workflow

::: figure One change from pull request to healthy rollout, with the exception path
```mermaid
sequenceDiagram
  participant E as Engineer
  participant G as Git host and CI
  participant A as Agent
  participant K as Cluster
  E->>G: Open pull request
  G->>G: CI renders, diffs, checks policy
  E->>G: Approve and merge
  G-->>A: Webhook (optional)
  A->>G: Pull new revision
  A->>K: Dry-run apply and diff
  A->>K: Apply and prune
  K-->>A: Rollout status
  alt Healthy
    A-->>G: Commit status: success
  else Unhealthy or failed
    A-->>G: Commit status: failure, alert
    E->>G: Merge a revert
    A->>K: Reconcile previous revision
  end
```
Text equivalent: The engineer opens a pull request; CI renders it, diffs it and checks policy; the engineer approves and merges, and
the Git host may send a webhook. The agent pulls the revision, dry-runs and diffs, applies and prunes, and reads the rollout status. It
then sets a success commit status, or a failure status with an alert, after which the engineer merges a revert and the agent
reconciles the previous revision.
:::

| Step | Trigger | Actor | Decision | System | Output and feedback |
|---|---|---|---|---|---|
| Propose | Need for change | Engineer or bot | What to change, where | Git host | Pull request; CI checks |
| Verify | Pull request opened | CI | Does it render, pass policy, use digests? | CI, renderer | Rendered diff and pass/fail, posted on the pull request |
| Approve | Checks green | Code owners | Is the risk acceptable now? | Git host | Merge commit; audit trail |
| Reconcile | New revision or interval | Agent | Is anything different? | Reconciler, API server | Applied objects; events, metrics |
| Assess | Apply finished | Agent | Did objects become healthy in time? | Health checks | Commit status, alert |
| Recover | Failure or alert | Engineer, on-call | Revert, fix forward or break glass? | Git host | Revert commit or exception record; incident review |

## Practices, patterns and tools

### Core practices: repository layout and environment promotion

**Separate configuration from application source.** Argo CD recommends a separate configuration repository: a replica change should not
trigger a build, the audit history stays readable, access can differ, and CI pushing manifests to the source repository can cause
build loops[^argo-best]. Flux documents four layouts: a **monorepo** (`apps/`, `infrastructure/`, `clusters/`, environments as
directories on one branch), a **repository per environment** when production access must be restricted (Git cannot restrict reads
per file), a **repository per team** and a **repository per app**[^flux-repo-structure].

**Model environments as directories, not branches.** Kustomize overlays exist to create "development, QA, staging and production
environment variants" of a shared base[^kustomize-glossary]. Kostis Kapelonis, writing on a GitOps vendor's blog, calls branch-per-
environment "an anti-pattern": merges carry unintended changes and the wrong order of changes, cherry-picks do not scale, and hand
edits in upstream branches are never back-ported[^kapelonis]. Directories make the difference between staging and production a
reviewable file diff.

**Promote by changing one value in the next environment.** CI records an immutable digest. Promotion is a pull request that sets that
digest in the next overlay. Automation can open the request: Flux image automation scans a registry, picks a tag by policy, updates
manifests and commits, optionally to a separate branch so a human merges it[^flux-image-update]. Kargo, an open-source project from
Akuity, orchestrates multi-stage promotion on top of GitOps[^kargo].

::: figure Promotion of one image digest through three environments
```mermaid
flowchart TB
  B["CI builds the image and records its digest"]:::neutral --> D["Automation updates the dev overlay"]:::core
  D -- "healthy, tests pass" --> SP["Bot opens PR: staging overlay to the same digest"]:::core
  D -. "fails" .-> FIX["Fix forward or revert"]:::risk
  SP -- "checks and review pass" --> PP["PR: production overlay to the same digest"]:::core
  SP -. "fails" .-> FIX
  PP -- "code owners approve, window open" --> REC["Production reconciles; health checked"]:::core
  PP -. "no approval or window closed" .-> WAIT["Wait for the window or escalate"]:::neutral
  REC -. "unhealthy" .-> RV["Revert the production commit"]:::risk
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: CI builds the image and records its digest; automation updates the development overlay. If development is healthy
and tests pass, a bot opens a pull request setting the staging overlay to the same digest. If staging checks and review pass, a pull
request sets the production overlay to the same digest. If code owners approve and the deployment window is open, production
reconciles and health is checked; otherwise the change waits or is escalated. A failure in development or staging leads to a fix
forward or a revert; an unhealthy production rollout leads to reverting the production commit.
:::

**Review what will run, not only the template.** A one-line change to Helm values can change hundreds of rendered lines. Either render in
CI and post the diff on the pull request, or commit rendered manifests. Argo CD's Source Hydrator (beta since 3.5.0) implements the
*rendered manifest pattern* by pushing hydrated manifests to Git before syncing them[^argo-hydrator].

### Secrets practice: three patterns

Never commit plain Kubernetes Secrets: their values are only base64-encoded, and even inside the cluster Secrets are, by default,
"stored unencrypted in the API server's underlying data store (etcd)"[^k8s-secret].

| Pattern | How it works | Strengths | Weaknesses |
|---|---|---|---|
| Encrypted in Git | SOPS "encrypts configuration files while keeping the structure visible"[^sops]; Flux decrypts in-cluster[^flux-sops]. Sealed Secrets can be decrypted "only by the controller running in the target cluster"[^sealed-secrets] | Secrets follow the same review and history as everything else | Key management and rotation; old ciphertext stays in history, so rotate values after key exposure |
| Reference in Git, value elsewhere | External Secrets Operator reads a secret manager and "automatically injects the values into a Kubernetes Secret"[^eso]; `refreshInterval` controls re-reads[^eso-api] | Values never touch Git; rotation happens in the manager | Another dependency at runtime; secret changes bypass pull-request review |
| Injected at render time | A plugin fetches values while the reconciler generates manifests | Familiar to Vault users | Argo CD "strongly caution[s] against" it: the reconciler needs secret access and caches generated manifests in plaintext in Redis[^argo-secrets] |

Argo CD recommends populating secrets on the destination cluster (Sealed Secrets, External Secrets Operator, Secrets Store CSI
Driver) so the reconciler never sees them[^argo-secrets]. For SOPS, Flux recommends age over OpenPGP where possible and encrypting
only `data` and `stringData`[^flux-sops]; who can read the decryption key is covered under the
[reference implementation](#reference-implementation).

### Drift practice: revert, tolerate or alert

"Correct all drift" sounds right and fails in practice, because other controllers legitimately write to the objects you declare.

| Situation | Policy | Mechanism |
|---|---|---|
| Hand edits to Deployments, Services, RBAC | Revert | Argo CD `selfHeal: true`[^argo-autosync]; Flux re-applies every `.spec.interval` and reverts `kubectl edit/patch/delete`[^flux-concepts] |
| `replicas` managed by an autoscaler | Tolerate | Leave `replicas` out of Git[^argo-best]; or Argo CD `ignoreDifferences` with `RespectIgnoreDifferences=true`[^argo-diffing][^argo-syncoptions]; Flux HelmRelease drift-detection ignore rules[^flux-helmrelease] |
| Fields filled in by mutating webhooks or other controllers | Tolerate narrowly | Argo CD diffing customization[^argo-diffing]; Flux `kustomize.toolkit.fluxcd.io/ssa: IfNotPresent` or `Merge`[^flux-kustomization] |
| Namespaces, persistent volume claims, CRDs | Protect from pruning | Argo CD `Prune=false` or `Prune=confirm`[^argo-syncoptions]; Flux `kustomize.toolkit.fluxcd.io/prune: disabled`[^flux-kustomization] |
| Helm releases | Decide explicitly | Flux HelmRelease drift detection is opt-in (`warn` or `enabled`)[^flux-helmrelease]; Argo CD inflates charts with `helm template`, so drift is handled like any manifest[^argo-helm] |
| Emergency change during an incident | Suspend, then reconcile back | Argo CD sync windows or disabling automation; Flux `suspend`[^argo-syncwindows][^flux-kustomization]; preconditions in [runbook step 1](#worked-runbook-a-refactor-that-prunes-production) |

::: note Helm versions are changing under your reconciler (as of October 2026)
Helm 4 (November 2025) installs new releases with server-side apply, while Helm 3 releases keep client-side apply after the
upgrade; Helm 3 gets security fixes until 2027-02-10[^helm-eol][^helm-overview]. Flux supports Helm 4 since v2.8.0 (February
2026)[^flux-release-28]. Check which Helm library your reconciler embeds before comparing drift with your local `helm`.
:::

### Ordering, health and dependencies

Declarations arrive all at once; clusters need order. Custom resource definitions must exist before custom resources, and a controller
must be ready before objects that depend on its webhooks. Flux expresses order between Kustomizations with `dependsOn` and gates on
readiness with `wait` or `healthChecks`[^flux-kustomization]. Argo CD orders within one sync by phases (PreSync, Sync, PostSync) and by
`argocd.argoproj.io/sync-wave` annotations, and prunes in reverse wave order[^argo-syncwaves]. "Healthy" means Kubernetes reports the
objects ready, not that the service is correct; add a smoke test (an Argo CD PostSync hook, or a Job checked by Flux health checks) when
readiness probes are weak.

### Methods, standards and frameworks

| Source | Type | What it requires or recommends | Binding? |
|---|---|---|---|
| OpenGitOps Principles and Glossary v1.0.0[^ogo-principles][^ogo-glossary] | Community specification (CNCF Sandbox) | The four principles; meaning of desired state, state store, drift, pull | Voluntary; the reference for "is it GitOps" |
| NIST SP 800-204C[^nist-204c] | US federal guidance | Pull-based GitOps workflow for microservices delivery; operators limited by platform authorisation | Guidance, not regulation |

No law or certification standard requires GitOps. Regulated organisations can use its pull-request and reconciliation evidence for
their own change-control requirements; mapping those belongs to [ComplianceOps](../complianceops/).

### Tooling landscape

Pick by job first, then by product. Versions and features below are as of October 2026.

| Job | Representative tools (verified) | Notes |
|---|---|---|
| Kubernetes reconciler | Argo CD (CNCF graduated; v3.5.5 on 2026-10-12)[^cncf-argo-page][^argo-releases]; Flux (CNCF graduated; v2.9.6 on 2026-10-01)[^cncf-flux-page][^flux-release-29] | Both satisfy all four principles when automated sync and drift correction are on |
| Managed or bundled reconciler | Flux extension for AKS and Azure Arc[^aks-flux]; EKS Capability for Argo CD[^eks-capabilities]; GitLab agent with Flux[^gitlab-gitops] | Fewer upgrades to run; feature set set by the provider |
| Multi-cluster templating | Argo CD ApplicationSet[^argo-appset]; Flux per-cluster paths[^flux-repo-structure] | ApplicationSet deletions cascade by default; see [failures](#common-failures-and-troubleshooting) |
| Promotion | Flux image automation[^flux-image-update]; Kargo[^kargo] | Automate lower environments; keep a review step where risk warrants |
| Secrets | SOPS (CNCF Sandbox)[^sops]; Sealed Secrets[^sealed-secrets]; External Secrets Operator[^eso] | See the patterns table above |
| Cloud and infrastructure resources | Crossplane (CNCF graduated 2025-11-06)[^crossplane]; tofu-controller, a community controller for Flux (flux-iac)[^tofu-controller] | GitOps beyond Kubernetes workloads; IaC fundamentals are on [InfraOps](../infraops/) |
| Progressive delivery | Flagger (Flux project)[^flux-concepts]; Argo Rollouts (Argo project)[^cncf-argo-grad] | Strategy belongs to [ReleaseOps](../releaseops/) |

### Argo CD and Flux compared on documented behaviour

| Criterion (as of October 2026) | Argo CD 3.5 | Flux 2.9 |
|---|---|---|
| Shape | API server, repo-server and application controller; web UI and CLI consume the API[^argo-arch] | Separate controllers for sources, Kustomize, Helm, notifications and image automation[^flux-components] |
| Sources | Git, Helm repositories and OCI, with webhook refresh[^argo-webhook] | Git, OCI, Helm repositories and S3 buckets[^flux-components][^flux-concepts] |
| Helm | Charts inflated with `helm template`; Argo CD owns the lifecycle, so `helm ls` shows nothing[^argo-helm] | Helm releases with history, opt-in drift detection, and configurable remediation of failed upgrades (strategy `rollback` by default)[^flux-helmrelease] |
| Drift correction | Optional `selfHeal`; automated sync and pruning are off until enabled[^argo-autosync] | Every Kustomization re-applies on its interval; prune is a required field[^flux-kustomization] |
| Tenancy | AppProjects restrict sources, destinations and kinds; syncs run with control-plane privileges unless the beta impersonation feature is enabled[^argo-projects][^argo-impersonation] | Kubernetes RBAC through impersonated service accounts; a fence only with the lockdown flags `--default-service-account` and `--no-cross-namespace-refs`[^flux-multitenancy][^flux-kustomization] |
| Source verification | Source Integrity in AppProjects: GnuPG signatures on Git sources only, not Helm or OCI; the legacy `signatureKeys` setting is deprecated[^argo-source-integrity] | GitRepository verifies PGP or SSH commit and tag signatures[^flux-gitrepo] |
| Multi-cluster | Typically one hub with stored cluster credentials[^argo-security] | Typically one installation per cluster[^aks-flux] |
| Tested Kubernetes versions | 1.33 to 1.36[^argo-tested] | 1.34 to 1.36[^flux-release-29] |

Neither column is "better": a shared UI and a central fleet view favour Argo CD; per-cluster autonomy and Kubernetes-native RBAC
favour Flux. CNCF's 2025 survey of Argo CD users found 42% running more than 500 applications per instance and 25% connecting more
than 20 clusters[^cncf-argo-survey]; as a survey of its own users, it says nothing about market share.

### Automation and AI

Automate the mechanical steps (pull-request checks, development image updates, promotion pull requests, statuses); keep human
judgement for high-risk production promotion, pruning protected kinds and break-glass.

AI assistants can draft manifests, summarise rendered diffs or explain why an application is out of sync. An assistant or agent that
can push to a watched path *is a deployer*: give it the same scoped, reviewable access as a CI bot, behind the same review and
policy checks. No evidence was found (as of October 2026) that AI-specific tooling improves GitOps outcomes.

## GitOps in practice

### Reference implementation

**Assumptions.** Two Kubernetes 1.36 clusters (staging and production), Flux 2.9 installed by bootstrap in each **with Flux's
multi-tenancy lockdown**, one configuration repository `fleet-config` on a Git host with branch protection, SOPS with one age key per
tenant and environment, and a tenant team "shop" that may change only its own namespace (hostnames are placeholders). An Argo CD
equivalent follows.

_Example status: layout only (text)._

```text
fleet-config/
├── clusters/
│   ├── staging/
│   │   ├── flux-system/          # written by flux bootstrap, plus the lockdown patch
│   │   ├── infrastructure.yaml   # platform add-ons, reconciled as cluster admin
│   │   └── tenants.yaml          # tenant onboarding, depends on infrastructure
│   └── production/               # same files, production paths
├── infrastructure/
│   ├── base/                     # ingress, cert-manager, policy controller
│   ├── staging/
│   └── production/
├── tenants/
│   ├── staging/shop.yaml
│   └── production/shop.yaml      # namespace, service account, Flux objects for "shop"
├── apps/
│   ├── base/shop/                # Deployment, Service, ServiceAccount
│   ├── staging/shop/             # image digest, replicas, encrypted secrets
│   └── production/shop/
└── .sops.yaml                    # one age recipient per tenant and environment
```

**Lock Flux down first.** By default a Kustomization that names no service account is applied as cluster-admin, and Flux grants the
built-in `admin` role rights over Flux objects[^flux-kustomization][^flux-multitenancy]. So a tenant holding `admin` in its namespace
could commit its own Kustomization without `serviceAccountName` and have Flux apply it as cluster-admin. Flux's documented lockdown
closes this. Such objects fall back to their namespace's `default` account, which has no permissions; cross-namespace source
references and Kustomize remote bases are refused (principle 2); `flux-system` keeps the controller's account[^flux-multitenancy].

_Example status: YAML syntax-checked; patches as in Flux's multi-tenancy guide[^flux-multitenancy], except that `source-watcher` is
left out of the `--default-service-account` patch, because the guide says it accepts that flag only from Flux 2.10.0; not applied._

```yaml
# clusters/production/flux-system/kustomization.yaml: bootstrap + lockdown (Flux 2.9)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-cross-namespace-refs=true
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller|notification-controller)"
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-cross-namespace-refs=true
    target:
      kind: Deployment
      name: "(image-reflector-controller|image-automation-controller|source-watcher)"
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-remote-bases=true
    target:
      kind: Deployment
      name: kustomize-controller
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --default-service-account=default
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller)"   # add source-watcher from Flux 2.10
  - patch: |
      - op: add
        path: /spec/serviceAccountName
        value: kustomize-controller
    target:
      kind: Kustomization
      name: flux-system
```

The platform's own Kustomizations install add-ons and tenant namespaces, so they name the controller's account explicitly:

_Example status: YAML syntax-checked by the validator; fields follow the Flux v1 Kustomization API[^flux-kustomization]; not applied to a cluster._

```yaml
# clusters/production/infrastructure.yaml and tenants.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  serviceAccountName: kustomize-controller   # platform scope: cluster-admin, on purpose
  interval: 30m          # drift check; new commits are picked up when the source changes
  retryInterval: 2m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/production
  prune: true
  wait: true             # health-check everything this applies
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenants
  namespace: flux-system
spec:
  serviceAccountName: kustomize-controller
  dependsOn:
    - name: infrastructure
  interval: 30m
  timeout: 3m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./tenants/production
  prune: true
```

The tenant file gives "shop" a namespace, a service account that Flux impersonates, and its own source. **With the lockdown set**, the
Kustomization below can do only what `shop-reconciler` may do in `shop`[^flux-multitenancy][^flux-kustomization]. Because that
account holds `admin`, commits under `apps/production/shop/` can also create Flux objects in `shop`; under the lockdown those run
only as accounts in `shop` too.

_Example status: YAML syntax-checked; not applied. The `trusted-signers`, `shop-config-auth` and `sops-age` Secrets are created out of
band during cluster bootstrap and are never committed._

```yaml
# tenants/production/shop.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    kustomize.toolkit.fluxcd.io/prune: disabled   # never garbage-collect the namespace
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: shop-reconciler
  namespace: shop
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: shop-reconciler-admin
  namespace: shop
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin                # namespace-scoped through the RoleBinding
subjects:
  - kind: ServiceAccount
    name: shop-reconciler
    namespace: shop
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: shop-config
  namespace: shop
spec:
  interval: 1m
  url: ssh://git@git.example.com/platform/fleet-config.git
  ref:
    branch: main
  secretRef:
    name: shop-config-auth     # read-only deploy key
  verify:
    mode: HEAD                 # the HEAD commit must carry a trusted signature
    secretRef:
      name: trusted-signers    # .asc (PGP) or .sshpub (SSH) public keys
  ignore: |                    # trims the artefact; NOT access control
    /*
    !/apps/production/shop/
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: shop
  namespace: shop
spec:
  serviceAccountName: shop-reconciler
  interval: 10m
  retryInterval: 1m
  timeout: 3m
  sourceRef:
    kind: GitRepository
    name: shop-config
  path: ./apps/production/shop
  targetNamespace: shop
  prune: true
  wait: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age
```

**Who can read these credentials.** `ignore` only trims the artefact Flux builds. The deploy key in `shop-config-auth` reads all of
`fleet-config`: in a monorepo every tenant can read every tenant's configuration, because Git "is not designed to restrict access
to certain files"[^flux-repo-structure]. Both Secrets sit in namespace `shop`, and "anyone who is authorized to create a Pod in a
namespace can use that access to read any Secret in that namespace"[^k8s-secret]. That covers the shop team, `shop-reconciler`
(`admin` allows "read/write access to most resources in a namespace"[^k8s-rbac], Pods included), and so any commit merged under
`apps/production/shop/`, plus cluster administrators. So the age key must decrypt nothing but shop's production Secrets: one key per
tenant and environment (see [Lab 3](#hands-on-labs)), or a per-tenant cloud KMS key reached through Flux's object-level workload
identity (behind a feature gate in Flux 2.9[^flux-kustomization]). If tenants must not read each other's configuration, use Flux's
repository-per-team layout with a deploy key per tenant repository[^flux-repo-structure].

**What a signature proves.** When merges happen on the Git host, the HEAD of `main` is a commit the host created. GitHub.com signs
web-interface commits with its own key (`https://github.com/web-flow.gpg`); GitHub Enterprise Server does so only if a site
administrator has enabled web commit signing, with the key at `https://HOSTNAME/web-flow.gpg`. Rebase-and-merge adds commits
"without commit signature verification"[^gh-sig]. So `trusted-signers` must include the host's key where the host signs, or a release
bot must sign merges or tags (`mode: Tag` or `TagAndHEAD`)[^flux-gitrepo]. The signature then proves the change came through the host or the bot; who approved
it comes from branch protection.

The same intent in Argo CD uses an AppProject to fence the tenant and an Application with automated sync. Cluster-scoped kinds are
allow-listed in a project, so leaving the list out permits none[^argo-projects]. Without a deletion finalizer, `kubectl delete` of
the Application does not cascade to the workloads, but `argocd app delete` cascades by default[^argo-deletion].

_Example status: YAML syntax-checked; fields follow the Argo CD 3.5 documentation[^argo-projects][^argo-autosync][^argo-syncwindows]; not applied._

```yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: shop
  namespace: argocd
spec:
  description: Shop team applications
  sourceRepos:
    - https://git.example.com/platform/fleet-config.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: shop
  syncWindows:
    - kind: deny                 # weekend freeze; on-call may still sync manually
      schedule: "0 18 * * 5"
      timeZone: "Europe/Amsterdam"
      duration: 62h
      applications:
        - "*"
      manualSync: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop-production
  namespace: argocd
spec:
  project: shop
  source:
    repoURL: https://git.example.com/platform/fleet-config.git
    targetRevision: main
    path: apps/production/shop
  destination:
    server: https://kubernetes.default.svc
    namespace: shop
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 3m
```

Pull-request checks need no cluster credentials. Render every directory that has a `kustomization.yaml`, including `tenants/` and
`clusters/`; run the image guard from [Lab 3](#hands-on-labs) on application and infrastructure renders; and refuse cluster-scoped
RBAC or CRDs in tenant paths. The lockdown would make such an apply fail; this check makes the pull request fail first.

_Example status: `bash -n` syntax-checked; logic **executed** on 2026-10-16 in a throw-away repository with a stub `kustomize` that
concatenates the resources each `kustomization.yaml` lists; not run with real kustomize._

```bash
#!/usr/bin/env bash
# ci/validate.sh: run on every pull request (needs kustomize and python3; no cluster)
set -euo pipefail
status=0
rendered="$(mktemp)"
trap 'rm -f "$rendered"' EXIT
mapfile -t dirs < <(find . -name kustomization.yaml -not -path './.git/*' \
  -exec dirname {} \; | sort)
for dir in "${dirs[@]}"; do
  echo "== ${dir}"
  if ! kustomize build "$dir" > "$rendered"; then
    echo "render failed: ${dir}"
    status=1
    continue
  fi
  case "$dir" in
    ./apps/*)
      python3 ci/check_images.py "$rendered" || status=1
      if grep -Eq '^kind: (ClusterRole(Binding)?|CustomResourceDefinition)$' "$rendered"
      then
        echo "cluster-scoped RBAC or CRD in a tenant path: ${dir}"
        status=1
      fi ;;
    ./infrastructure/*) python3 ci/check_images.py "$rendered" || status=1 ;;
  esac
done
exit "$status"
```

### Hands-on labs

**Labs 1–3 were not executed** here (no container runtime or Kubernetes tools); their commands follow the cited documentation and
pass syntax checks. The Python scripts in Labs 3 and 4 **were executed** (2026-10-16, Python 3.8.10); their outputs are real.

#### Lab 1 (beginner): watch Argo CD revert drift

**Objective.** See the reconciliation loop: declare an application, change it by hand, watch it come back, then turn self-heal off and
see drift persist. **Prerequisites.** Docker; kind v0.33.0[^kind-release]; kubectl 1.35 or later; about 4 GB of free memory; internet
access to GitHub and public registries. **Cost and risk.** Free and local. The cluster is disposable.

_Example status: `bash -n` and YAML syntax-checked; not executed. Pinned to Argo CD v3.5.5, which is tested on Kubernetes 1.33–1.36
[^argo-tested], hence the 1.36 node image._

1. Create the cluster and install Argo CD with server-side apply, which its CRDs need[^argo-getting-started]. Run these commands.

```bash
kind create cluster --name gitops-lab \
  --image kindest/node:v1.36.4@sha256:099e049362a1526b2db71494e1947aae99bd16290d7c895f2b7ea312e3cbfaed
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.5/manifests/install.yaml
kubectl -n argocd rollout status statefulset/argocd-application-controller --timeout=300s
kubectl -n argocd rollout status deployment/argocd-repo-server --timeout=300s
```

2. Save as `guestbook-app.yaml` and apply it with `kubectl apply -f guestbook-app.yaml`. The revision is a commit SHA, so the
   manifests cannot change under you; the image tag they name (`gb-frontend:v5`) still can, and the Lab 3 guard would flag it:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: 8088f4c0d970abb09e250248cc97e35623447cb5
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
```

3. Check status, create drift, and watch it disappear:

```bash
kubectl -n argocd get application guestbook \
  -o jsonpath='{.status.sync.status}{" "}{.status.health.status}{"\n"}'
kubectl -n guestbook scale deployment guestbook-ui --replicas=3
sleep 30
kubectl -n guestbook get deployment guestbook-ui -o jsonpath='{.spec.replicas}{"\n"}'
kubectl -n argocd get events --field-selector involvedObject.name=guestbook
```

4. Turn self-heal off, repeat the scale, and compare:

```bash
kubectl -n argocd patch application guestbook --type merge \
  -p '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":false}}}}'
kubectl -n guestbook scale deployment guestbook-ui --replicas=3
sleep 30
kubectl -n argocd get application guestbook -o jsonpath='{.status.sync.status}{"\n"}'
```

**Expected output.** After step 2, within a few minutes: `Synced Healthy`. In step 3 the replica count returns to `1` (self-heal retries
after a 5-second default timeout[^argo-autosync]) and the events list sync operations. In step 4 the count stays at `3` and the status is
`OutOfSync`: Argo CD sees the drift but, with self-heal off, live changes "will not trigger automated sync"[^argo-autosync].
**Validation.** You can explain why steps 3 and 4 differ using principle 4. **Troubleshooting.** CRD errors about annotation size mean
`--server-side` was missing; `Unknown` sync status usually means the repo-server cannot reach GitHub; `ImagePullBackOff` means the
node cannot reach the registry. **Teardown.** `kind delete cluster --name gitops-lab`.

#### Lab 2 (intermediate): Flux reconciliation, break-glass and a new revision

**Objective.** Reconcile a pinned release with Flux, see a `kubectl set image` reverted, practise suspend and resume as a break-glass
step, and change the desired state. **Prerequisites.** As Lab 1 (Flux 2.9 supports Kubernetes 1.34–1.36[^flux-release-29]). **Cost and
risk.** Free and local; this uses Flux's "dev install", which Flux documents for testing only[^flux-install].

_Example status: `bash -n` and YAML syntax-checked; not executed._

```bash
kind create cluster --name flux-lab \
  --image kindest/node:v1.36.4@sha256:099e049362a1526b2db71494e1947aae99bd16290d7c895f2b7ea312e3cbfaed
kubectl apply -f https://github.com/fluxcd/flux2/releases/download/v2.9.6/install.yaml
kubectl -n flux-system rollout status deployment/source-controller --timeout=180s
kubectl -n flux-system rollout status deployment/kustomize-controller --timeout=180s
```

Save as `podinfo.yaml` and apply it. The source is pinned to tag `6.15.0` of the podinfo demo application; its `kustomize/` directory
holds a Deployment, a Service and a HorizontalPodAutoscaler. Because the autoscaler owns `replicas`, this lab creates drift on the image
instead.

```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo
  ref:
    tag: "6.15.0"
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 2m
  timeout: 2m
  sourceRef:
    kind: GitRepository
    name: podinfo
  path: ./kustomize
  targetNamespace: default
  prune: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: podinfo
      namespace: default
```

```bash
kubectl apply -f podinfo.yaml
kubectl -n flux-system wait kustomization/podinfo --for=condition=ready --timeout=3m
img() {
  kubectl -n default get deployment podinfo \
    -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
}
reconcile() {
  kubectl -n flux-system annotate --field-manager=flux-client-side-apply --overwrite \
    kustomization/podinfo reconcile.fluxcd.io/requestedAt="$(date +%s)"
}
set_suspend() {  # $1 = true or false
  kubectl -n flux-system patch kustomization podinfo \
    --field-manager=flux-client-side-apply --type merge -p "{\"spec\":{\"suspend\":$1}}"
}
OLD=ghcr.io/stefanprodan/podinfo:6.14.1
img                                                    # 1. desired image
kubectl -n default set image deployment/podinfo podinfod="$OLD"
reconcile; sleep 20; img                               # 2. drift reverted
set_suspend true
kubectl -n default set image deployment/podinfo podinfod="$OLD"
sleep 20; img                                          # 3. break-glass: fix survives
set_suspend false
reconcile; sleep 20; img                               # 4. resume: declared image
kubectl -n flux-system patch gitrepository podinfo --type merge \
  -p '{"spec":{"ref":{"tag":"6.14.1"}}}'
sleep 90; img                                          # 5. new desired state
```

**Expected output.** Lines 1, 2 and 4 print `ghcr.io/stefanprodan/podinfo:6.15.0`; line 3 prints `…:6.14.1`, because suspension pauses
drift correction[^flux-kustomization]; line 5 prints `…:6.14.1` again, this time as the declared state. **Validation.** `kubectl -n flux-system get kustomization
podinfo` shows `READY True` and the applied revision. In a real setup step 5 is a merged pull request, because the GitRepository
itself lives in Git after bootstrap[^flux-concepts]; [GitLabOps](../gitlabops/) shows that loop through GitLab's agent for Kubernetes.
**Troubleshooting.** `kubectl -n flux-system describe kustomization podinfo` and `kubectl -n flux-system get events` show build, apply
and health-check errors; a health-check timeout usually means the image could not be pulled.
**Teardown.** `kind delete cluster --name flux-lab`.

#### Lab 3 (intermediate, no cluster): environments, an encrypted secret and a merge guard

**Objective.** Build a base-and-overlay layout, encrypt a Secret with SOPS and age, and block mutable images and plain Secrets before
merge. **Prerequisites.** kustomize 5.8, sops 3.13, age 1.3 (current releases in October 2026), Python 3.8 or later. **Cost and
risk.** Free and local, but the `.agekey` files are private keys: keep them out of Git. If one is ever committed, rotate it and every
value it encrypted, because Git history keeps the old ciphertext.

_Example status: shell steps `bash -n` syntax-checked and not executed (tools not installed here); `check_images.py` **executed** on
2026-10-16 on the sample described in step 3._

1. Create a base and a production overlay, a variant of the base[^kustomize-glossary]: save each document below as the file its first
   comment names, then copy the two overlay files to `apps/staging/shop/` with another `count` and digest. The base pins a digest
   too, because the reference implementation's `ci/validate.sh` runs the guard on every render under `apps/`; Kustomize's
   `images` field replaces it by image name, and `replicas` sets the count[^kustomize-fields].

_Example status: YAML syntax-checked; not rendered (kustomize not installed here). The digests are placeholders._

```yaml
# apps/base/shop/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
images:
  - name: registry.example.com/shop/web
    digest: sha256:1111111111111111111111111111111111111111111111111111111111111111
---
# apps/base/shop/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: registry.example.com/shop/web   # the kustomization pins the digest
          ports:
            - containerPort: 8080
---
# apps/base/shop/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080
---
# apps/production/shop/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop
resources:
  - ../../base/shop
  - secret-db.yaml
replicas:
  - name: web
    count: 3
images:
  - name: registry.example.com/shop/web
    digest: sha256:2222222222222222222222222222222222222222222222222222222222222222
---
# apps/production/shop/secret-db.yaml: plain until step 2 encrypts it
apiVersion: v1
kind: Secret
metadata:
  name: db
type: Opaque
stringData:
  password: lab-placeholder-not-a-real-secret
```

2. Create one key per environment for the shop tenant and a SOPS rule per path, then encrypt each `secret-db.yaml` so only the
   `stringData` values are encrypted[^flux-sops]. A file that matches no rule is refused, so nothing is encrypted to the wrong key:

```bash
for env in staging production; do
  age-keygen -o "shop-${env}.agekey"   # private key: outside Git, backed up offline
done
echo "*.agekey" >> .gitignore
pub_stg="$(age-keygen -y shop-staging.agekey)"
pub_prd="$(age-keygen -y shop-production.agekey)"
cat > .sops.yaml <<EOF
creation_rules:                             # one recipient per tenant and environment
  - path_regex: apps/staging/shop/secret.*\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: ${pub_stg}
  - path_regex: apps/production/shop/secret.*\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: ${pub_prd}
EOF
for env in staging production; do
  sops --encrypt --in-place "apps/${env}/shop/secret-db.yaml"
done
grep -c 'ENC\[' apps/production/shop/secret-db.yaml   # expect 1 or more
kustomize build apps/production/shop > rendered-production.yaml
```

3. Save the guard as `ci/check_images.py` and run it on your render: it should print `0 finding(s)`. The executed run below used a
   sample render that adds the mistakes the guard exists for: beside an encrypted Secret, a plain Secret, a Deployment with one pinned
   and one unpinned container, and a CronJob on `:latest`.

```python
"""Pre-merge guard for a GitOps repository: refuse mutable images and unencrypted Secrets.

Usage: kustomize build overlays/production | python3 check_images.py
   or: python3 check_images.py rendered.yaml
Reads rendered Kubernetes YAML (documents separated by ---). Standard library only and
line-based on purpose: it inspects renderer output, not hand-written YAML, so it needs no
YAML parser. Exit code 1 on any finding.
"""
import re
import sys

IMAGE = re.compile(r"^\s*-?\s*image:\s*['\"]?([^'\"\s]+)")
KIND = re.compile(r"^kind:\s*(\S+)")
NAME = re.compile(r"^\s{2}name:\s*(\S+)")


def documents(text):
    """Split a multi-document YAML stream on lines that are exactly '---'."""
    doc = []
    for line in text.splitlines():
        if line.strip() == "---":
            if doc:
                yield doc
            doc = []
        else:
            doc.append(line)
    if doc:
        yield doc


def check(text):
    findings = []
    for lines in documents(text):
        kind = next((m.group(1) for m in map(KIND.match, lines) if m), "?")
        name = next((m.group(1) for m in map(NAME.match, lines) if m), "?")
        where = "%s/%s" % (kind, name)
        if kind == "Secret" and not any(l.startswith("sops:") for l in lines):
            findings.append("%s: Secret is not SOPS-encrypted (no top-level sops: block)"
                            % where)
        for line in lines:
            m = IMAGE.match(line)
            if not m:
                continue
            ref = m.group(1)
            if "@sha256:" in ref:
                continue
            tag = ref.rsplit(":", 1)[1] if ":" in ref.rsplit("/", 1)[-1] else "(none)"
            reason = ("mutable tag 'latest'" if tag == "latest"
                      else "tag %s without digest" % tag)
            findings.append("%s: image %s uses %s" % (where, ref, reason))
    return findings


def main():
    if len(sys.argv) > 1:
        text = open(sys.argv[1], encoding="utf-8").read()
    else:
        text = sys.stdin.read()
    findings = check(text)
    for f in findings:
        print("FAIL " + f)
    print("%d finding(s)" % len(findings))
    return 1 if findings else 0


if __name__ == "__main__":
    sys.exit(main())
```

**Expected output** (actual output of `python3 check_images.py rendered-production.yaml` on the sample; exit code 1):

```text
FAIL Secret/legacy-api-key: Secret is not SOPS-encrypted (no top-level sops: block)
FAIL Deployment/checkout: image registry.example.com/infra/log-shipper:2.4.1 uses tag 2.4.1 without digest
FAIL CronJob/nightly-report: image registry.example.com/shop/report:latest uses mutable tag 'latest'
3 finding(s)
```

**Validation.** `SOPS_AGE_KEY_FILE=shop-production.agekey sops --decrypt apps/production/shop/secret-db.yaml` shows the plain values, and
the staging key fails on the same file, so the keys are separated (SOPS reads age keys from `SOPS_AGE_KEY_FILE`[^sops-age]). On the sample, pinning both images by digest and encrypting the second Secret brings the output to `0 finding(s)`, exit 0. **Troubleshooting.**
`sops` reporting "no matching creation rules" means the file path does not match `path_regex`; a Secret with encrypted `metadata` will
fail in Flux, which supports encrypting only `data` or `stringData`[^flux-sops]. **Teardown.** Delete the lab directory and the two
practice `.agekey` files.

#### Lab 4 (advanced, executed): measure a week of GitOps operations

**Objective.** Compute the [metrics](#metrics-and-kpis) of this page from one week of events for one cluster; in production, take the
inputs from the data sources listed in the metrics table. **Prerequisites.** Python 3.8 or later. **Cost and risk.** None.

_Example status: **executed** on 2026-10-16 with Python 3.8.10; JSON and Python syntax-checked by the validator. The data is invented
for the exercise._

Save as `gitops_events.json`:

```json
{
  "cluster": "prod-eu-1",
  "window": {"start": "2026-09-07T00:00:00Z", "end": "2026-09-14T00:00:00Z"},
  "managed_apps": 12,
  "changes": [
    {"pr": 101, "merged_at": "2026-09-07T09:02:00Z",
     "healthy_at": "2026-09-07T09:06:30Z", "outcome": "healthy"},
    {"pr": 102, "merged_at": "2026-09-07T13:40:00Z",
     "healthy_at": "2026-09-07T13:43:10Z", "outcome": "healthy"},
    {"pr": 103, "merged_at": "2026-09-08T10:15:00Z",
     "healthy_at": null, "outcome": "failed", "fixed_by_pr": 104},
    {"pr": 104, "merged_at": "2026-09-08T10:41:00Z",
     "healthy_at": "2026-09-08T10:45:00Z", "outcome": "healthy"},
    {"pr": 105, "merged_at": "2026-09-09T08:55:00Z",
     "healthy_at": "2026-09-09T09:03:40Z", "outcome": "healthy"},
    {"pr": 106, "merged_at": "2026-09-09T16:20:00Z",
     "healthy_at": "2026-09-09T16:22:50Z", "outcome": "healthy"},
    {"pr": 107, "merged_at": "2026-09-10T11:05:00Z",
     "healthy_at": "2026-09-10T11:31:00Z", "outcome": "healthy"},
    {"pr": 108, "merged_at": "2026-09-11T09:30:00Z",
     "healthy_at": "2026-09-11T09:33:20Z", "outcome": "healthy"},
    {"pr": 109, "merged_at": "2026-09-11T15:10:00Z",
     "healthy_at": null, "outcome": "failed", "fixed_by_pr": 110},
    {"pr": 110, "merged_at": "2026-09-11T15:52:00Z",
     "healthy_at": "2026-09-11T15:55:30Z", "outcome": "healthy"}
  ],
  "drift": [
    {"object": "Deployment/checkout", "cause": "human", "ticket": null,
     "detected_at": "2026-09-07T22:10:00Z", "resolved_at": "2026-09-07T22:10:20Z",
     "resolution": "reverted"},
    {"object": "ConfigMap/feature-flags", "cause": "controller", "ticket": null,
     "detected_at": "2026-09-08T03:00:00Z", "resolved_at": "2026-09-08T11:40:00Z",
     "resolution": "ignore-rule-added"},
    {"object": "Deployment/search", "cause": "human", "ticket": "INC-2291",
     "detected_at": "2026-09-09T19:45:00Z", "resolved_at": "2026-09-10T08:30:00Z",
     "resolution": "adopted-into-git"},
    {"object": "Service/payments", "cause": "human", "ticket": null,
     "detected_at": "2026-09-10T02:12:00Z", "resolved_at": "2026-09-10T02:12:30Z",
     "resolution": "reverted"},
    {"object": "HorizontalPodAutoscaler/web", "cause": "human", "ticket": null,
     "detected_at": "2026-09-12T14:00:00Z", "resolved_at": "2026-09-12T14:00:10Z",
     "resolution": "reverted"},
    {"object": "Deployment/search", "cause": "human", "ticket": null,
     "detected_at": "2026-09-13T21:05:00Z", "resolved_at": "2026-09-13T21:05:20Z",
     "resolution": "reverted"}
  ],
  "out_of_sync": [
    {"app": "web", "from": "2026-09-08T03:00:00Z", "to": "2026-09-08T11:40:00Z"},
    {"app": "search", "from": "2026-09-09T19:45:00Z", "to": "2026-09-10T08:30:00Z"},
    {"app": "checkout", "from": "2026-09-08T10:15:00Z", "to": "2026-09-08T10:45:00Z"},
    {"app": "web", "from": "2026-09-11T15:10:00Z", "to": "2026-09-11T15:55:30Z"}
  ]
}
```

Save as `gitops_metrics.py` and run `python3 gitops_metrics.py gitops_events.json`:

```python
"""Compute GitOps operating metrics for ONE cluster from a weekly event log.

Usage: python3 gitops_metrics.py gitops_events.json
Metric definitions are this Atlas's (see the Metrics and KPIs section).
Standard library only; Python 3.8+.
"""
import json
import statistics
import sys
from datetime import datetime


def ts(value):
    """Parse an ISO-8601 UTC timestamp such as 2026-09-07T09:02:00Z."""
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def minutes(start, end):
    return (ts(end) - ts(start)).total_seconds() / 60


def p90(values):
    """Nearest-rank 90th percentile (small samples: report it, do not over-read it)."""
    ordered = sorted(values)
    rank = max(1, -(-9 * len(ordered) // 10))  # ceil(0.9 * n)
    return ordered[rank - 1]


def main(path):
    with open(path, encoding="utf-8") as fh:
        data = json.load(fh)
    start, end = data["window"]["start"], data["window"]["end"]
    window_min = minutes(start, end)

    changes = data["changes"]
    lags = [minutes(c["merged_at"], c["healthy_at"])
            for c in changes if c["outcome"] == "healthy"]
    failed = [c for c in changes if c["outcome"] == "failed"]
    print("Cluster %s, %s to %s" % (data["cluster"], start[:10], end[:10]))
    print("Changes merged: %d, reached healthy: %d, failed: %d"
          % (len(changes), len(lags), len(failed)))
    print("Reconciliation lag (merge -> healthy): "
          "median %.1f min, p90 %.1f min, max %.1f min"
          % (statistics.median(lags), p90(lags), max(lags)))
    print("Apply failure rate: %d/%d = %.0f%%"
          % (len(failed), len(changes), 100.0 * len(failed) / len(changes)))

    drift = data["drift"]
    human = [d for d in drift if d["cause"] == "human"]
    unticketed = [d for d in human if not d["ticket"]]
    resolve = [minutes(d["detected_at"], d["resolved_at"]) for d in drift]
    mix = {}
    for d in drift:
        mix[d["resolution"]] = mix.get(d["resolution"], 0) + 1
    print("Drift events: %d (human %d, controller %d); median time to resolve %.1f min"
          % (len(drift), len(human), len(drift) - len(human), statistics.median(resolve)))
    print("Resolution mix: "
          + ", ".join("%s %d" % (k, v) for k, v in sorted(mix.items())))
    print("Out-of-band human changes: %d, of which without a break-glass ticket: %d"
          % (len(human), len(unticketed)))

    oos = sum(minutes(o["from"], o["to"]) for o in data["out_of_sync"])
    share = 100.0 * oos / (data["managed_apps"] * window_min)
    print("Time out of sync: %.0f app-minutes of %.0f = %.2f%%"
          % (oos, data["managed_apps"] * window_min, share))


if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.exit("usage: python3 gitops_metrics.py gitops_events.json")
    main(sys.argv[1])
```

**Expected output** (actual output of the executed run):

```text
Cluster prod-eu-1, 2026-09-07 to 2026-09-14
Changes merged: 10, reached healthy: 8, failed: 2
Reconciliation lag (merge -> healthy): median 3.8 min, p90 26.0 min, max 26.0 min
Apply failure rate: 2/10 = 20%
Drift events: 6 (human 5, controller 1); median time to resolve 0.4 min
Resolution mix: adopted-into-git 1, ignore-rule-added 1, reverted 4
Out-of-band human changes: 5, of which without a break-glass ticket: 4
Time out of sync: 1360 app-minutes of 120960 = 1.12%
```

**Validation.** Recompute one number by hand (the worked calculation in [Metrics and KPIs](#metrics-and-kpis) does this). **Questions.** Why does the p90 equal the maximum? Why does a median time-to-resolve of 0.4 minutes hide the most important drift events?
**Troubleshooting.** `ValueError` from `fromisoformat` means a timestamp is not ISO-8601; Python older than 3.8 is not supported.
**Teardown.** None.

### Use cases and scenarios

All five scenarios are **illustrative**: they combine documented tool behaviour into realistic situations, not reports from named
organisations.

| Scenario | Context and problem | Method | Trade-offs and pitfalls | Success measure |
|---|---|---|---|---|
| **1. Platform add-ons across a fleet** | A platform team runs 40 clusters; ingress, certificate and policy controllers differ by cluster | Shared `infrastructure/base` with per-cluster paths (Flux) or an ApplicationSet cluster generator (Argo CD)[^flux-repo-structure][^argo-appset]; two canary clusters first | One bad base change reaches every cluster; ApplicationSet deletions cascade by default (guards and their preconditions under [failures](#common-failures-and-troubleshooting))[^argo-appset-deletion] | Number of distinct add-on versions across the fleet; time to roll a security patch to all clusters |
| **2. Change evidence in a regulated firm** | Auditors sample production changes: who approved, what applied, when | Protected production paths with code-owner review and signed commits, the host's key trusted[^gh-protected]; reconciler events kept with pull requests; sync windows for freezes[^argo-syncwindows] | Merge and apply are separate events[^argo-security]; break-glass must be just as traceable | Share of sampled changes with a complete evidence chain; unticketed out-of-band changes per month |
| **3. Rebuild a cluster or region** | A cluster is lost or a new region is needed | Bootstrap a reconciler at the same path, restore data from backups, reconcile in dependency order (`dependsOn`, sync waves)[^flux-kustomization][^argo-syncwaves] | Desired state excludes data[^ogo-glossary]; decryption keys and bootstrap credentials must be recoverable outside the cluster | Rebuild time from empty cluster to all applications healthy, measured in a quarterly drill |
| **4. Disconnected edge sites** | Hundreds of shops with outbound-only, unreliable links | One agent per site pulling small, signed OCI artefacts ("Gitless GitOps")[^flux-concepts][^flux-components]; site overlays only for true differences | Large artefacts over thin links; overlay sprawl; a hub cannot reach sites behind NAT | Share of sites on the current revision within 24 hours; sites stale for more than 7 days |
| **5. Cloud resources beside the workload** | Teams click databases and buckets into existence; production differs from staging | Declare cloud resources with Crossplane[^crossplane] or tofu-controller, starting in drift-detection-only mode[^tofu-controller] | A field change can force replacement of a stateful resource; keep deletion protection and approval for destructive plans | Cloud resources created outside Git per month; drift detected versus corrected |

## Measure, secure and pay

### Metrics and KPIs

GitOps changes *how* changes reach production, so measure the mechanism here and leave delivery outcomes to DORA's metrics on
[DevOps](../devops/). The definitions below are **this Atlas's**; no standard defines GitOps metrics.

| KPI | Definition and formula | Data source | Pitfall | Review |
|---|---|---|---|---|
| Reconciliation lag | Time from merge (or artefact publication) to every object healthy at that revision; report median and p90 per cluster | Merge time from the Git host; applied revision and health from reconciler status or events | Polling and health timeouts dominate; webhooks cut polling[^argo-webhook][^flux-receivers] | Weekly |
| Apply failure rate | Merged changes that did not reach healthy without a further commit ÷ merged changes | Commit statuses, reconciler events, `argocd_app_sync_total{phase}`[^argo-metrics] | Automatic retries hide flapping; count per revision, not per attempt | Weekly |
| Drift rate | Drift events per managed application per week, split by cause (human, controller) and resolution (reverted, adopted into Git, ignore rule added) | Self-heal events, drift alerts | Ignore rules hide drift from the count; review the rule list too | Weekly |
| Out-of-band changes | Write requests by human identities to managed objects; target zero without a break-glass ticket | Kubernetes audit logs (who, what, when, on which object)[^k8s-audit] | Shared admin accounts make "who" useless | Weekly, monthly trend |
| Time out of sync | Σ out-of-sync durations ÷ (managed applications × window) | `argocd_app_info{sync_status}`[^argo-metrics]; Flux `gotk_resource_info{ready}`[^flux-metrics] | Suspended applications may report ready; count suspension separately | Monthly |
| Rebuild time | Time from an empty cluster to all applications healthy in a drill, including data restore | Drill records | A drill without data restore measures the easy half | Quarterly |
| Reconciler health | Reconcile duration, errors, controller restarts, source fetch failures | `gotk_reconcile_duration_seconds`[^flux-metrics]; `argocd_app_reconcile`[^argo-metrics] | A stopped reconciler causes stale clusters, not outages, so nobody notices | Alert continuously |

#### Worked calculation

Take the [Lab 4](#hands-on-labs) week. Eight of ten merged changes became healthy. Their lags, in minutes, sorted: 2.8, 3.2, 3.3,
3.5, 4.0, 4.5, 8.7, 26.0.

- **Median lag** = (3.5 + 4.0) ÷ 2 = **3.75 min** (printed as 3.8).
- **p90** by nearest rank = value number ⌈0.9 × 8⌉ = 8 = **26.0 min**. With eight samples the p90 *is* the maximum; report it, but
  do not set targets on a week of data.
- **Apply failure rate** = 2 ÷ 10 = **20%**. Both failures were fixed by follow-up pull requests merged 26 and 42 minutes later.
- **Time out of sync** = 520 + 765 + 30 + 45.5 = 1,360.5 app-minutes, over 12 applications × 10,080 minutes = 120,960
  app-minutes, which is **1.12%**.
- **The story behind the numbers.** Two drift events account for (520 + 765) ÷ 1,360.5 = **94%** of the out-of-sync time: a
  controller rewriting `feature-flags`, tolerated only after an ignore rule was reviewed and merged 8 hours 40 minutes later, and the
  hotfix on `search`, adopted into Git the next morning. Yet the median time to resolve drift is 0.4 minutes, because self-heal
  reverted the four other events within seconds. The median says "drift is a non-issue"; the distribution says "two exceptions ran
  for hours outside Git". Review the tail, and review the four unticketed human changes.

**Targets.** No published benchmarks exist for these measures. Measure four weeks, then set targets from your own baseline and from
what users of the platform accept (for example, "p90 lag under 10 minutes for application changes").

**Vanity metrics.** "Percentage of services on GitOps", "syncs per day" and "number of Applications" grow with effort, not with control.

#### Queries

_Example status: PromQL written for this page and not executed; metric and label names are from the Argo CD and Flux documentation
[^argo-metrics][^flux-metrics]. The Flux resource metric requires the kube-state-metrics configuration from Flux's monitoring example._

```promql
# Argo CD: applications currently out of sync, per project
count by (project) (argocd_app_info{sync_status="OutOfSync"})

# Argo CD: share of sync operations that ended in Error or Failed over 7 days
sum(increase(argocd_app_sync_total{phase=~"Error|Failed"}[7d]))
  / sum(increase(argocd_app_sync_total[7d]))

# Flux: Kustomizations that are not ready, per namespace
count by (exported_namespace) (
  gotk_resource_info{customresource_kind="Kustomization", ready="False"})

# Flux: p90 reconcile duration by kind over the last hour
histogram_quantile(0.9,
  sum by (le, kind) (rate(gotk_reconcile_duration_seconds_bucket[1h])))
```

### Security and compliance: who can really deploy

The most useful security question in GitOps is not "is the cluster locked down?" but "who can change what the reconciler reads?".
Each row below is a path to production.

| What the reconciler reads or trusts | Who can change it | Control |
|---|---|---|
| The watched branch and paths | Anyone who can push or merge | Required reviews with code owners, required checks, signed commits, no administrator bypass[^gh-protected]; Flux commit verification[^flux-gitrepo]; Argo CD source integrity[^argo-source-integrity]. A signature shows who merged, not who approved ([what a signature proves](#reference-implementation)) |
| The reconciler's own objects (Applications, AppProjects, Kustomizations, GitRepositories) | Whoever can edit them in Git or in the cluster | Keep them in a platform-owned path; restrict Kubernetes RBAC on these kinds |
| Images, charts and remote bases referenced by the manifests | Registry pushers and upstream maintainers | Digests and exact versions; Argo CD's repository allow-list "only restricts the initial repository which is cloned", not remote bases or chart dependencies[^argo-security] |
| Secret-decryption keys | Key holders, and anyone who can create Pods where the key Secret lives[^k8s-secret] | One key per tenant and environment (this Atlas's recommendation; Flux's guide scopes a `.sops.yaml` to a cluster or namespace directory[^flux-sops]); cloud KMS or offline age keys; rotation |
| Cluster credentials held by a central instance | Hub administrators | Narrow the `argocd-manager` role on each cluster; Argo CD needs cluster-wide read but not cluster-wide write[^argo-security] |
| Suspend, sync and rollback permissions | Users with reconciler roles or UI access | SSO groups and project roles[^argo-projects]; audit these actions like deployments |

**Shrink the reconciler.** In Flux, impersonate tenant service accounts under the lockdown shown in the
[reference implementation](#reference-implementation)[^flux-multitenancy]. In Argo CD, use
dedicated AppProjects (the `default` project permits any source, destination and kind[^argo-projects]), limit the controller's write
rights by namespace and kind[^argo-security], and evaluate the beta impersonation feature, whose documentation warns that without it a
compromised Argo CD gives attackers `cluster-admin` access[^argo-impersonation].

**Keep the renderer narrow.** Disable unused config-management tools[^argo-security] and inject no secrets at render time
([secrets](#secrets-practice-three-patterns)).

**Evidence and retention.** Keep the [evidence chain](#operating-model-and-governance) for your audit period, with audit-log
entries for any out-of-band action[^k8s-audit]; Argo CD's guide recommends persisting its Kubernetes Events with an exporter if you
need them for longer[^argo-security]. Retention periods, and whether
pull-request approval satisfies segregation-of-duties rules, depend on your regulator; map them with [ComplianceOps](../complianceops/).

Supply-chain controls that act *before* a change reaches the state store (scanning, signing images, admission policies) belong to
[DevSecOps](../devsecops/); cluster hardening belongs to [KubernetesSecOps](../kubernetessecops/).

### Cost, capacity and sustainability

- **Reconciler capacity.** Argo CD's repo-server forks Helm or Kustomize per render (`--parallelismlimit` avoids out-of-memory
  kills), the application controller can be sharded, and the HA manifests need at least three nodes[^argo-ha]. Flux's cost grows with
  the number of objects and each `interval`[^flux-kustomization]; shorter intervals buy faster drift correction with more API and Git
  traffic.
- **Git host and registry load.** Every source is polled on its interval; with hundreds of applications, prefer webhooks for speed and
  longer intervals for drift checks (Flux's recommended production example uses one minute for the source and an hour for drift
  detection)[^flux-kustomization].
- **Monitoring cost.** Exposing Application labels as metrics multiplies series; Argo CD warns about cardinality[^argo-metrics].
- **People cost.** Review time is the real price of "every change is a pull request"; automate low-risk promotions.
- **Sustainability.** This page found no evidence that GitOps changes energy use materially. Carbon and cost optimisation of
  workloads belong to [GreenOps](../greenops/) and [FinOps](../finops/).

### Assurance of the operating model

A GitOps setup can look healthy while quietly failing. Five checks keep it honest:

1. **Is the loop running?** Alert when a source or Kustomization is not ready, or when an application has not reconciled within
   several intervals; a dead reconciler leaves clusters stale, not down.
2. **Is Git the only path?** Review out-of-band human writes from audit logs weekly; every one needs a break-glass record.
3. **Are exceptions shrinking?** Review every ignore rule and every `prune: disabled` annotation quarterly; remove the stale ones.
4. **Can we rebuild?** Run a quarterly drill: new cluster, bootstrap, restore data, measure rebuild time.
5. **Does the evidence chain hold?** Sample ten production changes a quarter and trace each from pull request to approval to applied
   revision to healthy status.

## When things go wrong

### Common failures and troubleshooting

| Symptom | Likely cause | Diagnosis | Remediation | Prevention |
|---|---|---|---|---|
| Application `OutOfSync` right after a successful sync | A mutating webhook or controller changes fields; templates with random values; HPA reordering metrics[^argo-diffing] | Diff live against desired; find the field that differs | Fix the manifest, or ignore that one field | Avoid random values in templates; keep ignore rules narrow |
| Replica count flaps; endless syncs | `replicas` is in Git and an autoscaler also sets it | Events show repeated syncs of one Deployment | Remove `replicas` from Git or ignore it[^argo-best] | Lint for `replicas` on autoscaled workloads |
| Kustomization never becomes ready | Image pull failure, failing probes, or a custom resource applied before its CRD | `kubectl describe` the Kustomization; read its events | Fix the workload; add `dependsOn` or sync waves | Health checks with realistic timeouts[^flux-kustomization][^argo-syncwaves] |
| Secret decryption fails | The age key Secret is missing or rotated; encrypted metadata | Reconciler events name the decryption error | Restore the key Secret; re-encrypt with current recipients | Key-rotation runbook; encrypt only `data`/`stringData`[^flux-sops] |
| Merged changes take minutes to appear | Polling interval (up to three minutes by default in Argo CD) | Compare merge time with the last fetched revision | Configure webhooks or a Flux Receiver[^argo-webhook][^flux-receivers] | Make lag a tracked KPI |
| Nothing deploys and nobody notices | Expired Git credential or a crashed controller | Source objects not ready; reconcile metrics stop | Renew the credential; restart and investigate | Alert on staleness; track credential expiry |
| Apps vanish from many clusters at once | An ApplicationSet generator stopped matching clusters; the default `sync` policy deletes Applications and the finalizer cascades[^argo-appset-deletion] | ApplicationSet events and history | Restore the generator input; resync | Controller-wide `--policy create-update`, or per-ApplicationSet `applicationsSync: create-update`, which is ignored unless `applicationsetcontroller.enable.policy.override` is `"true"` (default false); plus `preserveResourcesOnDeletion: true`, which also protects workloads when the ApplicationSet itself is deleted[^argo-appset-deletion] |
| Apply fails on an immutable field | A Deployment selector or Job template changed | Apply error names the field | Set Flux `force` once, or the per-object force annotation, then remove it[^flux-kustomization] | Review changes to immutable fields explicitly |

### Worked runbook: a refactor that prunes production

::: example Illustrative incident
A pull request "tidies" `tenants/production/kustomization.yaml`. During a merge-conflict resolution, the line listing `payments.yaml`,
which declares the `payments` namespace, disappears. Review focuses on another tenant's quota change in the same pull request, and the
team has not yet added the prune-protection label shown in the [reference implementation](#reference-implementation). The `tenants`
Kustomization has pruning enabled, so on the next revision it deletes objects that are no longer declared[^flux-kustomization] (Argo CD
behaves the same with `prune: true`[^argo-autosync]). Deleting a namespace "deletes *everything* under the namespace"[^k8s-namespaces],
including the payments ledger's PersistentVolumeClaim. That volume was dynamically provisioned, so it inherited the `Delete` reclaim
policy, which removes the storage asset too[^k8s-pv].
:::

::: figure Runbook for an accidental prune
```mermaid
flowchart TB
  A["Alert: applications missing or namespace terminating"]:::risk --> B["Step 1: freeze reconciliation for the affected scope"]:::core
  B --> C{"Step 2: data-bearing objects deleted?"}:::decision
  C -- "Yes, in parallel" --> D["Step 4: restore data from backup"]:::risk
  C -- "Yes or no" --> E["Step 3: find the commit and revert it by pull request"]:::core
  E --> G["Step 5: resume reconciliation and watch health"]:::core
  G --> H{"All healthy and data verified?"}:::decision
  D --> H
  H -- "Yes" --> I["Step 6: prune protection, CI deletion check"]:::neutral
  H -- "No" --> R["Keep the freeze; repeat steps 3 and 5"]:::risk
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: Step numbers match the runbook below. After the alert, step 1 freezes reconciliation for the affected scope and
step 2 checks whether data-bearing objects were deleted. Either way, step 3 finds and reverts the removing commit by pull request; if
data was deleted, step 4 restores it from backup in parallel. Step 5 resumes reconciliation and watches health. If anything is
unhealthy or the data is unverified, keep the freeze and repeat steps 3 and 5; otherwise step 6 adds prune protection and a CI
deletion check.
:::

**1. Freeze (minutes 0–5).** Stop the reconciler from acting on the bad revision. A Flux suspension pauses new revisions and drift
correction; it holds because `suspend` is not declared in Git (a declared `suspend: false` "does not allow for 'hot patching'"
[^flux-kustomization]). Turning off Argo CD's automated sync holds only if nothing restores it: a parent application with self-heal,
or an ApplicationSet, puts the child's `syncPolicy` back, and self-heal then reverts your live fix. Prepare the exemption in advance:
in the parent, `ignoreDifferences` on `/spec/syncPolicy/automated` with `RespectIgnoreDifferences=true`[^argo-bootstrap]; in an
ApplicationSet, `ignoreApplicationDifferences` on `/spec/syncPolicy`, documented for "temporarily toggling auto-sync"
[^argo-appset-deletion]. A deny sync window blocks all syncs without editing the Application[^argo-syncwindows], but its AppProject
usually lives in Git too.

_Example status: `bash -n` syntax-checked; not executed. Flux names match the reference implementation; the Argo CD application name
is illustrative and assumes the exemption above._

```bash
# Flux: pause the platform's tenants Kustomization
kubectl -n flux-system patch kustomization tenants \
  --field-manager=flux-client-side-apply --type merge -p '{"spec":{"suspend":true}}'
# Argo CD equivalent: stop automated sync for the application that manages tenants
argocd app set tenants-production --sync-policy none
```

**2. Assess (minutes 5–15).** List what was deleted and whether it held data. Reconciler events and Argo CD's sync history name the
revision and the pruned objects; Git shows the change.

```bash
argocd app history tenants-production                        # Argo CD: revisions synced
# Flux: what the tenants Kustomization did
kubectl -n flux-system get events --field-selector involvedObject.name=tenants
git log --oneline -5 -- tenants/production
SUSPECT_SHA=abc1234   # placeholder: the commit that dropped payments.yaml
git show --stat "$SUSPECT_SHA" -- tenants/production
kubectl get namespace payments -o jsonpath='{.status.phase}{"\n"}'
```

**3. Restore configuration (minutes 15–30).** Revert the offending commit through a pull request (an emergency approval path is fine;
skipping Git is not). If the namespace is still `Terminating`, wait for deletion to finish before resuming, because a terminating
namespace "cannot be used for new objects"[^k8s-namespaces].

```bash
git revert --no-edit "$SUSPECT_SHA"
git push origin HEAD:revert-tenants-prune   # then merge it via the emergency path
```

**4. Restore data (in parallel).** Configuration comes back from Git; data does not[^ogo-glossary]. Restore the ledger volume from the
latest backup and reconcile any transactions since, following [DisasterRecoveryOps](../disasterrecoveryops/).

**5. Resume and verify.** Resume reconciliation (`suspend: false`, or re-enable automated sync), then confirm every application is
`Synced` and healthy and the restored data passes application-level checks.

**6. Prevent.** Mark namespaces and claims as not prunable (Argo CD `Prune=false` or `Prune=confirm`[^argo-syncoptions]; Flux
`kustomize.toolkit.fluxcd.io/prune: disabled`[^flux-kustomization]); use a `Retain` reclaim policy for critical data classes[^k8s-pv]; and
add a CI check that renders every directory with a `kustomization.yaml` (including `tenants/`, where this incident happened) and
fails when a pull request removes a protected kind:

_Example status: `bash -n` syntax-checked; logic **executed** on 2026-10-16 in a throw-away repository shaped like the reference
layout, with a stub `kustomize` that concatenates the resources each `kustomization.yaml` lists; not run with real kustomize._

```bash
#!/usr/bin/env bash
# ci/deletion-check.sh: fail if a PR removes Namespaces, PVCs or CRDs from any render.
# Usage: ci/deletion-check.sh origin/main   (run from the repo root; needs git, kustomize)
set -euo pipefail
base_ref="${1:-origin/main}"
work="$(mktemp -d)"
trap 'git worktree remove --force "$work/base" >/dev/null 2>&1 || true
      rm -rf "$work"' EXIT
git worktree add --quiet --detach "$work/base" "$base_ref"

dirs() {  # every directory with a kustomization.yaml, tenants/ and clusters/ included
  find . -name kustomization.yaml -not -path './.git/*' -exec dirname {} \;
}
render() {  # $1 = directory, $2 = output; empty if the directory is absent on that side
  : > "$2"
  [ -f "$1/kustomization.yaml" ] || return 0
  kustomize build "$1" \
    | awk '/^kind:/ {k=$2; next} k && /^  name:/ {print k "/" $2; k=""}' | sort -u > "$2"
}

all="$( { (cd "$work/base" && dirs); dirs; } | sort -u)"
status=0
for dir in $all; do
  if ! render "$work/base/$dir" "$work/before" || ! render "$dir" "$work/after"; then
    echo "render failed: ${dir}"   # a broken render must not pass as "nothing removed"
    status=1
    continue
  fi
  removed="$(comm -23 "$work/before" "$work/after" \
    | grep -E '^(Namespace|PersistentVolumeClaim|CustomResourceDefinition)/' || true)"
  if [ -n "$removed" ]; then
    printf 'Protected objects removed from %s:\n%s\n' "$dir" "$removed"
    status=1
  fi
done
exit "$status"
```

On the test repository, the incident's change (the `payments.yaml` line dropped from `tenants/production/kustomization.yaml`)
printed `Protected objects removed from ./tenants/production:` and `Namespace/payments`, and exited 1. Removing
`apps/production/shop/` reported `PersistentVolumeClaim/orders-db`; a broken render on either side printed `render failed` and exited
1; no change exited 0. Because a broken base render fails too, a pull request that repairs a broken `main` needs a code owner's
override. The `awk` step assumes renderer output with `kind` before `metadata`, which kustomize produces; it is not a YAML parser.

### Anti-patterns and myths

Each has a home section above; this list is the short form.

- **"We use Git, so we do GitOps."** Without a pulling, reconciling agent it is versioned push deployment
  ([four-question test](#the-is-it-gitops-test-this-atlass-checklist)).
- **Branch per environment.** Merges drag unrelated changes along[^kapelonis]; use directories
  ([layout](#core-practices-repository-layout-and-environment-promotion)).
- **Mutable references in production.** Production changes without a commit[^argo-best]; pin digests and versions, and check in CI.
- **Self-heal without break-glass.** Responders fight the reconciler, or turn GitOps off for good
  ([judgement call](#a-judgement-call-self-heal-during-an-incident)).
- **Myth: pull is automatically secure.** The repository becomes the credential and the reconciler is highly privileged
  ([push versus pull](#push-versus-pull-where-the-credentials-live)).

## Deciding and adopting

### When to use GitOps and when not to

**Good fit.** The target exposes a declarative API with a reconciler you can run (Kubernetes above all; cloud resources through
Kubernetes-hosted controllers). Several people or teams change the same environments. You need evidence of who approved what. Drift
has hurt you. You run more than a handful of clusters or environments that should stay alike.

**Poor fit, or only partly.**

- *Imperative or one-off work* (schema migrations, data fixes, certificate ceremonies): GitOps assumes repeatable convergence; design
  these as runbooks, triggered from declared Jobs if you must.
- *Objects created by other controllers*: operators, autoscalers and admission webhooks own parts of the state. Do not declare what they
  manage.
- *Targets with no agent*: SaaS settings or appliances without an API a reconciler can drive. Push from CI with scoped credentials and
  an audit log, and say honestly that it is push.
- *A single small service on one host*: the reconciler and review discipline may cost more than they save.
- *Teams without review capacity*: pull requests that nobody reads add delay without adding control.

**Hidden responsibilities.** Running and upgrading reconcilers, key management, the exception list, break-glass drills, and on-call for
the delivery system itself.

### Decision framework

::: figure Should this component be managed with GitOps?
```mermaid
flowchart TB
  S["A component you want to manage"]:::neutral --> Q1{"Configuration exposed through a declarative API?"}:::decision
  Q1 -- "No" --> N1["Do not force it: scripts or push, with an audit log"]:::risk
  Q1 -- "Yes" --> Q2{"Can an agent run near it and reach the store?"}:::decision
  Q2 -- "No" --> N2["Push from CI with scoped credentials, or ship artefacts to a local agent"]:::neutral
  Q2 -- "Yes" --> Q3{"Configuration, not data or one-off actions?"}:::decision
  Q3 -- "No" --> N3["Keep data and migrations out: runbooks and backups"]:::risk
  Q3 -- "Yes" --> Q4{"Can the team review changes and own break-glass?"}:::decision
  Q4 -- "No" --> N4["Start with manual sync and drift alerts"]:::neutral
  Q4 -- "Yes" --> Y["Adopt: automated sync, self-heal and guarded prune"]:::core
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: Four questions in sequence. No declarative API for the configuration: do not force GitOps; use scripts or push
automation with an audit log. No agent near it that can reach the state store: push from CI with scoped credentials, or ship
artefacts to a local agent. Data or one-off actions rather than configuration: keep them out; use runbooks and backups. A team that
cannot review changes or own break-glass: start with manual sync and drift alerts. Otherwise adopt automated sync, self-heal and
guarded pruning.
:::

**Weighted comparison (illustrative).** Scores 1 (poor) to 5 (good) are this Atlas's judgement for the stated assumptions, not
measurements; change the weights to fit your situation.

*Assumptions:* a multi-team Kubernetes platform with ten clusters, regulated change evidence, and a 24/7 on-call rotation.

| Criterion | Weight | Push CD from CI | Pull, manual sync | Pull, automated with self-heal |
|---|---|---|---|---|
| Credential exposure | 25 | 2 | 4 | 4 |
| Drift control | 20 | 1 | 3 | 5 |
| Audit evidence | 20 | 3 | 4 | 5 |
| Incident flexibility | 15 | 4 | 4 | 3 |
| Operational overhead (higher = lighter) | 10 | 4 | 3 | 3 |
| Learning curve (higher = easier) | 10 | 5 | 4 | 3 |
| **Weighted score (÷ 100)** | | **2.80** | **3.70** | **4.05** |

For a four-person start-up with one cluster, weights might be 10, 15, 5, 20, 25 and 25. The scores become 3.55, 3.60 and 3.50: a
near tie. At that size, choose by what you expect to grow into.

### Maturity model

**This Atlas's model** (no official GitOps maturity model exists). Move up only when the exit criteria hold for a full quarter.

| Level | Observable behaviour | Evidence | Exit criteria |
|---|---|---|---|
| 0 Scripted | Deploys by scripts or pipelines; hand fixes common | Pipeline logs | All production configuration exists as declarations |
| 1 Declared | Manifests in Git and reviewed; CI pushes them | Pull requests | Every overlay renders in CI; mutable tags flagged |
| 2 Reconciled | Agents pull; drift is visible; production may use manual sync | Reconciler status, drift alerts | CI holds no cluster credentials for managed environments |
| 3 Governed | Automated sync with self-heal; least-privilege tenants; digests; secrets pattern; prune guards; break-glass procedure | Projects or impersonation config; exception list; break-glass records | Zero unticketed out-of-band changes; no tenant reconciles as cluster-admin |
| 4 Assured | Metrics reviewed; rebuild drills pass; evidence sampled; promotion automated with policy | Dashboards, drill reports, sampling results | Rebuild time within target; sampling finds complete evidence chains |

### Adoption roadmap: 30/60/90 days

| Phase | Actions | Owner | Milestone and metric |
|---|---|---|---|
| Days 1–30 | Write the repository-and-promotion ADR; pick one non-critical service and the staging cluster; bootstrap the reconciler; manual sync; CI rendering and the guard script; drift alerts | Platform team with one app team | The staging service changes only through pull requests; reconciliation-lag baseline recorded |
| Days 31–60 | Automated sync and self-heal in staging; choose the secrets pattern; AppProjects or the Flux lockdown for tenants; onboard the service in production with manual sync; write and rehearse break-glass; remove CI's cluster credentials for that service | Platform, security, on-call | First production change through GitOps; break-glass rehearsal done |
| Days 61–90 | Automated sync in production for low-risk services; prune guards and the deletion check; promotion automation for development and staging; metrics dashboard; first staging rebuild drill; plan the next wave | Platform, app teams, SRE | Three or more services managed; 30 days without unticketed out-of-band changes; measured rebuild time |

Quick win: drift alerts alone (manual sync) show hand changes nobody knew about. Dependencies: branch protection, an artefact
registry with digests, and an owner for the reconciler on-call.

## Toolkit

### Checklists

**Quick start**

- [ ] Write down what is in scope (clusters, namespaces, kinds) and what stays out (data, operator-owned objects)
- [ ] Choose a state store and layout; record it in an ADR
- [ ] Install the reconciler so it manages itself (bootstrap)
- [ ] Render every overlay in CI and block mutable images and plain Secrets
- [ ] Turn on drift alerts before automatic correction

**Application onboarding preflight**

- [ ] Images referenced by digest; charts and remote bases pinned
- [ ] No `replicas` for autoscaled workloads; known mutated fields listed as narrow ignore rules
- [ ] Secrets follow the chosen pattern; no plain Secret manifests
- [ ] Health checks or readiness probes reflect real readiness
- [ ] Namespaces and claims carry prune protection
- [ ] Owner, code owners and on-call contact recorded

**Reconciler production readiness**

- [ ] Version pinned and within the supported Kubernetes range[^argo-tested][^flux-release-29]
- [ ] Tenants fenced: AppProjects, not the `default` project[^argo-projects], or the Flux lockdown flags, so no tenant
  Kustomization or HelmRelease runs as cluster-admin[^flux-multitenancy]
- [ ] Webhooks configured; intervals chosen deliberately
- [ ] Metrics scraped; alerts for not-ready sources and stale reconciliation
- [ ] Credential expiry dates tracked; decryption keys backed up outside the cluster
- [ ] Upgrade procedure tested in staging

**Security review**

- [ ] Branch protection on watched paths, with no administrator bypass[^gh-protected]
- [ ] Commit or source verification on production sources[^flux-gitrepo][^argo-source-integrity]
- [ ] CI cannot write to production paths without review; bots scoped to one repository and path
- [ ] Remote bases refused (Flux `--no-remote-bases`) or pinned by commit[^flux-multitenancy][^argo-security]
- [ ] Unused config-management tools disabled on the renderer[^argo-security]
- [ ] No render-time secret injection[^argo-secrets]

**Incident and break-glass**

- [ ] Who may suspend reconciliation, for which scope, for how long
- [ ] Suspension cannot be undone by Git or a parent (preconditions in
  [runbook step 1](#worked-runbook-a-refactor-that-prunes-production))
- [ ] Suspension creates a ticket and an alert if it exceeds the time limit
- [ ] The fix reaches Git before reconciliation resumes
- [ ] Revert-by-pull-request rehearsed; emergency approval path known

**Monthly review**

- [ ] Reconciliation lag, apply failures, drift and out-of-band changes reviewed
- [ ] Exception list (ignore rules, prune-disabled objects) reviewed and pruned
- [ ] Reconciler and controller versions checked against support windows

### Templates

**1. Architecture decision record: repository and promotion strategy**

```text
ADR-NNN: GitOps repository and promotion strategy
Status: proposed | accepted | superseded by ADR-NNN        Date: YYYY-MM-DD
Context: number of clusters, teams, environments; regulatory evidence needs;
  Git host; registry.
Decision:
  State store: Git repository <name> (or OCI artefacts in <registry>)
  Layout: monorepo with apps/, infrastructure/, clusters/ | repo per environment
          | repo per team
  Environments: directories with Kustomize overlays (no branch per environment)
  Promotion: PR that sets the image digest in the next overlay;
             automation allowed up to <environment>
  Reconciler: <Argo CD | Flux> <version>; topology: hub | per cluster | managed
  Tenancy: <AppProjects | impersonated service accounts>; platform paths owned by <team>
  Secrets: <SOPS+age | External Secrets Operator | Sealed Secrets>
  Drift policy: revert by default; tolerated fields listed in <file>
Consequences: what gets easier, what gets harder, what we must operate.
Review date: YYYY-MM-DD
```

**2. Break-glass record**

```text
Break-glass record BG-YYYY-NNN
Incident: INC-NNNN            Opened by: <name>            Approved by: <on-call lead>
Scope suspended: <cluster>/<namespace>/<Kustomization or Application>  (smallest possible)
Suspended at: YYYY-MM-DD HH:MM UTC        Time limit: <hours>   Alert if exceeded: yes
Manual changes made (exact commands or diff):
Pull request that brings the change into Git: <link>          Merged at:
Reconciliation resumed at:                 Post-resume state: Synced/Healthy (yes/no)
Follow-up actions:
```

**3. Application onboarding request**

```text
Application: <name>        Team: <team>        Code owners: <group>
Environments and clusters: dev / staging / production on <clusters>
Source of manifests: <path in config repo> | <app repo path or chart + version>
Images: registry path; digest pinning confirmed (yes/no)
Secrets: pattern used; keys or secret-manager paths
Autoscaled or operator-managed fields: <list>
Data-bearing objects: <PVCs, databases>; backup owner: <team>
Health: readiness probes; extra smoke test (yes/no)
Promotion: automated up to <environment>; production approval by <group>
Risk class: low | medium | high        On-call contact: <rota>
```

**4. Drift exception record**

```text
Exception DX-NNN    Object: <kind>/<namespace>/<name>    Field(s): <JSON pointer>
Why it differs: <controller, webhook, autoscaler, vendor operator>
Mechanism: ignoreDifferences | ssa annotation | omitted from Git | prune disabled
Owner: <team>    Created: YYYY-MM-DD    Review by: YYYY-MM-DD
Removal condition: <what must change for this exception to go away>
```

### Deep comparisons

GitOps against push CD is covered in [push versus pull](#push-versus-pull-where-the-credentials-live) and the weighted comparison.
Three other approaches compete with it:

| Approach | Who applies changes | Drift handling | Credentials | Best for |
|---|---|---|---|---|
| **IaC with pull-request automation** | A CI job running `plan` and `apply` | Detected at the next plan | Cloud credentials in CI | Cloud infrastructure with careful plan review ([InfraOps](../infraops/)) |
| **Configuration-management agents** | An agent on each host pulling playbooks or manifests[^ansible-pull] | Re-applied on schedule; reporting varies | On the host | Virtual machines and bare metal |
| **Platform-managed deploys** | A PaaS or managed service | The provider's model | Provider IAM | Teams that want the platform to own delivery ([PlatformOps](../platformops/), [NoOps](../noops/)) |

### FAQ

**Do I need Kubernetes to do GitOps?** No, but it helps. The principles are tool-neutral[^ogo-principles]. Outside Kubernetes you need a
reconciler for the target: Crossplane or the OpenTofu/Terraform controller for cloud resources, or pull agents such as `ansible-pull` for
hosts, with less mature drift reporting[^crossplane][^tofu-controller][^ansible-pull].

**How do I roll back?** Revert the commit and let the reconciler apply it. Argo CD's own rollback cannot be used while automated sync is
enabled[^argo-autosync]. Flux can roll back a failed Helm upgrade automatically when remediation is configured[^flux-helmrelease]; when it does, Git still declares the
failed version, so follow with a revert so that Git tells the truth.

**Can CI still deploy to development?** Yes: push or automated image updates in development, pull with review for production. Be
explicit about which environments meet which principles.

**What about database migrations?** Treat them as application concerns: backward-compatible migrations run by the application or by a
declared Job with ordering (sync waves or `dependsOn`)[^argo-syncwaves][^flux-kustomization]. Never rely on GitOps to undo a migration;
reverting a manifest does not revert data.

**Does GitOps satisfy a change advisory board?** It provides evidence (approvals, applied revisions), but deciding which changes need
which approval is change management ([ITOps](../itops/)): automate low-risk standard changes and keep human approval where policy
demands it.

## Going further

### Advanced topics and emerging direction

Status as of October 2026; each topic is introduced earlier on the page.

| Topic | What to know |
|---|---|
| Rendered manifests in Git (Argo CD Source Hydrator, beta)[^argo-hydrator] | Reviewers see exactly what will run; storage and pull-request noise grow |
| OCI artefacts as the state store ("Gitless GitOps")[^flux-concepts] | Removes the Git host from the production path; pair with signature verification |
| Promotion orchestration (Kargo)[^kargo] | Moves promotion logic out of ad-hoc scripts; another component to run |
| Pull-based multi-cluster for Argo CD (argocd-agent, pre-1.0)[^argocd-agent] | Central view without central credentials; evaluate maturity |
| Declarative cloud control planes (Crossplane)[^crossplane] | Extends reconciliation to cloud resources; guard destructive changes |

### Learning path

- **Prerequisites.** Git and pull requests; Kubernetes objects and `kubectl`; Kustomize or Helm basics; CI concepts from
  [DevOps](../devops/). [KubernetesOps](../kubernetesops/) helps.
- **One day.** Read the four principles and the glossary[^ogo-principles][^ogo-glossary]; do [Lab 1](#hands-on-labs); run the
  four-question test on your current setup.
- **One week.** Labs 2 and 3; design a repository layout for two environments; write the drift policy and break-glass procedure for one
  service.
- **Thirty days.** Run the first 30 days of the [roadmap](#adoption-roadmap-306090-days) on a real non-critical service; compute the
  [metrics](#metrics-and-kpis) with Lab 4's method from your own events.
- **Specialisations.** Multi-tenancy and reconciler security ([DevSecOps](../devsecops/), [KubernetesSecOps](../kubernetessecops/));
  fleet management ([PlatformOps](../platformops/)); progressive delivery ([ReleaseOps](../releaseops/)); cloud resources
  ([InfraOps](../infraops/)); GitLab wiring ([GitLabOps](../gitlabops/)).
- **Portfolio projects.** A public repository with overlays, SOPS-encrypted secrets, a CI guard and a documented drift policy; a
  written rebuild drill with measured times.

## Wrap-up

### Executive recap and cheat sheet

| | |
|---|---|
| **One sentence** | Declare the desired state in a versioned store and let agents pull and reconcile it continuously |
| **The test** | Declarative · versioned and immutable · pulled automatically · continuously reconciled[^ogo-principles] |
| **Decide first** | State store and layout; hub or per-cluster agents; tenancy; secrets pattern; drift policy; break-glass |
| **Essential practices** | Directories, not branches, for environments; promote by changing one digest; pin everything; protect namespaces and claims from pruning; render and check in CI |
| **Measure** | Reconciliation lag (median, p90), apply failure rate, drift by cause, unticketed out-of-band changes, time out of sync, rebuild time; delivery outcomes via DORA on [DevOps](../devops/) |
| **Tools (as of October 2026)** | Argo CD 3.5, Flux 2.9; SOPS, External Secrets Operator, Sealed Secrets; Kargo; Crossplane; managed options on AKS and EKS |
| **Watch-outs** | Cluster-admin reconcilers (no lockdown); Git write equals deploy; mutable references; render-time secrets; ApplicationSet cascades; break-glass undone by a parent |
| **Next actions** | Run the four-question test; write the ADR; onboard one service with drift alerts; schedule the first rebuild drill |

### A judgement call: self-heal during an incident

::: example Illustrative scenario
Saturday, 02:10. Checkout latency is climbing. The on-call engineer finds that a connection-pool limit in a ConfigMap must be raised
now. Production runs automated sync with self-heal; the emergency pull-request path needs a second approver, and the Git host reports
degraded performance.
:::

**Option A — keep self-heal on and fix only through an emergency pull request.** The invariant holds, the evidence chain stays intact,
and nobody has to remember to undo anything. It fails when the path to Git *is* the bottleneck (approver asleep, Git host degraded) or
when the reconciler is part of the problem, for example by re-applying a bad revision.

**Option B — suspend reconciliation for the one affected application, patch live, then land the fix in Git before resuming.** Service
recovers faster. The risk is the forgotten suspension: drift becomes permanent, the next merge silently overwrites the fix, or the
application stays outside GitOps for weeks.

**Assumptions.** The emergency path usually takes about ten minutes (illustrative; measure your own); suspension is possible
per application or per object (Flux supports both a suspended Kustomization and a per-object `kustomize.toolkit.fluxcd.io/reconcile:
disabled` annotation[^flux-kustomization]; Argo CD can switch off automation for one application or block syncs with a sync window
[^argo-autosync][^argo-syncwindows]); nothing undoes the suspension (the preconditions in
[runbook step 1](#worked-runbook-a-refactor-that-prunes-production)); suspension actions are audited.

**Why the attractive choice can be wrong.** "Never bypass GitOps" sounds principled, but it turns a reliability control into an outage
multiplier when approvals or the Git host are slow. "Just patch it" sounds pragmatic, but as a habit it stops Git describing
production.

**Decision.** Choose B, *designed in advance*: the smallest possible scope, a time limit with an alert (for example four hours), a
break-glass record, a fix pull request opened during the incident, and resumption only after it merges. Count the action as an
out-of-band change with a ticket in your [metrics](#metrics-and-kpis). Prefer A whenever the emergency path is reliably faster than your
recovery target and the Git host is healthy.

**Uncertainty.** No published data compares recovery times for the two approaches; the answer depends on your approval latency. Measure
both in game days and revisit the rule.

### Learning objectives and mastery check

After this page you should be able to:

- **Explain** the four OpenGitOps principles and why "pull" enables continuous reconciliation.
- **Classify** a delivery setup with the four-question test.
- **Design** a repository layout, promotion flow, secrets pattern and drift policy for several teams.
- **Operate** a reconciler safely: least privilege, prune guards, break-glass, metrics.
- **Evaluate** when GitOps is the wrong tool, and defend the choice to a leader or auditor.

::: answer 1. "We do GitOps: our CI workflow runs kubectl apply after every merge." Which principles does this meet?
It is **declarative** and probably **versioned** (manifests in Git), though tags may be mutable. It is **not pulled automatically**
(CI pushes) and **not continuously reconciled**: after the job ends, nothing compares the cluster with Git. CI also holds cluster
credentials, the push model's security downside in NIST's guidance[^nist-204c]. An in-cluster agent with automated sync and drift
correction closes both gaps.
:::

::: answer 2. Argo CD runs automated sync with selfHeal off. A colleague scales a Deployment by hand. What happens?
The new replica count stays and the application shows `OutOfSync`: with self-heal off, live changes "will not trigger automated
sync"[^argo-autosync]. The next commit's sync overwrites the drift, so it is detected but only incidentally corrected. In a
multi-source application a commit to any source does this, and Argo CD warns that disabling self-heal "does not guarantee
that live cluster changes in multi-source applications will persist"[^argo-autosync].
:::

::: answer 3. A staging overlay uses the image nginx:1.27 and a remote base with ?ref=main. Which principle is at risk, and what is the fix?
**Versioned and immutable.** Both references can change meaning without any commit in your repository: the tag can be re-pushed and the
branch moves. Argo CD's best practices warn about exactly this[^argo-best]. Pin the image by digest and the base to a tag or commit SHA,
and enforce it with a CI check such as the Lab 3 guard.
:::

::: answer 4. Two teams share a cluster. How do you stop team A's manifests from creating cluster-wide RBAC?
In **Flux**, run team A's Kustomization under an impersonated service account bound only in its namespace, and set the lockdown flags:
`--default-service-account`, so that no Kustomization or HelmRelease without a service account runs as cluster-admin, and
`--no-cross-namespace-refs`, so team A cannot use another namespace's sources[^flux-kustomization][^flux-multitenancy]. In **Argo CD**,
put the team's applications in an AppProject that restricts destinations and allow-lists no cluster-scoped kinds[^argo-projects]; consider the beta impersonation feature, because without it syncs run with the controller's privileges
[^argo-impersonation]. In both, add code-owner review on the platform paths.
:::

::: answer 5. Eight changes merged this week. Seven became healthy after 2, 3, 3, 4, 5, 6 and 30 minutes; one failed. Compute the median lag, the nearest-rank p90 and the apply failure rate.
Median of seven sorted values = the 4th = **4 minutes**. p90 rank = ⌈0.9 × 7⌉ = 7, so **p90 = 30 minutes**, which is also the maximum.
Apply failure rate = 1 ÷ 8 = **12.5%**. Report the p90 but do not set targets on one week; investigate the 30-minute change rather
than the average.
:::

::: answer 6. After an incident you discover the reconciler's Git token expired nine days ago. What failed, and what would have caught it?
The delivery system failed silently: the clusters were stale, not down, so no user-facing alert fired. The missing
[assurance checks](#assurance-of-the-operating-model): an alert on not-ready sources or Kustomizations (for example
`gotk_resource_info{ready="False"}` for Flux[^flux-metrics], or application conditions in Argo CD[^argo-metrics]), an alert when an
application has not reconciled within several intervals, and a calendar of credential expiry dates.
:::

::: answer 7. An auditor asks for proof that production change X was approved and is what actually ran. What do you provide, and which gap must you close?
The pull request and its approvals, the merge commit, the reconciler event naming the applied revision and its health, and
audit-log entries for any manual action. The gap: Git history "does not necessarily correlate one-to-one with events that happen in
a cluster"[^argo-security], so export and retain reconciler events and audit logs for your audit period.
:::

::: answer 8. The CTO proposes GitOps for the company's single monolith running on one virtual machine. When would you advise against it?
Run the [decision tree](#decision-framework). With no declarative API for the host, one service and a small team, the reconciler,
layout and review discipline may cost more than they return; a well-logged push pipeline or a configuration-management agent may be
enough. Revisit when the workload moves to a declarative platform, more teams change it, or audit evidence becomes a requirement.
:::

### Conclusion

GitOps is a small idea with large consequences: write down what production should be, keep every version, and let a control loop
close the gap. Its definition is now stable and vendor-neutral, two mature reconcilers implement it, and both standards guidance and
cloud providers treat the pull model as normal.

The idea is also easy to misread. Pull removes cluster credentials from CI, but it turns the repository into the credential and puts a
cluster-admin reconciler inside the cluster unless you narrow it. Self-heal stops quiet drift, but it will fight your responders unless
you design break-glass in advance. Git recreates configuration, not data. Most GitOps failures are failures of these edges: mutable
references, unguarded pruning, forgotten suspensions, stale reconcilers, and guarantees taken without their preconditions.

Adopt it where the target is declarative and many hands change it; start with drift alerts and manual sync; earn automation by
measuring lag, failures and out-of-band changes; and say plainly when a system is better served by an honest push pipeline. GitOps is
not "using Git". It is choosing to make the declared state the only path to production, and then doing the work to keep that true.

## Sources and further reading

All sources were opened on 2026-10-16 (those changed in the post-score polish again on 2026-10-17); retrieval notes,
quotation sources and the gaps that could not be checked are in the evidence ledger (`research/gitops-ledger.md`). Version-specific facts are as of October 2026.

[^ogo-principles]: [GitOps Principles v1.0.0](https://github.com/open-gitops/documents/blob/main/PRINCIPLES.md) — OpenGitOps, released 2021-10-08 ([release list](https://github.com/open-gitops/documents/releases)). Checked 2026-10-16.
[^ogo-glossary]: [GitOps Glossary v1.0.0](https://github.com/open-gitops/documents/blob/main/GLOSSARY.md) — OpenGitOps, 2021. Checked 2026-10-16.
[^ogo-project]: [OpenGitOps project README](https://github.com/open-gitops/project) — OpenGitOps (CNCF Sandbox), undated. Checked 2026-10-16.
[^cncf-ogo]: [OpenGitOps project page](https://www.cncf.io/projects/opengitops/) — CNCF, accepted 2021-01-26. Checked 2026-10-16.
[^wg-charter]: [GitOps Working Group charter](https://github.com/cncf/tag-app-delivery/blob/master/gitops-wg/charter.md) — CNCF TAG App Delivery, undated. Checked 2026-10-16.
[^tag-readme]: [TAG App Delivery repository README (working groups table)](https://github.com/cncf/tag-app-delivery) — CNCF, undated. Checked 2026-10-16.
[^cncf-tags]: [CNCF Technical Advisory Groups](https://contribute.cncf.io/community/tags/) — CNCF, 2025 structure. Checked 2026-10-16.
[^weave-2017]: [GitOps - Operations by Pull Request (archived copy)](http://web.archive.org/web/20180107011605/https://www.weave.works/blog/gitops-operations-by-pull-request) — Alexis Richardson, Weaveworks, 2017-08-07 (Wayback Machine snapshot 2018-01-07; the original domain no longer hosts it). Checked 2026-10-16.
[^cncf-flux-page]: [Flux project page](https://www.cncf.io/projects/flux/) — CNCF (accepted 2019-07-15, incubating 2021-03-12, graduated 2022-11-30). Checked 2026-10-16.
[^cncf-argo-page]: [Argo project page](https://www.cncf.io/projects/argo/) — CNCF (accepted 2020-03-26, graduated 2022-12-06). Checked 2026-10-16.
[^cncf-argo-grad]: [The Cloud Native Computing Foundation announces Argo has graduated](https://www.cncf.io/announcements/2022/12/06/the-cloud-native-computing-foundation-announces-argo-has-graduated/) — CNCF, 2022-12-06. Checked 2026-10-16.
[^sdtimes-weave]: [State of GitOps now in Flux as Weaveworks shuts down](https://sdtimes.com/softwaredev/state-of-gitops-now-in-flux-as-weaveworks-shuts-down/) — SD Times, 2024-02-09. Checked 2026-10-16.
[^gitlab-flux-2024]: [The continued support of FluxCD at GitLab](https://about.gitlab.com/blog/the-continued-support-of-fluxcd-at-gitlab/) — GitLab, 2024-03-05. Checked 2026-10-16.
[^gitlab-gitops]: [GitOps with the agent for Kubernetes and Flux](https://docs.gitlab.com/user/clusters/agent/gitops/) — GitLab Docs, current release. Checked 2026-10-16.
[^nist-204c]: [SP 800-204C, Implementation of DevSecOps for a Microservices-based Application with Service Mesh](https://csrc.nist.gov/pubs/sp/800/204/c/final) — R. Chandramouli, NIST, March 2022, §4.7–4.7.1. Checked 2026-10-16.
[^k8s-controller]: [Controllers](https://kubernetes.io/docs/concepts/architecture/controller/) — Kubernetes documentation. Checked 2026-10-16.
[^k8s-secret]: [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) — Kubernetes documentation. Checked 2026-10-16.
[^k8s-audit]: [Auditing](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/) — Kubernetes documentation. Checked 2026-10-16.
[^k8s-namespaces]: [Share a Cluster with Namespaces](https://kubernetes.io/docs/tasks/administer-cluster/namespaces/) — Kubernetes documentation. Checked 2026-10-16.
[^k8s-pv]: [Persistent Volumes: reclaiming](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming) — Kubernetes documentation. Checked 2026-10-16.
[^k8s-rbac]: [Using RBAC Authorization: user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) — Kubernetes documentation. Checked 2026-10-16.
[^kustomize-glossary]: [Kustomize glossary](https://kubectl.docs.kubernetes.io/references/kustomize/glossary/) — Kubernetes SIG CLI. Checked 2026-10-16.
[^kustomize-fields]: [images](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/images/) and [replicas](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replicas/) — Kustomize reference, Kubernetes SIG CLI. Checked 2026-10-17.
[^argo-arch]: [Architecture](https://argo-cd.readthedocs.io/en/stable/operator-manual/architecture/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-autosync]: [Automated Sync Policy](https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/) — Argo CD documentation (3.5). Checked 2026-10-17.
[^argo-best]: [Best Practices](https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-secrets]: [Secret Management](https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-security]: [Security](https://argo-cd.readthedocs.io/en/stable/operator-manual/security/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-impersonation]: [Application Sync using impersonation (beta since v3.5.0)](https://argo-cd.readthedocs.io/en/stable/operator-manual/app-sync-using-impersonation/) — Argo CD documentation. Checked 2026-10-16.
[^argo-projects]: [Projects](https://argo-cd.readthedocs.io/en/stable/user-guide/projects/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-syncwindows]: [Sync Windows](https://argo-cd.readthedocs.io/en/stable/user-guide/sync_windows/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-syncwaves]: [Sync Phases and Waves](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-syncoptions]: [Sync Options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-diffing]: [Diffing Customization](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-deletion]: [App Deletion](https://argo-cd.readthedocs.io/en/stable/user-guide/app_deletion/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-bootstrap]: [Cluster Bootstrapping](https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/) — Argo CD documentation (3.5): ApplicationSets recommended, app of apps "an admin-only tool", ignoring differences in child applications. Checked 2026-10-16.
[^argo-appset]: [Introduction to ApplicationSet controller](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-appset-deletion]: [Controlling if/when the ApplicationSet controller modifies Applications](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Controlling-Resource-Modification/) and [Application Pruning & Resource Deletion](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Application-Deletion/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-metrics]: [Metrics](https://argo-cd.readthedocs.io/en/stable/operator-manual/metrics/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-webhook]: [Webhook Configuration](https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/) — Argo CD documentation (3.5). Checked 2026-10-17.
[^argo-faq]: [FAQ: how often does Argo CD check for changes](https://argo-cd.readthedocs.io/en/stable/faq/) — Argo CD documentation (3.5). Checked 2026-10-17.
[^argo-getting-started]: [Getting Started](https://argo-cd.readthedocs.io/en/stable/getting_started/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-helm]: [Helm](https://argo-cd.readthedocs.io/en/stable/user-guide/helm/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-hydrator]: [Source Hydrator (beta since v3.5.0)](https://argo-cd.readthedocs.io/en/stable/user-guide/source-hydrator/) — Argo CD documentation. Checked 2026-10-16.
[^argo-source-integrity]: [Source Integrity Verification](https://argo-cd.readthedocs.io/en/stable/user-guide/source-integrity/) — Argo CD documentation (3.5); Git GnuPG verification is its only method, the legacy `signatureKeys` setting is deprecated. Checked 2026-10-16.
[^argo-ha]: [High Availability](https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-tested]: [Installation: tested Kubernetes versions](https://argo-cd.readthedocs.io/en/stable/operator-manual/installation/) — Argo CD documentation (3.5). Checked 2026-10-16.
[^argo-releases]: [Argo CD releases](https://github.com/argoproj/argo-cd/releases) — Argo project, v3.5.5 published 2026-10-12. Checked 2026-10-17.
[^argocd-agent]: [argocd-agent](https://github.com/argoproj-labs/argocd-agent) — argoproj-labs, v0.10.0 published 2026-08-26. Checked 2026-10-16.
[^flux-concepts]: [Core Concepts](https://fluxcd.io/flux/concepts/) — Flux documentation (2.9). Checked 2026-10-16.
[^flux-components]: [GitOps Toolkit components](https://fluxcd.io/flux/components/) — Flux documentation (2.9). Checked 2026-10-16.
[^flux-kustomization]: [Kustomization API](https://fluxcd.io/flux/components/kustomize/kustomizations/) — Flux documentation (kustomize.toolkit.fluxcd.io/v1). Checked 2026-10-16.
[^flux-gitrepo]: [GitRepository API: verification](https://fluxcd.io/flux/components/source/gitrepositories/) — Flux documentation (source.toolkit.fluxcd.io/v1). Checked 2026-10-16.
[^flux-helmrelease]: [HelmRelease API: drift detection and remediation](https://fluxcd.io/flux/components/helm/helmreleases/) — Flux documentation (helm.toolkit.fluxcd.io/v2). Checked 2026-10-16.
[^flux-multitenancy]: [Flux multi-tenancy](https://fluxcd.io/flux/installation/configuration/multitenancy/) — Flux documentation (2.9). Checked 2026-10-16.
[^flux-repo-structure]: [Ways of structuring your repositories](https://fluxcd.io/flux/guides/repository-structure/) — Flux documentation (2.9). Checked 2026-10-16.
[^flux-sops]: [Manage Kubernetes secrets with SOPS](https://fluxcd.io/flux/guides/mozilla-sops/) — Flux documentation (2.9). Checked 2026-10-16.
[^flux-image-update]: [Automate image updates to Git](https://fluxcd.io/flux/guides/image-update/) — Flux documentation (2.9). Checked 2026-10-16.
[^flux-metrics]: [Flux Prometheus metrics](https://fluxcd.io/flux/monitoring/metrics/) — Flux documentation (2.9). Checked 2026-10-16.
[^flux-install]: [Flux installation (dev install)](https://fluxcd.io/flux/installation/) — Flux documentation (2.9). Checked 2026-10-16.
[^flux-providers]: [Notification Provider API: Git commit status updates](https://fluxcd.io/flux/components/notification/providers/) — Flux documentation. Checked 2026-10-16.
[^flux-receivers]: [Notification Receiver API](https://fluxcd.io/flux/components/notification/receivers/) — Flux documentation. Checked 2026-10-16.
[^flux-release-29]: [Flux v2.9.0 release notes (Kubernetes compatibility)](https://github.com/fluxcd/flux2/releases/tag/v2.9.0) — Flux project, 2026-06-30; latest patch v2.9.6 published 2026-10-01. Checked 2026-10-17.
[^helm-eol]: [Helm 3 End of Life](https://helm.sh/blog/helm-v3-end-of-life) — Helm project, 2026-06-02. Checked 2026-10-16.
[^helm-overview]: [Helm documentation overview (Helm 4)](https://helm.sh/docs/overview/) — Helm documentation. Checked 2026-10-16.
[^flux-release-28]: [Flux v2.8.0 release notes](https://github.com/fluxcd/flux2/releases/tag/v2.8.0) — Flux project, 2026-02-24. Checked 2026-10-16.
[^kind-release]: [kind v0.33.0 release notes](https://github.com/kubernetes-sigs/kind/releases/tag/v0.33.0) — Kubernetes SIGs, 2026-08-26. Checked 2026-10-16.
[^sops]: [SOPS: Secrets OPerationS](https://getsops.io/) — SOPS project (CNCF Sandbox). Checked 2026-10-16.
[^sops-age]: [SOPS: age identities](https://getsops.io/docs/usage/identities/age/) — SOPS documentation. Checked 2026-10-16.
[^eso]: [External Secrets Operator](https://external-secrets.io/latest/) — External Secrets project. Checked 2026-10-16.
[^eso-api]: [ExternalSecret API](https://external-secrets.io/latest/api/externalsecret/) — External Secrets project (external-secrets.io/v1). Checked 2026-10-16.
[^sealed-secrets]: [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) — Bitnami Labs. Checked 2026-10-16.
[^gh-sig]: [About commit signature verification](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification) and its [Enterprise Server 3.22 version](https://docs.github.com/en/enterprise-server@3.22/authentication/managing-commit-signature-verification/about-commit-signature-verification) — GitHub Docs (web-interface signing and rebase-and-merge notes). Checked 2026-10-17.
[^gh-protected]: [About protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) — GitHub Docs. Checked 2026-10-16.
[^kapelonis]: [Stop using branches for deploying to different GitOps environments](https://octopus.com/blog/stop-using-branches-deploying-different-gitops-environments) — Kostis Kapelonis, Octopus Deploy blog (originally Codefresh), 2021-12-17. Checked 2026-10-16.
[^kargo]: [Kargo](https://github.com/akuity/kargo) — Akuity, Apache-2.0; v1.12.4 published 2026-10-12. Checked 2026-10-16.
[^aks-flux]: [Application deployments with GitOps (Flux v2)](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-gitops-flux2) — Microsoft Learn, updated 2026-03-24. Checked 2026-10-16.
[^eks-capabilities]: [Announcing Amazon EKS Capabilities](https://aws.amazon.com/about-aws/whats-new/2025/11/amazon-eks-capabilities) — AWS, 2025-11-30. Checked 2026-10-16.
[^crossplane]: [Cloud Native Computing Foundation announces graduation of Crossplane](https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/) — CNCF, 2025-11-06. Checked 2026-10-16.
[^tofu-controller]: [tofu-controller](https://github.com/flux-iac/tofu-controller) — flux-iac (community controller for Flux, previously Weave TF-Controller), v0.16.5 published 2026-08-06. Checked 2026-10-16.
[^ansible-pull]: [ansible-pull](https://docs.ansible.com/ansible/latest/cli/ansible-pull.html) — Ansible documentation. Checked 2026-10-16.
[^cncf-argo-survey]: [CNCF End User Survey finds Argo CD as majority adopted GitOps solution for Kubernetes](https://www.cncf.io/announcements/2025/07/24/cncf-end-user-survey-finds-argo-cd-as-majority-adopted-gitops-solution-for-kubernetes/) — CNCF, 2025-07-24 (survey of Argo CD users). Checked 2026-10-16.
[^cncf-annual-2026]: [CNCF Annual Cloud Native Survey announcement](https://www.cncf.io/announcements/2026/01/20/kubernetes-established-as-the-de-facto-operating-system-for-ai-as-production-use-hits-82-in-2025-cncf-annual-cloud-native-survey/) — CNCF, 2026-01-20. Checked 2026-10-16.

**Further reading**

- [opengitops.dev](https://opengitops.dev/) — the principles' home page and community links.
- [flux2-kustomize-helm-example](https://github.com/fluxcd/flux2-kustomize-helm-example) and [flux2-multi-tenancy](https://github.com/fluxcd/flux2-multi-tenancy) — reference repositories linked from Flux's repository-structure guide.
- [argocd-example-apps](https://github.com/argoproj/argocd-example-apps) — the sample applications used in Lab 1.

**Changelog**

- 2026-10-16 — First published draft (managing author): research, ledger, 13 parts, 8 diagrams, 4 labs (2 scripts executed). Awaiting independent fact-check.
- 2026-10-16 — Revision after fact-check r1 (`reviews/gitops-r1-response.md`): Flux lockdown and credential-reach notes; per-tenant, per-environment SOPS keys; CI scripts render `tenants/` and `clusters/` (re-tested with a stub); ApplicationSet and break-glass preconditions; signing, webhook, attribution and Lab 4 fixes; repetition cut. Argo CD 3.6 was at rc3 on 2026-10-15: refresh the 3.5 lines when it ships.
- 2026-10-17, post-score polish (`reviews/gitops-polish.md`): GitHub Enterprise Server signing condition; polling range; multi-source self-heal caveat; self-contained Lab 3; runbook figure with step numbers and parallel restore; shorter decision-tree labels; restatements cut. Argo CD 3.6 still rc3 on 2026-10-17.