---
title: "ReleaseOps"
slug: releaseops
category: delivery
term_status: ambiguous
evidence_status: verified
summary: "ReleaseOps is an informal label for turning deployed code into a deliberate, reversible change for users: versioning, cadence, progressive exposure, feature flags, rollback and release records. The exact name is also a ServiceNow product."
audience: [Beginners, Release managers, DevOps and platform engineers, Tech leads, Engineering leaders]
last_verified: 2026-10-17
key_takeaways:
  - "\"ReleaseOps\" is not a standardised discipline: the exact name is best documented as a ServiceNow product, and the work it describes is known as release engineering and release management."
  - "Deploying installs code; releasing exposes behaviour to users. Separating them turns a release into a decision you can stage, measure and reverse."
  - "A progressive rollout only protects you if each step is large and long enough to detect the failure you fear: size steps from traffic, not habit, and set the alarm at the critical value, below the effect itself."
  - "Configuration, flags and data are releases too: in three well-documented failures, something that changed behaviour reached every server or region at once — a reused flag on an incompletely deployed fleet, a content file, a replicated policy."
  - "Rollback is a design property, not a button: make each step reversible (expand and contract, two-phase changes) and check what state a rollback recreates."
  - "Feature flags are inventory with a carrying cost: give each one an owner and an expiry date, and fail CI when it is overdue."
  - "Keep approval proportional: DORA found no evidence that external change boards lower change fail rates; peer review plus automated evidence usually serves better."
related: [devops, gitops, testops, itops, sre, mobileops, dbops, gitlabops]
---

## What ReleaseOps is

### In 30 seconds

**ReleaseOps is an informal name for the work that turns a finished change into something users experience — on purpose, in
steps, and with a way back.** It covers versions and notes, cadence, gradual exposure of a change already in production
(canaries, rings, flags), the decision to widen or abort, and the record of why.

Its central idea: **deploying and releasing are different events.** Deploying installs a version; releasing makes behaviour
available to people. Separated, a release becomes a risk decision to stage, measure and reverse. It is for release managers,
platform engineers, tech leads and leaders sizing release machinery.

::: note Term status: ambiguous — read this first
"ReleaseOps" has no standard definition. The exact name is best documented as a **ServiceNow product** (new in the Zurich release)
for deploying changes, customisations and custom apps on the ServiceNow platform[^sn-rn][^sn-explore]. The generic sense
appears mainly in a consultancy explainer[^opsio]; Mozilla's "Release Operations (RelOps) team" builds and operates the worker fleet
behind Firefox CI[^mozilla-relops], a different meaning. The work itself is established as **release engineering**, "building and
delivering software" in Google's SRE book[^sre-releng], and **release management**, the ITIL 4 practice of making new and changed
services available for use[^itil-rm]. This Atlas uses "ReleaseOps" as an **editorial label** for the release-strategy layer between
them.
:::

### At a glance

| Question | Answer |
|---|---|
| Goal | Changes reach users in small, observable, reversible steps |
| Users | Release managers, platform engineers, service owners, product owners who decide exposure |
| In scope | Deploy versus release, versioning, release branching and cadence, exposure mechanisms, flags, rollback, gates, approval and freezes, release records |
| Prerequisites | A pipeline producing versioned artefacts; telemetry that can compare two versions |
| Outputs | Release policy, versioned artefacts with notes, rollout definitions, flag registry, release records |
| Term status | `ambiguous`: informal label and a ServiceNow product name; the practices are established |

### The problem it solves

Most release pain comes from coupling three things with different risks: **installing** a version, **exposing** its behaviour and
**announcing** it. Coupled, every release is all-or-nothing:

- A defect reaches every user at once.
- The only remedy is a redeploy, unsafe if the new version has written data the old one cannot read.
- Marketing dates force unready code out, or hold ready code back.
- Heavy approval and long freezes compensate, enlarging batches.

Decoupled, the new version runs with no users, then staff, then a slice of traffic compared with the current version,
widening only on evidence. Google's SRE book describes almost all updates proceeding gradually, "according to a
defined process, with appropriate verification steps interspersed"[^sre-launch].

### What it is not

- **Not the pipeline** ([DevOps](../devops/)); ReleaseOps starts once an artefact exists.
- **Not the deployment mechanism** — reconciliation belongs to [GitOps](../gitops/); a canary controller sits on top.
- **Not ITIL change enablement** — change advisory boards (CABs) and change records are on [ITOps](../itops/).
- **Not test engineering** — whether a gate's signal is trustworthy is [TestOps](../testops/); whether to gate is here.
- **Not SAFe's "Agile Release Train"**, "a long-lived team of Agile teams", generally 50–125 people[^safe-art] — an
  organisational unit, not a schedule ([AgileOps](../agileops/)).

::: caution Editorial distinction
These boundaries are this Atlas's choice, so each topic is taught once; no standard draws them, and one team often does all of
it.
:::

## Where it comes from and where it sits

### Origins and evolution

The practices come from several traditions; there is no founding moment.

- **Configuration management.** Wingerd and Seiwald (1998) advised branching "only when necessary, on
  incompatible policy, late, and instead of freeze"[^trunk-release].
- **Continuous delivery.** Fowler (2010) credits the name blue-green to Daniel Terhorst-North and Jez Humble[^bluegreen];
  Sato's canary note cites Humble and Farley's *Continuous Delivery*[^canary-sato].
- **Release engineering at web scale.** Google's SRE book (2016) calls it "a relatively new and fast-growing discipline"[^sre-releng];
  the SRE Workbook (2018) adds canarying[^sre-canary].
- **Flags and progressive delivery.** Hodgson's 2017 toggle categories[^toggles]; RedMonk's "progressive delivery", August
  2018[^redmonk].

### Glossary and mental model

| Term | Meaning on this page | Example |
|---|---|---|
| Deploy / deploy dark | Install a version / install it with no users exposed yet | New pods at 0% traffic |
| Release | Make new behaviour available to some or all users | Flag `new-checkout` on for 10% |
| Canary | "A partial and time-limited deployment of a change in a service and its evaluation"[^sre-canary] | 5% of traffic on the new version |
| Progressive delivery | Widening exposure in steps with automated checks; RedMonk's term[^redmonk] | 5% → 25% → 50% → 100% |
| Ring, wave, cell, one-box | Exposure groups widened in order; a cell is a service's internal shard, a one-box stage its smallest deployable unit[^aws-handsoff] | Staff → one cell → one region → all |
| Bake time | Waiting after a step for slow-burning problems[^aws-handsoff] | One hour at 10% |
| SLI, SLO, error budget | Measured service level, its target, the unreliability it allows ([SRE](../sre/)) | 99.9% success over 28 days |
| p95 | 95th-percentile latency | p95 = 300 ms |
| DORA, CAB | DevOps Research and Assessment, part of Google Cloud since 2018[^dora-google]; change advisory board | — |
| Feature flag, kill switch | Runtime switch for behaviour (release, experiment, ops, permissioning kinds[^toggles]); a long-lived ops flag that disables a feature fast | `new-checkout` |
| Rollback / roll forward | Return to the last good state / fix with a newer version | `kubectl rollout undo` / `v1.4.1` |
| Expand and contract | Incompatible change in three phases — expand, migrate, contract[^parallel] | Add column, dual-write, backfill, drop |

**The mental model: two dials.** Every service has a *deployment dial* (which version runs where) and an *exposure dial* (who
experiences which behaviour). ReleaseOps turns the second deliberately — in steps sized to detect harm, with a named person allowed
to turn it back. Flags turn the exposure dial inside one version; canaries and blue-green turn both dials, using traffic.

### How ReleaseOps relates to other Ops

| Neighbour | It owns | ReleaseOps adds |
|---|---|---|
| [DevOps](../devops/) | CI/CD, trunk-based development, the five DORA metrics | Exposure strategy once the artefact exists |
| [GitOps](../gitops/) | Reconciliation, promotion between environments | Steps, analysis and abort rules on top |
| [TestOps](../testops/) | A fast, trustworthy test signal | Whether and where to gate |
| [DevSecOps](../devsecops/) | Scanner policy and exceptions | Security findings as a gate input |
| [ITOps](../itops/) | ITIL change enablement, CAB, CMDB | Proportional approval inside the release path |
| [SRE](../sre/) | SLOs and error-budget policies | Budget status as a release input |
| [MobileOps](../mobileops/) | Store review, phased app updates, old-version tail | General strategy |
| [DBOps](../dbops/) | Migration safety, locks, replication lag | Sequencing schema and code releases |
| [IncidentOps](../incidentops/) | Incident roles, communication, follow-ups | The release-side lever (abort, flag off) |

::: figure Where ReleaseOps sits among its neighbours
```mermaid
flowchart LR
  D["DevOps: versioned artefact"] --> R
  T["TestOps: trusted gate signal"] --> R
  S["SRE: SLOs and error budget"] --> R
  R["ReleaseOps: who gets which change, when, and how it is undone"]
  R --> G["GitOps: desired state to clusters"]
  R --> I["ITOps: change approval records"]
  R --> M["MobileOps: store and device limits"]
  R --> B["DBOps: schema changes in step"]
  R --> X["IncidentOps: when a release fails"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class R core
  class D,T,S,G,I,M,B,X neutral
```
Text equivalent: DevOps (artefact), TestOps (gate signal) and SRE (SLOs, error budget) feed ReleaseOps, which decides who gets which
change, when, and how it is undone; GitOps, ITOps, MobileOps, DBOps and IncidentOps carry out or record those decisions.
:::

## Principles and operating model

### Charter and principles

This charter is **this Atlas's proposal**; no official ReleaseOps manifesto exists.

> **Mission:** every change that alters what users experience reaches them in steps small enough to be safe, observable enough to
> judge and reversible enough to undo.

1. **Separate deploy from release** — in Hodgson's words, "separating [feature] release from [code] deployment"[^toggles].
2. **Every step must be able to see what you fear**; size steps from traffic ([worked calculation](#metrics-and-kpis)).
3. **Reversibility is designed**: each step, data included, is undoable or declared one-way in advance.
4. **Everything that changes behaviour is a release** — code, configuration, flags, content, policy data.
5. **Smallest sensible batch**: smaller release artefacts are cheaper and easier to roll back[^sre-canary].
6. **Explicit decision rights**: widening needs evidence and an owner; anyone on call may abort.
7. **Proportional control**: approval, freezes and bake time scale with risk.

**Non-goals:** zero-risk releases; a committee per change; flags as permanent configuration.

### Fundamentals: building blocks

#### Versioning: SemVer and CalVer

**Semantic Versioning 2.0.0** increments MAJOR for incompatible API changes, MINOR for compatible features and PATCH for compatible
fixes; in `0.y.z` "Anything MAY change at any time", and released contents must not change[^semver]. **CalVer** numbers by the
calendar — Ubuntu's `26.04` is `YY.0M`[^calver]. Use SemVer where consumers judge compatibility (libraries, APIs, agents), CalVer
where cadence matters more; a continuously deployed service needs neither publicly.

#### Branching as it affects releases

Trunk-based development is taught on [DevOps](../devops/). For releases:

- **Release from trunk, roll forward** — high-throughput teams keep no release branches[^trunk-release].
- **Release branches cut late** — the branch "should not receive continued development work"[^trunk-release]; fixes land on
  trunk and are cherry-picked (as at Google[^sre-releng]).
- **GitFlow** — its author now suggests a simpler workflow for continuous delivery, while it may still fit software with several
  supported versions[^gitflow].

#### Cadence: continuous, trains and long-term support

A **release train** leaves on schedule with whatever is ready: Chrome moved to four-week milestones in 2021, with an eight-week
Extended Stable option[^chrome]; Kubernetes releases about three times a year and patches each branch for about 12
months[^k8s-cycle]. Trains suit software customers install; continuous release suits services you operate.

#### Exposure mechanisms

| Mechanism | How exposure moves | Undo speed | Extra cost | Watch out for |
|---|---|---|---|---|
| Rolling update | Instances replaced in batches (Kubernetes default: 25% surge, 25% unavailable[^k8s-deploy]) | Minutes | Small surge | No traffic control; old and new serve side by side |
| Blue-green | Router switches all traffic between two environments | Seconds | Double capacity | Schema must suit both versions[^bluegreen]; judged before/after |
| Canary | A slice of traffic runs the new version and is compared | Seconds to minutes | Replicas or a router | Too little traffic |
| Rings or waves | Fixed groups widened in order with bake time | Varies | Pipeline time | Slow if every wave bakes fully |
| Feature flag | Code path enabled per user, cohort or percentage | Seconds | Flag service | Flag debt; untested combinations |

Old and new versions always share data and clients, side by side or in turn, so both must work with them. Blue-green fits when
users must never meet a mix of versions; unless traffic is split, it is "effectively performing a before/after canary"[^sre-canary].

### Roles, ownership and RACI

Titles vary; the decisions do not. One person may hold several roles, but on-call always holds the **abort** right.

| Activity | Feature owner | Service owner | Release manager | Platform team | On-call |
|---|---|---|---|---|---|
| Write and maintain the release policy | C | A | R | C | C |
| Version, notes and release record | I | A | R | C | I |
| Approve the change (peer review) | I | A/R | I | I | I |
| Start a progressive rollout | I | A | R | C | I |
| Widen exposure to the next step | A | R | C | I | C |
| Abort, roll back or flip a kill switch | I | I | I | I | A/R |
| Remove the flag and dead code | A | R | C | I | I |
| Maintain rollout and flag tooling | I | C | C | A/R | I |
| Post-release review | C | A | R | C | C |

R = responsible, A = accountable, C = consulted, I = informed. Kubernetes' release team, for example, leaves branch and patch
management to release managers[^k8s-team].

### Release lifecycle

**Day 0** designs for release, **Day 1** widens exposure step by step, **Day 2** cleans up.

::: figure The release lifecycle as an exposure ladder
```mermaid
flowchart TB
  A["Day 0: design for release (flag, compatible schema, stop signals)"] --> B["Deploy dark: new version running, 0% exposure"]
  B --> C["First ring: staff or one cell"]
  C --> Q1{"Signals healthy for the whole step?"}
  Q1 -- "Yes" --> E["Widen: 5% then 25% then 50%, with bake time"]
  E --> Q2{"Healthy at every step?"}
  Q2 -- "Yes" --> F["100%: released"]
  F --> H["Day 2: remove flag, contract schema, close record"]
  Q1 -- "No" --> K["Abort: flag off or traffic back to stable"]
  Q2 -- "No" --> K
  K --> L["Fix forward, then restart from the first ring"]
  L --> B
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class A,B,C,E,F,H core
  class Q1,Q2 decision
  class K,L risk
```
Text equivalent: Day 0 adds a flag, a compatible schema and stop signals. The version is deployed dark, exposed to staff or one
cell, then widened through 5%, 25% and 50% to 100% while each whole step stays healthy; Day 2 removes the flag, contracts the
schema and closes the record. An unhealthy step aborts (flag off or traffic back to stable); a fix is rolled forward and the
ladder restarts.
:::

### Operating model and governance

**Approval.** DORA's research (its capability page cites the 2019 State of DevOps Report) found approvals work best as peer review
plus automation; heavyweight external approval hurt delivery, with "no evidence" that it lowers change fail rates[^dora-change].
A CAB keeps a role in coordination, process improvement and business trade-offs. Make the normal path fast enough for emergencies;
reserve board review for risks automation cannot judge ([ITOps](../itops/)).

**Freezes.** Four different controls share the word:

| Control | What stops | Evidence and trade-off |
|---|---|---|
| Code freeze | Merging to the main branch | DORA advises "Don't have code freezes"[^dora-trunk]; Kubernetes still merges only critical fixes in its last ~4 weeks[^k8s-cycle] |
| Change freeze | Production changes for a period (peak sales, thin holiday cover) | Less change risk, but a larger next release; keep it short and keep deploying dark |
| Time-window blockers | Automatic deploys outside agreed hours or during alarms | Amazon uses both; windows too small make changes pile up[^aws-handsoff] |
| Error-budget freeze | Feature releases while reliability is below target | Google's example policy halts "all changes and releases other than P0 issues or security fixes"[^sre-ebp] |

**Cadence.** Review the policy quarterly, pauses, aborts and overrides weekly, flag debt monthly.

#### Gate policy

A gate is any check that can stop a release. The policy gives each an **effect** — *block*, *pause* (a person decides) or
*record* — and an override rule. Whether a verdict deserves that power is a [TestOps](../testops/) question; which security
findings block is set by [DevSecOps](../devsecops/). An Atlas proposal:

| Gate | Effect | Who may override | Recorded | Override expires |
|---|---|---|---|---|
| Build, unit and contract tests | Block | Nobody: fix or revert | CI run | — |
| Post-merge or nightly suite red | Block promotion of later builds | Service owner | Failing run, ticket | Next green run, ≤ 3 days |
| Security findings | As the DevSecOps policy says | Its exception approver | Exception ID | The exception's expiry |
| Canary analysis | *Failed* aborts; *Inconclusive* pauses | On-call resumes; a retry after *Failed* needs the service owner | Analysis run | This release |
| Check quarantined as flaky | Record | Owner of the quarantine | Quarantine ticket | Quarantine expiry |
| Freeze or spent error budget | Block feature releases | Service owner with SRE | Reason | End of freeze |

An override is a named person's decision, recorded with reason and expiry — never a pipeline variable anyone can set.

## Architecture and workflow

### Reference architecture

A release path has two control loops — **deployment** (which version runs, at what weight) and **exposure** (which users get which
behaviour) — each with an owner, both feeding one record.

::: figure Reference architecture of a progressive release path
```mermaid
flowchart TB
  subgraph B["Build and version"]
    SRC["Trunk with reviewed commits"] --> CI["CI: build once, test"] --> ART["Immutable artefact plus notes"]
  end
  subgraph C["Release control"]
    DEF["Release definition in Git: version, steps, stop rules"] --> CTRL["Rollout controller"]
    OC["Service owner and on-call"] --> FLAG["Flag service"]
  end
  subgraph P["Production"]
    RT["Traffic router"] --> ST["Stable version"]
    RT --> CN["Canary version"]
  end
  ART --> DEF
  CTRL --> RT
  ST --> TEL["Telemetry: SLIs per version"]
  CN --> TEL
  TEL --> AN["Analysis: canary against stable"]
  AN --> CTRL
  OC --> CTRL
  FLAG --> P
  C --> REC["Release record"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class SRC,CI,ART,ST,CN,RT neutral
  class DEF,CTRL,FLAG,TEL,REC core
  class AN,OC decision
```
Text equivalent: reviewed commits are built once into an immutable artefact with notes. A release definition in Git (version,
steps, stop rules) drives a rollout controller that weights a router in front of stable and canary versions; per-version
telemetry feeds an analysis telling it to continue, pause or abort. Service owner and on-call can override the controller and
control a flag service that switches behaviour per user in either version; the release-control layer writes the release record.
:::

**Responsibilities and trust boundaries.**

| Component | Responsibility | Who may change it |
|---|---|---|
| Artefact and notes | One immutable build per version | CI only; people edit notes |
| Release definition | Version, steps, stop rules | Reviewed merge, applied by [GitOps](../gitops/) |
| Rollout controller | Moves weights, runs analysis, aborts | Platform team; service team owns each rollout |
| Traffic router | Splits requests by weight or header (Gateway API `HTTPRoute`[^gw-split]) | Platform team |
| Flag service | Per-user exposure in any version | Feature owner widens; on-call switches off |
| Analysis | Compares versions on a few signals | Service owner sets thresholds |
| Release record | The decision trail | Automation writes; people add decisions |

**Alternatives.** Without Kubernetes, use weighted load-balancer targets, weighted function aliases, or flags plus rolling
deploys. Installed software has no router: exposure moves through rings of devices, and rollback means shipping a new version
([MobileOps](../mobileops/)).

### End-to-end workflow

A happy path, and exception paths named in advance.

::: figure Release workflow with its exception paths
```mermaid
sequenceDiagram
  participant P as Pipeline
  participant O as On-call
  participant R as Controller
  participant A as Analysis
  Note over P: Build, version, update Git
  Note over R: Deploy dark, canary at 5%
  R->>A: Start tripwire and step checks
  alt Healthy
    A-->>R: Successful
    Note over R: Widen to 25%, 50%, 100%
  else Errors over threshold
    A-->>R: Failed
    Note over R: Abort to stable
    R-->>O: Alert with analysis link
    O->>P: Revert in Git, fix forward
  else Few requests or unclear
    A-->>R: Inconclusive
    R-->>O: Paused, decision needed
  end
```
Text equivalent: the pipeline builds a merged change and records the new version in Git; the controller deploys it dark, sends
5% of traffic to the canary and starts the tripwire and step checks. Healthy: it widens to 25%, 50% and 100%. Errors over a
threshold: it aborts to stable and alerts on-call, who reverts the version in Git and has a fix rolled forward. Too few requests
or a value between thresholds: it pauses for on-call.
:::

**Exception paths.**

1. **Failed analysis.** Argo Rollouts aborts and stays *Degraded* until the desired state changes back[^argo-gs] (with GitOps, a
   revert commit).
2. **Inconclusive analysis.** It pauses for a person to resume or abort[^argo-analysis]; give every pause an owner and deadline.
3. **Stalled deployment.** "Kubernetes takes no action on a stalled Deployment other than to report a status
   condition"[^k8s-deploy]; something you own must act on `ProgressDeadlineExceeded`.
4. **Emergency fix.** Same path, shorter steps ([judgement call](#a-judgement-call-push-the-urgent-update-everywhere-now)).

## Practices, frameworks and tools

### Core practices and playbooks

| Practice | Trigger | Owner | Verification |
|---|---|---|---|
| Cut a release | Release PR merged or train departs | Release manager | Notes match the diff; version follows policy |
| Progressive rollout | New desired version | Service owner | Each step passed analysis or was decided by a person |
| Flag release | Code deployed dark | Feature owner | Business and error metrics per cohort |
| Hotfix on a release branch | Defect in a supported version | Release manager | Fix on trunk first, then cherry-picked |
| Expand and contract | Incompatible schema or API change | Service owner with [DBOps](../dbops/) | Both versions work between phases |

**Playbook: a standard progressive rollout.**

1. Check the record names version, risk class, steps, stop signals and abort owner; deploy dark and check health.
2. Expose the first ring (staff or one cell) for at least one bake period and the request minimum.
3. Widen in steps, comparing canary with stable over the same period — "time is one of the biggest sources of change in observed
   metrics", so before/after comparisons mislead[^sre-canary].
4. Watch the top few user-facing signals ("perhaps no more than a dozen"), one canary at a time per service[^sre-canary].
5. At 100%, schedule flag removal and schema contraction; close the record.

**Contingency.** Before step 1, record whether rollback is safe. If the release writes data the previous version cannot read,
split it: Amazon describes a "Prepare" version that reads the new format while still writing the old, then an "Activate" version
that writes it, so each step can be rolled back[^aws-rollback] — the same idea as *expand, migrate, contract*[^parallel].

### Methods, standards and frameworks

| Name | Kind | Governs | Status |
|---|---|---|---|
| Semantic Versioning 2.0.0[^semver], CalVer[^calver] | Specification, convention | Versions | Voluntary |
| Keep a Changelog 1.1.0[^kac], Conventional Commits 1.0.0[^cc] | Conventions | Changelogs; commit messages that map to SemVer | Voluntary; enable automation |
| OpenFeature[^openfeature] | Open specification (CNCF incubating) | Vendor-neutral flag API | Voluntary |
| ITIL 4 release and deployment management[^itil-rm][^itil-dm] | Service-management framework | Release and deployment practices | Voluntary unless contracted |
| DORA capabilities[^dora-change][^dora-trunk] | Research-based recommendations | Change approval, trunk-based development | Evidence, not a rule |
| Google SRE release engineering and canarying[^sre-releng][^sre-canary] | One organisation's documented practice | Branching, canary design | Adapt, do not copy |

None is mandatory by itself; auditors need evidence that changes were authorised and tested, not a particular process
([compliance](#quality-security-and-compliance)).

### Tooling landscape

Names are examples, not endorsements. Versions were checked on 2026-10-17 against each project's GitHub releases: re-check
there before copying an example, and review the table quarterly.

| Job | Representative options (as of October 2026) | Strengths | Limits |
|---|---|---|---|
| Version and notes | release-please (Release PRs; does not publish packages)[^release-please]; semantic-release (version, notes, publishing)[^semrel]; GitHub generated notes[^gh-notes] | No manual bumping | Only as good as commit discipline; notes need human editing |
| Progressive delivery on Kubernetes | Argo Rollouts (v1.10.1, 2026-10-12)[^argo]; Flagger (v1.45.0)[^flagger]; plain Deployments (rolling only) | Declarative steps, analysis, automatic abort | Another controller; only as good as the metrics |
| Traffic splitting | Gateway API `HTTPRoute` weights[^gw-split]; service meshes; load balancers | Precise percentages, header routing | Extra infrastructure |
| Feature flags | OpenFeature SDKs with flagd or another provider[^openfeature][^flagd-quick]; Unleash[^unleash]; commercial services | Per-user exposure, kill switches | Flag debt; a new runtime dependency |
| Platform-specific | ServiceNow ReleaseOps[^sn-explore]; GitLab ([GitLabOps](../gitlabops/)) | Integrated | Lock-in |

**Selection questions.** Do per-version metrics exist (if not, invest in observability first)? Who owns controller and flag
service? Are flag changes audited? Is abort one step for on-call?

### Automation and AI

Automate the mechanical and time-critical: version bumps, note drafts, analysis and abort. Keep people on thresholds, risky
widenings and *Inconclusive* results, which Argo Rollouts leaves to "human judgement"[^argo-analysis]. Automation fails too: noisy
aborts get overridden. Language models can draft notes; a person checks
breaking changes and security fixes, and no model decides the SemVer bump.

## ReleaseOps in practice

### Reference implementation

**Assumptions.** A stateless HTTP service on Kubernetes at about 50 requests per second, released from GitHub, with Argo Rollouts
v1.10.1 and its Gateway API traffic-router plugin (v0.17.0, 2026-09-01), Prometheus and flagd. Illustrative and syntax-checked; **none was executed for
this page.**

**Artefact 1 — the release policy as data**, one file per service, reviewed like code. A step ends only when both its bake time
and its request count are met, so here the `high` class's 1% step lasts about 2.2 hours, not 60 minutes
([sizing](#metrics-and-kpis)).

_Example status: YAML syntax-checked; illustrative policy, not consumed by any specific tool._

```yaml
# release-policy.yaml (illustrative)
service: checkout-api
owner: team-checkout
risk_classes:
  low:        # copy, styling, paths already behind a flag
    exposure_steps_percent: [10, 100]
    min_bake_minutes: 15
    approval: peer-review
  standard:   # most code changes
    exposure_steps_percent: [5, 25, 50, 100]
    min_bake_minutes: 30
    approval: peer-review
  high:       # payments, authentication, schema contraction, global config
    exposure_steps_percent: [1, 5, 25, 50, 100]
    min_bake_minutes: 60
    approval: peer-review plus service owner
    requires: [kill-switch, rollback-rehearsed, traffic-router]
step_evidence:  # a step ends when BOTH min_bake_minutes and this count are met
  min_canary_requests: 4000   # 80% chance to catch 0.2% -> 0.4% (see sizing)
  too_few_requests: pause     # a person extends the step or aborts
stop_signals:   # alarm at the critical value, below the effect to be caught
  - "step end: canary 5xx minus stable over 0.12 points pauses, over 0.2 aborts"
  - "any 5-minute window at 5% or more: over 1 point aborts (gross failure)"
  - "canary p95 latency above 1.2 x stable p95"
abort_rights: [on-call, service-owner, release-manager]
gates:          # effect: block | pause | record (see Gate policy)
  tests-pre-merge:   {effect: block, override: none}
  post-merge-suite:  {effect: block-promotion, override: service-owner,
                      max_days: 3}
  security-findings: {effect: per-devsecops-policy,
                      override: exception-approver}
  canary-analysis:   {failed: abort, inconclusive: pause, retry: service-owner}
  quarantined-check: {effect: record, until: quarantine-expiry}
override_record: [person, gate, release, reason, expires]
freezes:
  error_budget_spent: "block feature releases; allow fixes and security changes"
  deploy_windows: "automatic starts Mon-Thu 08:00-16:00; emergencies exempt"
flags:
  release_flag_max_age_days: 30
  required_metadata: [owner, kind, expires]
```

**Artefact 2 — the `standard` class as a canary that fails closed at every step.** Two error-rate checks, two jobs (the
policy's latency signal is omitted for brevity):

- A **background tripwire**, from the first pause to promotion, compares canary and stable 5xx rates every two minutes over
  five-minute windows and aborts only on gross failures: canary errors over 1 percentage point above stable (about six
  times the baseline).
- A **step-end check** after each pause requires the policy's 4,000 canary requests (about 27 minutes at 5% of 50 requests per
  second) and judges the whole step at the [sized](#metrics-and-kpis) alarm threshold: up to 0.12 points passes, over 0.2
  aborts; in between, or with too few requests, the run is *Inconclusive* and pauses for a person[^argo-analysis].

Simulated (10,000 runs per case, Poisson errors; not a cluster test): no healthy rollout tripped the tripwire and about 5%
stopped once, mostly as a pause; a 0.4% canary was stopped at the 5% step in 80% of runs and never reached 100%; a 2% canary
aborted seven minutes in. At a 1% step two errors in five minutes cross 1 point, aborting 57% of healthy simulated steps, so use
30-minute windows there. `or vector(0)` reads "no errors" as zero; `len(result) > 0` fails an empty result; `failureLimit: 1`
aborts on the second failed measurement[^argo-analysis]. The Gateway API plugin[^argo-gwapi] sets exact weights on an
`HTTPRoute`; without a router, Argo Rollouts uses replica counts, so the smallest step is one pod — 10% of 10
replicas[^argo-canary].

_Example status: YAML syntax-checked; PromQL not executed; thresholds checked by simulation only. Assumes the plugin is
registered with the controller, an `HTTPRoute` `checkout-api` whose `backendRefs` are the two Services, and request metrics
labelled `rollouts_pod_template_hash` (copied from the pod label)._

```yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-vs-stable   # background tripwire: gross failures only
spec:
  args:
    - name: canary             # pod-template hashes of the two versions
    - name: stable
  metrics:
    - name: error-rate-delta   # sized for steps of 5% or more at ~50 rps
      initialDelay: 5m         # wait for a full window of canary data
      interval: 2m             # no count: runs until promotion or failure
      failureLimit: 1          # the second failed measurement aborts
      successCondition: len(result) > 0 && result[0] <= 0.01
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            (
              (sum(rate(http_requests_total{app="checkout-api",
                code=~"5..", rollouts_pod_template_hash="{{args.canary}}"}[5m]))
                or vector(0))
              / sum(rate(http_requests_total{app="checkout-api",
                rollouts_pod_template_hash="{{args.canary}}"}[5m]))
            ) - (
              (sum(rate(http_requests_total{app="checkout-api",
                code=~"5..", rollouts_pod_template_hash="{{args.stable}}"}[5m]))
                or vector(0))
              / sum(rate(http_requests_total{app="checkout-api",
                rollouts_pod_template_hash="{{args.stable}}"}[5m]))
            )
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: step-evidence          # inline: no interval, one measurement per step
spec:
  args:
    - name: canary
    - name: stable
  metrics:
    - name: enough-canary-requests  # 0 to 3,999: Inconclusive, rollout pauses
      successCondition: len(result) > 0 && result[0] >= 4000
      failureCondition: len(result) == 0
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            sum(increase(http_requests_total{app="checkout-api",
                rollouts_pod_template_hash="{{args.canary}}"}[30m]))
    - name: error-rate-delta-over-step
      # alarm at the critical value for 4,000 requests, not at the 0.2 effect:
      # <= 0.12 points passes; 0.12-0.2 is Inconclusive (pause); > 0.2 aborts
      successCondition: len(result) > 0 && result[0] <= 0.0012
      failureCondition: len(result) == 0 || result[0] > 0.002
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            (
              (sum(increase(http_requests_total{app="checkout-api",
                code=~"5..", rollouts_pod_template_hash="{{args.canary}}"}[30m]))
                or vector(0))
              / sum(increase(http_requests_total{app="checkout-api",
                rollouts_pod_template_hash="{{args.canary}}"}[30m]))
            ) - (
              (sum(increase(http_requests_total{app="checkout-api",
                code=~"5..", rollouts_pod_template_hash="{{args.stable}}"}[30m]))
                or vector(0))
              / sum(increase(http_requests_total{app="checkout-api",
                rollouts_pod_template_hash="{{args.stable}}"}[30m]))
            )
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
spec:
  replicas: 10
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: api
          image: registry.example.com/checkout-api:1.4.0
          ports:
            - containerPort: 8080
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: checkout-api
      analysis:                # background, from step index 1 (the first pause)
        startingStep: 1
        templates:
          - templateName: error-rate-vs-stable
        args: &hashes          # YAML anchor, reused by every step-end check
          - name: canary
            valueFrom: {podTemplateHashValue: Latest}
          - name: stable
            valueFrom: {podTemplateHashValue: Stable}
      steps:                   # standard class: 5, 25, 50, then 100
        - setWeight: 5
        - pause: {duration: 30m}
        - analysis: {templates: [{templateName: step-evidence}], args: *hashes}
        - setWeight: 25
        - pause: {duration: 30m}
        - analysis: {templates: [{templateName: step-evidence}], args: *hashes}
        - setWeight: 50
        - pause: {duration: 30m}
        - analysis: {templates: [{templateName: step-evidence}], args: *hashes}
```

**Artefact 3 — a release flag with owner and expiry.** flagd allows flag-level metadata[^flagd-schema], where this page stores
`owner`, `kind` and `expires` for CI (Lab 2). `fractional` buckets users deterministically on flag key plus user ID[^flagd-frac];
staff are matched by an attribute, not an e-mail address, so the flag service gets no personal data.

_Example status: JSON syntax-checked; evaluated only in the unexecuted Lab 2._

```json
{
  "$schema": "https://flagd.dev/schema/v0/flags.json",
  "flags": {
    "new-checkout": {
      "state": "ENABLED",
      "variants": { "on": true, "off": false },
      "defaultVariant": "off",
      "targeting": {
        "if": [
          { "==": [{ "var": "staff" }, true] },
          "on",
          {
            "fractional": [
              { "cat": [{ "var": "$flagd.flagKey" }, { "var": "userId" }] },
              ["on", 10],
              ["off", 90]
            ]
          }
        ]
      },
      "metadata": {
        "owner": "team-checkout",
        "kind": "release",
        "expires": "2026-11-30"
      }
    }
  }
}
```

**Artefact 4 — versions and notes on GitHub.** release-please maintains a Release PR from Conventional Commits; merging it updates
the changelog, tags the commit and creates a GitHub release[^release-please]. The action's README grants `contents`, `issues` and
`pull-requests` write permissions and warns that releases and PRs created with the default `GITHUB_TOKEN` trigger no further
workflows (current major: v5, 2026-04-22)[^rp-action]. For third-party actions GitHub now calls a workflow lockfile "the
strongest protection available", with full-commit-SHA pinning as the fallback[^gh-secure].

_Example status: YAML syntax-checked; not executed._

```yaml
# .github/workflows/release-please.yml
name: release-please
on:
  push:
    branches: [main]
permissions:
  contents: write
  issues: write
  pull-requests: write
jobs:
  release-please:
    runs-on: ubuntu-latest
    steps:
      # Production: use a workflow lockfile, or pin v5's full commit SHA.
      - uses: googleapis/release-please-action@v5
        with:
          release-type: simple
```

_Example status: YAML syntax-checked; not executed._

```yaml
# .github/release.yml: categories for GitHub's generated release notes
changelog:
  exclude:
    labels: [ignore-for-release]
  categories:
    - title: Breaking changes
      labels: [breaking-change]
    - title: New features
      labels: [enhancement]
    - title: Other changes
      labels: ["*"]
```

::: warning Who can release?
Whoever merges to `main` shapes the Release PR; whoever merges it creates a tag. If tags deploy to production, merge rights *are*
release rights: require reviews, protect tags, grant only the three permissions above, and if another token replaces `GITHUB_TOKEN`,
scope it to this repository and expose it only to this job.
:::

### Hands-on labs

**Lab 1 and Lab 2's linter were executed** (last on 2026-10-17, Ubuntu, git 2.43.0, Python 3.8.10); **the rest was not**. Use
throwaway directories and clusters only.

#### Lab 1 (beginner): decide the version and draft the notes

**Objective.** Turn commit conventions into a SemVer decision and a changelog section[^cc][^kac]. **Prerequisites.** git 2.28+,
Python 3.8+. **Cost and risk.** None (temporary directory).

1. Save this script as `next_release.py`:

_Example status: executed on 2026-10-17 (Python 3.8.10)._

```python
#!/usr/bin/env python3
"""Propose the next SemVer version and a changelog section.

Reads Conventional Commits since the last vX.Y.Z tag.
"""
import re
import subprocess
import sys
from datetime import date

TAG_RE = re.compile(r"^v(\d+)\.(\d+)\.(\d+)$")
HEADER_RE = re.compile(
    r"^(?P<type>\w+)(?:\((?P<scope>[^)]+)\))?(?P<bang>!)?: (?P<desc>.+)$"
)
RANK = {"patch": 1, "minor": 2, "major": 3}


def git(*args):
    done = subprocess.run(
        ["git", *args], check=True, capture_output=True, text=True
    )
    return done.stdout


def last_tag():
    listed = git("tag", "--list", "v*", "--sort=-v:refname").split()
    tags = [t for t in listed if TAG_RE.match(t)]
    return tags[0] if tags else None


def commits_since(tag):
    span = f"{tag}..HEAD" if tag else "HEAD"
    raw = git("log", span, "--format=%h%x1f%B%x1e")
    for entry in filter(None, (e.strip() for e in raw.split("\x1e"))):
        sha, _, body = entry.partition("\x1f")
        yield sha, body.strip()


def main():
    tag = last_tag()
    major, minor, patch = (
        map(int, TAG_RE.match(tag).groups()) if tag else (0, 0, 0)
    )
    bump, skipped = None, []
    # Keep a Changelog headings
    sections = {"Changed": [], "Added": [], "Fixed": []}
    for sha, body in commits_since(tag):
        m = HEADER_RE.match(body.splitlines()[0])
        if not m:
            skipped.append(sha)
            continue
        footer = re.search(r"^BREAKING[ -]CHANGE: ", body, re.M)
        breaking = bool(m["bang"]) or footer is not None
        kinds = {"feat": "minor", "fix": "patch"}
        level = "major" if breaking else kinds.get(m["type"])
        if level is None:  # docs:, chore:, ci: ... do not trigger a release
            continue
        bump = level if bump is None or RANK[level] > RANK[bump] else bump
        if breaking:
            sections["Changed"].append(f"- BREAKING: {m['desc']} ({sha})")
        else:
            heading = "Added" if m["type"] == "feat" else "Fixed"
            sections[heading].append(f"- {m['desc']} ({sha})")
    if bump is None:
        print(f"No releasable changes since {tag or 'the first commit'}.")
        return 0
    if major == 0 and bump == "major":
        # team policy for 0.y.z, where SemVer says anything may change
        bump = "minor"
    nxt = {
        "major": (major + 1, 0, 0),
        "minor": (major, minor + 1, 0),
        "patch": (major, minor, patch + 1),
    }[bump]
    version = ".".join(map(str, nxt))
    print(f"Last release: {tag or 'none'} -> next: v{version} ({bump})")
    print(f"\n## [{version}] - {date.today().isoformat()}")
    for title, lines in sections.items():
        if lines:
            print(f"### {title}")
            print("\n".join(lines))
    if skipped:
        print("\nNot Conventional Commits, review by hand: "
              + ", ".join(skipped), file=sys.stderr)
    return 0


if __name__ == "__main__":
    sys.exit(main())
```

2. Save this driver as `run.sh` beside it and run `bash run.sh`:

_Example status: executed on 2026-10-17 (git 2.43.0)._

```bash
#!/usr/bin/env bash
set -euo pipefail
SCRIPT="$(cd "$(dirname "$0")" && pwd)/next_release.py"
LAB="$(mktemp -d)"
cd "${LAB}"
git init -q -b main demo && cd demo
git config user.email "lab@example.com" && git config user.name "Release Lab"
echo "v1" > app.txt && git add app.txt
git commit -qm "feat: first public API"
git tag v1.0.0
echo "a" >> app.txt && git commit -qam "fix: handle an empty basket"
echo "b" >> app.txt && git commit -qam "feat(search): add fuzzy matching"
echo "c" >> app.txt && git commit -qam "docs: clarify local setup"
echo "d" >> app.txt && git commit -qam "Update stuff"
python3 "${SCRIPT}"
echo "----- after a breaking change -----"
echo "e" >> app.txt
git commit -qam "feat(api)!: remove the v1 search endpoint"
python3 "${SCRIPT}"
echo "LAB=${LAB}"
```

**Expected output** (hashes and date differ):

```text
Last release: v1.0.0 -> next: v1.1.0 (minor)

## [1.1.0] - 2026-10-16
### Added
- add fuzzy matching (22b7a6f)
### Fixed
- handle an empty basket (2734245)
----- after a breaking change -----
Last release: v1.0.0 -> next: v2.0.0 (major)

## [2.0.0] - 2026-10-16
### Changed
- BREAKING: remove the v1 search endpoint (55c8e41)
...
```

Standard error also lists "Update stuff" as not conventional. **Validation.** The `docs:` commit causes no bump; the breaking change
moves `1.1.0` to `2.0.0`. **Troubleshooting.** `unknown switch 'b'`: git older than 2.28. **Teardown.** `rm -rf "$LAB"`.
**Reflect.** A fix hidden in "Update stuff" would ship unannounced — lint commit messages in CI.

#### Lab 2 (intermediate): release a feature with a flag, then kill it

**Objective.** Release to staff, then 10% of users, then switch off, with flagd; make flag debt fail CI. **Prerequisites.** Docker,
`curl`, Python 3.8+. **Cost and risk.** None.

1. Save this as `flags/demo.flagd.json`: Artefact 3 with `null` in place of the `fractional` expression (staff only), plus an
   expired flag for step 5.

_Example status: JSON syntax-checked; the linter in step 5 was executed against this file._

```json
{
  "$schema": "https://flagd.dev/schema/v0/flags.json",
  "flags": {
    "new-checkout": {
      "state": "ENABLED",
      "variants": { "on": true, "off": false },
      "defaultVariant": "off",
      "targeting": {
        "if": [{ "==": [{ "var": "staff" }, true] }, "on", null]
      },
      "metadata": {
        "owner": "team-checkout", "kind": "release", "expires": "2026-11-30"
      }
    },
    "legacy-search": {
      "state": "ENABLED",
      "variants": { "on": true, "off": false },
      "defaultVariant": "on",
      "metadata": {
        "owner": "team-search", "kind": "release", "expires": "2026-08-31"
      }
    }
  }
}
```

2. Start flagd (pinned to the version whose quick start this follows[^flagd-quick]), bound to loopback so other machines cannot
   reach it:

_Example status: `bash -n` syntax-checked; not executed (no Docker on the authoring host)._

```bash
#!/usr/bin/env bash
set -euo pipefail
docker run --rm -d --name flagd -p 127.0.0.1:8013:8013 \
  -v "$(pwd)/flags:/flags" \
  ghcr.io/open-feature/flagd:v0.17.1 start --uri file:./flags/demo.flagd.json
URL="http://127.0.0.1:8013/flagd.evaluation.v2.Service/ResolveBoolean"
eval_flag() {  # usage: eval_flag <true|false: staff> <userId>
  local ctx="{\"staff\":$1,\"userId\":\"$2\"}"
  curl -s -X POST "${URL}" -H "Content-Type: application/json" \
    -d "{\"flagKey\":\"new-checkout\",\"context\":${ctx}}"
  echo
}
eval_flag true "u-0"     # staff
eval_flag false "u-1"    # customer
```

3. Restore the `fractional` expression and save (flagd reloads the file); count `true` across 200 customers:

_Example status: `bash -n` syntax-checked; not executed._

```bash
#!/usr/bin/env bash
set -euo pipefail
URL="http://127.0.0.1:8013/flagd.evaluation.v2.Service/ResolveBoolean"
on=0
for i in $(seq 1 200); do
  ctx="{\"staff\":false,\"userId\":\"u-${i}\"}"
  req="{\"flagKey\":\"new-checkout\",\"context\":${ctx}}"
  v=$(curl -s -X POST "${URL}" -H "Content-Type: application/json" \
        -d "${req}" \
      | python3 -c 'import json,sys; print(json.load(sys.stdin).get("value"))')
  [ "${v}" = "True" ] && on=$((on + 1))
done
echo "on for ${on} of 200 customers"
```

4. Kill switch: set `"defaultVariant": "off"`, delete `targeting`, and repeat step 2's two evaluations.
5. Save this linter as `flag_lint.py`; run `python3 flag_lint.py flags/demo.flagd.json 2026-10-16`.

_Example status: executed on 2026-10-17 against the lab file (exit code 1, one problem reported)._

```python
#!/usr/bin/env python3
"""Fail when a release flag lacks an owner or expiry, or is past its expiry.

A 'time bomb' check for CI.
"""
import json
import sys
from datetime import date

# kill switches and entitlements are reviewed, not expired
LONG_LIVED = {"ops", "permission"}


def main(path, today):
    with open(path, encoding="utf-8") as fh:
        flags = json.load(fh)["flags"]
    problems = []
    for key, flag in sorted(flags.items()):
        meta = flag.get("metadata", {})
        kind = meta.get("kind", "release")
        if not meta.get("owner"):
            problems.append(f"{key}: no owner")
        if kind in LONG_LIVED:
            continue
        expires = meta.get("expires")
        if not expires:
            problems.append(f"{key}: {kind} flag without an expiry date")
        elif date.fromisoformat(expires) < today:
            problems.append(
                f"{key}: expired on {expires}"
                " - remove the flag and its dead code path"
            )
    for p in problems:
        print("FLAG-DEBT", p)
    print(f"{len(flags)} flags checked, {len(problems)} problem(s)")
    return 1 if problems else 0


if __name__ == "__main__":
    arg = sys.argv[2] if len(sys.argv) > 2 else None
    when = date.fromisoformat(arg) if arg else date.today()
    sys.exit(main(sys.argv[1], when))
```

**Expected output.** Step 2: staff `true` (reason `TARGETING_MATCH`), customer `false` (`DEFAULT`). Step 3: roughly 20 of 200, the
*same* users on every re-run, because fractional assignment is deterministic[^flagd-frac]. Step 4: everyone `false` (`STATIC`).
Step 5: `FLAG-DEBT legacy-search: expired on 2026-08-31 …`, exit code 1. **Validation.** Changing a customer's `userId` can move
them between cohorts; adding other attributes does not. **Troubleshooting.** Connection refused: check
`docker logs flagd`; `FLAG_NOT_FOUND`: wrong key or file path. **Teardown.** `docker stop flagd`; delete `flags/`.

#### Lab 3 (advanced): a canary that aborts itself

**Objective.** Watch analysis stop a bad version at 20% exposure. **Prerequisites.** Docker, kind v0.33.0, kubectl, the Argo
Rollouts kubectl plugin, about 4 GB of free memory. **Cost and risk.** Local only.

_Example status: `bash -n` and YAML syntax-checked; not executed. Images: `argoproj/rollouts-demo` (colour tags; `bad-*` tags
return a high rate of errors)[^rollouts-demo] and `curlimages/curl:8.23.0`._

```bash
#!/usr/bin/env bash
set -euo pipefail
kind create cluster --name release-lab
kubectl create namespace argo-rollouts
REL="https://github.com/argoproj/argo-rollouts/releases/download/v1.10.1"
kubectl apply -n argo-rollouts -f "${REL}/install.yaml"
kubectl -n argo-rollouts rollout status deployment/argo-rollouts --timeout=180s
kubectl apply -f canary-lab.yaml
kubectl argo rollouts status demo --timeout 180s
# A good release; press Ctrl-C when the rollout shows Healthy.
kubectl argo rollouts set image demo demo=argoproj/rollouts-demo:yellow
kubectl argo rollouts get rollout demo --watch
# A bad release.
kubectl argo rollouts set image demo demo=argoproj/rollouts-demo:bad-red
kubectl argo rollouts get rollout demo --watch
```

`canary-lab.yaml`:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: smoke
spec:
  metrics:
    - name: http-200
      count: 2
      interval: 30s
      failureLimit: 0
      provider:
        job:
          spec:
            backoffLimit: 0
            template:
              spec:
                restartPolicy: Never
                containers:
                  - name: smoke
                    image: curlimages/curl:8.23.0
                    command: ["sh", "-c"]
                    args:
                      - |
                        fails=0; i=0; u=http://demo-canary/color
                        while [ "$i" -lt 40 ]; do
                          code=`curl -s -o /dev/null -w '%{http_code}' "$u"`
                          [ "$code" = "200" ] || fails=`expr "$fails" + 1`
                          i=`expr "$i" + 1`
                        done
                        echo "non-200 responses: $fails of 40"
                        [ "$fails" -le 2 ]
---
apiVersion: v1
kind: Service
metadata:
  name: demo-canary
spec:
  selector:
    app: demo
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: demo-stable
spec:
  selector:
    app: demo
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: demo
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: demo
          image: argoproj/rollouts-demo:blue
          ports:
            - containerPort: 8080
          readinessProbe:
            tcpSocket:
              port: 8080
  strategy:
    canary:
      canaryService: demo-canary
      stableService: demo-stable
      steps:
        - setWeight: 20
        - analysis:
            templates:
              - templateName: smoke
        - setWeight: 60
        - pause:
            duration: 30s
```

**Expected output.** `yellow`: smoke Jobs pass and the rollout becomes Healthy. `bad-red`: one canary pod (20% of 5), the smoke Job
fails, the rollout aborts to stable `yellow` and shows Degraded[^argo-gs]. **Validation.** `kubectl get analysisrun` shows the failed
run. **Troubleshooting.** *Inconclusive*: the Job image could not be pulled[^argo-job]. The TCP readiness probe is deliberate: an HTTP probe on `/color` would
hide the faulty pod from analysis. **Recover.** Set the image back to `yellow` (GitOps: revert). **Teardown.** `kind delete cluster --name release-lab`.

### Use cases and scenarios

All three are **illustrative scenarios** (composites, not real organisations); sourced cases follow
[later](#three-documented-failures).

::: example Scenario 1 — a SaaS checkout redesign with a fixed marketing date
**Problem.** A redesigned checkout launches in six weeks; payments are high-risk. **Method.** Merges deploy dark behind
`new-checkout`; staff use it for two weeks; the flag goes 1% → 5% → 25% → 100% in the last week, while each binary still passes a
canary. **Trade-offs.** Two code paths to test. **Success criteria.** Conversion and payment errors per cohort within bounds;
launch day needs no deploy. **Pitfall.** Keeping the old path beyond 30 days "just in case".
:::

::: example Scenario 2 — an on-premises product and its SDK
**Problem.** Customers upgrade a few times a year; several versions are supported. **Method.** SemVer, a quarterly train with
late release branches and cherry-picked fixes, `[YANKED]` for withdrawn versions[^kac], published support windows.
**Trade-offs.** CI per supported branch[^trunk-release]; backports. **Success criteria.** Upgrade success rate; time from trunk fix
to patch. **Pitfall.** Copying the train to services you operate only adds waiting.
:::

::: example Scenario 3 — a payments platform with a change advisory board
**Problem.** A weekly CAB makes releases bundle 40 changes, which fail often. **Method.** A *standard change* route agreed with the
ITIL owners ([ITOps](../itops/)) pre-approves reviewed, tested changes released progressively with recorded analysis; the CAB
reviews the route and still sees novel or high-risk changes. **Trade-offs.** Evidence work; auditors must accept recorded review.
**Success criteria.** Smaller batches; better change fail rate and recovery time. **Pitfall.** Auto-filling the old ticket for a
40-change bundle automates the form, not the control.
:::

## Measure, govern, pay

### Metrics and KPIs

DORA's five delivery metrics are defined on [DevOps](../devops/)[^dora-metrics], incident durations on
[IncidentOps](../incidentops/). Release-specific measures:

| KPI | Definition and formula | Data source | Pitfall | Review |
|---|---|---|---|---|
| Early-detection ratio | Release-caused defects stopped at ≤ 10% exposure ÷ all release-caused defects | Release records | Small counts | Quarterly |
| Exposure per bad release | Σ (affected users × minutes until safe state) | Analysis runs, flag log | "Affected" is an estimate | Per event |
| Time to safe state | Detection → exposure removed | Controller, flag log | Not time to repair | Weekly |
| False-abort rate | Aborts not caused by the release ÷ aborts; also the Inconclusive share | Controller history | Teaches people to override | Monthly |
| Release lead time | Merge → 100% exposure | Git, controller, flags | Not DORA's lead time (ends at deployment) | Monthly |
| Flag debt | Overdue release flags ÷ release flags | Flag registry, linter | Kill switches are not debt | Monthly |

**Worked calculation (illustrative month).** Of 10 release-caused defects, 7 stopped at ≤ 10%: early detection **70%**. One defect reached 5% of 200,000 users for 20 minutes:
10,000 × 20 = **200,000 user-minutes**, against 4,000,000 at full exposure. Five of 18 release flags are overdue: flag debt
**28%**.

**Worked calculation: how long must a canary step last?** The stable version fails 0.2% of requests; you want to detect a canary
failing 0.4%, accepting a 5% false-alarm chance and an 80% detection chance. A normal approximation that treats the stable rate as
known gives

n ≈ ((1.645 × √(0.002 × 0.998) + 0.84 × √(0.004 × 0.996)) ÷ 0.002)² ≈ **4,000 canary requests**.

**The alarm threshold is not the effect size.** Alarm at the critical value — stable plus
1.645 × √(0.002 × 0.998 ÷ 4,000) ≈ **0.12 percentage points** — not at the 0.2-point effect you fear: an alarm at the effect
itself catches a true doubling only about half the time, at any traffic. The cost is about one false alarm in twenty, so that band
should pause for a person, not abort ([Artefact 2](#reference-implementation)).

At 50 requests per second in total, a 1% canary needs about **2.2 hours**, a 10% canary about **13 minutes**; detecting 0.2% → 0.3%
needs about 14,300. A five-minute 1% step sees 150 requests and 0.3 expected errors: it flags only failures around ten times the
baseline (about 2% errors), not a doubling. Use larger steps, longer bake, staff traffic or synthetic checks. This ignores
correlated errors and multiple metrics: a sanity check, not a statistical design.

### Quality, security and compliance

- **Decision rights are access rights.** Map who may widen, abort, override, and change the policy or flags (audited).
- **Fail-safe flag defaults.** OpenFeature evaluation "must always return the default value in the event of abnormal
  execution"[^of-spec], so the default in code is what users get when the flag service is down: make it the safe behaviour.
- **Supply chain.** Release the artefact that was tested — one build, signed, with provenance ([DevSecOps](../devsecops/)).
- **Privacy.** Send the flag service only the attributes its rules need; prefer opaque IDs to e-mail addresses.
- **Compliance evidence.** DORA notes that peer review in the development platform can meet segregation of
  duties[^dora-change]; a record linking pull request, pipeline run, analysis verdicts and flag log beats meeting minutes.
  Controls vary by industry and jurisdiction; confirm with auditors ([ComplianceOps](../complianceops/)).

### Cost and capacity

- **Double capacity.** Blue-green runs two environments. Argo Rollouts with traffic routing keeps stable fully scaled — instant
  abort, but double the pods until the update ends; `dynamicStableScale` saves capacity at the cost of a slower abort[^argo-canary].
- **Duplicated CI.** Each active release branch needs its own CI[^trunk-release] and backports.
- **Waiting.** Amazon's default pipelines take about four or five business days to reach all Regions[^aws-handsoff] — right for
  global services, wrong for an internal tool.

### Assurance of the release process

Watch the release system itself: complete records, pauses older than a day, gate overrides, freeze exceptions, flag debt and
rollback rehearsals. Each quarter, trace five releases from pull request to full exposure; where the trail breaks, fix the
automation, not the paperwork.

## When releases go wrong

### Common failures and troubleshooting

| Symptom | Likely cause | Diagnosis | Remediation | Prevention |
|---|---|---|---|---|
| Canary passed, users complain at 100% | Signal missing; step too small or short; slow burn | Compare the metric across steps | Flag off or roll back | Size steps from traffic; user-facing SLIs |
| Analysis Inconclusive or "no data" | Too few requests; value between thresholds; label mismatch | Run the query by hand for both versions | Fix query; resume or abort deliberately | `len(result) > 0`; `or vector(0)` |
| Rollback made it worse | Old version cannot read new data; flag meaning changed; partial fleet | Diff data formats and flag semantics | Roll forward | Expand and contract; never reuse a flag |
| Flag change caused an outage | Flag change not staged; global propagation | Flag audit log vs timeline | Revert the flag | Stage flag and config changes |
| Same bug returns next release | Fixed on the release branch only | Search trunk history | Land on trunk, cherry-pick | Trunk first[^trunk-release] |

### Worked runbook: a stop signal fires during a rollout

**Trigger:** analysis fails or pauses, an alert fires, or users report a problem mid-rollout. **Owner:** on-call, without
asking.

::: figure Decision tree for backing out a release
```mermaid
flowchart TB
  S["Stop signal or user report during a rollout"] --> W["Freeze widening; open an incident if users are hurt"]
  W --> F{"Is the change behind a run-time flag?"}
  F -- "Yes" --> OFF["Turn the flag off for everyone"]
  OFF --> V1{"Metrics back to baseline?"}
  V1 -- "Yes" --> RECF["Record the flag state; fix before re-enabling"]
  V1 -- "No: defect outside the flag" --> D
  F -- "No" --> D{"Has the new version written data the old one cannot read?"}
  D -- "No" --> RB["Abort the rollout or roll back"]
  D -- "Yes" --> RF["Roll forward: fix through a short canary"]
  RB --> V2{"Metrics back to baseline everywhere?"}
  RF --> V2
  V2 -- "Yes" --> REC["Record; revert desired state in Git by pull request"]
  V2 -- "No" --> INC["Escalate the incident: the cause may lie elsewhere"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class S,W,OFF,RB,REC,RECF core
  class F,D,V1,V2 decision
  class RF,INC risk
```
Text equivalent: freeze widening, opening an incident if users are hurt. If the change is behind a run-time flag,
turn it off; if metrics recover, record the flag state and fix before re-enabling. Otherwise — no flag, or a defect outside it —
abort or roll back if the new version has written no data the old one cannot read, else roll forward. If metrics then recover
everywhere, record it and revert the desired state in Git by pull request; if not, escalate: the cause may lie elsewhere.
:::

**Steps.**

1. **Stop widening** for the service; start no other deployments "to compare".
2. **Confirm by comparison** with stable, and check the canary had traffic.
3. **Choose the lever** with the tree: flag first, then abort or rollback, then roll forward.
4. **Execute** the one alternative that matches your setup (adapt names; syntax-checked, not executed). On Argo Rollouts with
   GitOps, run `argo` and then `gitops`, or the next reconciliation re-applies the bad version.

```bash
#!/usr/bin/env bash
# Usage: back-out.sh argo | deployment | gitops
set -euo pipefail
case "${1:?choose argo, deployment or gitops}" in
  argo)        # traffic back to stable; Degraded until desired state changes
    kubectl argo rollouts abort checkout-api ;;
  deployment)  # previous revision of a Deployment (revisionHistoryLimit > 0)
    kubectl rollout undo deployment/checkout-api ;;
  gitops)      # make Git agree with what now runs, through expedited review
    bad="${BAD_COMMIT:?set BAD_COMMIT to the release commit}"
    git switch -c "revert-${bad}"
    git revert --no-edit "${bad}"
    git push -u origin HEAD
    echo "Open a pull request and ask for expedited review." ;;
  *) echo "unknown lever: $1" >&2; exit 2 ;;
esac
```

5. **Verify reach.** Check version labels per pod, region and cell; partial rollbacks and unreachable clients recur in the cases
   below.
6. **Communicate and record** in the release record; open the review ([IncidentOps](../incidentops/)).
7. **Fix forward** through the same path with shorter steps — never a side door; any break-glass push to the protected branch uses
   a named identity and is recorded.

### Three documented failures

Each case comes from the organisation's own or a regulator's report; the "release lesson" is this Atlas's reading.

**Knight Capital, 1 August 2012 (SEC order).** The SEC found that new trading code "repurposed a flag that was formerly used to
activate" an unused function, Power Peg; that a technician, unreviewed, did not copy the new code to one of eight servers during a
staged deployment; and that flagged orders made that server run the old code — about 4 million executions in about 45 minutes and,
ultimately, a loss of more than $460 million. Knight then "uninstalled the new RLP code from the seven servers where it had been
deployed correctly", which "worsened the problem"[^sec-knight]; it settled for $12 million without admitting or denying the
findings[^sec-knight-pr]. *Release lesson:* never reuse a flag's meaning, verify a deployment's reach, and know what state a
rollback recreates.

**CrowdStrike, 19 July 2024 (company reviews).** A Rapid Response Content update — "configuration data; it is not code or a kernel
driver" — reached Windows sensors at 04:09 UTC and was reverted at 05:27 UTC; it shipped on "trust in the checks performed in the
Content Validator" and past successes. The root cause: "The sensor expected 20 input fields, while
the update provided 21 input fields", an out-of-bounds read that crashed hosts. Content now passes "successive deployment rings
before rollout into production"[^crowdstrike]. *Release lesson:* configuration that changes behaviour needs staged
exposure, and a revert cannot reach crashed machines.

**Google Cloud, 12 June 2025 (provider report).** A Service Control quota feature added on 29 May passed a region-by-region
rollout without its failing path being exercised; it "did not have appropriate error handling nor was it feature flag
protected". Policy data with blank fields, replicated globally within seconds, triggered it everywhere. Its "red-button" off
switch finished rolling out about 40 minutes in; restarts without randomised backoff then overloaded a dependency, and full
recovery in us-central1 took up to about 2 h 40 min. Google writes: "If this had been flag protected, the issue would have been caught in
staging"[^gcp-2025]. *Release lesson:* a staged binary protects nothing if something unstaged switches its behaviour on, and a
kill switch that needs its own rollout is not a seconds-scale lever.

### Anti-patterns and myths

- **Ceremonial canaries** — five minutes at 1% on a quiet service ([sizing](#metrics-and-kpis)).
- **Flags forever, or reused.** Release toggles should "generally not stick around much longer than a week or two"[^toggles];
  Knight's dormant code woke through a reused flag.
- **"It's only config."** Configuration, policy data and content change behaviour; stage them like code.
- **Rollback as a belief.** An unrehearsed rollback, or one against data in a new format, is a hope, not a plan.
- **The gatekeeper release manager.** Relaying approvals adds queue time; the role earns its keep by improving the path.
- **Myths:** "GitFlow is the standard" (its author suggests a simpler flow for continuous delivery[^gitflow]); "automated rollback
  makes us safe" (only for reversible changes and reachable instances).

## Deciding and adopting

### When to use and when not to use

**Invest in progressive release machinery when** you operate the software, deploy several times a week, have traffic to compare
versions within an hour, and a bad release would be costly.

**Keep it light, or skip it, when:**

- **The service is quiet.** An internal tool with a few hundred users needs rolling deploys, a flag on the risky part and a
  tried rollback, not a canary with nothing to measure.
- **The team is tiny.** Three engineers need versioned artefacts, a tried rollback and expiring flags, not a controller.
- **Users install the software.** Rings, staged distribution and support windows replace traffic canaries
  ([MobileOps](../mobileops/)).
- **The change is one-way.** Data migrations and public announcements need expand and contract, rehearsal and explicit sign-off.

**Hidden responsibilities:** running the controller and flag service, per-version metrics, flag removal, and reviewing pauses,
aborts and overrides.

### Decision framework

::: figure Choosing a release strategy, including the light option
```mermaid
flowchart TB
  Q1{"Do you operate the running software?"} -- "No: users install it" --> A1["Rings, staged distribution, SemVer, support windows"]
  Q1 -- "Yes" --> Q2{"Can old and new versions share data and clients?"}
  Q2 -- "Yes" --> Q3{"Must users never meet a mix of versions?"}
  Q2 -- "No" --> C["Make them compatible: expand and contract, or Prepare then Activate"]
  C -- "Then ask again" --> Q2
  Q3 -- "Yes" --> BG["Blue-green if two environments are affordable; if not, deploy dark, then flip a flag for everyone"]
  Q3 -- "No" --> Q4{"Traffic to compare versions within an hour?"}
  Q4 -- "Yes" --> Q5{"Must the business choose when users see it?"}
  Q4 -- "No" --> A2["Keep it light: rolling deploy, flags for staff, rehearsed rollback"]
  Q5 -- "Yes" --> A4["Canary for the binary plus a release flag"]
  Q5 -- "No" --> A5["Automated canary with analysis"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class Q1,Q2,Q3,Q4,Q5 decision
  class BG,A4,A5 core
  class A1,A2 neutral
  class C risk
```
Text equivalent: software users install gets rings, staged distribution, SemVer and support windows. For software you operate,
first make old and new versions work against the same data and clients (expand and contract, or Prepare then Activate). If
users must never meet a mix of versions: blue-green when two environments are affordable, else deploy dark and flip a flag for
everyone at once. Otherwise, without traffic to compare versions within an hour, keep it light (rolling deploys, flags for
staff, a rehearsed rollback); with enough, a canary plus a release flag when the business must choose the moment, else an
automated canary with analysis.
:::

Assumptions: compatibility comes first because every branch runs or swaps versions against shared data[^aws-rollback];
"within an hour" comes from the sizing; a flag keeps users on one behaviour only for the code it covers, and flipping it
for everyone is unstaged: watch closely, switch back at once.

### Maturity model

**This Atlas's model**, not an industry standard; levels describe a service, not a company.

| Level | Observable behaviour | Exit evidence | Next step |
|---|---|---|---|
| 0 Ad hoc | Deploy equals release; nobody knows exactly what is live | — | Version artefacts; write down the rollback |
| 1 Repeatable | Versioned artefacts and notes; rolling deploys | Every live version traceable; rollback tried | Deploy dark; flag risky changes |
| 2 Decoupled | Risky features flagged with owner and expiry; a record per release | Flag debt measured; launch dates no longer move deploys | Per-version metrics; manual canary |
| 3 Progressive | Automated canary and abort on key services; written gate policy | Early detection and time to safe state reported | Stage config and flag changes |
| 4 Adaptive | Code, config, flags and data staged alike; steps sized from traffic; false aborts tracked | Quarterly data-driven policy review | Remove steps that never catch anything |

### Adoption roadmap: 30/60/90 days

| Window | Actions | Owner | Milestone |
|---|---|---|---|
| Days 1–30 | List every path that changes production behaviour; write policy v1 for one key service; start a flag registry; record baselines; quick wins: a kill switch for the riskiest feature, the flag-debt check in CI | Service owner, release manager | Policy merged |
| Days 31–60 | Deploy dark plus a manual canary on that service; per-version metrics; rehearse a rollback; automated notes for one library; a standard-change route with [ITOps](../itops/) | Platform team, service owner | A rehearsed abort |
| Days 61–90 | Automated analysis and abort there; two more services; one configuration pipeline on staged exposure; first quarterly review | Platform team, release manager | Three services at level 3 |

**Dependencies.** Per-version telemetry ([ObservabilityOps](../observabilityops/)), platform capacity, and leaders agreeing that
on-call may abort without approval.

## Toolkit

### Checklists

#### Release readiness (before the deploy)

- [ ] Notes generated; breaking changes and security fixes checked by a person
- [ ] Risk class, steps and bake time taken from the policy
- [ ] Two or three stop signals visible per version
- [ ] Schema changes expand-only; rollback checked for data compatibility, or a roll-forward plan written
- [ ] Risky behaviour flagged (owner, expiry); the code default is the safe behaviour
- [ ] On-call knows about the rollout and holds the abort right
- [ ] No freeze or spent error budget, or a recorded exception

#### Flag hygiene (monthly)

- [ ] Flag-debt check passes in CI
- [ ] Flags at 100% for over two weeks scheduled for removal
- [ ] Kill switches exercised outside production

#### Security and governance audit (quarterly)

- [ ] Rights to widen, abort, override and change flags match the RACI; leavers removed
- [ ] Main branch and tags protected; direct pushes only by the named break-glass identity
- [ ] Release tokens scoped to one repository and job; workflow dependencies locked or SHA-pinned
- [ ] Flag and controller audit logs reviewed; every override has a reason and expiry
- [ ] Expired overrides and waivers closed

#### Post-release review (after any pause, abort or release-caused incident)

- [ ] Timeline rebuilt from the release record
- [ ] Which signal caught it, at what exposure; time to safe state; what would have caught it earlier
- [ ] Follow-ups owned and dated

### Templates

#### Release record

_Example status: YAML syntax-checked; illustrative field names._

```yaml
release: checkout-api 1.4.0
date: 2026-10-16
owner: team-checkout
risk_class: standard
summary: "New address validation; payment retry logic"
links:
  pull_requests: ["https://git.example.com/checkout-api/pulls/812"]
  pipeline_run: "https://ci.example.com/runs/5531"
  notes: "https://git.example.com/checkout-api/releases/v1.4.0"
rollback:
  safe: true
  method: "abort rollout; no new data formats written"
  checked_by: "a.reviewer"
flags:
  - key: new-checkout
    owner: team-checkout
    expires: 2026-11-30
exposure_log:   # standard class: >= 30 min and 4,000 canary requests per step
  - time: "2026-10-16T09:35:00Z"
    step: "5%"
    verdict: Successful
    decided_by: rollout-controller
  - time: "2026-10-16T10:05:00Z"
    step: "25%"
    verdict: Successful
    decided_by: rollout-controller
  - time: "2026-10-16T10:35:00Z"
    step: "50%"
    verdict: Successful
    decided_by: rollout-controller
outcome: released   # released | aborted | rolled-forward
follow_ups: []
```

#### Architecture decision record: release strategy

```text
Title:   Release strategy for <service>
Status:  proposed | accepted | superseded
Date:    YYYY-MM-DD
Owner:   <name>

Context
  Who uses the service, traffic per hour, cost of a bad release,
  data and schema constraints, regulatory or contractual change controls,
  team size and on-call coverage.

Decision
  Exposure mechanism (rolling / blue-green / canary / flags / rings),
  step sizes, bake time and request minimum, stop signals, abort rights,
  gate effects and overrides, versioning scheme, branching model, cadence.

Options considered
  At least two, including "keep it light".
  For each: protection gained, cost, operational load.

Consequences
  New dependencies (controller, flag service), capacity,
  flag-removal duty, evidence for audits.

Review
  Date, and the data that would change the decision
  (for example early-detection ratio, false aborts).
```

### Deep comparisons

Mechanisms are compared under [fundamentals](#fundamentals-building-blocks); in an incident, the back-out levers matter:

| Lever | Speed | Works when | Fails when |
|---|---|---|---|
| Flag off | Seconds, if evaluated at run time | The behaviour is flagged and the old path still works | The defect is outside the flagged path (a new library, a leak); the switch itself needs a rollout |
| Abort or roll back | Seconds to minutes | Versions are compatible and the rollback reaches every instance | New data formats exist; clients are unreachable |
| Roll forward | Minutes to hours | Going back is unsafe or slower than a small fix | The fix skips the staged path |

### FAQ

**Is "ReleaseOps" a job title or a team?**
Rarely; release manager and release engineer are the usual titles, with platform engineers and service owners sharing the work.

**If we have canaries, do we still need feature flags?**
They turn different dials: a canary tests a new *binary* on a slice of traffic; a flag controls *behaviour* inside one binary. A
leak in a new library is caught by a canary; a launch timed by marketing is controlled by a flag.

**Rollback or roll forward — which is better?**
Roll back (or flag off) while the previous state is still valid; roll forward when it is not (table above), and design two-phase
changes so that this is rare[^aws-rollback].

**Does progressive delivery replace testing?**
No. It limits the damage of what tests miss; test signal quality is on [TestOps](../testops/).

**What does this have to do with ServiceNow's ReleaseOps product?**
Only the name. That product moves changes and custom apps through pipelines on the ServiceNow platform; as of October 2026 it
is not supported in regulated environments or on-premises, and entitlements must be checked[^sn-explore].

## Going further

### Advanced topics and emerging direction

- **Configuration pipelines, waves and cells.** Amazon described dedicated configuration and feature-flag pipelines, and waves of
  increasing size, the first in one low-traffic Region, one zone or cell at a time[^aws-handsoff].
- **Statistical and AI judges.** Many metrics checked repeatedly raise false alarms; judges such as Kayenta (an Argo Rollouts
  provider[^argo-analysis]) address this. Trust no judge until you have measured its false aborts and missed regressions.
- **Bucketing stability.** flagd v0.15.0 (2026-04-01) warned that "fractional pseudorandom assignments will change"[^flagd-v015], and
  v0.18.0 (2026-10-15) that "pseudorandom bucket assignment may change"[^flagd-v018], so an upgrade mid-rollout can move users between
  cohorts.

### Learning path

| Stage | What to do |
|---|---|
| Prerequisites | CI/CD on [DevOps](../devops/); your platform's deployment model |
| 1-day primer | Hodgson on toggles[^toggles], the SRE Workbook on canarying[^sre-canary]; Lab 1 |
| 1 week | Labs 2 and 3; a policy and canary step size for one real service |
| 30 days | The roadmap's first month; one rehearsed rollback |
| Advanced specialisation | Statistical canary analysis; staged configuration pipelines; supporting several released versions |
| Portfolio | A flag-debt check in CI; a fail-closed canary analysis; a policy revised with a quarter's data |

**Paths through this page.** *Beginners:* [in 30 seconds](#in-30-seconds), [glossary](#glossary-and-mental-model), Lab 1,
[FAQ](#faq). *Practitioners:* [reference implementation](#reference-implementation), [labs](#hands-on-labs),
[runbook](#worked-runbook-a-stop-signal-fires-during-a-rollout), [checklists](#checklists). *Leaders and architects:*
[governance](#operating-model-and-governance), [decision framework](#decision-framework), [maturity](#maturity-model),
[roadmap](#adoption-roadmap-306090-days), [judgement call](#a-judgement-call-push-the-urgent-update-everywhere-now).

## Wrap-up

### Executive recap and cheat sheet

| Area | Remember |
|---|---|
| Decide | Separate deploy from release; use the [decision framework](#decision-framework); keep it light for quiet services |
| Practise | Deploy dark, widen in steps, compare with stable, expand and contract |
| Control | On-call may always abort; widening needs evidence; overrides are named and expire |
| Measure | Early detection, exposure per bad release, time to safe state, false aborts, flag debt |
| Watch out | Ceremonial canaries, immortal or reused flags, "it's only config", untested rollbacks |

### A judgement call: push the urgent update everywhere now?

::: decision Illustrative dilemma
**Situation.** Your agent runs on 50,000 customer machines; an attack is spreading and a detection update — configuration, not
code — is ready. Policy says 1% → 10% → 100% with one-hour bakes. Security asks to push it everywhere now.

**Option A — push globally now:** full protection in minutes. **Option B — follow the policy:** two hours to full coverage, but a
bad update reaches at most 1%, then 10%, before it is caught.

**Assumptions.** The chance this update is faulty is unknown but not zero, and the *kind* of failure matters: an update that
crashes the host fails fleet-wide, and a revert cannot reach crashed machines[^crowdstrike]. The attack reaches machines one at a
time.

**Reasoning.** Option A feels responsible because the threat is visible and the update's risk is not, yet it accepts a small chance
of an unrecoverable, fleet-wide outcome. Option B treats an emergency as routine. A third option beats both: a **compressed staged
rollout** — internal machines for 15 minutes, 5% for 15 minutes, then everyone, halting on crash reports or missing heartbeats (a
crashed host cannot report its own failure). Amazon describes a similar path for urgent changes: shorter bake only after Principal
Engineer scrutiny, minimal changes in flight, deployments actively watched[^aws-handsoff].

**Uncertainty.** If the attack destroyed machines faster than a 30-minute delay allows, the balance would shift; write that
threshold into the policy *before* the emergency.
:::

### Learning objectives and mastery check

You should now be able to **explain** deploy versus release, **choose** an exposure mechanism, **design** a canary step from
traffic, **operate** the back-out levers and **evaluate** a release process with release metrics.

::: answer 1. Give an example of a change that is deployed but not released.
New checkout code runs in every pod while `new-checkout` serves `off` to everyone. Conversely, a flag change releases without a
deployment.
:::

::: answer 2. 20 requests per second, 0.5% errors, a 1% canary for 10 minutes. Useful?
Barely: about 120 requests, so 0.6 expected errors normally and 1.2 if the rate doubles — noise. Only failures several times the
baseline (about 3–4% errors) would show reliably; use larger or longer steps.
:::

::: answer 3. Why is the success condition `len(result) == 0 || result[0] >= 0.95` risky?
An empty result counts as success, so a canary with no traffic or mismatched labels passes untested. Require data; read zero
errors as zero (`or vector(0)`).
:::

::: answer 4. A step-end check aborts when the canary's error rate exceeds stable by 0.2 points, the doubling you fear. What is wrong?
The measured difference scatters around 0.2 points, so a real doubling crosses that line only about half the time. Alarm at the
critical value (about 0.12 points at 4,000 requests) and let the band up to 0.2 pause for a person.
:::

::: answer 5. Version 2 writes a new record format and is at 30% when errors appear. Roll back?
No. The errors probably come from the 70% still on version 1 failing to read version 2's records[^aws-rollback], so pausing does
not stop them and rolling back spreads them. Turn off the new-format writes if they are flagged; otherwise roll forward quickly to a
version that reads both formats. Next time, ship that reader before the writer.
:::

::: answer 6. Last night's end-to-end suite failed on a commit that is in today's release candidate. Can it ship?
Not by default: the [gate policy](#gate-policy) blocks promotion until a green run or a service-owner override, which expires
within three days. A test quarantined as flaky only records; whether it deserves trust is a [TestOps](../testops/) decision.
:::

### Conclusion

"ReleaseOps" is a borrowed name for an old problem: how to let change reach people without betting everything on one moment. The
field's answer is to separate installing from exposing, and to treat exposure as a dial with evidence at every step and a named
hand ready to turn it back.

The documented failures share a pattern: something that changed behaviour reached everything at once. Release discipline
means one staged, reversible path for every kind of change, with honestly sized steps; for quiet services it means less
machinery, not more. Judge a release process by how little damage a bad change can do before someone notices, and how fast
users are back on the safe path.

## Sources and further reading

[^sn-explore]: [Exploring ReleaseOps](https://www.servicenow.com/docs/r/guL1zqmJkV9eywMq_6qu~w/GJM3yULVV_dQs6SRwZSFdg) — ServiceNow Docs (Zurich), updated 2026-06-09. Checked 2026-10-16.
[^sn-rn]: [ReleaseOps release notes](https://www.servicenow.com/docs/r/zurich/release-notes/release-ops-rn-2.html) — ServiceNow Docs, updated 2025-07-31. Checked 2026-10-16.
[^opsio]: [ReleaseOps Overview](https://opsiocloud.com/knowledge-base/what-is-releaseops/) — Opsio, 2025-09-05. Checked 2026-10-16.
[^mozilla-relops]: [Mozilla Platform Operations](https://github.com/mozilla-platform-ops) — Mozilla, GitHub README. Checked 2026-10-16.
[^sre-releng]: [Release Engineering](https://sre.google/sre-book/release-engineering/) — Dinah McNutt, *Site Reliability Engineering*, 2016. Checked 2026-10-16.
[^sre-canary]: [Canarying Releases](https://sre.google/workbook/canarying-releases/) — Warner, Davidovič et al., *The Site Reliability Workbook*, 2018. Checked 2026-10-16.
[^sre-launch]: [Reliable Product Launches at Scale](https://sre.google/sre-book/reliable-product-launches/) — Singh, Kirsch, Rau, *Site Reliability Engineering*, 2016. Checked 2026-10-16.
[^sre-ebp]: [Example Error Budget Policy](https://sre.google/workbook/error-budget-policy/) — Steven Thurgood, SRE Workbook, 2018. Checked 2026-10-16.
[^itil-rm]: [ITIL 4 Practitioner: Release Management](https://www.peoplecert.org/en/browse-certifications/it-governance-and-service-management/itil-1/itil-4-practitioner-release-management-3798/) — PeopleCert. Checked 2026-10-16.
[^itil-dm]: [ITIL 4 Practitioner: Deployment Management](https://www.peoplecert.org/browse-certifications/it-governance-and-service-management/ITIL-1/itil-4-practitioner-deployment-management-3796) — PeopleCert. Checked 2026-10-16.
[^redmonk]: [Towards Progressive Delivery](https://redmonk.com/jgovernor/towards-progressive-delivery/) — James Governor, RedMonk, 2018-08-06. Checked 2026-10-16.
[^toggles]: [Feature Toggles (aka Feature Flags)](https://martinfowler.com/articles/feature-toggles.html) — Pete Hodgson, 2017-10-09. Checked 2026-10-16.
[^bluegreen]: [BlueGreenDeployment](https://martinfowler.com/bliki/BlueGreenDeployment.html) — Martin Fowler, 2010-03-01. Checked 2026-10-16.
[^canary-sato]: [CanaryRelease](https://martinfowler.com/bliki/CanaryRelease.html) — Danilo Sato, 2014-06-25. Checked 2026-10-16.
[^parallel]: [ParallelChange](https://martinfowler.com/bliki/ParallelChange.html) — Danilo Sato, 2014-05-13. Checked 2026-10-16.
[^trunk-release]: [Branch for release](https://trunkbaseddevelopment.com/branch-for-release/) — Paul Hammant, trunkbaseddevelopment.com. Checked 2026-10-16.
[^gitflow]: [A successful Git branching model](https://nvie.com/posts/a-successful-git-branching-model/) — Vincent Driessen, 2010, reflection note 2020-03-05. Checked 2026-10-16.
[^chrome]: [Speeding up Chrome's release cycle](https://blog.chromium.org/2021/03/speeding-up-release-cycle.html) — Chromium Blog, 2021-03-04. Checked 2026-10-16.
[^k8s-cycle]: [Kubernetes Release Cycle](https://kubernetes.io/releases/release/) — Kubernetes, updated 2026-06-25. Checked 2026-10-16.
[^k8s-team]: [Release Team README](https://github.com/kubernetes/sig-release/blob/master/release-team/README.md) — Kubernetes SIG Release. Checked 2026-10-16.
[^safe-art]: [Agile Release Train](https://framework.scaledagile.com/agile-release-train) — Scaled Agile. Checked 2026-10-16.
[^semver]: [Semantic Versioning 2.0.0](https://semver.org/) — Tom Preston-Werner. Checked 2026-10-16.
[^calver]: [Calendar Versioning](https://calver.org/) — calver.org. Checked 2026-10-16.
[^kac]: [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/) — keepachangelog.com. Checked 2026-10-16.
[^cc]: [Conventional Commits 1.0.0](https://www.conventionalcommits.org/en/v1.0.0/) — conventionalcommits.org. Checked 2026-10-16.
[^release-please]: [release-please](https://github.com/googleapis/release-please) — googleapis, v17.11.2 (2026-08-24). Checked 2026-10-16.
[^rp-action]: [release-please-action](https://github.com/googleapis/release-please-action) — googleapis, README and v5.0.0 notes (2026-04-22). Checked 2026-10-16.
[^semrel]: [semantic-release](https://github.com/semantic-release/semantic-release) — v25.0.9 (2026-08-05). Checked 2026-10-16.
[^gh-secure]: [Secure use reference](https://docs.github.com/en/actions/reference/security/secure-use) — GitHub Docs. Checked 2026-10-16.
[^gh-notes]: [Automatically generated release notes](https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes) — GitHub Docs. Checked 2026-10-16.
[^k8s-deploy]: [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) — Kubernetes, updated 2026-07-07. Checked 2026-10-16.
[^argo]: [Argo Rollouts](https://argo-rollouts.readthedocs.io/en/stable/) — Argo Project; [v1.10.1](https://github.com/argoproj/argo-rollouts/releases/tag/v1.10.1), 2026-10-12. Checked 2026-10-16.
[^argo-canary]: [Canary Deployment Strategy](https://argo-rollouts.readthedocs.io/en/stable/features/canary/) — Argo Rollouts docs. Checked 2026-10-16.
[^argo-analysis]: [Analysis & Progressive Delivery](https://argo-rollouts.readthedocs.io/en/stable/features/analysis/) — Argo Rollouts docs. Checked 2026-10-16.
[^argo-job]: [Job Metrics](https://argo-rollouts.readthedocs.io/en/stable/analysis/job/) — Argo Rollouts docs. Checked 2026-10-16.
[^argo-gwapi]: [Quick start](https://rollouts-plugin-trafficrouter-gatewayapi.readthedocs.io/en/latest/quick-start/) — Argo Rollouts Gateway API plugin (argoproj-labs); latest release [v0.17.0](https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/releases/tag/v0.17.0), 2026-09-01. Checked 2026-10-17.
[^argo-gs]: [Getting Started](https://argo-rollouts.readthedocs.io/en/stable/getting-started/) — Argo Rollouts docs. Checked 2026-10-16.
[^rollouts-demo]: [rollouts-demo](https://github.com/argoproj/rollouts-demo) — Argo Project. Checked 2026-10-16.
[^flagger]: [Flagger](https://docs.flagger.app/) — Flux project, v1.45.0 (2026-09-01). Checked 2026-10-16.
[^gw-split]: [HTTP traffic splitting](https://gateway-api.sigs.k8s.io/guides/user-guides/traffic-splitting/) — Kubernetes Gateway API. Checked 2026-10-16.
[^openfeature]: [Introduction](https://openfeature.dev/docs/reference/intro/) — OpenFeature. Checked 2026-10-16.
[^of-spec]: [Flag Evaluation API, requirement 1.4.10](https://openfeature.dev/specification/sections/flag-evaluation/) — OpenFeature specification. Checked 2026-10-16.
[^flagd-quick]: [Quick Start](https://flagd.dev/quick-start/) — flagd; same commands at tag [flagd/v0.17.1](https://github.com/open-feature/flagd/blob/flagd/v0.17.1/docs/quick-start.md). Checked 2026-10-16.
[^flagd-frac]: [Fractional operation](https://flagd.dev/reference/custom-operations/fractional-operation/) — flagd. Checked 2026-10-16.
[^flagd-schema]: [Flag definition schema](https://flagd.dev/schema/v0/flags.json) and [Metadata](https://flagd.dev/concepts/metadata/) — flagd. Checked 2026-10-16.
[^flagd-v015]: [flagd v0.15.0 release notes](https://github.com/open-feature/flagd/releases/tag/flagd%2Fv0.15.0) — OpenFeature, 2026-04-01. Checked 2026-10-16.
[^flagd-v018]: [flagd v0.18.0 release notes](https://github.com/open-feature/flagd/releases/tag/flagd%2Fv0.18.0) — OpenFeature, 2026-10-15. Checked 2026-10-16.
[^unleash]: [Unleash Documentation](https://docs.getunleash.io/) — Unleash. Checked 2026-10-16.
[^aws-handsoff]: [Automating safe, hands-off deployments](https://aws.amazon.com/builders-library/automating-safe-hands-off-deployments/) — Clare Liguori, Amazon Builders' Library, 2020-06-18. Checked 2026-10-17.
[^aws-rollback]: [Ensuring rollback safety during deployments](https://aws.amazon.com/builders-library/ensuring-rollback-safety-during-deployments/) — Sandeep Pokkunuri, Amazon Builders' Library, 2019-12-05. Checked 2026-10-16.
[^dora-change]: [Streamlining change approval](https://dora.dev/capabilities/streamlining-change-approval/) — DORA, updated 2025-10-30. Checked 2026-10-16.
[^dora-metrics]: [DORA's software delivery performance metrics](https://dora.dev/guides/dora-metrics/) — DORA, updated 2026-01-05. Checked 2026-10-16.
[^dora-google]: [DORA Joins Google Cloud](https://dora.dev/news/dora-joins-google-cloud/) — DORA, 2018-12-20. Checked 2026-10-16.
[^dora-trunk]: [Trunk-based development](https://dora.dev/capabilities/trunk-based-development/) — DORA. Checked 2026-10-16.
[^sec-knight]: [Order, Release No. 34-70694: Knight Capital Americas LLC](https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf) — U.S. SEC, 2013-10-16. Checked 2026-10-16.
[^sec-knight-pr]: [Press release 2013-222](https://www.sec.gov/newsroom/press-releases/2013-222) — U.S. SEC, 2013-10-16. Checked 2026-10-16.
[^crowdstrike]: [Falcon Content Update Remediation and Guidance Hub](https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/) — CrowdStrike, preliminary review 2024-07-24, root-cause summary 2024-08-06. Checked 2026-10-16.
[^gcp-2025]: [Incident report, 12 June 2025](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW) — Google Cloud Service Health, 2025-06-13. Checked 2026-10-16.

**Changelog**

- 2026-10-16 — r0: first version; Lab 1 and the Lab 2 linter executed (`research/releaseops-ledger.md`).
- 2026-10-16 — r1: review fixes (decision tree, per-step analysis, gate policy, back-out, cases, sizing); labs re-run
  (`reviews/releaseops-r1-response.md`).
- 2026-10-17 — r2: canary thresholds recalibrated and simulated; alarm-versus-effect lesson; Figures 3, 4 and 6; plugin
  version; re-check cadence (`reviews/releaseops-r2-response.md`).
- 2026-10-17, post-score polish: release-record steps, latency note, flag-flip caveat, Figure 3 crossings, Amazon's
  first wave (`reviews/releaseops-polish.md`).