ReleaseOps

ReleaseOps is an informal name for the work that turns a finished change into something users experience — on purpose, in steps, and with a way back. It covers versions and notes, cadence, gradual exposure of a change already in production (canaries, rings, flags), the decision to widen or abort, and the record of why.

Its central idea: deploying and releasing are different events. Deploying installs a version; releasing makes behaviour available to people. Separated, a release becomes a risk decision to stage, measure and reverse. It is for release managers, platform engineers, tech leads and leaders sizing release machinery.

Most release pain comes from coupling three things with different risks: installing a version, exposing its behaviour and announcing it. Coupled, every release is all-or-nothing:

• A defect reaches every user at once.
• The only remedy is a redeploy, unsafe if the new version has written data the old one cannot read.
• Marketing dates force unready code out, or hold ready code back.
• Heavy approval and long freezes compensate, enlarging batches.

Decoupled, the new version runs with no users, then staff, then a slice of traffic compared with the current version, widening only on evidence. Google’s SRE book describes almost all updates proceeding gradually, “according to a defined process, with appropriate verification steps interspersed”^[7].

• Not the pipeline (DevOps); ReleaseOps starts once an artefact exists.
• Not the deployment mechanism — reconciliation belongs to GitOps; a canary controller sits on top.
• Not ITIL change enablement — change advisory boards (CABs) and change records are on ITOps.
• Not test engineering — whether a gate’s signal is trustworthy is TestOps; whether to gate is here.
• Not SAFe’s “Agile Release Train”, “a long-lived team of Agile teams”, generally 50–125 people^[8] — an organisational unit, not a schedule (AgileOps).

The practices come from several traditions; there is no founding moment.

• Configuration management. Wingerd and Seiwald (1998) advised branching “only when necessary, on incompatible policy, late, and instead of freeze”^[9].
• Continuous delivery. Fowler (2010) credits the name blue-green to Daniel Terhorst-North and Jez Humble^[10]; Sato’s canary note cites Humble and Farley’s Continuous Delivery^[11].
• Release engineering at web scale. Google’s SRE book (2016) calls it “a relatively new and fast-growing discipline”^[5]; the SRE Workbook (2018) adds canarying^[12].
• Flags and progressive delivery. Hodgson’s 2017 toggle categories^[13]; RedMonk’s “progressive delivery”, August 2018^[14].

The mental model: two dials. Every service has a deployment dial (which version runs where) and an exposure dial (who experiences which behaviour). ReleaseOps turns the second deliberately — in steps sized to detect harm, with a named person allowed to turn it back. Flags turn the exposure dial inside one version; canaries and blue-green turn both dials, using traffic.

flowchart LR
  D["DevOps: versioned artefact"] --> R
  T["TestOps: trusted gate signal"] --> R
  S["SRE: SLOs and error budget"] --> R
  R["ReleaseOps: who gets which change, when, and how it is undone"]
  R --> G["GitOps: desired state to clusters"]
  R --> I["ITOps: change approval records"]
  R --> M["MobileOps: store and device limits"]
  R --> B["DBOps: schema changes in step"]
  R --> X["IncidentOps: when a release fails"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class R core
  class D,T,S,G,I,M,B,X neutral

This charter is this Atlas’s proposal; no official ReleaseOps manifesto exists.

Mission: every change that alters what users experience reaches them in steps small enough to be safe, observable enough to judge and reversible enough to undo.

1. Separate deploy from release — in Hodgson’s words, “separating [feature] release from [code] deployment”^[13].
2. Every step must be able to see what you fear; size steps from traffic (worked calculation).
3. Reversibility is designed: each step, data included, is undoable or declared one-way in advance.
4. Everything that changes behaviour is a release — code, configuration, flags, content, policy data.
5. Smallest sensible batch: smaller release artefacts are cheaper and easier to roll back^[12].
6. Explicit decision rights: widening needs evidence and an owner; anyone on call may abort.
7. Proportional control: approval, freezes and bake time scale with risk.

Non-goals: zero-risk releases; a committee per change; flags as permanent configuration.

Semantic Versioning 2.0.0 increments MAJOR for incompatible API changes, MINOR for compatible features and PATCH for compatible fixes; in 0.y.z “Anything MAY change at any time”, and released contents must not change^[18]. CalVer numbers by the calendar — Ubuntu’s 26.04 is YY.0M^[19]. Use SemVer where consumers judge compatibility (libraries, APIs, agents), CalVer where cadence matters more; a continuously deployed service needs neither publicly.

Trunk-based development is taught on DevOps. For releases:

• Release from trunk, roll forward — high-throughput teams keep no release branches^[9].
• Release branches cut late — the branch “should not receive continued development work”^[9]; fixes land on trunk and are cherry-picked (as at Google^[5]).
• GitFlow — its author now suggests a simpler workflow for continuous delivery, while it may still fit software with several supported versions^[20].

A release train leaves on schedule with whatever is ready: Chrome moved to four-week milestones in 2021, with an eight-week Extended Stable option^[21]; Kubernetes releases about three times a year and patches each branch for about 12 months^[22]. Trains suit software customers install; continuous release suits services you operate.

Old and new versions always share data and clients, side by side or in turn, so both must work with them. Blue-green fits when users must never meet a mix of versions; unless traffic is split, it is “effectively performing a before/after canary”^[12].

Titles vary; the decisions do not. One person may hold several roles, but on-call always holds the abort right.

R = responsible, A = accountable, C = consulted, I = informed. Kubernetes’ release team, for example, leaves branch and patch management to release managers^[24].

Day 0 designs for release, Day 1 widens exposure step by step, Day 2 cleans up.

flowchart TB
  A["Day 0: design for release (flag, compatible schema, stop signals)"] --> B["Deploy dark: new version running, 0% exposure"]
  B --> C["First ring: staff or one cell"]
  C --> Q1{"Signals healthy for the whole step?"}
  Q1 -- "Yes" --> E["Widen: 5% then 25% then 50%, with bake time"]
  E --> Q2{"Healthy at every step?"}
  Q2 -- "Yes" --> F["100%: released"]
  F --> H["Day 2: remove flag, contract schema, close record"]
  Q1 -- "No" --> K["Abort: flag off or traffic back to stable"]
  Q2 -- "No" --> K
  K --> L["Fix forward, then restart from the first ring"]
  L --> B
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class A,B,C,E,F,H core
  class Q1,Q2 decision
  class K,L risk

Approval. DORA’s research (its capability page cites the 2019 State of DevOps Report) found approvals work best as peer review plus automation; heavyweight external approval hurt delivery, with “no evidence” that it lowers change fail rates^[25]. A CAB keeps a role in coordination, process improvement and business trade-offs. Make the normal path fast enough for emergencies; reserve board review for risks automation cannot judge (ITOps).

Freezes. Four different controls share the word:

Cadence. Review the policy quarterly, pauses, aborts and overrides weekly, flag debt monthly.

A gate is any check that can stop a release. The policy gives each an effect — block, pause (a person decides) or record — and an override rule. Whether a verdict deserves that power is a TestOps question; which security findings block is set by DevSecOps. An Atlas proposal:

An override is a named person’s decision, recorded with reason and expiry — never a pipeline variable anyone can set.

A release path has two control loops — deployment (which version runs, at what weight) and exposure (which users get which behaviour) — each with an owner, both feeding one record.

flowchart TB
  subgraph B["Build and version"]
    SRC["Trunk with reviewed commits"] --> CI["CI: build once, test"] --> ART["Immutable artefact plus notes"]
  end
  subgraph C["Release control"]
    DEF["Release definition in Git: version, steps, stop rules"] --> CTRL["Rollout controller"]
    OC["Service owner and on-call"] --> FLAG["Flag service"]
  end
  subgraph P["Production"]
    RT["Traffic router"] --> ST["Stable version"]
    RT --> CN["Canary version"]
  end
  ART --> DEF
  CTRL --> RT
  ST --> TEL["Telemetry: SLIs per version"]
  CN --> TEL
  TEL --> AN["Analysis: canary against stable"]
  AN --> CTRL
  OC --> CTRL
  FLAG --> P
  C --> REC["Release record"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class SRC,CI,ART,ST,CN,RT neutral
  class DEF,CTRL,FLAG,TEL,REC core
  class AN,OC decision

Responsibilities and trust boundaries.

Alternatives. Without Kubernetes, use weighted load-balancer targets, weighted function aliases, or flags plus rolling deploys. Installed software has no router: exposure moves through rings of devices, and rollback means shipping a new version (MobileOps).

A happy path, and exception paths named in advance.

sequenceDiagram
  participant P as Pipeline
  participant O as On-call
  participant R as Controller
  participant A as Analysis
  Note over P: Build, version, update Git
  Note over R: Deploy dark, canary at 5%
  R->>A: Start tripwire and step checks
  alt Healthy
    A-->>R: Successful
    Note over R: Widen to 25%, 50%, 100%
  else Errors over threshold
    A-->>R: Failed
    Note over R: Abort to stable
    R-->>O: Alert with analysis link
    O->>P: Revert in Git, fix forward
  else Few requests or unclear
    A-->>R: Inconclusive
    R-->>O: Paused, decision needed
  end

Exception paths.

1. Failed analysis. Argo Rollouts aborts and stays Degraded until the desired state changes back^[29] (with GitOps, a revert commit).
2. Inconclusive analysis. It pauses for a person to resume or abort^[30]; give every pause an owner and deadline.
3. Stalled deployment. “Kubernetes takes no action on a stalled Deployment other than to report a status condition”^[23]; something you own must act on ProgressDeadlineExceeded.
4. Emergency fix. Same path, shorter steps (judgement call).

Playbook: a standard progressive rollout.

1. Check the record names version, risk class, steps, stop signals and abort owner; deploy dark and check health.
2. Expose the first ring (staff or one cell) for at least one bake period and the request minimum.
3. Widen in steps, comparing canary with stable over the same period — “time is one of the biggest sources of change in observed metrics”, so before/after comparisons mislead^[12].
4. Watch the top few user-facing signals (“perhaps no more than a dozen”), one canary at a time per service^[12].
5. At 100%, schedule flag removal and schema contraction; close the record.

Contingency. Before step 1, record whether rollback is safe. If the release writes data the previous version cannot read, split it: Amazon describes a “Prepare” version that reads the new format while still writing the old, then an “Activate” version that writes it, so each step can be rolled back^[31] — the same idea as expand, migrate, contract^[17].

None is mandatory by itself; auditors need evidence that changes were authorised and tested, not a particular process (compliance).

Names are examples, not endorsements. Versions were checked on 2026-10-17 against each project’s GitHub releases: re-check there before copying an example, and review the table quarterly.

Selection questions. Do per-version metrics exist (if not, invest in observability first)? Who owns controller and flag service? Are flag changes audited? Is abort one step for on-call?

Automate the mechanical and time-critical: version bumps, note drafts, analysis and abort. Keep people on thresholds, risky widenings and Inconclusive results, which Argo Rollouts leaves to “human judgement”^[30]. Automation fails too: noisy aborts get overridden. Language models can draft notes; a person checks breaking changes and security fixes, and no model decides the SemVer bump.

Assumptions. A stateless HTTP service on Kubernetes at about 50 requests per second, released from GitHub, with Argo Rollouts v1.10.1 and its Gateway API traffic-router plugin (v0.17.0, 2026-09-01), Prometheus and flagd. Illustrative and syntax-checked; none was executed for this page.

Artefact 1 — the release policy as data, one file per service, reviewed like code. A step ends only when both its bake time and its request count are met, so here the high class’s 1% step lasts about 2.2 hours, not 60 minutes (sizing).

Example status: YAML syntax-checked; illustrative policy, not consumed by any specific tool.

# release-policy.yaml (illustrative)
service: checkout-api
owner: team-checkout
risk_classes:
  low:        # copy, styling, paths already behind a flag
    exposure_steps_percent: [10, 100]
    min_bake_minutes: 15
    approval: peer-review
  standard:   # most code changes
    exposure_steps_percent: [5, 25, 50, 100]
    min_bake_minutes: 30
    approval: peer-review
  high:       # payments, authentication, schema contraction, global config
    exposure_steps_percent: [1, 5, 25, 50, 100]
    min_bake_minutes: 60
    approval: peer-review plus service owner
    requires: [kill-switch, rollback-rehearsed, traffic-router]
step_evidence:  # a step ends when BOTH min_bake_minutes and this count are met
  min_canary_requests: 4000   # 80% chance to catch 0.2% -> 0.4% (see sizing)
  too_few_requests: pause     # a person extends the step or aborts
stop_signals:   # alarm at the critical value, below the effect to be caught
  - "step end: canary 5xx minus stable over 0.12 points pauses, over 0.2 aborts"
  - "any 5-minute window at 5% or more: over 1 point aborts (gross failure)"
  - "canary p95 latency above 1.2 x stable p95"
abort_rights: [on-call, service-owner, release-manager]
gates:          # effect: block | pause | record (see Gate policy)
  tests-pre-merge:   {effect: block, override: none}
  post-merge-suite:  {effect: block-promotion, override: service-owner,
                      max_days: 3}
  security-findings: {effect: per-devsecops-policy,
                      override: exception-approver}
  canary-analysis:   {failed: abort, inconclusive: pause, retry: service-owner}
  quarantined-check: {effect: record, until: quarantine-expiry}
override_record: [person, gate, release, reason, expires]
freezes:
  error_budget_spent: "block feature releases; allow fixes and security changes"
  deploy_windows: "automatic starts Mon-Thu 08:00-16:00; emergencies exempt"
flags:
  release_flag_max_age_days: 30
  required_metadata: [owner, kind, expires]

Artefact 2 — the standard class as a canary that fails closed at every step. Two error-rate checks, two jobs (the policy’s latency signal is omitted for brevity):

• A background tripwire, from the first pause to promotion, compares canary and stable 5xx rates every two minutes over five-minute windows and aborts only on gross failures: canary errors over 1 percentage point above stable (about six times the baseline).
• A step-end check after each pause requires the policy’s 4,000 canary requests (about 27 minutes at 5% of 50 requests per second) and judges the whole step at the sized alarm threshold: up to 0.12 points passes, over 0.2 aborts; in between, or with too few requests, the run is Inconclusive and pauses for a person^[30].

Simulated (10,000 runs per case, Poisson errors; not a cluster test): no healthy rollout tripped the tripwire and about 5% stopped once, mostly as a pause; a 0.4% canary was stopped at the 5% step in 80% of runs and never reached 100%; a 2% canary aborted seven minutes in. At a 1% step two errors in five minutes cross 1 point, aborting 57% of healthy simulated steps, so use 30-minute windows there. or vector(0) reads “no errors” as zero; len(result) > 0 fails an empty result; failureLimit: 1 aborts on the second failed measurement^[30]. The Gateway API plugin^[43] sets exact weights on an HTTPRoute; without a router, Argo Rollouts uses replica counts, so the smallest step is one pod — 10% of 10 replicas^[44].

Example status: YAML syntax-checked; PromQL not executed; thresholds checked by simulation only. Assumes the plugin is registered with the controller, an HTTPRoute checkout-api whose backendRefs are the two Services, and request metrics labelled rollouts_pod_template_hash (copied from the pod label).

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-vs-stable   # background tripwire: gross failures only
spec:
  args:
    - name: canary             # pod-template hashes of the two versions
    - name: stable
  metrics:
    - name: error-rate-delta   # sized for steps of 5% or more at ~50 rps
      initialDelay: 5m         # wait for a full window of canary data
      interval: 2m             # no count: runs until promotion or failure
      failureLimit: 1          # the second failed measurement aborts
      successCondition: len(result) > 0 && result[0] <= 0.01
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            (
              (sum(rate(http_requests_total{app="checkout-api",
                code=~"5..", rollouts_pod_template_hash="{{args.canary}}"}[5m]))
                or vector(0))
              / sum(rate(http_requests_total{app="checkout-api",
                rollouts_pod_template_hash="{{args.canary}}"}[5m]))
            ) - (
              (sum(rate(http_requests_total{app="checkout-api",
                code=~"5..", rollouts_pod_template_hash="{{args.stable}}"}[5m]))
                or vector(0))
              / sum(rate(http_requests_total{app="checkout-api",
                rollouts_pod_template_hash="{{args.stable}}"}[5m]))
            )
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: step-evidence          # inline: no interval, one measurement per step
spec:
  args:
    - name: canary
    - name: stable
  metrics:
    - name: enough-canary-requests  # 0 to 3,999: Inconclusive, rollout pauses
      successCondition: len(result) > 0 && result[0] >= 4000
      failureCondition: len(result) == 0
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            sum(increase(http_requests_total{app="checkout-api",
                rollouts_pod_template_hash="{{args.canary}}"}[30m]))
    - name: error-rate-delta-over-step
      # alarm at the critical value for 4,000 requests, not at the 0.2 effect:
      # <= 0.12 points passes; 0.12-0.2 is Inconclusive (pause); > 0.2 aborts
      successCondition: len(result) > 0 && result[0] <= 0.0012
      failureCondition: len(result) == 0 || result[0] > 0.002
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            (
              (sum(increase(http_requests_total{app="checkout-api",
                code=~"5..", rollouts_pod_template_hash="{{args.canary}}"}[30m]))
                or vector(0))
              / sum(increase(http_requests_total{app="checkout-api",
                rollouts_pod_template_hash="{{args.canary}}"}[30m]))
            ) - (
              (sum(increase(http_requests_total{app="checkout-api",
                code=~"5..", rollouts_pod_template_hash="{{args.stable}}"}[30m]))
                or vector(0))
              / sum(increase(http_requests_total{app="checkout-api",
                rollouts_pod_template_hash="{{args.stable}}"}[30m]))
            )
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
spec:
  replicas: 10
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: api
          image: registry.example.com/checkout-api:1.4.0
          ports:
            - containerPort: 8080
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: checkout-api
      analysis:                # background, from step index 1 (the first pause)
        startingStep: 1
        templates:
          - templateName: error-rate-vs-stable
        args: &hashes          # YAML anchor, reused by every step-end check
          - name: canary
            valueFrom: {podTemplateHashValue: Latest}
          - name: stable
            valueFrom: {podTemplateHashValue: Stable}
      steps:                   # standard class: 5, 25, 50, then 100
        - setWeight: 5
        - pause: {duration: 30m}
        - analysis: {templates: [{templateName: step-evidence}], args: *hashes}
        - setWeight: 25
        - pause: {duration: 30m}
        - analysis: {templates: [{templateName: step-evidence}], args: *hashes}
        - setWeight: 50
        - pause: {duration: 30m}
        - analysis: {templates: [{templateName: step-evidence}], args: *hashes}

Artefact 3 — a release flag with owner and expiry. flagd allows flag-level metadata^[45], where this page stores owner, kind and expires for CI (Lab 2). fractional buckets users deterministically on flag key plus user ID^[46]; staff are matched by an attribute, not an e-mail address, so the flag service gets no personal data.

Example status: JSON syntax-checked; evaluated only in the unexecuted Lab 2.

{
  "$schema": "https://flagd.dev/schema/v0/flags.json",
  "flags": {
    "new-checkout": {
      "state": "ENABLED",
      "variants": { "on": true, "off": false },
      "defaultVariant": "off",
      "targeting": {
        "if": [
          { "==": [{ "var": "staff" }, true] },
          "on",
          {
            "fractional": [
              { "cat": [{ "var": "$flagd.flagKey" }, { "var": "userId" }] },
              ["on", 10],
              ["off", 90]
            ]
          }
        ]
      },
      "metadata": {
        "owner": "team-checkout",
        "kind": "release",
        "expires": "2026-11-30"
      }
    }
  }
}

Artefact 4 — versions and notes on GitHub. release-please maintains a Release PR from Conventional Commits; merging it updates the changelog, tags the commit and creates a GitHub release^[36]. The action’s README grants contents, issues and pull-requests write permissions and warns that releases and PRs created with the default GITHUB_TOKEN trigger no further workflows (current major: v5, 2026-04-22)^[47]. For third-party actions GitHub now calls a workflow lockfile “the strongest protection available”, with full-commit-SHA pinning as the fallback^[48].

Example status: YAML syntax-checked; not executed.

# .github/workflows/release-please.yml
name: release-please
on:
  push:
    branches: [main]
permissions:
  contents: write
  issues: write
  pull-requests: write
jobs:
  release-please:
    runs-on: ubuntu-latest
    steps:
      # Production: use a workflow lockfile, or pin v5's full commit SHA.
      - uses: googleapis/release-please-action@v5
        with:
          release-type: simple

Example status: YAML syntax-checked; not executed.

# .github/release.yml: categories for GitHub's generated release notes
changelog:
  exclude:
    labels: [ignore-for-release]
  categories:
    - title: Breaking changes
      labels: [breaking-change]
    - title: New features
      labels: [enhancement]
    - title: Other changes
      labels: ["*"]

Lab 1 and Lab 2’s linter were executed (last on 2026-10-17, Ubuntu, git 2.43.0, Python 3.8.10); the rest was not. Use throwaway directories and clusters only.

Objective. Turn commit conventions into a SemVer decision and a changelog section^[33][32]. Prerequisites. git 2.28+, Python 3.8+. Cost and risk. None (temporary directory).

1. Save this script as next_release.py:

Example status: executed on 2026-10-17 (Python 3.8.10).

#!/usr/bin/env python3
"""Propose the next SemVer version and a changelog section.

Reads Conventional Commits since the last vX.Y.Z tag.
"""
import re
import subprocess
import sys
from datetime import date

TAG_RE = re.compile(r"^v(\d+)\.(\d+)\.(\d+)$")
HEADER_RE = re.compile(
    r"^(?P<type>\w+)(?:\((?P<scope>[^)]+)\))?(?P<bang>!)?: (?P<desc>.+)$"
)
RANK = {"patch": 1, "minor": 2, "major": 3}


def git(*args):
    done = subprocess.run(
        ["git", *args], check=True, capture_output=True, text=True
    )
    return done.stdout


def last_tag():
    listed = git("tag", "--list", "v*", "--sort=-v:refname").split()
    tags = [t for t in listed if TAG_RE.match(t)]
    return tags[0] if tags else None


def commits_since(tag):
    span = f"{tag}..HEAD" if tag else "HEAD"
    raw = git("log", span, "--format=%h%x1f%B%x1e")
    for entry in filter(None, (e.strip() for e in raw.split("\x1e"))):
        sha, _, body = entry.partition("\x1f")
        yield sha, body.strip()


def main():
    tag = last_tag()
    major, minor, patch = (
        map(int, TAG_RE.match(tag).groups()) if tag else (0, 0, 0)
    )
    bump, skipped = None, []
    # Keep a Changelog headings
    sections = {"Changed": [], "Added": [], "Fixed": []}
    for sha, body in commits_since(tag):
        m = HEADER_RE.match(body.splitlines()[0])
        if not m:
            skipped.append(sha)
            continue
        footer = re.search(r"^BREAKING[ -]CHANGE: ", body, re.M)
        breaking = bool(m["bang"]) or footer is not None
        kinds = {"feat": "minor", "fix": "patch"}
        level = "major" if breaking else kinds.get(m["type"])
        if level is None:  # docs:, chore:, ci: ... do not trigger a release
            continue
        bump = level if bump is None or RANK[level] > RANK[bump] else bump
        if breaking:
            sections["Changed"].append(f"- BREAKING: {m['desc']} ({sha})")
        else:
            heading = "Added" if m["type"] == "feat" else "Fixed"
            sections[heading].append(f"- {m['desc']} ({sha})")
    if bump is None:
        print(f"No releasable changes since {tag or 'the first commit'}.")
        return 0
    if major == 0 and bump == "major":
        # team policy for 0.y.z, where SemVer says anything may change
        bump = "minor"
    nxt = {
        "major": (major + 1, 0, 0),
        "minor": (major, minor + 1, 0),
        "patch": (major, minor, patch + 1),
    }[bump]
    version = ".".join(map(str, nxt))
    print(f"Last release: {tag or 'none'} -> next: v{version} ({bump})")
    print(f"\n## [{version}] - {date.today().isoformat()}")
    for title, lines in sections.items():
        if lines:
            print(f"### {title}")
            print("\n".join(lines))
    if skipped:
        print("\nNot Conventional Commits, review by hand: "
              + ", ".join(skipped), file=sys.stderr)
    return 0


if __name__ == "__main__":
    sys.exit(main())

2. Save this driver as run.sh beside it and run bash run.sh:

Example status: executed on 2026-10-17 (git 2.43.0).

#!/usr/bin/env bash
set -euo pipefail
SCRIPT="$(cd "$(dirname "$0")" && pwd)/next_release.py"
LAB="$(mktemp -d)"
cd "${LAB}"
git init -q -b main demo && cd demo
git config user.email "lab@example.com" && git config user.name "Release Lab"
echo "v1" > app.txt && git add app.txt
git commit -qm "feat: first public API"
git tag v1.0.0
echo "a" >> app.txt && git commit -qam "fix: handle an empty basket"
echo "b" >> app.txt && git commit -qam "feat(search): add fuzzy matching"
echo "c" >> app.txt && git commit -qam "docs: clarify local setup"
echo "d" >> app.txt && git commit -qam "Update stuff"
python3 "${SCRIPT}"
echo "----- after a breaking change -----"
echo "e" >> app.txt
git commit -qam "feat(api)!: remove the v1 search endpoint"
python3 "${SCRIPT}"
echo "LAB=${LAB}"

Expected output (hashes and date differ):

Last release: v1.0.0 -> next: v1.1.0 (minor)

## [1.1.0] - 2026-10-16
### Added
- add fuzzy matching (22b7a6f)
### Fixed
- handle an empty basket (2734245)
----- after a breaking change -----
Last release: v1.0.0 -> next: v2.0.0 (major)

## [2.0.0] - 2026-10-16
### Changed
- BREAKING: remove the v1 search endpoint (55c8e41)
...

Standard error also lists “Update stuff” as not conventional. Validation. The docs: commit causes no bump; the breaking change moves 1.1.0 to 2.0.0. Troubleshooting. unknown switch 'b': git older than 2.28. Teardown. rm -rf "$LAB". Reflect. A fix hidden in “Update stuff” would ship unannounced — lint commit messages in CI.

Objective. Release to staff, then 10% of users, then switch off, with flagd; make flag debt fail CI. Prerequisites. Docker, curl, Python 3.8+. Cost and risk. None.

1. Save this as flags/demo.flagd.json: Artefact 3 with null in place of the fractional expression (staff only), plus an expired flag for step 5.

Example status: JSON syntax-checked; the linter in step 5 was executed against this file.

{
  "$schema": "https://flagd.dev/schema/v0/flags.json",
  "flags": {
    "new-checkout": {
      "state": "ENABLED",
      "variants": { "on": true, "off": false },
      "defaultVariant": "off",
      "targeting": {
        "if": [{ "==": [{ "var": "staff" }, true] }, "on", null]
      },
      "metadata": {
        "owner": "team-checkout", "kind": "release", "expires": "2026-11-30"
      }
    },
    "legacy-search": {
      "state": "ENABLED",
      "variants": { "on": true, "off": false },
      "defaultVariant": "on",
      "metadata": {
        "owner": "team-search", "kind": "release", "expires": "2026-08-31"
      }
    }
  }
}

2. Start flagd (pinned to the version whose quick start this follows^[41]), bound to loopback so other machines cannot reach it:

Example status: bash -n syntax-checked; not executed (no Docker on the authoring host).

#!/usr/bin/env bash
set -euo pipefail
docker run --rm -d --name flagd -p 127.0.0.1:8013:8013 \
  -v "$(pwd)/flags:/flags" \
  ghcr.io/open-feature/flagd:v0.17.1 start --uri file:./flags/demo.flagd.json
URL="http://127.0.0.1:8013/flagd.evaluation.v2.Service/ResolveBoolean"
eval_flag() {  # usage: eval_flag <true|false: staff> <userId>
  local ctx="{\"staff\":$1,\"userId\":\"$2\"}"
  curl -s -X POST "${URL}" -H "Content-Type: application/json" \
    -d "{\"flagKey\":\"new-checkout\",\"context\":${ctx}}"
  echo
}
eval_flag true "u-0"     # staff
eval_flag false "u-1"    # customer

3. Restore the fractional expression and save (flagd reloads the file); count true across 200 customers:

Example status: bash -n syntax-checked; not executed.

#!/usr/bin/env bash
set -euo pipefail
URL="http://127.0.0.1:8013/flagd.evaluation.v2.Service/ResolveBoolean"
on=0
for i in $(seq 1 200); do
  ctx="{\"staff\":false,\"userId\":\"u-${i}\"}"
  req="{\"flagKey\":\"new-checkout\",\"context\":${ctx}}"
  v=$(curl -s -X POST "${URL}" -H "Content-Type: application/json" \
        -d "${req}" \
      | python3 -c 'import json,sys; print(json.load(sys.stdin).get("value"))')
  [ "${v}" = "True" ] && on=$((on + 1))
done
echo "on for ${on} of 200 customers"

4. Kill switch: set "defaultVariant": "off", delete targeting, and repeat step 2’s two evaluations.
5. Save this linter as flag_lint.py; run python3 flag_lint.py flags/demo.flagd.json 2026-10-16.

Example status: executed on 2026-10-17 against the lab file (exit code 1, one problem reported).

#!/usr/bin/env python3
"""Fail when a release flag lacks an owner or expiry, or is past its expiry.

A 'time bomb' check for CI.
"""
import json
import sys
from datetime import date

# kill switches and entitlements are reviewed, not expired
LONG_LIVED = {"ops", "permission"}


def main(path, today):
    with open(path, encoding="utf-8") as fh:
        flags = json.load(fh)["flags"]
    problems = []
    for key, flag in sorted(flags.items()):
        meta = flag.get("metadata", {})
        kind = meta.get("kind", "release")
        if not meta.get("owner"):
            problems.append(f"{key}: no owner")
        if kind in LONG_LIVED:
            continue
        expires = meta.get("expires")
        if not expires:
            problems.append(f"{key}: {kind} flag without an expiry date")
        elif date.fromisoformat(expires) < today:
            problems.append(
                f"{key}: expired on {expires}"
                " - remove the flag and its dead code path"
            )
    for p in problems:
        print("FLAG-DEBT", p)
    print(f"{len(flags)} flags checked, {len(problems)} problem(s)")
    return 1 if problems else 0


if __name__ == "__main__":
    arg = sys.argv[2] if len(sys.argv) > 2 else None
    when = date.fromisoformat(arg) if arg else date.today()
    sys.exit(main(sys.argv[1], when))

Expected output. Step 2: staff true (reason TARGETING_MATCH), customer false (DEFAULT). Step 3: roughly 20 of 200, the same users on every re-run, because fractional assignment is deterministic^[46]. Step 4: everyone false (STATIC). Step 5: FLAG-DEBT legacy-search: expired on 2026-08-31 …, exit code 1. Validation. Changing a customer’s userId can move them between cohorts; adding other attributes does not. Troubleshooting. Connection refused: check docker logs flagd; FLAG_NOT_FOUND: wrong key or file path. Teardown. docker stop flagd; delete flags/.

Objective. Watch analysis stop a bad version at 20% exposure. Prerequisites. Docker, kind v0.33.0, kubectl, the Argo Rollouts kubectl plugin, about 4 GB of free memory. Cost and risk. Local only.

Example status: bash -n and YAML syntax-checked; not executed. Images: argoproj/rollouts-demo (colour tags; bad-* tags return a high rate of errors)^[49] and curlimages/curl:8.23.0.

#!/usr/bin/env bash
set -euo pipefail
kind create cluster --name release-lab
kubectl create namespace argo-rollouts
REL="https://github.com/argoproj/argo-rollouts/releases/download/v1.10.1"
kubectl apply -n argo-rollouts -f "${REL}/install.yaml"
kubectl -n argo-rollouts rollout status deployment/argo-rollouts --timeout=180s
kubectl apply -f canary-lab.yaml
kubectl argo rollouts status demo --timeout 180s
# A good release; press Ctrl-C when the rollout shows Healthy.
kubectl argo rollouts set image demo demo=argoproj/rollouts-demo:yellow
kubectl argo rollouts get rollout demo --watch
# A bad release.
kubectl argo rollouts set image demo demo=argoproj/rollouts-demo:bad-red
kubectl argo rollouts get rollout demo --watch

canary-lab.yaml:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: smoke
spec:
  metrics:
    - name: http-200
      count: 2
      interval: 30s
      failureLimit: 0
      provider:
        job:
          spec:
            backoffLimit: 0
            template:
              spec:
                restartPolicy: Never
                containers:
                  - name: smoke
                    image: curlimages/curl:8.23.0
                    command: ["sh", "-c"]
                    args:
                      - |
                        fails=0; i=0; u=http://demo-canary/color
                        while [ "$i" -lt 40 ]; do
                          code=`curl -s -o /dev/null -w '%{http_code}' "$u"`
                          [ "$code" = "200" ] || fails=`expr "$fails" + 1`
                          i=`expr "$i" + 1`
                        done
                        echo "non-200 responses: $fails of 40"
                        [ "$fails" -le 2 ]
---
apiVersion: v1
kind: Service
metadata:
  name: demo-canary
spec:
  selector:
    app: demo
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: demo-stable
spec:
  selector:
    app: demo
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: demo
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: demo
          image: argoproj/rollouts-demo:blue
          ports:
            - containerPort: 8080
          readinessProbe:
            tcpSocket:
              port: 8080
  strategy:
    canary:
      canaryService: demo-canary
      stableService: demo-stable
      steps:
        - setWeight: 20
        - analysis:
            templates:
              - templateName: smoke
        - setWeight: 60
        - pause:
            duration: 30s

Expected output. yellow: smoke Jobs pass and the rollout becomes Healthy. bad-red: one canary pod (20% of 5), the smoke Job fails, the rollout aborts to stable yellow and shows Degraded^[29]. Validation. kubectl get analysisrun shows the failed run. Troubleshooting. Inconclusive: the Job image could not be pulled^[50]. The TCP readiness probe is deliberate: an HTTP probe on /color would hide the faulty pod from analysis. Recover. Set the image back to yellow (GitOps: revert). Teardown. kind delete cluster --name release-lab.

All three are illustrative scenarios (composites, not real organisations); sourced cases follow later.

DORA’s five delivery metrics are defined on DevOps^[51], incident durations on IncidentOps. Release-specific measures:

Worked calculation (illustrative month). Of 10 release-caused defects, 7 stopped at ≤ 10%: early detection 70%. One defect reached 5% of 200,000 users for 20 minutes: 10,000 × 20 = 200,000 user-minutes, against 4,000,000 at full exposure. Five of 18 release flags are overdue: flag debt 28%.

Worked calculation: how long must a canary step last? The stable version fails 0.2% of requests; you want to detect a canary failing 0.4%, accepting a 5% false-alarm chance and an 80% detection chance. A normal approximation that treats the stable rate as known gives

n ≈ ((1.645 × √(0.002 × 0.998) + 0.84 × √(0.004 × 0.996)) ÷ 0.002)² ≈ 4,000 canary requests.

The alarm threshold is not the effect size. Alarm at the critical value — stable plus 1.645 × √(0.002 × 0.998 ÷ 4,000) ≈ 0.12 percentage points — not at the 0.2-point effect you fear: an alarm at the effect itself catches a true doubling only about half the time, at any traffic. The cost is about one false alarm in twenty, so that band should pause for a person, not abort (Artefact 2).

At 50 requests per second in total, a 1% canary needs about 2.2 hours, a 10% canary about 13 minutes; detecting 0.2% → 0.3% needs about 14,300. A five-minute 1% step sees 150 requests and 0.3 expected errors: it flags only failures around ten times the baseline (about 2% errors), not a doubling. Use larger steps, longer bake, staff traffic or synthetic checks. This ignores correlated errors and multiple metrics: a sanity check, not a statistical design.

• Decision rights are access rights. Map who may widen, abort, override, and change the policy or flags (audited).
• Fail-safe flag defaults. OpenFeature evaluation “must always return the default value in the event of abnormal execution”^[52], so the default in code is what users get when the flag service is down: make it the safe behaviour.
• Supply chain. Release the artefact that was tested — one build, signed, with provenance (DevSecOps).
• Privacy. Send the flag service only the attributes its rules need; prefer opaque IDs to e-mail addresses.
• Compliance evidence. DORA notes that peer review in the development platform can meet segregation of duties^[25]; a record linking pull request, pipeline run, analysis verdicts and flag log beats meeting minutes. Controls vary by industry and jurisdiction; confirm with auditors (ComplianceOps).

• Double capacity. Blue-green runs two environments. Argo Rollouts with traffic routing keeps stable fully scaled — instant abort, but double the pods until the update ends; dynamicStableScale saves capacity at the cost of a slower abort^[44].
• Duplicated CI. Each active release branch needs its own CI^[9] and backports.
• Waiting. Amazon’s default pipelines take about four or five business days to reach all Regions^[15] — right for global services, wrong for an internal tool.

Watch the release system itself: complete records, pauses older than a day, gate overrides, freeze exceptions, flag debt and rollback rehearsals. Each quarter, trace five releases from pull request to full exposure; where the trail breaks, fix the automation, not the paperwork.

Trigger: analysis fails or pauses, an alert fires, or users report a problem mid-rollout. Owner: on-call, without asking.

flowchart TB
  S["Stop signal or user report during a rollout"] --> W["Freeze widening; open an incident if users are hurt"]
  W --> F{"Is the change behind a run-time flag?"}
  F -- "Yes" --> OFF["Turn the flag off for everyone"]
  OFF --> V1{"Metrics back to baseline?"}
  V1 -- "Yes" --> RECF["Record the flag state; fix before re-enabling"]
  V1 -- "No: defect outside the flag" --> D
  F -- "No" --> D{"Has the new version written data the old one cannot read?"}
  D -- "No" --> RB["Abort the rollout or roll back"]
  D -- "Yes" --> RF["Roll forward: fix through a short canary"]
  RB --> V2{"Metrics back to baseline everywhere?"}
  RF --> V2
  V2 -- "Yes" --> REC["Record; revert desired state in Git by pull request"]
  V2 -- "No" --> INC["Escalate the incident: the cause may lie elsewhere"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class S,W,OFF,RB,REC,RECF core
  class F,D,V1,V2 decision
  class RF,INC risk

Steps.

1. Stop widening for the service; start no other deployments “to compare”.
2. Confirm by comparison with stable, and check the canary had traffic.
3. Choose the lever with the tree: flag first, then abort or rollback, then roll forward.
4. Execute the one alternative that matches your setup (adapt names; syntax-checked, not executed). On Argo Rollouts with GitOps, run argo and then gitops, or the next reconciliation re-applies the bad version.

#!/usr/bin/env bash
# Usage: back-out.sh argo | deployment | gitops
set -euo pipefail
case "${1:?choose argo, deployment or gitops}" in
  argo)        # traffic back to stable; Degraded until desired state changes
    kubectl argo rollouts abort checkout-api ;;
  deployment)  # previous revision of a Deployment (revisionHistoryLimit > 0)
    kubectl rollout undo deployment/checkout-api ;;
  gitops)      # make Git agree with what now runs, through expedited review
    bad="${BAD_COMMIT:?set BAD_COMMIT to the release commit}"
    git switch -c "revert-${bad}"
    git revert --no-edit "${bad}"
    git push -u origin HEAD
    echo "Open a pull request and ask for expedited review." ;;
  *) echo "unknown lever: $1" >&2; exit 2 ;;
esac

5. Verify reach. Check version labels per pod, region and cell; partial rollbacks and unreachable clients recur in the cases below.
6. Communicate and record in the release record; open the review (IncidentOps).
7. Fix forward through the same path with shorter steps — never a side door; any break-glass push to the protected branch uses a named identity and is recorded.

Each case comes from the organisation’s own or a regulator’s report; the “release lesson” is this Atlas’s reading.

Knight Capital, 1 August 2012 (SEC order). The SEC found that new trading code “repurposed a flag that was formerly used to activate” an unused function, Power Peg; that a technician, unreviewed, did not copy the new code to one of eight servers during a staged deployment; and that flagged orders made that server run the old code — about 4 million executions in about 45 minutes and, ultimately, a loss of more than $460 million. Knight then “uninstalled the new RLP code from the seven servers where it had been deployed correctly”, which “worsened the problem”^[53]; it settled for $12 million without admitting or denying the findings^[54]. Release lesson: never reuse a flag’s meaning, verify a deployment’s reach, and know what state a rollback recreates.

CrowdStrike, 19 July 2024 (company reviews). A Rapid Response Content update — “configuration data; it is not code or a kernel driver” — reached Windows sensors at 04:09 UTC and was reverted at 05:27 UTC; it shipped on “trust in the checks performed in the Content Validator” and past successes. The root cause: “The sensor expected 20 input fields, while the update provided 21 input fields”, an out-of-bounds read that crashed hosts. Content now passes “successive deployment rings before rollout into production”^[55]. Release lesson: configuration that changes behaviour needs staged exposure, and a revert cannot reach crashed machines.

Google Cloud, 12 June 2025 (provider report). A Service Control quota feature added on 29 May passed a region-by-region rollout without its failing path being exercised; it “did not have appropriate error handling nor was it feature flag protected”. Policy data with blank fields, replicated globally within seconds, triggered it everywhere. Its “red-button” off switch finished rolling out about 40 minutes in; restarts without randomised backoff then overloaded a dependency, and full recovery in us-central1 took up to about 2 h 40 min. Google writes: “If this had been flag protected, the issue would have been caught in staging”^[56]. Release lesson: a staged binary protects nothing if something unstaged switches its behaviour on, and a kill switch that needs its own rollout is not a seconds-scale lever.

• Ceremonial canaries — five minutes at 1% on a quiet service (sizing).
• Flags forever, or reused. Release toggles should “generally not stick around much longer than a week or two”^[13]; Knight’s dormant code woke through a reused flag.
• “It’s only config.” Configuration, policy data and content change behaviour; stage them like code.
• Rollback as a belief. An unrehearsed rollback, or one against data in a new format, is a hope, not a plan.
• The gatekeeper release manager. Relaying approvals adds queue time; the role earns its keep by improving the path.
• Myths: “GitFlow is the standard” (its author suggests a simpler flow for continuous delivery^[20]); “automated rollback makes us safe” (only for reversible changes and reachable instances).

Invest in progressive release machinery when you operate the software, deploy several times a week, have traffic to compare versions within an hour, and a bad release would be costly.

Keep it light, or skip it, when:

• The service is quiet. An internal tool with a few hundred users needs rolling deploys, a flag on the risky part and a tried rollback, not a canary with nothing to measure.
• The team is tiny. Three engineers need versioned artefacts, a tried rollback and expiring flags, not a controller.
• Users install the software. Rings, staged distribution and support windows replace traffic canaries (MobileOps).
• The change is one-way. Data migrations and public announcements need expand and contract, rehearsal and explicit sign-off.

Hidden responsibilities: running the controller and flag service, per-version metrics, flag removal, and reviewing pauses, aborts and overrides.

flowchart TB
  Q1{"Do you operate the running software?"} -- "No: users install it" --> A1["Rings, staged distribution, SemVer, support windows"]
  Q1 -- "Yes" --> Q2{"Can old and new versions share data and clients?"}
  Q2 -- "Yes" --> Q3{"Must users never meet a mix of versions?"}
  Q2 -- "No" --> C["Make them compatible: expand and contract, or Prepare then Activate"]
  C -- "Then ask again" --> Q2
  Q3 -- "Yes" --> BG["Blue-green if two environments are affordable; if not, deploy dark, then flip a flag for everyone"]
  Q3 -- "No" --> Q4{"Traffic to compare versions within an hour?"}
  Q4 -- "Yes" --> Q5{"Must the business choose when users see it?"}
  Q4 -- "No" --> A2["Keep it light: rolling deploy, flags for staff, rehearsed rollback"]
  Q5 -- "Yes" --> A4["Canary for the binary plus a release flag"]
  Q5 -- "No" --> A5["Automated canary with analysis"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class Q1,Q2,Q3,Q4,Q5 decision
  class BG,A4,A5 core
  class A1,A2 neutral
  class C risk

Assumptions: compatibility comes first because every branch runs or swaps versions against shared data^[31]; “within an hour” comes from the sizing; a flag keeps users on one behaviour only for the code it covers, and flipping it for everyone is unstaged: watch closely, switch back at once.

This Atlas’s model, not an industry standard; levels describe a service, not a company.

Dependencies. Per-version telemetry (ObservabilityOps), platform capacity, and leaders agreeing that on-call may abort without approval.

• Notes generated; breaking changes and security fixes checked by a person
• Risk class, steps and bake time taken from the policy
• Two or three stop signals visible per version
• Schema changes expand-only; rollback checked for data compatibility, or a roll-forward plan written
• Risky behaviour flagged (owner, expiry); the code default is the safe behaviour
• On-call knows about the rollout and holds the abort right
• No freeze or spent error budget, or a recorded exception

• Flag-debt check passes in CI
• Flags at 100% for over two weeks scheduled for removal
• Kill switches exercised outside production

• Rights to widen, abort, override and change flags match the RACI; leavers removed
• Main branch and tags protected; direct pushes only by the named break-glass identity
• Release tokens scoped to one repository and job; workflow dependencies locked or SHA-pinned
• Flag and controller audit logs reviewed; every override has a reason and expiry
• Expired overrides and waivers closed

• Timeline rebuilt from the release record
• Which signal caught it, at what exposure; time to safe state; what would have caught it earlier
• Follow-ups owned and dated

Example status: YAML syntax-checked; illustrative field names.

release: checkout-api 1.4.0
date: 2026-10-16
owner: team-checkout
risk_class: standard
summary: "New address validation; payment retry logic"
links:
  pull_requests: ["https://git.example.com/checkout-api/pulls/812"]
  pipeline_run: "https://ci.example.com/runs/5531"
  notes: "https://git.example.com/checkout-api/releases/v1.4.0"
rollback:
  safe: true
  method: "abort rollout; no new data formats written"
  checked_by: "a.reviewer"
flags:
  - key: new-checkout
    owner: team-checkout
    expires: 2026-11-30
exposure_log:   # standard class: >= 30 min and 4,000 canary requests per step
  - time: "2026-10-16T09:35:00Z"
    step: "5%"
    verdict: Successful
    decided_by: rollout-controller
  - time: "2026-10-16T10:05:00Z"
    step: "25%"
    verdict: Successful
    decided_by: rollout-controller
  - time: "2026-10-16T10:35:00Z"
    step: "50%"
    verdict: Successful
    decided_by: rollout-controller
outcome: released   # released | aborted | rolled-forward
follow_ups: []

Title:   Release strategy for <service>
Status:  proposed | accepted | superseded
Date:    YYYY-MM-DD
Owner:   <name>

Context
  Who uses the service, traffic per hour, cost of a bad release,
  data and schema constraints, regulatory or contractual change controls,
  team size and on-call coverage.

Decision
  Exposure mechanism (rolling / blue-green / canary / flags / rings),
  step sizes, bake time and request minimum, stop signals, abort rights,
  gate effects and overrides, versioning scheme, branching model, cadence.

Options considered
  At least two, including "keep it light".
  For each: protection gained, cost, operational load.

Consequences
  New dependencies (controller, flag service), capacity,
  flag-removal duty, evidence for audits.

Review
  Date, and the data that would change the decision
  (for example early-detection ratio, false aborts).

Mechanisms are compared under fundamentals; in an incident, the back-out levers matter:

Is “ReleaseOps” a job title or a team? Rarely; release manager and release engineer are the usual titles, with platform engineers and service owners sharing the work.

If we have canaries, do we still need feature flags? They turn different dials: a canary tests a new binary on a slice of traffic; a flag controls behaviour inside one binary. A leak in a new library is caught by a canary; a launch timed by marketing is controlled by a flag.

Rollback or roll forward — which is better? Roll back (or flag off) while the previous state is still valid; roll forward when it is not (table above), and design two-phase changes so that this is rare^[31].

Does progressive delivery replace testing? No. It limits the damage of what tests miss; test signal quality is on TestOps.

What does this have to do with ServiceNow’s ReleaseOps product? Only the name. That product moves changes and custom apps through pipelines on the ServiceNow platform; as of October 2026 it is not supported in regulated environments or on-premises, and entitlements must be checked^[2].

• Configuration pipelines, waves and cells. Amazon described dedicated configuration and feature-flag pipelines, and waves of increasing size, the first in one low-traffic Region, one zone or cell at a time^[15].
• Statistical and AI judges. Many metrics checked repeatedly raise false alarms; judges such as Kayenta (an Argo Rollouts provider^[30]) address this. Trust no judge until you have measured its false aborts and missed regressions.
• Bucketing stability. flagd v0.15.0 (2026-04-01) warned that “fractional pseudorandom assignments will change”^[57], and v0.18.0 (2026-10-15) that “pseudorandom bucket assignment may change”^[58], so an upgrade mid-rollout can move users between cohorts.

Paths through this page. Beginners: in 30 seconds, glossary, Lab 1, FAQ. Practitioners: reference implementation, labs, runbook, checklists. Leaders and architects: governance, decision framework, maturity, roadmap, judgement call.

You should now be able to explain deploy versus release, choose an exposure mechanism, design a canary step from traffic, operate the back-out levers and evaluate a release process with release metrics.

“ReleaseOps” is a borrowed name for an old problem: how to let change reach people without betting everything on one moment. The field’s answer is to separate installing from exposing, and to treat exposure as a dial with evidence at every step and a named hand ready to turn it back.

The documented failures share a pattern: something that changed behaviour reached everything at once. Release discipline means one staged, reversible path for every kind of change, with honestly sized steps; for quiet services it means less machinery, not more. Judge a release process by how little damage a bad change can do before someone notices, and how fast users are back on the safe path.

Changelog

• 2026-10-16 — r0: first version; Lab 1 and the Lab 2 linter executed (research/releaseops-ledger.md).
• 2026-10-16 — r1: review fixes (decision tree, per-step analysis, gate policy, back-out, cases, sizing); labs re-run (reviews/releaseops-r1-response.md).
• 2026-10-17 — r2: canary thresholds recalibrated and simulated; alarm-versus-effect lesson; Figures 3, 4 and 6; plugin version; re-check cadence (reviews/releaseops-r2-response.md).
• 2026-10-17, post-score polish: release-record steps, latency note, flag-flip caveat, Figure 3 crossings, Amazon’s first wave (reviews/releaseops-polish.md).