DevOps

DevOps is a way of organising software delivery so that building a service and running it are one continuous responsibility. The people who change the system share goals, tools and feedback with the people who keep it running, and often they are the same people. Small changes reach users quickly and safely, failures are noticed and fixed fast, and the organisation learns from both.

Developers are often rewarded for shipping change and operations staff for stability, so each side optimises against the other. DevOps aligns both around the flow of value to users, automates the path from commit to production, makes the results visible, and treats incidents as information rather than occasions for blame. Success is measured by delivery outcomes: DORA, Google Cloud’s long-running research programme, uses five metrics for how fast changes flow and how often deployments go wrong.^[1]

There is no single official definition, and the commonly cited ones stress different things:

• Microsoft describes DevOps as combining development and operations “to unite people, process, and technology in application planning, development, delivery, and operations”.^[2]
• Bass, Weber and Zhu, in DevOps: A Software Architect’s Perspective (2015),^[3] define it as “a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality” (wording checked as quoted by Wikipedia).^[4]
• Jez Humble, co-author of Accelerate,^[5] put its essence as designing “a system in which people are held responsible for the consequences of their actions”.^[6]

This page uses all three: shared ownership, the commit-to-production path and the whole lifecycle.

The classic pathology is a wall between development and operations. Werner Vogels, Amazon’s CTO, described the traditional model in 2006 as one where “you take your software to the wall that separates development and operations” and then stop thinking about it.^[10] The symptoms:

• Large, infrequent releases. Each release bundles months of change, so each is risky; risk invites more approval steps, which make releases larger still. DORA describes this loop: responding to instability with more process increases lead times and batch sizes.^[11]
• Integration hell. Long-lived branches diverge, and merging them needs code freezes and “integration phases”.^[12]
• Hand-offs without context. Operations runs software it did not help design: DevOps Topologies’ anti-type A, “Dev and Ops silos”.^[13]
• Blame-driven incident response. When failure leads to scapegoating, people hide bad news and the organisation stops learning: the “pathological” culture in Westrum’s typology, which DORA uses.^[14]
• Manual, unrepeatable deployments, run late at night by tired people.

flowchart TB
  subgraph BEFORE["Before: functional silos"]
    direction TB
    D1["Developers rewarded for change"] -->|"release bundle every few months"| W{{"Hand-off wall"}}
    W --> O1["Operations rewarded for stability"]
    O1 -->|"tickets and blame"| D1
    C1["Change advisory board (CAB)"] -.->|"adds wait time"| W
  end
  subgraph AFTER["After: one team owns the service end to end"]
    direction TB
    T["Product team builds and runs the service"] -->|"small change, many times a week"| P["Automated pipeline tests, checks, deploys"]
    P --> PR["Production"]
    PR -->|"telemetry and incidents"| T
    PL["Platform team self-service paved road"] -.->|"as a service"| P
  end
  BEFORE ~~~ AFTER
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class D1,O1,W,C1 risk
  class T,P,PR core
  class PL neutral

What it costs not to change. Published savings figures vary too much by context to quote, so build your own case from four costs: delay (cost of delay); failure, because large batches fail more expensively; people, because DORA reports that strong continuous delivery predicts lower burnout and less “deployment pain”;^[15] and learning, because a quarterly release gives four chances a year to learn whether ideas work.

Not a tool, product or job title you can buy. John Willis put culture first: “People and process first. If you don’t have culture, all automation attempts will be fruitless”.^[16] DORA warns that modern tooling without the technical practices and process change “won’t produce the expected benefits”.^[15]

Not developers doing operations so that Ops can be cut. DevOps Topologies calls this anti-type C, “Dev don’t need Ops”, which underestimates how complex operations is.^[13] Operational expertise moves and is shared; it does not disappear (see NoOps for that aspiration).

Not a separate team between Dev and Ops. Humble argued in 2012 that a “DevOps team” placed between development and operations creates exactly the silo DevOps exists to remove; he accepted the label only for people who build a self-service platform and toolchain and coach other teams.^[6] DevOps Topologies lists the “DevOps team silo” as anti-type B, sensible only when it is temporary (“less than (say) 12 or 18 months”) and mandated to make itself unnecessary.^[13]

Not one job description. A “DevOps engineer” advertisement can mean a build, cloud or platform engineer or an SRE, so read the responsibilities. The rebranded sysadmin (anti-type E) and the DevOps tools team building pipelines in isolation (anti-type D)^[13] are fine as job categories and wrong as transformation strategies, because the hand-off remains.

Not just CI/CD, and not a synonym. DORA’s capability catalogue (34 capabilities in October 2026) also covers culture, leadership, documentation, team structure and well-being.^[17] DevOps extends Agile thinking through release and operation; the SRE Workbook frames SRE as “class SRE implements interface DevOps”;^[18] platform engineering makes DevOps practices cheap to adopt.^[19]

Not the EU regulation. In finance, “DORA” also means the EU Digital Operational Resilience Act, Regulation (EU) 2022/2554, applicable since 17 January 2025 to financial entities and their ICT third-party providers.^[20] It is unrelated to the DORA metrics, so say which one you mean.

Sourced milestones only; not a complete timeline.

• 1968. Mel Conway’s “How Do Committees Invent?” (Datamation): a system’s design copies its organisation’s communication structure.^[21]
• 2000. Martin Fowler’s “Continuous Integration” (revised in 2024) popularises merging at least daily with a self-testing build.^[22]
• 2006. Werner Vogels: “You build it, you run it.” Introducing the interview, AWS’s Jeff Barr added that a new developer’s manager “may very well hand you a pager that same day”.^[10]
• 2008. Patrick Debois runs an “Agile Infrastructure” session at the Agile conference in Toronto and meets Andrew Shafer there (Debois’s own account).^[23]
• 2009. John Allspaw and Paul Hammond’s Velocity talk “10+ Deploys A Day: Devs & Ops Cooperation at Flickr”;^[24] the first devopsdays, in Ghent, founded by Patrick Debois.^[8]
• 2010. John Willis’s “What DevOps Means to Me” sets out CAMS (Culture, Automation, Measurement, Sharing);^[16] Lean was added later to make CALMS.^[18]
• 2012. Gene Kim’s article on “The Three Ways”, the principles from which DevOps practices are derived.^[25]
• 2014–2025. DORA’s annual reports turn DevOps from anecdote into measured, if correlational, evidence.^[7]
• 2018. Accelerate by Forsgren, Humble and Kim;^[5] DORA joins Google Cloud (announced 2018-12-20).^[26]
• 2023–2024. DORA renames “time to restore” as failed deployment recovery time, narrowed to change-caused failures (2023), and adds deployment rework rate as a fifth metric (2024).^[27]
• 2025. DORA’s State of AI-assisted Software Development: AI acts as “an amplifier, magnifying an organization’s existing strengths and weaknesses”.^[28]

The mental model: a delivery system with two loops. Picture a pipe with a feedback loop: work flows from idea to production to user, and information flows back from telemetry, incidents and user behaviour. DevOps makes the pipe short and smooth (flow) and the loop fast and loud (feedback), in a culture that learns from it: Gene Kim’s Three Ways.^[25] Almost every practice on this page removes friction from the pipe or speeds up the loop.

Key terms. The five delivery metrics are defined once, in the metrics part.

• Value stream: all the steps from a customer need to its fulfilment in production (idea → code → review → test → deploy → monitor).
• %C/A (percent complete and accurate): recorded at each step, the share of the work it receives that it can use without rework;^[29] a low value points upstream.
• Batch size: how much change moves through the system at once: a 50-line pull request, not a 5,000-line branch.
• Trunk-based development: developers merge small changes to the trunk (main) at least daily, with three or fewer active branches.^[12]
• Continuous integration (CI): everyone merges to the mainline at least daily, and every push triggers a self-testing build.^[22]
• Continuous delivery (CD): software is kept deployable, so any version can be released on demand;^[30] release becomes a business decision.
• Deployment versus release: deployment installs code in production; release exposes it to users, for example through a flag.
• Canary: “A partial and time-limited deployment of a change in a service and its evaluation”^[31] (strategies: ReleaseOps).
• Feature toggle (flag): a runtime switch that changes behaviour without a deployment.^[32]
• Artifact: the immutable, versioned build output; DORA advises “the same packages for every environment”.^[33]
• Smoke test: a quick post-deployment check that the system works and its dependencies are reachable.^[33]
• Shift left: moving quality and security checks earlier, where problems are cheaper to fix.
• Percentile (p50, p90): the value below which 50% (the median) or 90% of observations fall; p90 shows the slow tail.
• Roll back versus fix forward: redeploy the previous artifact, or ship a new change that repairs the fault.
• Generative culture: Westrum’s “performance-oriented” culture: high cooperation, messengers trained, failure leads to inquiry.^[14]
• Transformational leadership: vision, inspirational communication, intellectual stimulation, support and recognition; a DORA capability.^[34]
• J-curve: early gains, then a dip, then recovery to a higher level; DORA describes this shape for both continuous delivery and platform engineering, with different causes of the dip.^[15][19]
• Segregation of duties (SoD): the author of a change is not its only approver (risk controls); a change advisory board (CAB) is one way of approving changes (ITOps).

DevOps is the hub of the Atlas’s delivery pages: it owns the culture, the pipeline and the measurement model, and neighbours specialise. The boundaries are editorial distinctions, not industry standards; one real team often does several of these jobs.

flowchart LR
  AG["AgileOps: planned work enters the value stream"]
  IT["ITOps: change-governance interface (ITIL)"]
  DEV["DevOps: culture, flow, CI/CD, DORA metrics"]
  SRE["SRE: takes reliability targets (SLOs, error budgets)"]
  SEC["DevSecOps: security gates in the pipeline"]
  REL["ReleaseOps: how changes are exposed"]
  GIT["GitOps: one pull-based deployment mechanism"]
  INF["InfraOps: environments as code"]
  TEST["TestOps: a trustworthy test signal"]
  PLAT["PlatformOps: builds the paved road as a product"]
  DX["DevExOps: finds developer friction"]
  AG --> DEV
  IT --> DEV
  DEV --> SRE
  DEV --> SEC
  DEV --> REL
  DEV --> GIT
  DEV --> INF
  DEV --> TEST
  DEV --> PLAT
  DX --> PLAT
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class DEV core
  class SRE,PLAT,DX,SEC,REL,GIT,INF,TEST,AG,IT neutral

IncidentOps (live incidents) and ObservabilityOps (telemetry) are neighbours too; DevOps uses both and teaches neither. These are not competitors. DevOps defines the delivery system and its measures, with the value stream and the owning team as its unit; SRE sets reliability targets inside it, platform engineering makes it cheap to adopt, DevSecOps secures it, GitOps is one way to deploy within it, and IT service management connects it to enterprise governance.

CAMS and CALMS. John Willis’s 2010 post set out Culture, Automation, Measurement and Sharing, calling sharing “the loopback in the CAMS cycle”.^[16] Google’s SRE Workbook credits Willis, Damon Edwards and Jez Humble with CALMS, which adds Lean.^[18] As a diagnostic, one warning sign per pillar: incidents end with a name (Culture); one person can deploy (Automation); work waits days for review (Lean); dashboards nobody discusses (Measurement); every team writes its own pipeline (Sharing).

The Three Ways (Gene Kim, 2012).^[25] (1) Flow: optimise “the performance of the entire system, as opposed to the performance of a specific silo”. (2) Amplify feedback: create “the right to left feedback loops”. (3) Continual experimentation and learning: take risks, learn from failure, and practise, because repetition is “the prerequisite to mastery”.

---
config:
  flowchart:
    nodeSpacing: 30
    rankSpacing: 35
---
flowchart TB
  IDEA["Idea or hypothesis"] --> CODE["Small change committed to trunk"]
  CODE --> PIPE["Automated build, test and deploy"]
  PIPE --> PROD["Running in production"]
  PROD --> USER["Users and business outcome"]
  IDEA ~~~ UF
  CODE ~~~ TEL
  PIPE ~~~ LEARN
  USER -.-> UF["Second Way: usage feedback"]
  UF -.-> IDEA
  PROD -.-> TEL["Second Way: telemetry and alerts"]
  TEL -.-> CODE
  PROD -.-> LEARN["Third Way: blameless reviews and experiments"]
  LEARN -.-> PIPE
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class IDEA,CODE,PIPE,PROD,USER core
  class TEL,UF neutral
  class LEARN decision

Continuous delivery principles (Jez Humble): build quality in; work in small batches; computers perform repetitive tasks and people solve problems; relentlessly pursue continuous improvement; everyone is responsible.^[35] There is no official “DevOps Manifesto” comparable to the Agile Manifesto, and this page does not invent one.

Invariants. Every production change is traceable to a reviewed commit, a build and a deployment record; the same artifact is promoted through every environment;^[33] rollback is possible within minutes, and practised; a deployment without monitoring is not finished. The main trade-offs:

A DevOps capability rests on four layers, each depending on the one below:

1. Foundations: version control for everything, including pipeline and monitoring configuration;^[36][38] a single trunk; code review (repository, CODEOWNERS, branch protection).
2. Flow: CI, test automation, trunk-based development, small batches, feature toggles, deployment automation (pipeline definition, test suites, versioned artifacts).
3. Feedback: observability, deployment records, incident reviews, delivery metrics, value stream maps.
4. Organisation: stream-aligned teams with clear ownership, platform and enabling teams, generative culture, leadership that funds improvement (service catalogue, on-call rotas, improvement backlog).

Culture is not a soft extra: DORA’s 2023 report found that generative cultures “correlate with 30% higher organizational performance”, and user-focused teams achieve 40% higher organisational performance (survey correlations).^[39][14] Continuous delivery alone draws on 14 technical capabilities in DORA’s catalogue;^[15] find the one holding you back.

Architecture matters as much as tooling. Conway’s law predicts that a system’s design copies the organisation’s communication structure,^[21] so if five teams must coordinate to ship one change, the architecture is the bottleneck. DORA’s loose-coupling test: can a team test and deploy its service independently, in business hours, with negligible downtime?^[40] Citing its 2021 report, DORA adds that “elite” teams (that year’s top cluster) meeting their reliability targets were three times more likely to have adopted such an architecture.^[15]

This page uses Team Topologies’ vocabulary: four team types (stream-aligned, the default, aligned to a flow of work from a segment of the business; enabling; complicated-subsystem; platform), three interaction modes (collaboration, X-as-a-Service, facilitation), and cognitive load as a design constraint, because “overloaded teams make poor decisions and move slowly”.^[37]

---
config:
  flowchart:
    nodeSpacing: 30
    wrappingWidth: 160
---
flowchart TB
  SECT["Security and compliance: policy as automated checks"]
  LEAD["Engineering leadership: funds improvement, removes blockers"]
  ENAB["Enabling team: time-boxed facilitation on testing, CI, incident reviews"]
  PLATF["Platform team: pipeline, runtime and observability X-as-a-Service"]
  SRET["SRE specialists: collaboration on SLOs"]
  S1["Stream-aligned team: checkout (builds and runs it)"]
  S2["Stream-aligned team: search (builds and runs it)"]
  SECT --> PLATF
  LEAD ~~~ PLATF
  ENAB -.-> S1
  PLATF --> S1
  PLATF --> S2
  SRET -.-> S2
  LEAD --> S1
  LEAD --> S2
  S2 -.->|"feedback"| PLATF
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class S1,S2 core
  class PLATF,ENAB,SRET,SECT neutral
  class LEAD decision

The RACI below (R = responsible, A = accountable, C = consulted, I = informed) is an illustrative default; in small organisations one person holds several roles.

Microsoft describes the loop as plan, develop, deliver and operate, and stresses that “the phases aren’t role-specific”.^[2] For one service:

A costly mistake is to skip Day 0 and bolt the pipeline on after months of untested code and manual steps.

Ownership. Every service has exactly one owning team in a service catalogue, including on-call, security patches and cost.

Flow policies (defaults any exception must justify): small pull requests reviewed promptly, ideally synchronously, because days-long review pushes developers away from small batches; branches under a day old, three or fewer active, no code freezes;^[12] a broken trunk fixed or reverted within minutes.^[36]

Ceremonies (a minimal set).

Escalation. A constraint a team cannot fix (a shared database, a central approval, a platform gap) goes to the delivery review owner, then after two reviews to engineering leadership, with the metric evidence attached.

Prefer controls that run on every change and leave evidence. This is the page’s one explanation of segregation of duties.

• Segregation of duties through recorded peer review. Many regimes require that a change’s author is not its only approver. DORA notes that peer review recorded in the development platform can satisfy this, so a separate external approval body is not needed for that purpose.^[11] Whether your auditor or regulator agrees is their decision: bring evidence (each review linked to its build and deployment), and remember that a rubber-stamp review destroys the control.
• Automated checks: tests, security and policy checks in the pipeline (DevSecOps).
• Risk tiers. Standard changes promote automatically; high-risk ones (schema migrations on critical data, authentication) get human review through a deployment approval or equivalent gate that your platform plan actually provides (see the reference implementation).
• A re-chartered CAB focuses on coordination, process improvement and business trade-offs, not routine approvals.^[11] Enterprise change enablement: ITOps.

The delivery system below is technology-neutral: every box is a responsibility, and the tooling landscape lists ways to fill it. The four zones are trust boundaries, so every arrow that crosses between zones needs its own access control and audit trail. Dashed arrows show feedback and runtime control rather than the flow of the change.

---
config:
  flowchart:
    nodeSpacing: 30
    rankSpacing: 40
---
flowchart TB
  subgraph DEVZONE["Developer zone"]
    IDE["Owning team's workstation: code, local tests"]
  end
  subgraph CIZONE["Source and build zone"]
    VCS["Version control: trunk, reviews, branch protection"]
    CI["CI runners: build, unit and integration tests"]
    CHK["Policy and security checks"]
    REG["Artifact registry: immutable, versioned"]
  end
  subgraph RUNZONE["Runtime zone"]
    CD["Deployment orchestrator"]
    STG["Staging environment"]
    PRD["Production environment"]
    FLAG["Feature flag service: a production control"]
  end
  subgraph FBZONE["Feedback zone"]
    OBS["Observability: metrics, logs, traces, alerts"]
    REC["Delivery records: deployments, incidents"]
    MET["Five delivery metrics per service"]
  end
  IDE -->|"push small change"| VCS
  VCS -->|"webhook"| CI
  CI --> CHK
  CHK -->|"publish once"| REG
  REG -->|"same artifact"| CD
  CD --> STG
  CD -->|"after checks pass"| PRD
  FLAG -.->|"runtime exposure"| PRD
  PRD --> OBS
  CD -->|"deployment event"| REC
  OBS -->|"incident and recovery events"| REC
  REC --> MET
  OBS -.->|"alerts"| IDE
  MET -.->|"monthly review"| IDE
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class VCS,CI,REG,CD core
  class IDE,STG,OBS,REC,MET neutral
  class CHK,PRD,FLAG risk

One change from commit to production through a push-based pipeline with a canary step, including the failure path. Release strategies belong to ReleaseOps.

---
config:
  sequence:
    wrap: true
    width: 104
    actorMargin: 10
---
sequenceDiagram
  actor Dev as Developer
  participant VCS as Version control
  participant CI as CI pipeline
  participant Rev as Peer reviewer
  participant CD as Deployment orchestrator
  participant Prod as Production
  participant Obs as Observability
  Dev->>VCS: 1. Push short-lived branch, open pull request
  VCS->>CI: 2. Build and fast tests
  CI-->>Dev: 3. Result in minutes
  Rev->>VCS: 4. Review and approve the small diff
  Dev->>VCS: 5. Merge to trunk
  VCS->>CI: 6. Build artifact once, full tests, checks
  CI->>CD: 7. Promote artifact to staging, smoke test passes
  CD->>Prod: 8. Deploy a canary slice first
  Prod->>Obs: 9. Metrics, logs, traces
  alt Health checks pass
    Obs-->>CD: Canary healthy
    CD->>Prod: Complete rollout, record deployment
  else Error rate or latency breaches threshold
    Obs-->>CD: Canary unhealthy
    CD->>Prod: Roll back to previous artifact
    CD-->>Dev: Page owning team with deployment ID
    Dev->>VCS: Fix forward or revert in a new small change
  end

Each step leaves a measurable trace (branch age, CI duration, review wait, smoke failures, deployment time, recovery), and the approval is the segregation-of-duties evidence.

Exception paths to design up front. Pipeline down: an audited break-glass path, reviewed every time it is used. Urgent security fix: the same pipeline and checks; in this Atlas’s view, an emergency path that skips tests removes the checks exactly when people are most rushed. Failed migration: backward-compatible database steps, so the application can roll back without a data rollback (Playbook 3 below).^[41]

The core practices are CI, trunk-based development, test automation (with developers as the primary authors of acceptance tests^[42]), deployment automation, small batches (work taking “longer than a week to complete and check is too big”^[43]), feature toggles, team-owned observability, value stream mapping (Lab 1), streamlined change approval and blameless incident review. Their measures are the supporting indicators; below are the test stages and three playbooks for what teams most often get wrong.

DORA’s ordering: “faster-running unit tests run before slower-running acceptance tests, and both are run before any manual testing”, with developer feedback in under ten minutes.^[42] Budgets after the first row are this Atlas’s suggestions.

DORA’s advice is “don’t tolerate flaky tests”: a pass should mean the software is releasable and a failure a real defect;^[42] flaky tests also teach people to rerun until green. Quarantine mechanics belong to TestOps.

Prerequisites: CI runs on every push; a feature-toggle mechanism exists (even a config switch); the team agrees.

1. Measure the baseline for two weeks: active branches, median branch age, merge conflicts, review wait.
2. Make CI fast (about 10 minutes or less) first;^[36] on a slow pipeline, trunk-based development queues at the merge button.
3. Agree: branches live less than a day, reviews are answered within an agreed window, incomplete work hides behind toggles.
4. Drain the old branches one at a time, behind a toggle.
5. Protect trunk: review and a green build to merge; no force-pushes.
6. Review after four weeks with the same measures.

Example status: syntax-checked by bash -n, not executed against a remote repository. gh is the GitHub CLI; on other platforms, open the pull request in the web interface.

# A day in trunk-based development: one small change, merged the same day.
git switch main && git pull --ff-only
git switch -c fix/checkout-rounding        # short-lived branch, hours not weeks
# ...edit code and tests...
git add -A && git commit -m "Round totals half-even in checkout (flagged off)"
git fetch origin && git rebase origin/main  # stay current with trunk
git push -u origin fix/checkout-rounding    # pushes the branch; the reference workflow runs only on a PR
gh pr create --fill                         # open the pull request; CI runs on it
# after a green build and a prompt review, merge via the platform, then:
git switch main && git pull --ff-only && git branch -d fix/checkout-rounding

Acceptance criteria: three or fewer active branches; median branch age under a day; no code freeze in the period. Rollback: if trunk is repeatedly broken, pause and invest in tests rather than returning to long branches. Variants: pair or ensemble programming can replace asynchronous review; DORA recommends making review synchronous.^[12]

Count every manual step of today’s deployment, including undocumented ones;^[33] script them in version control; use the same script and artifact for every environment, with configuration kept separate;^[33] add a smoke test that fails the deployment and a rehearsed rollback; emit a deployment record; then move deployment into the pipeline (the reference implementation shows the end state). Acceptance criteria: zero manual steps for standard changes, rollback rehearsed this quarter, every deployment recorded, so that “releases are performed in a low-risk way during normal business hours”.^[15]

DORA recommends versioned database change scripts, decoupled from application changes so the application can roll back “without requiring a database rollback”.^[41] The technique is parallel change, or expand and contract: expand to support old and new, migrate the clients, then contract.^[44] Illustrative example: renaming orders.cust_name to orders.customer_name takes five small deployments, each passing the pipeline on its own: (1) add a nullable customer_name; (2) write both columns, still reading cust_name; (3) copy old values with an idempotent batch job checked by row counts; (4) read customer_name, still writing both; (5) stop writing cust_name and drop it after a soak period. Steps 1–4 can each be undone by redeploying the previous application version, because the old column stays complete; step 5 is irreversible, so back up and schedule it deliberately. The common failure is steps 1 and 5 in one migration (a plain rename). Migration tooling, locks and replication lag belong to DBOps.

DevOps has no normative standard of the ISO kind; none of these sources is mandatory unless a regulator or contract makes it so.

Named tools are examples verified to exist and to describe themselves as stated (October 2026), not endorsements; features and pricing change.

Pull-based deployment, infrastructure as code and observability tooling are covered on GitOps, InfraOps and ObservabilityOps.

Selection heuristics: prefer the CI system native to where your code lives unless isolation or regulation forces self-hosting; confirm you can export pipeline definitions and deployment history; prove it on your two hardest pipelines.

Automate what is repetitive, rule-based and frequent: builds, tests, deployments, environments, dependency updates, evidence and metrics. Keep humans on novel, high-impact judgement: architecture, risky migrations, incident command, choosing the next constraint; in Humble’s words, “computers perform repetitive tasks, people solve problems”.^[35] Automating a bad process makes waste faster, unowned automation becomes the new bottleneck, and a failing auto-remediation loop can hide a real incident.

What the evidence says about AI (as of October 2026). DORA’s 2024 report found that AI adoption “significantly increases individual productivity, flow, and job satisfaction”, yet “negatively impacts software delivery stability and throughput”.^[59] Its 2025 research calls AI an amplifier of existing strengths and weaknesses.^[28] DORA’s AI Capabilities Model (78 in-depth interviews, a survey of almost 5,000 respondents) names seven capabilities that “amplify or unlock the benefits of AI”: a clear and communicated AI stance; healthy data ecosystems; AI-accessible internal data; strong version control practices; working in small batches; a user-centric focus; quality internal platforms. Without a user-centric focus, “AI adoption can have a negative impact on team performance”.^[60] Reviewing machine-generated code can take more effort per line, so small batches matter more.^[43]

Each AI use needs its own control: generated tests and pipeline code are reviewed and run in CI (risk: tests that pass trivially); build and incident summaries are checked against raw logs; auto-remediation gets a narrow scope, a kill switch and an audit log (risk: loops that mask incidents).

Scenario (illustrative). One team owns checkout-api, a containerised Python service on GitHub, and wants every pull request tested, every merge to main built once, and the same image deployed to staging then production, with a smoke test, automatic rollback and a machine-generated deployment record. It uses GitHub Actions (any orchestrator works) and GitHub’s container registry; scripts/deploy.sh is yours to write for your runtime. Environments staging and production exist in the repository settings;^[61] for a human approval, add required reviewers to production (up to six, with an option to prevent self-review).^[55] Placeholders such as example.com are fake.

Example status: YAML syntax-checked by the Atlas validator; keys checked against GitHub’s workflow syntax, expressions and OIDC documentation on 2026-10-16;^[49][62][63] not executed (it needs a repository, a registry, a cloud role and a deploy target). Action major versions match the actions’ READMEs as of October 2026;^[64][65] pin them to full commit SHAs in production.

# .github/workflows/delivery.yml
# Test every pull request; on main: build once, deploy the SAME image to staging, then production.
name: delivery

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

permissions:
  contents: read            # least privilege by default; jobs widen only what they need

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  # Cancel superseded pull-request runs, but never cancel a deployment already running on main.
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

env:
  IMAGE: ghcr.io/example-org/checkout-api   # registry paths must be lowercase

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: '3.13'
      - run: python -m pip install -r requirements.txt -r requirements-dev.txt
      - run: python -m pytest -q

  build:
    if: github.event_name == 'push'
    needs: test
    runs-on: ubuntu-latest
    timeout-minutes: 20
    permissions:
      contents: read
      packages: write       # push to the registry and nothing else
    outputs:
      image: ${{ steps.meta.outputs.image }}
    steps:
      - uses: actions/checkout@v7
      - id: meta
        run: echo "image=${IMAGE}:${GITHUB_SHA}" >> "$GITHUB_OUTPUT"
      - name: Build once, tag by commit SHA, push
        env:
          REGISTRY_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          REGISTRY_USER: ${{ github.actor }}
          IMAGE_REF: ${{ steps.meta.outputs.image }}
        run: |
          echo "$REGISTRY_TOKEN" | docker login ghcr.io -u "$REGISTRY_USER" --password-stdin
          docker build --label "org.opencontainers.image.revision=${GITHUB_SHA}" -t "$IMAGE_REF" .
          docker push "$IMAGE_REF"

  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    timeout-minutes: 20
    permissions:
      contents: read        # job-level permissions replace the default, so restate it
      id-token: write       # may request a short-lived OIDC token; grants no other access
    environment:
      name: staging
      url: https://staging.checkout.example.com
    steps:
      - uses: actions/checkout@v7
      # Log in to your cloud here with its official OIDC login action. The cloud role trusts only the
      # subject repo:example-org/checkout-api:environment:staging. No long-lived cloud keys in secrets.
      - name: Deploy the image built above
        env:
          IMAGE_REF: ${{ needs.build.outputs.image }}
        run: ./scripts/deploy.sh staging "$IMAGE_REF"
      - name: Smoke test
        run: ./scripts/smoke-test.sh https://staging.checkout.example.com "$GITHUB_SHA"

  deploy-production:
    needs: [build, deploy-staging]
    runs-on: ubuntu-latest
    timeout-minutes: 30
    permissions:
      contents: read
      id-token: write
    environment:
      name: production      # protection rules (reviewers, allowed branches) are set on the environment
      url: https://checkout.example.com
    steps:
      - uses: actions/checkout@v7
      # OIDC cloud login here; this role trusts only repo:example-org/checkout-api:environment:production.
      - id: start
        run: echo "at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$GITHUB_OUTPUT"
      - name: Deploy the same image
        env:
          IMAGE_REF: ${{ needs.build.outputs.image }}
        run: ./scripts/deploy.sh production "$IMAGE_REF"
      - name: Smoke test
        run: ./scripts/smoke-test.sh https://checkout.example.com "$GITHUB_SHA"
      - name: Roll back to the previous image
        if: ${{ failure() }}
        run: ./scripts/rollback.sh production
      - name: Record the deployment event
        if: ${{ !cancelled() }}
        env:
          OUTCOME: ${{ job.status }}
          STARTED_AT: ${{ steps.start.outputs.at }}
          EVENTS_URL: ${{ secrets.DELIVERY_EVENTS_URL }}
          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
        run: |
          printf '{"service":"checkout-api","environment":"production","version":"%s","started_at":"%s","finished_at":"%s","outcome":"%s","run_url":"%s"}\n' \
            "$GITHUB_SHA" "$STARTED_AT" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$OUTCOME" "$RUN_URL" > deployment-event.json
          curl --fail --silent --show-error -X POST -H 'Content-Type: application/json' \
            --data @deployment-event.json "$EVENTS_URL"

Why it is built this way:

• Build once, deploy many. Only build creates an image; both deploy jobs use the same commit-SHA tag.^[33]
• Least privilege, including the deployment credential. Default contents: read; only build writes packages; only the deploy jobs may request an OIDC token, which the cloud exchanges for “a short-lived access token that is only valid for a single job”, so “you won’t need to duplicate your cloud credentials as long-lived GitHub secrets”.^[66] For jobs that reference an environment, the token’s subject is repo:ORG-NAME/REPO-NAME:environment:ENVIRONMENT-NAME:^[63] make each cloud role trust only its own repository and environment. That alone does not stop a feature branch: environments allow every branch to deploy unless you set deployment branches, and by default administrators can bypass protection rules,^[55] so limit production to main and disallow bypassing before the production role trusts that subject. Fallback if the target cannot use OIDC: environment secrets, readable only by jobs that reference the environment^[55] (Pro, Team or Enterprise for private repositories), kept narrow and rotated. Secrets reach scripts through env, never ${{ }} in shell text, which prevents script injection.
• Never cancel a deployment midway. cancel-in-progress is true only for pull requests; a concurrency group keeps one pending run and cancels an older pending one,^[49] so on main the latest change wins.
• Failure is handled in the pipeline. A failed smoke test triggers rollback (failure()^[62]), and the deployment is recorded whatever the outcome, so the change fail rate cannot be hidden.

Example status: executed on 2026-10-16 (GNU bash, curl) against a local HTTP server, covering three cases: correct version (exit 0), wrong version (exit 1) and connection refused (exit 1).

#!/usr/bin/env bash
# smoke-test.sh: fail a deployment unless /healthz answers with the expected version in time.
# Usage: smoke-test.sh <base-url> [expected-version]
set -euo pipefail

BASE_URL="${1:?usage: smoke-test.sh <base-url> [expected-version]}"
EXPECTED_VERSION="${2:-}"
ATTEMPTS="${ATTEMPTS:-10}"
SLEEP_SECONDS="${SLEEP_SECONDS:-3}"

for attempt in $(seq 1 "$ATTEMPTS"); do
  if body="$(curl --silent --show-error --fail --max-time 5 "${BASE_URL}/healthz")"; then
    if [[ -z "$EXPECTED_VERSION" || "$body" == *"\"version\": \"${EXPECTED_VERSION}\""* ]]; then
      echo "smoke test passed on attempt ${attempt}: ${body}"
      exit 0
    fi
    echo "attempt ${attempt}: healthy but running the wrong version: ${body}" >&2
  else
    echo "attempt ${attempt}: health check failed" >&2
  fi
  sleep "$SLEEP_SECONDS"
done

echo "smoke test FAILED after ${ATTEMPTS} attempts against ${BASE_URL}" >&2
exit 1

Checking the version, not just HTTP 200, catches the silent failure where the old version still serves because the new one never started. /healthz must report its build version, for example {"status": "ok", "version": "<commit-sha>"}; the string match assumes that spacing, so parse JSON (for example with jq) in real services.

The pipeline posts a minimal event; your metrics store adds the commits (git log <previous-version>..<version>) and links incidents, as in the richer record below. The schema is this Atlas’s own, not claimed to conform to CDEvents, which is worth evaluating if several tools must exchange delivery events.^[48]

Example status: JSON syntax-checked by the validator. The SHA is a fake full-length value, because the pipeline records the full 40-character $GITHUB_SHA.

{
  "service": "checkout-api",
  "environment": "production",
  "version": "3f9c2e1d4b7a8c6e5f0a1b2c3d4e5f6a7b8c9d0e",
  "started_at": "2026-09-08T15:52:10Z",
  "finished_at": "2026-09-08T16:00:00Z",
  "outcome": "success",
  "commits": [
    {"sha": "3f9c2e1d4b7a8c6e5f0a1b2c3d4e5f6a7b8c9d0e", "committed_at": "2026-09-08T10:00:00Z"}
  ],
  "failed": true,
  "failure": {
    "detected_at": "2026-09-08T16:07:00Z",
    "recovered_at": "2026-09-08T19:10:00Z",
    "incident_id": "INC-0420",
    "recovery": "fix-forward"
  },
  "unplanned": false,
  "run_url": "https://ci.example.com/runs/12345"
}

outcome: success and failed: true are both correct: the pipeline succeeded, but the deployment later needed immediate intervention. Change fail rate counts the second; counting only red pipelines under-reports it.

Three labs, from beginner to advanced. Labs 1 and 2 need no cloud account and cost nothing; Lab 3 uses a public GitHub repository.

• Objective: find where a change spends its time, and why, with DORA’s value stream mapping method.^[29]
• Prerequisites: a whiteboard or spreadsheet; 60–90 minutes; people from every part of one service’s stream (business, design, development, testing, operations, support). Cost and risk: none, except politics: present findings as system problems.

Steps

1. Choose one service and one typical change type, for example “a small feature”.
2. List the steps from “work item accepted” to “running in production”, keeping to 5–15 process blocks.
3. For each block, record process time (hands-on work), lead time (elapsed, including waiting) and %C/A from recent real examples; %C/A is recorded by the receiving step: the share of the work it receives that it can use without sending it back.^[29] Use one unit for lead times (here, working days of 8 hours).
4. Compute the totals: lead time = Σ lead times; process time = Σ process times; activity ratio = process time ÷ lead time; rolled %C/A = the product of all the %C/A values.
5. Circle the block with the largest waiting time (lead minus process) and the block with the lowest %C/A, then draw a future-state map for 6 months to 2 years ahead.^[29]

Worked example (illustrative data)

At 8 hours per working day, 18 days are 144 hours, so the activity ratio is 13.75 ÷ 144 ≈ 9.5%: more than 90% of the elapsed time is waiting (in 24-hour days it would be 3.2%, so keep one unit from map to map). Rolled %C/A = 0.80 × 0.90 × 0.85 × 0.70 × 0.95 × 0.90 ≈ 0.366: only about one change in three passes every step without rework.

• Expected output: a map, totals and one or two named constraints: here the CAB wait (5 days for 15 minutes of work) and QA (4 days, with 30% of incoming changes needing rework, so the fix lies upstream in CI tests and review).
• Validation: participants agree the numbers within about 20%; otherwise pull data from tickets and version control. Troubleshooting: an unknown lead time is itself a finding (add timestamps); more than 15 blocks means you are mapping tasks.
• Teardown: photograph and date the map; repeat in about six months.^[29]

• Objective: turn raw deployment records into DORA’s five metrics and understand what each number does not tell you.^[1]
• Prerequisites: Python 3.8 or later, standard library only; 30 minutes; no cost or risk (illustrative data).

Steps

1. Save the illustrative two-week log below as deployments.json. It covers one service, which is the level at which DORA says the metrics belong.^[1]

Example status: JSON syntax-checked; this exact file was used for the executed run below.

{
  "service": "checkout-api",
  "window": {"start": "2026-09-01T00:00:00Z", "end": "2026-09-15T00:00:00Z"},
  "deployments": [
    {"id": "d01", "deployed_at": "2026-09-01T10:00:00Z", "commits": ["2026-08-31T15:00:00Z", "2026-09-01T08:30:00Z"], "failed": false, "unplanned": false},
    {"id": "d02", "deployed_at": "2026-09-02T11:00:00Z", "commits": ["2026-09-01T16:00:00Z"], "failed": true, "recovered_at": "2026-09-02T11:45:00Z", "unplanned": false},
    {"id": "d03", "deployed_at": "2026-09-02T14:00:00Z", "commits": ["2026-09-02T12:30:00Z"], "failed": false, "unplanned": true},
    {"id": "d04", "deployed_at": "2026-09-04T09:30:00Z", "commits": ["2026-09-03T17:00:00Z", "2026-09-04T08:00:00Z"], "failed": false, "unplanned": false},
    {"id": "d05", "deployed_at": "2026-09-07T10:15:00Z", "commits": ["2026-09-04T15:00:00Z", "2026-09-07T09:00:00Z"], "failed": false, "unplanned": false},
    {"id": "d06", "deployed_at": "2026-09-08T16:00:00Z", "commits": ["2026-09-08T10:00:00Z"], "failed": true, "recovered_at": "2026-09-08T19:10:00Z", "unplanned": false},
    {"id": "d07", "deployed_at": "2026-09-08T19:00:00Z", "commits": ["2026-09-08T18:20:00Z"], "failed": false, "unplanned": true},
    {"id": "d08", "deployed_at": "2026-09-10T11:00:00Z", "commits": ["2026-09-09T14:00:00Z", "2026-09-10T09:45:00Z"], "failed": false, "unplanned": false},
    {"id": "d09", "deployed_at": "2026-09-11T15:30:00Z", "commits": ["2026-09-11T10:00:00Z"], "failed": false, "unplanned": false},
    {"id": "d10", "deployed_at": "2026-09-14T10:00:00Z", "commits": ["2026-09-11T16:00:00Z", "2026-09-14T08:30:00Z"], "failed": false, "unplanned": false}
  ]
}

2. Save the script as dora_metrics.py.

Example status: executed on 2026-10-16 with Python 3.8.10; syntax-checked by the validator.

"""Compute DORA's five software delivery metrics for ONE service from a deployment log.

Usage: python3 dora_metrics.py deployments.json
Definitions follow dora.dev/guides/dora-metrics (updated 2026-01-05). Standard library only.
"""
import json
import statistics
import sys
from datetime import datetime


def ts(value):
    """Parse an ISO-8601 UTC timestamp such as 2026-09-01T10:00:00Z."""
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def hours(delta):
    return delta.total_seconds() / 3600


def main(path):
    with open(path, encoding="utf-8") as fh:
        data = json.load(fh)
    deps = sorted(data["deployments"], key=lambda d: d["deployed_at"])
    if not deps:
        sys.exit("no deployments in window")
    start, end = ts(data["window"]["start"]), ts(data["window"]["end"])
    days = (end - start).total_seconds() / 86400

    # Throughput 1: deployment frequency (and the gap between deployments).
    per_day = len(deps) / days
    gaps = [hours(ts(b["deployed_at"]) - ts(a["deployed_at"])) for a, b in zip(deps, deps[1:])]

    # Throughput 2: change lead time = commit -> running in production, per change.
    lead = [hours(ts(d["deployed_at"]) - ts(c)) for d in deps for c in d["commits"]]

    # Instability 1: change fail rate = deployments needing immediate intervention / deployments.
    failed = [d for d in deps if d["failed"]]
    # Throughput 3: failed deployment recovery time, change-caused failures only.
    # The clock starts at deployed_at (this Atlas's convention; starting at detection is the alternative).
    recovery = [hours(ts(d["recovered_at"]) - ts(d["deployed_at"])) for d in failed]
    # Instability 2: deployment rework rate = unplanned deployments caused by a production incident / deployments.
    rework = [d for d in deps if d["unplanned"]]

    print(f"service: {data['service']}  window: {days:.0f} days  deployments: {len(deps)}")
    print(f"deployment frequency: {per_day:.2f}/day ({per_day * 7:.1f}/week); median gap {statistics.median(gaps):.1f} h")
    print(f"change lead time: median {statistics.median(lead):.1f} h; p90 {statistics.quantiles(lead, n=10)[-1]:.1f} h; n={len(lead)} changes")
    print(f"change fail rate: {len(failed)}/{len(deps)} = {len(failed) / len(deps):.0%}")
    if recovery:
        print(f"failed deployment recovery time: median {statistics.median(recovery) * 60:.0f} min (n={len(recovery)})")
    print(f"deployment rework rate: {len(rework)}/{len(deps)} = {len(rework) / len(deps):.0%}")


if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "deployments.json")

3. Run python3 dora_metrics.py deployments.json.

Expected output (copied from the real run):

service: checkout-api  window: 14 days  deployments: 10
deployment frequency: 0.71/day (5.0/week); median gap 29.8 h
change lead time: median 5.5 h; p90 66.5 h; n=15 changes
change fail rate: 2/10 = 20%
failed deployment recovery time: median 118 min (n=2)
deployment rework rate: 2/10 = 20%

Validation: check one number by hand. The lead times sorted are 0.7, 1.25, 1.25, 1.5, 1.5, 1.5, 1.5, 5.5, 6, 16.5, 19, 19, 21, 66 and 67.25 hours; the eighth of 15 values is 5.5 hours, matching the median. Then compare your interpretation with reading the numbers.

Exercises. (a) Mark d06 as "failed": false and rerun: what changes, and what should a reviewer ask before accepting that edit? (b) Move d05 and d10 to Friday evening: lead time falls, but is a Friday-evening deployment a good idea? Troubleshooting: ValueError: Invalid isoformat string means a malformed timestamp; statistics.StatisticsError means an empty list (you need at least two deployments and two changes). Teardown: delete the two files.

• Objective: build the reference implementation and prove that a bad deployment is caught and recorded.
• Prerequisites: a public GitHub repository (on Free, Pro and Team, required reviewers work only in public repositories, as of October 2026);^[55] Docker; any deploy target that your deploy.sh and rollback.sh drive (a free-tier VM, a local Kubernetes cluster). Cost and risk: possibly small cloud costs; use a throwaway project, never a real production system.
• Status: not executed for this page (it needs your account and target); the smoke test was executed locally.

Steps

1. Create a minimal service whose /healthz returns {"status": "ok", "version": "<full-commit-sha>"}, with the 40-character SHA injected at build time.
2. Add delivery.yml, scripts/smoke-test.sh, deploy.sh and rollback.sh (which redeploys the last successful image tag). For a cloud target, create one role per environment trusting GitHub’s OIDC provider for repo:<you>/<repo>:environment:staging or …:production only; no long-lived cloud keys in repository secrets.^[66]
3. Create the staging and production environments, with yourself as a required reviewer on production.
4. Merge a harmless change: test → build → staging → production, all green, one event delivered.
5. Failure drill: make /healthz report a wrong version and merge.

• Expected output: in step 5 the production smoke test fails after its retries, rollback runs, the job is red, the previous version still serves, and the event records "outcome": "failure".
• Validation: mapped to Lab 2’s format (outcome: failure → "failed": true), the Lab 2 script counts one failed deployment. Troubleshooting: docker push denied → check packages: write and a lowercase image name; job waiting forever → a required reviewer has not approved; event post failing → check the DELIVERY_EVENTS_URL secret (curl --fail turns HTTP errors into a failed step).
• Teardown: delete the environments, secrets, images and cloud roles with their OIDC trust, destroy the target, and archive the repository.

Case A is sourced history; B and C are illustrative composites. The regulated change-approval case is the judgement call.

A. Flickr, 2009 (sourced). At the Velocity conference, John Allspaw and Paul Hammond spoke on “10+ Deploys A Day: Devs & Ops Cooperation at Flickr”. The contemporary report quotes the session summary: “on a normal day there are 10 full deployments of the site to our servers”.^[24] It was an early public demonstration that frequent deployment is possible when the two groups share goals and tooling; the report gives no failure or recovery figures, so it is not a benchmark.

This page owns the delivery measurement model for the Atlas; other pages link here rather than redefining these metrics.

DORA (metrics guide updated 2026-01-05) groups its five metrics into throughput (how many changes move through the system) and instability (how well deployments go), and applies them “at the application or service level”.^[1]

Data comes from deployment records, version control (commit times) and incident records linked to deployments. Four notes:

• Recovery time covers only change-caused failures. Before 2023 it was “time to restore service” for any outage; the 2024 model moved it into throughput.^[27] Generic incident MTTR is taught in IncidentOps.
• “Lead time” is ambiguous. DORA’s change lead time starts at commit; Kanban lead time starts at the request (AgileOps). Say which one you mean.
• Rework rate is the newest metric (2024). DORA added it to test whether change fail rate was a good proxy for the rework a team must do.^[27]
• DORA does not fix when the recovery clock starts. This page starts it at deployment, which counts the time a failure went unnoticed; starting at detection isolates response speed. Choose one, write it down, and keep it stable.

---
config:
  flowchart:
    nodeSpacing: 30
    wrappingWidth: 170
---
flowchart TB
  subgraph CAP["Capabilities you can change"]
    C1["Technical: CI, trunk-based, test and deploy automation, loose coupling"]
    C2["Process: small batches, streamlined approval, value stream visibility"]
    C3["Cultural: generative culture, transformational leadership"]
  end
  subgraph PERF["Delivery performance, per service"]
    T["Throughput: lead time, deployment frequency, recovery time"]
    I["Instability: change fail rate, rework rate"]
  end
  subgraph OUT["Outcomes"]
    O1["Organisational performance"]
    O2["Well-being: less burnout, less deployment pain"]
  end
  C1 --> T
  C2 --> T
  C3 --> T
  C1 --> I
  C2 --> I
  C3 --> I
  T --> O1
  I --> O1
  T --> O2
  I --> O2
  O1 -.->|"monthly review"| C1
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class C1,C2,C3 neutral
  class T,I core
  class O1,O2 decision

The Lab 2 output (one service, 14 days, 10 deployments, 15 changes): 0.71 deployments a day; median lead time 5.5 hours, p90 66.5 hours; change fail rate 20% (d02, d06); recovery time 117.5 minutes (shown as 118), the median of 45 (d02, rolled back) and 190 (d06, fixed forward); rework rate 20% (d03 and d07, the unplanned follow-ups).

• Throughput is healthy for a team deploying in working hours. The p90 lead time comes from two commits made on a Friday afternoon and deployed on Monday morning (d05, d10): the weekend is the constraint, not the pipeline, so ask whether that matters before “fixing” it.
• Instability needs attention. One deployment in five fails, and both failures were followed by an unplanned deployment. Rework rate exposes this fix-forward work, which a team that tracks only rollbacks would miss.^[27]
• Small samples are noisy. With 10 deployments, one more failure moves the change fail rate by 10 points; watch trends over months.
• Experiments. Adopt “roll back first, fix forward afterwards”, because d06’s fix forward took four times longer than d02’s rollback, and look at what d02 and d06 had in common (size, test coverage, area of code); see the worked runbook.

These are not DORA’s five. Most come from DORA’s capability guides, as ways to measure the capabilities behind the five.

Start from your own baseline. The metrics guide gives no numeric thresholds, only that top performers do well across all five and low performers do poorly.^[1] Relying on one metric is a listed pitfall: “Teams should identify multiple metrics, including some with a healthy amount of tension between them”.^[1] Set direction, not quotas (lead time down with change fail rate flat, not “deploy 10 times a day”). For an industry comparison, the Quick Check (five multiple-choice questions, under a minute) compares you “to the rest of the industry”.^[45]

“Elite” is a survey cluster, not a standard. The language comes from DORA’s annual reports, not the metrics guide. In 2021 DORA “used four metrics to classify teams as elite, high, medium or low performers”, and high and elite made up two-thirds of respondents.^[67] Clusters are recomputed yearly: results “tend to cluster into four groups”, yet the 2022 data had no “Elite” cluster.^[68] The 2025 report adds “a cluster analysis that reveals seven common team profiles or archetypes” combining performance, stability and well-being, including “Foundational challenges” and “Harmonious high achievers”;^[69] the full report was not opened, so this page does not name the other five, say whether they replace the clusters, or give thresholds. If a tool calls your team “elite”, ask which year, service and window.

Data collection traps. Write down what “requires immediate intervention” means and apply it consistently. Some change failures surface late (leaks, batch jobs, cache expiry): link each change-caused incident to its deployment and mark it failed whenever it was detected, or the change fail rate is undercounted. Squash merges hide commit times, so decide whether lead time starts at the first commit or when the pull request opens. For apps and packaged software, measure the release you control. Below about 20 deployments a period, percentages swing widely; use rolling 90-day windows.

DevOps moves controls into the pipeline, where they run on every change and leave evidence. Security engineering belongs to DevSecOps, audit evidence programmes to ComplianceOps and change governance to ITOps; segregation of duties is under risk controls.

Jurisdiction caveat. No legal advice here. EU financial entities must also comply with the Digital Operational Resilience Act (see what DevOps is not); map your obligations to automated controls with your compliance function.

Cloud unit economics belong to FinOps and the carbon side to GreenOps.

Cost of delay (illustrative). Cost of delay ≈ value per week in production × weeks of waiting. Twelve features a quarter, each worth an estimated 5,000 currency units a week once live and each waiting 3 weeks after “code complete”, leave about 12 × 5,000 × 3 = 180,000 units a quarter unearned. If two engineers for a quarter could cut the wait to 1 week, the recurring saving is about 120,000 units a quarter; compare it with their fully loaded cost, using ranges, as a decision aid rather than a forecast.

Capacity and the J-curve. Improvement needs explicit capacity, and a dip after the first gains is normal. In DORA’s CD J curve, “typical of transformation programs”, quick wins and “an initial improvement” come first; then “automation increases test requirements, which are dealt with manually” and technical debt blocks progress, and getting out takes process redesign, architectural improvement and skills alongside the automation.^[15] For platforms, “initial performance gains” are “followed by a dip as complexity increases”, then recovery.^[19] Agree in advance how long a dip is acceptable and which metrics will show recovery.

How do you know the operating model works and nobody is quietly bypassing it?

• Feedback loops with owners and cadences: pipeline and alerts (minutes), incident reviews (days), the monthly delivery review, a quarterly culture pulse, value stream maps every six months.^[29]
• Exception signals: production versions not from the pipeline; break-glass or admin-bypass merges; a rising share of “emergency” changes, which means the normal path is too slow; toggles past expiry;^[32] checks disabled “temporarily”.
• Metric integrity and audit sampling: each quarter, trace about ten random changes from commit to verification and confirm that failed deployments were counted; each broken link is a finding against the system, not the person.
• Risk indicators: rising CI duration, approval wait, flaky-test rate or pages per person; a falling Westrum score.

This runbook covers change-caused failures; organisation-wide incident command belongs to IncidentOps. The first question is a judgement, not a timer: “symptoms began within 30 minutes of a deployment” is only an example threshold, and slow-burn failures are linked to their deployment in the review (data traps).

flowchart TB
  A["Alert or smoke test fails"] --> B{"Is a recent deployment a plausible cause?"}
  B -->|"no"| X["General incident process (IncidentOps)"]
  B -->|"yes"| C["Declare: deployment ID, service, owner, start time"]
  C --> D{"Is rollback safe? No irreversible migration, previous artifact available"}
  D -->|"yes"| E["Roll back to the previous artifact"]
  D -->|"no"| F["Disable the feature toggle, or fix forward via the normal pipeline"]
  E --> G{"Symptoms resolved?"}
  F --> G
  G -->|"no"| H["Escalate: page secondary, widen investigation"]
  H --> D
  G -->|"yes"| I["Record recovered_at; mark deployment failed"]
  I --> J["Blameless review within 5 working days"]
  J --> K["System action: test, guard, smaller batch or pipeline check"]
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A,H risk
  class B,D,G decision
  class C,E,F,I,J,K core
  class X neutral

Worked example: d06 from Lab 2 (illustrative).

Recovery took 190 minutes instead of perhaps 20 because rollback was unpractised and the smoke test was blind to the critical journey: properties of the system, not failings of a person.

Example status: syntax-checked by bash -n; illustrative commands; deploy.sh and rollback.sh are your own scripts and were not executed here.

# 1. Confirm what is running and what ran before (from your deployment records).
./scripts/deployments.sh list --service checkout-api --env production --last 3
# 2. Roll back to the previous known-good artifact using the SAME deploy path.
./scripts/rollback.sh production
# 3. Verify with the same smoke test the pipeline uses.
./scripts/smoke-test.sh https://checkout.example.com "<previous-version-sha>"
# 4. Record the recovery so the metrics stay honest.
./scripts/deployments.sh mark-failed --id d06 --recovered-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)"

Myths. DevOps trades stability for speed: top performers do well on all five metrics.^[1] Only web start-ups can do it: regulated firms adapt the controls, and DORA names “using industry as a shield against improving” as a pitfall, with regulated industries as its example.^[1] DevOps means microservices and Kubernetes: what matters is independent testing and deployment;^[40] a modular monolith with a fast pipeline can beat a distributed monolith (KubernetesOps covers one runtime option).

Strong fit. The software changes regularly, changes matter to users, more than a handful of people contribute, and you operate it or can get production feedback. Weak fit, or a reduced form:

Hidden responsibilities. On-call for what you own; pipelines and tests maintained as production code; platform funding; time for reviews; the discomfort of seeing your real metrics. Exit condition per investment: stop once it is no longer the constraint (for example, CI already under 10 minutes) and move on.

flowchart TB
  Q0["Do you operate the software and change it more than a few times a year?"]
  Q0 -->|"no"| R0["Reduced form: version control, CI, scripted release. Skip full CD"]
  Q0 -->|"yes"| Q1["Is deployment manual or risky?"]
  Q1 -->|"yes"| R1["Deployment automation and smoke tests"]
  Q1 -->|"no"| Q2["Is CI over 10 minutes, or trunk often red?"]
  Q2 -->|"yes"| R2["CI and test automation; fix flaky tests"]
  Q2 -->|"no"| Q3["Do changes wait days for approval or review?"]
  Q3 -->|"yes"| R3["Streamline change approval; smaller PRs; synchronous review"]
  Q3 -->|"no"| Q4["Must several teams coordinate to deploy one change?"]
  Q4 -->|"yes"| R4["Loose coupling: architecture and team boundaries"]
  Q4 -->|"no"| Q5["Is change fail rate or rework rate rising?"]
  Q5 -->|"yes"| R5["Smaller batches, better tests, observability, rollback drills"]
  Q5 -->|"no"| Q6["Do many teams rebuild the same pipeline work?"]
  Q6 -->|"yes"| R6["Platform as a product: see PlatformOps"]
  Q6 -->|"no"| R7["Map the value stream again; constraint may be upstream, see AgileOps"]
  R0 ~~~ R1 ~~~ R2 ~~~ R3 ~~~ R4 ~~~ R5 ~~~ R6
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q0,Q1,Q2,Q3,Q4,Q5,Q6 decision
  class R1,R2,R3,R4,R5,R6,R7 core
  class R0 neutral

The order of the questions is an editorial heuristic that puts first the constraints blocking everything downstream; your value stream map overrides it.

Assumptions (illustrative): a 150-engineer organisation with about 15 teams, mostly manual deployments, a supportive CTO and no platform team. The weights reflect this context, so change them for yours.

Calculation for A: 0.25×5 + 0.20×4 + 0.20×3 + 0.15×4 + 0.20×5 = 1.25 + 0.80 + 0.60 + 0.60 + 1.00 = 4.25. B: 0.25×2 + 0.20×2 + 0.20×4 + 0.15×2 + 0.20×2 = 0.50 + 0.40 + 0.80 + 0.30 + 0.40 = 2.40. C: 0.25×3 + 0.20×2 + 0.20×4 + 0.15×1 + 0.20×2 = 0.75 + 0.40 + 0.80 + 0.15 + 0.40 = 2.50.

Sensitivity check. With scalability at 40%, time-to-improvement at 15% and learning transfer at 10%, A = 3.85, B = 2.80 and C = 2.80: A still leads. Once practices are proven, B, reframed as a platform product with the pilot teams as first customers, is the next step (PlatformOps); C pays the cost of change everywhere before learning anywhere.

flowchart TB
  L1["Level 1 Siloed: manual, infrequent, hand-offs"] -->|"pipeline exists, trunk protected"| L2["Level 2 Automated islands: CI on trunk, scripted deploys"]
  L2 -->|"daily merges, fast tests, metrics baseline"| L3["Level 3 Integrated: trunk-based, one pipeline to production"]
  L3 -->|"on-demand deploys, risk-tiered approvals"| L4["Level 4 Continuous delivery: small batches, team owns production"]
  L4 -->|"constraint-driven improvement loop, culture measured"| L5["Level 5 Learning system: experiments, platform as product"]
  L5 -.->|"new constraint or reorganisation can move you back"| L3
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class L1 risk
  class L2,L3 neutral
  class L4,L5 core

For one pilot value stream with one team, a part-time coach and a sponsor; regulated environments often need a longer phase 3 to agree evidence with audit.

Dependencies: a sponsor protecting improvement capacity; production telemetry access; a security and compliance contact from day 30, not day 89.

Quick start (any team, this week)

• Everything needed to build and deploy is in version control, including pipeline and monitoring configuration^[36][38]
• Every push triggers a build and fast tests; trunk is protected (review and a green build required)
• The service exposes a health endpoint that reports its version
• Deployments are recorded automatically

Pipeline preflight (before calling a pipeline production-ready)

• Builds once; the same artifact is promoted through all environments^[33]
• Fast feedback in about 10 minutes or less^[42]
• Smoke test checks version and one critical user journey; rollback is automated and rehearsed
• Pipeline credentials are least-privilege and short-lived; pipeline definition changes require review

Monthly delivery review

• Five metrics per service, trend over at least three months
• Every failed deployment reviewed and its system action tracked; unplanned follow-ups marked as rework
• One constraint chosen and one experiment defined, with its expected metric movement
• Exceptions reviewed: break-glass use, bypassed checks, emergency changes, expired toggles

1. One-page DevOps charter for a value stream

Value stream:            <service or product name>
Sponsor:                 <name, role>          Team lead: <name>
Mission (1 sentence):    Deliver small, safe changes to <users> on demand and learn from every change.
Current baseline (date): lead time <median>, deploy frequency <n/period>, change fail rate <%>,
                         recovery time <median>, rework rate <%>
Biggest constraint:      <from the value stream map>
90-day goals:            <direction for 2-3 metrics, e.g. lead time down, instability flat>
Improvement capacity:    <% of team time, agreed with sponsor>
Controls we must keep:   <e.g. segregation of duties, data residency>; evidence: <how automated>
Non-goals:               <e.g. reorganising other teams, tool replacement>
Review cadence:          monthly delivery review on <day>; next value stream map on <date>

2. Architecture decision record: adopting trunk-based development

ADR-0007: Adopt trunk-based development for checkout-api
Status:       Accepted (2026-10-16)
Context:      Median branch age 9 days; 2-day merge integration before each release; code freeze 3 days/month.
Decision:     Branches live < 1 day; <= 3 active branches; merge to main at least daily; release toggles for
              incomplete work; required review + green build to merge.
Evidence:     DORA trunk-based development capability (practices and measures); our value stream map (date).
Consequences: + smaller reviews, no freeze, faster feedback
              - toggle carrying cost; requires CI < ~10 min; requires prompt reviews
Review date:  2027-01-15 (compare branch age, freeze days, change fail rate with baseline)

3. Delivery review scorecard (monthly, per service)

Service: <name>      Month: <YYYY-MM>      Reviewed by: <team, platform rep, manager>
Item                                   Last month   This month   3-month trend   Action
Change lead time (median / p90)
Deployment frequency
Change fail rate
Failed deployment recovery time (median)
Deployment rework rate
CI feedback time (p90)
Approval wait time (median)
Constraint chosen and experiment

4. Team working agreement (delivery)

1. We merge small changes to main at least daily; no branch lives longer than a day.
2. Pull requests get a first review within <agreed window>; we prefer pairing for risky changes.
3. A red main is everyone's top priority: fix or revert within 15 minutes.
4. Every deployment goes through the pipeline. Break-glass use is reviewed the next working day.
5. If a deployment fails, we roll back first unless rollback is unsafe, then learn.
6. Every release toggle has an owner and an expiry date; we remove expired toggles in the next sprint.
7. Incident reviews are blameless and end with system actions, not names.

The deployment event record and the smoke test in the reference implementation are reusable too.

Established (use now). Platform engineering as the scaling mechanism for DevOps, which “often follows” a J-curve too and has possible throughput and stability costs if poorly managed.^[19][59] Socio-technical design: shaping team boundaries deliberately to get the architecture you want (Conway’s law, Team Topologies).^[21][37] Rework as a first-class signal alongside change fail rate.^[27]

Fast-moving (as of October 2026). AI-assisted delivery: re-check DORA’s 2024–2025 research before relying on it.^[59][28][60] Agentic tools that open pull requests and run pipelines raise questions about review load, provenance and permissions: treat the agent as a contributor whose changes go through the same pipeline and controls as anyone else’s. Interoperable delivery events: CDEvents, an incubated CD Foundation project extending CloudEvents, aims to give CI/CD tools a shared event vocabulary, which would ease metrics across mixed toolchains.^[48]

Speculative or marketing. “Autonomous DevOps” or “self-driving pipelines” that need no human judgement, and products promising “DevOps in a box”: no source opened for this page shows either working as described.

Foundational reading: Accelerate (2018)^[5]; The DevOps Handbook, 2nd edition (2021)^[9]; Fowler on continuous integration^[22] and continuous delivery^[30]; DORA’s capability guides.^[17]

Decide which value stream to pilot and who owns it end to end, what share of capacity is protected for improvement, which controls must be kept and how they are evidenced automatically, and whether to fund a platform team (only once one team has proven the practices). Measure per service: change lead time, deployment frequency, failed deployment recovery time (throughput); change fail rate, deployment rework rate (instability); improve them together and never rank teams.^[1] Watch for the DevOps team silo, title rebranding, tool-first programmes, metrics as targets, approvals added after every incident and AI-inflated pull requests. Start with the 90-day roadmap.

Situation (illustrative). An insurer’s policy-administration team sends every change, from typo fixes to premium-calculation logic, to a weekly CAB; the median approval wait is four days. Last year the CAB rejected 3 of about 400 changes, and two serious incidents came from approved changes. The CTO wants speed; the head of risk fears that removing the CAB looks like removing a control.

Option 1: keep the CAB for everything, but meet twice a week. It looks safe, is quick to agree, and avoids a conversation with the auditors.

Option 2: risk-tiered automated change control. Standard changes are promoted automatically with recorded peer review, passing tests and checks, and a machine-generated deployment record. High-risk changes (premium calculation, data migrations, access control) get explicit review by named risk owners, through an approval gate the platform plan actually provides (see the plan caveat). The CAB becomes a forum for cross-team coordination, trends and exceptions.

Assumptions. Reliable automated tests for the standard tier; an audit function willing to evaluate automated evidence; genuine peer review; regulations that specify outcomes (segregation of duties, traceability) rather than a meeting.

Evidence. DORA found that heavyweight approval processes hurt delivery performance, and found no evidence that more formal external review is associated with lower change fail rates; recorded peer review can carry the segregation-of-duties control (risk controls).^[11] Locally, the team’s own data shows a 0.75% rejection rate (3 of about 400) and incidents slipping past the board anyway: the CAB adds wait time without adding much detection.

Why the attractive choice can be wrong. Option 1 halves the wait but keeps the cause: every change still queues for people who lack the context to judge it, batches stay large, and DORA notes that adding process in response to instability tends to raise lead times and batch sizes.^[11] Why the other choice can also be wrong. Option 2 fails if the tests are thin, the peer review is a formality, or audit is informed after the fact: a visible control is replaced by an invisible gap.

Decision process. Pilot Option 2 on one service for a quarter, with audit involved from day 1 and the tiers defined in writing; measure approval wait, automated-promotion share, change fail and rework rates against the baseline; sample ten changes a month for evidence quality; keep the CAB for the pilot’s high-risk tier.

Expected result, with uncertainty. Wait time should fall sharply for standard changes; the effect on change fail rate is uncertain in the first quarter (the J-curve may apply).^[15] If instability rises and evidence quality is poor, stop and fix tests and reviews first. If audit rejects the evidence model, you have learned that cheaply on one service.

After working through this page you should be able to:

1. Explain DevOps as a socio-technical system using the Three Ways and CALMS, and say why a title or tool is not DevOps.
2. Define and compute DORA’s five delivery metrics for one service, and interpret them responsibly.
3. Design a delivery pipeline that builds once, promotes the same artifact, verifies with a smoke test and records deployments.
4. Diagnose a delivery constraint with a value stream map and choose the next improvement.
5. Evaluate team structures and change-governance options against the evidence and the anti-patterns.

Answer these before opening the answers.

DevOps began as a complaint about a wall between two departments and, through more than a decade of DORA research, became one of the most studied ideas in software delivery.^[7] Its lasting insight is simple to state and hard to live by: the people who change a system should feel, quickly and without blame, the consequences of their changes, and the path from idea to user should be short, automated and observable enough that this feedback is cheap.

Teams that buy the tools without the ownership ship the same problems faster, and AI amplifies whichever delivery system it lands in. You will not “finish” DevOps: see your value stream, measure it per service, fix the biggest constraint, and repeat. Organisations that keep doing that tend to deliver faster and more safely.^[1][15]

All sources were opened and checked on 2026-10-16; the evidence ledger (research/devops-ledger.md) maps each claim to its source and lists what could not be fetched automatically.

Further reading (not cited above).

• DORA AI Capabilities Model report page — DORA, updated 2025-11-25.
• Continuous delivery — Jez Humble’s site accompanying the book Continuous Delivery.

Changelog.

• 2026-10-16, r3 follow-up (orchestrator): OIDC subject tied to deployment-branch limits and no administrator bypass; DORA’s flaky-test advice quoted with its stated reason.
• 2026-10-16, r3: regrouped into 14 parts with nested sub-sections (Blueprint v2); cut from 17,818 to 14,876 words by removing restatements, with schema-migration detail handed to DBOps; Jenkins approval cell corrected (submitter); Figure 3 redrawn without floating edge labels; Figure 9’s text equivalent corrected.
• 2026-10-16, r2: ten figures redrawn for a narrow column; the J-curve described accurately for both continuous delivery and platform engineering; rolled %C/A rounding corrected.
• 2026-10-16, r1: GitHub plan limits dated; OIDC deploy credentials; DORA’s %C/A definition; clusters, team profiles, test stages and a schema-change playbook added; quotations made verbatim.
• 2026-10-16: first edition (pilot); Lab 2 and the smoke test executed locally.