--- title: "IncidentOps" slug: incidentops category: reliability term_status: overlapping-synonym evidence_status: verified summary: "IncidentOps is a rare, informal label for incident management: declaring, coordinating, communicating and learning from unplanned service disruptions. This page teaches that established practice, with a focus on human coordination under pressure." audience: [On-call engineers, Incident commanders, Support and communications leads, Engineering managers, Beginners] last_verified: 2026-10-17 key_takeaways: - "\"IncidentOps\" is not a recognised discipline. We found a consultancy article, a Slack app and hobby projects using the word; the practice it names, incident management, is long established." - "Declare early on written criteria. A declared incident that turns out small costs minutes; a late declaration costs hours of uncoordinated work." - "Split the work: the incident commander coordinates and decides but does not fix; a communications lead updates people on a fixed cadence." - "Mitigate before you explain. After a quick size-up (attacker? site lost?), a blunt generic mitigation such as a rollback or a traffic drain usually beats waiting for a precise fix." - "MTTR has at least four meanings, and incident durations are too skewed and variable to show trends, even across a year of incidents. Track phases, distributions and SLO impact instead." - "Follow-up actions need one owner, a due date and tracked closure, but a review that only counts tickets has stopped learning." - "Security incidents, disaster-recovery invocation and SLO policy belong to neighbouring disciplines; design those handoffs before you need them." related: [sre, observabilityops, responseops, chatops, itops, disasterrecoveryops, supportops, aiops] --- ## What IncidentOps is ### In 30 seconds **IncidentOps** is an informal name for **incident management**: how an organisation recognises that a live service has gone wrong, organises people to restore it, keeps everyone affected informed, and learns afterwards. The label adds nothing to a mature practice documented by Google, PagerDuty, Atlassian and ITIL. **In one line: IncidentOps is coordinated decision-making under time pressure, with explicit roles, a communication rhythm and a follow-up loop.** Incidents rarely go badly for lack of skill; they go badly when nobody is in charge, two people change the same system, customers hear nothing, or follow-up actions are never done. Google names three hazards of an unmanaged incident: a sharp focus on the technical problem, poor communication, and "freelancing" (changing production without coordination).[^sre-managing] ### At a glance | Question | Answer | |---|---| | Goal | Shorter, smaller customer impact; coordinated response; timely communication; learning that changes the system | | Users | On-call engineers, incident commanders (ICs), communications and support leads, service owners | | Scope | Declaration, severity, on-call, roles, communication, status pages, mitigation, reviews, follow-up actions, incident metrics | | Out of scope | Attacker-driven incidents ([ResponseOps](../responseops/)), SLOs ([SRE](../sre/)), telemetry and alert design ([ObservabilityOps](../observabilityops/)), invoking DR ([DisasterRecoveryOps](../disasterrecoveryops/)) | | Prerequisites | A service with an owner, monitoring that can page someone, a shared chat or call tool | | Key outputs | Severity policy, rotas and escalation, role cards, comms templates, incident records, reviews, tracked actions | | Term status | **Overlapping synonym**: a rare label for an established practice, taught here under its standard name | ### The problem it solves Without a process, the first engineer to notice becomes investigator, fixer, spokesperson and historian at once; helpers change things in parallel; customers learn from social media; and nobody can reconstruct the timeline. In Grafana Labs' 2026 survey (1,363 self-selected respondents; vendor-run), 30 % named alert fatigue as their biggest obstacle to faster incident response and 16 % named painful coordination across teams.[^grafana-survey] Incident management replaces improvisation with prepared decisions: what counts as an incident, who leads, who may change production, where people gather, how often updates go out, and what happens afterwards. Google's SRE book claims a written playbook "produces roughly a 3x improvement in MTTR as compared to the strategy of 'winging it'" (MTTR: mean time to repair),[^sre-intro] without publishing a method ([Metrics and KPIs](#metrics-and-kpis)). | Before | After | |---|---| | The incident starts when someone feels it is serious | It starts when written criteria are met; anyone may declare | | The most senior engineer fixes and directs at once | An IC directs, operators fix; one channel, one record | | Updates when someone remembers | First public post within minutes, then on a fixed cadence | ### What it is not: boundaries and naming - **Not a separate discipline.** No standards body, book or foundation defines "IncidentOps"; our searches on 2026-10-16 found one consultancy article, a Slack app sold as Incident/Ops[^incidentopsio] and unrelated hobby repositories. Opsio presents IncidentOps as a framework it "developed" that "emphasizes proactive prevention alongside reactive response", extended to workplace safety.[^opsio] Neither shows a wider movement. - **Not security incident response.** An adversary brings evidence, legal duties and containment: that is [ResponseOps](../responseops/), which teaches NIST SP 800-61 Rev. 3.[^nist61] This page covers the shared coordination and handoff. - **Not problem management or SRE.** Removing causes over weeks is problem management ([ITOps](../itops/)); reliability targets are [SRE](../sre/). - **Not every alert.** Google's incidents are escalated and need an immediate, organised response; a planned, announced outage is not one.[^anatomy] ::: note Editorial distinction IncidentOps (live handling, review mechanics), [SRE](../sre/) (targets, postmortem culture), [ITOps](../itops/) (ITIL practices) and [ResponseOps](../responseops/) (adversaries) is this Atlas's division of labour; no standard draws these lines, and often one team does all four. ::: ## Where it comes from and where it sits ### Origins: from fireground command to software incidents Google's incident management "is based on the Incident Command System" (ICS), which US emergency services use for large multi-agency responses.[^sre-managing] Sources date ICS differently: the SRE Workbook says it "was established in 1968 by firefighters",[^wb-ir] while FEMA's NIMS doctrine places the work in the 1970s, in California's FIRESCOPE programme.[^nims] Software borrowed the portable parts: one commander, a few named roles, a manageable span of control, common vocabulary and explicit handovers. Researchers of complex failures had challenged "root cause" thinking by 2000;[^cook] Google published its model in 2016 and 2018, and PagerDuty opened its process (Apache License 2.0).[^pd-about] ### Glossary and mental model | Term | Meaning on this page | |---|---| | Incident | An unplanned disruption that needs an immediate, organised response[^anatomy] | | Major incident | PagerDuty: "Any incident that requires a coordinated response between multiple teams";[^pd-what] all its SEV-1 and SEV-2 count, and lower severities can[^pd-sev] | | Severity | Size of the impact, used to choose the response; not blame | | Incident commander (IC) | Holds the picture, assigns work, decides; does not fix | | Ops lead / SMEs | Change the system; "the only group modifying the system during an incident"[^sre-managing] | | Comms lead | Owns internal and external updates on a cadence | | Mitigate / resolve | Mitigate = impact stops; resolve = incident closed | | Generic mitigation | A prepared blunt action, such as rollback or drain, used before the cause is known[^wb-ir] | | MTTx, MTTR | Mean time to *x*: detect, acknowledge, mitigate, resolve. MTTR's "R" varies, so always say which *x* | | IMAG | Incident Management At Google, Google's ICS-based protocol[^wb-ir] | | SLI / SLO | A measured signal of user experience / its target, taught on [SRE](../sre/) | | Routing key / dedup key | Integration key routing events to one paging service / key tying a trigger to its resolve[^pd-events] | **Mental model: an incident is a temporary organisation**, with a leader, a structure, a meeting place, a record and a communication plan, dissolved deliberately. As STELLA observed, "during high tempo times the usually trivial costs of coordination become significant";[^stella] good incident management keeps those costs low so responders can think about the system. ### How IncidentOps relates to other Ops ::: figure Where IncidentOps sits among its neighbours ```mermaid %%{init: {"flowchart": {"wrappingWidth": 156, "nodeSpacing": 16, "rankSpacing": 30, "padding": 8}}}%% flowchart TB IN["Signals in: alerts (ObservabilityOps), correlated events (AIOps), customer reports (SupportOps)"] --> INC CHT["ChatOps: channels"] --- INC INC["IncidentOps: command, comms, follow-up"] INC --> SRE["SRE: SLOs, culture"] INC --> ITO["ITOps: problem, change"] SRE ~~~ RSP ITO ~~~ DRO INC -.-> RSP["ResponseOps: if attacker"] INC -.-> DRO["DisasterRecoveryOps: if site lost"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class INC core class IN,SRE,ITO,CHT neutral class RSP,DRO risk ``` Text equivalent: alerts (ObservabilityOps), correlated events (AIOps) and customer reports (SupportOps) flow into IncidentOps, which commands the response and passes outputs to SRE and ITOps; ChatOps provides the channels. Dashed handoffs go to ResponseOps for an attacker and, for a lost site or region, to DisasterRecoveryOps, whose invocation authority decides on DR. ::: | Neighbour | Shared ground | IncidentOps focus | Their focus | |---|---|---|---| | [SRE](../sre/) | Postmortems, on-call | Response, review mechanics, action tracking | SLOs, error budgets, blameless culture | | [ObservabilityOps](../observabilityops/) | Paging alerts | Who is paged, escalation, declaration | Telemetry and alert design | | [ResponseOps](../responseops/) | Command and comms roles | Any operational incident | Attackers, evidence, notification duties | | [ChatOps](../chatops/) | Incident channels | Roles and rules in the channel | Bots, commands, permissions | | [ITOps](../itops/) | ITIL incident practice | Live technique | ITIL processes, problem and change control | | [DisasterRecoveryOps](../disasterrecoveryops/) | Major outages | Spotting DR criteria, escalating | RTO/RPO, invocation, failover | | [SupportOps](../supportops/) | Customers in outages | Status page, customer liaison | Tickets, SLAs, queues | ## Principles and operating model ### A proposed charter and principles No official IncidentOps manifesto exists; this charter is **this Atlas's proposal**, distilled from the practices cited here. **Mission:** when a live service fails, restore it quickly and safely, keep everyone affected accurately informed, and change the system so the failure is less likely or less harmful next time. 1. **Declare early, on written criteria.** "It is better to declare an incident early and then find a simple fix and close out the incident than to have to spin up the incident management framework hours into a burgeoning problem."[^sre-managing] 2. **One commander, many hands.** The IC decides and coordinates; the IC is "NOT a resolver".[^pd-roles] 3. **Mitigate before you explain.**[^wb-ir] 4. **Communicate on a clock.** Silence reads as inaction.[^wb-ir] 5. **Keep one record:** one channel, one live document, one timeline. 6. **Hand over explicitly**, with spoken confirmation.[^sre-managing] 7. **Learn, then act:** contributing factors without blame; owners and deadlines for the few actions that matter. 8. **Keep on-call sustainable:** load, pay and rest are part of reliability.[^sre-oncall] **Non-goals:** blame; root cause during the incident; replacing problem management or security response. ### Fundamentals: the building blocks | Building block | Minimal version | Mature version | |---|---|---| | Severity policy | Three levels with examples | Metric-based criteria per service, reviewed yearly | | On-call and escalation | One rota; escalate after 5–10 minutes | Primary, secondary and IC rotas as code; follow-the-sun | | Declaration and meeting place | A chat command; one channel and call | Automatic channel, record and roles; working and updates channels | | Communication plan | Status page and a template | Pre-approved templates; contact lists outside production | | Mitigation catalogue | "Roll back the last deploy" | Tested rollback, drain, feature-kill and failover per service | | Review and actions | Template and tracker label | Deadlines by action priority; closure reporting | ### Roles, ownership and RACI Roles are hats, not job titles: the IC "assumes all roles that have not been delegated yet".[^wb-ir] PagerDuty adds a **deputy** (a "hot standby" IC), a **scribe**, subject-matter experts who report in CAN form (Condition, Actions, Needs), and **customer and internal liaisons**.[^pd-roles] | Activity | IC | On-call, ops lead, SMEs | Comms lead | Scribe | Service owner | |---|---|---|---|---|---| | Declare and set severity | A (once named) | R (anyone may declare) | I | I | C | | Change production | A | R | I | I | C | | Status page and customer messages | A (approves) | C | R | I | C | | Timeline and decision log | A | C | I | R | I | | Security stream to ResponseOps (Atlas proposal) | A, R | C | C | I | C | | Escalate to the DR invocation authority | R (escalates) | C | C | I | C | | Complete follow-up actions | I | R | I | I | A | A = accountable, R = responsible, C = consulted, I = informed. The DR invocation authority, outside this table, is accountable for invoking DR ([DisasterRecoveryOps](../disasterrecoveryops/)). Follow-up moves to the service owner because incident roles end. **Span of control.** FEMA's doctrine gives "one supervisor to five subordinates" as the optimal span, as a guideline.[^nims] Beyond about five direct reports, form sub-teams with their own leads; PagerDuty pre-names them Alpha, Bravo and Charlie.[^pd-ic] ::: figure How the role structure grows with the incident ```mermaid %%{init: {"flowchart": {"wrappingWidth": 76, "nodeSpacing": 8, "rankSpacing": 34, "padding": 5}, "themeVariables": {"fontSize": "14px"}}}%% flowchart TB IC["Incident commander"] --- DEP["Deputy"] IC --> OPS["Ops lead"] IC --> CL["Comms lead"] IC --> SCR["Scribe"] OPS --> A["Alpha team: lead, SMEs"] OPS --> B["Bravo team: lead, SMEs"] CL --> CUS["Customer liaison"] CL --> INT["Internal liaison"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class IC core class DEP,OPS,CL,SCR,A,B,CUS,INT neutral ``` Text equivalent: the incident commander sits at the top with a deputy as hot standby; the ops lead, comms lead and scribe report to the IC. As the incident grows, the ops lead splits work into sub-teams (Alpha, Bravo), each with a lead and its own experts, so nobody supervises more than about five people; the comms lead adds customer and internal liaisons. In a small incident one person holds several boxes. ::: ### Lifecycle of an incident Google describes a continuous lifecycle of preparedness, response, and mitigation and recovery, with the postmortem inside recovery.[^anatomy] One incident, with the exception paths that most often go wrong: ::: figure Incident lifecycle with exception paths ```mermaid %%{init: {"flowchart": {"wrappingWidth": 100, "nodeSpacing": 12, "rankSpacing": 28, "padding": 8}}}%% flowchart TB P["Prepare: rotas, policies, drills"] --> D["Detect: alert or report"] D --> T{{"Incident criteria met?"}} T -- "Yes or unsure" --> DC["Declare: severity, IC, channel"] T -- "No" --> Q["Ticket or monitor"] DC --> M["Mobilise roles; first update"] M --> A{{"Size up: attacker?"}} A -- "No" --> L{{"Site or region lost?"}} A -- "Yes" --> RS["Security stream to ResponseOps (Atlas proposal)"] L -- "No" --> MI["Mitigate: stop the impact"] L -- "Yes" --> DR["Escalate to DR authority"] RS -->|"evidence rules"| MI DR -->|"failover if invoked"| MI MI --> V{{"Impact stopped?"}} V -- "Yes" --> R["Resolve and close"] V -- "No" --> A R --> RV["Review: timeline, factors"] RV --> FA["Follow-up actions tracked"] FA --> P classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class P,D,DC,M,MI,R,RV,FA core class T,A,L,V decision class RS,DR,Q risk ``` Text equivalent: an alert or report is detected. If the criteria are met, or the person is unsure, the incident is declared with a severity, an IC and a channel; otherwise it becomes a ticket. The team mobilises, posts a first update and sizes up before choosing a mitigation. A suspected attacker sends the security workstream to ResponseOps (Atlas proposal), whose evidence rules constrain the mitigation; a lost site or region is escalated to the DR invocation authority, and failover is the mitigation if DR is invoked. If the impact has not stopped, the loop returns to size-up, where a security cause can still appear; otherwise the incident is resolved and reviewed, and tracked actions feed preparation. ::: ### Operating model and governance | Cadence | Forum | Outputs | |---|---|---| | Every shift | On-call handover | Written handover note[^wb-oncall] | | Weekly | Review of all incidents and near misses | Severity corrections, review owners, alert fixes | | Within days | Post-incident review | Agreed factors and actions (playbook below) | | Monthly | Action review with service owners | Escalation of overdue actions | | Quarterly | On-call health and process review | Policy, staffing, training | Governance is a few written policies: severity, on-call, communication (who approves public statements), reviews and an action SLO. Atlassian gives "priority actions" an SLO "which is either 4 or 8 weeks, depending on the service", and engineering managers review the open list.[^atl-pm] ## Architecture and workflow ### Reference architecture The architecture is mostly organisational, but its tools form a chain that must work when production does not. Two rules matter most: the **paging path and the status page must not depend on the systems they report on**, and the **incident record is written once and reused** for updates, metrics and the review. ::: figure Reference architecture: from signal to learning ```mermaid %%{init: {"flowchart": {"wrappingWidth": 100, "nodeSpacing": 12, "rankSpacing": 30, "padding": 8}}}%% flowchart TB SIG["Alerts"] --> RT["Alert routing and grouping"] RT --> PG["Paging: schedules and escalation"] HUM["People and tickets: manual page or chat command"] --> CO PG --> CO["Coordination: channel, call, roles"] CO --> REC["Incident record and timeline"] CO --> COM["Comms: status page, internal updates"] REC --> REV["Review and action tracker"] REV --> ENG["Engineering backlog and alert fixes"] ENG --> SIG HB["Paging-path heartbeat"] -.-> PG IND["Independent hosting"] -.-> COM classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class SIG,HUM,RT,PG,CO,REC,COM,REV,ENG core class HB,IND risk ``` Text equivalent: alerts pass through routing to paging, which reaches a human through schedules and escalation; support tickets and people reach coordination directly, by a manual page or chat command. Coordination in a channel and call produces the incident record and the communications. The record feeds the review and action tracker, whose actions improve the alerts. Safeguards: a heartbeat on the paging path and independent hosting for communication tools. ::: The Amazon S3 case in [Use cases and scenarios](#use-cases-and-scenarios) shows what fate-sharing costs. ### End-to-end workflow **Happy path** (illustrative timings for a SEV-2): 1. **T+0 detect.** An SLO-based alert pages the primary; PagerDuty escalates after 5 minutes without acknowledgement.[^pd-oncall] 2. **T+3 triage** against Google's criteria: second team needed, visible to customers, unsolved after an hour's concentrated analysis?[^sre-managing] If in doubt, declare. 3. **T+5 declare.** A chat command opens a channel and a live document, sets severity and names the IC. 4. **T+8 mobilise.** Ops and comms assigned; first status post ([cadence](#core-practices-and-playbooks)).[^pd-ext] 5. **T+10 onwards: the command loop** (size up, stabilise, update, verify);[^pd-ic] updates on the cadence. 6. **T+35 mitigate**, for example by rollback. 7. **T+60 resolve.** Recovery and any data loss confirmed, final update posted, review owner named.[^pd-ext][^pd-pm] 8. **Within days: review**, with actions in the owning teams' backlogs. **Exception paths:** - **Severity changes.** "If you are unsure which level an incident is …, treat it as the higher one"; argue later, not on the call.[^pd-sev] - **Long incidents.** Hand over command at shift boundaries with Google's "You're now the incident commander, okay?" and wait for acknowledgement.[^sre-managing] - **A security cause appears**, at size-up or later. *Atlas proposal:* the IC keeps overall coordination, moves security discussion to a restricted channel, pages the security on-call and hands the security workstream to the [ResponseOps](../responseops/) lead; evidence preservation then constrains restores, and legal review gates statements. PagerDuty also keeps an IC for security incidents and says to trigger the process "anyway" when unsure.[^pd-security] - **A site or region is lost.** The IC checks the activation criteria in the DR policy and escalates, with a recommendation, to the invocation authority it names ([DisasterRecoveryOps](../disasterrecoveryops/)); NIST gives "the ultimate authority to activate the plan" to "A senior management official, such as the CIO".[^nist34] The incident stays the coordination and communication layer. - **A supplier or cloud provider is the cause.** Your customers' impact is still your incident: declare on your own criteria, without waiting for the provider. Watch its account-specific feed (AWS Health names your affected resources; Google Cloud calls Personalized Service Health its primary channel for incidents relevant to your projects)[^aws-health][^gcp-psh] and open a support case; AWS advises the highest severity your plan allows only for cases that "can't be worked around or that directly affect production applications".[^aws-support] Mitigate what you control (fail over, degrade the dependent feature, queue work) and keep your own updates on cadence. Provider pages can lag: Statuspage, which can mirror a vendor's status as a "third-party component", lets you override it "if the external service has not yet updated their status page during an outage".[^sp-third-party] Record the supplier's case number and times for the review. - **False alarm.** Close it with a note; that is the process working. ## Practices, frameworks and tools ### Core practices and playbooks | Practice | Trigger | Owner | Output | Verification | |---|---|---|---|---| | Declare on criteria | Criteria met, or unsure | Anyone; IC confirms | Channel, record, severity, IC | Time to declare | | Command loop | Every few minutes | IC | Decisions, time-boxed tasks | Decision log | | Mitigation first | Fault location plausible | Ops lead | Rollback, drain, feature kill, failover | Impact graph recovers | | Comms cadence | Declaration, then a timer | Comms lead | Status posts, internal summaries | Gaps between updates | | Crowd control | Over about five active responders | IC, deputy | Sub-teams, updates channel | Direct reports per lead | **The command loop.** PagerDuty's IC training runs command as a loop: *size up*, *stabilise*, *update* and *verify*, which asks "Have you finished?", switches to a backup plan if an action failed and, "If problems persist, begin again from the size-up step".[^pd-ic] Its phrases remove ambiguity: "We're proceeding with…"; "Are there any strong objections?" rather than "Does everyone agree?"; "A, please perform B, I'll come back to you in X minutes. Understood?"; and "Clear is better than concise". **Mitigation first.** The Workbook says "first responders must prioritize mitigation above all else", and that "you only need to know the location of the root cause".[^wb-ir] Keep a catalogue of **generic mitigations** per service (rollback, drain, feature-flag kill, failover, load shedding), each with a tested procedure, an owner and known side effects ([Templates](#templates)): they are "blunt instruments".[^wb-ir] **Crowd control.** Laura Maguire's field research found incident channels that "can host several hundred participants": useful for anticipating who is needed, but "a distraction" that can force "the core team into private channels".[^maguire] PagerDuty lets anyone observe the call or chat, asks non-experts to route discussion through their service's primary SME, and keeps a channel "For executive summary updates only".[^pd-during] This page proposes generalising it: a **working channel** for responders and an **updates channel** where the comms lead posts and everyone else reads. An executive who starts directing gets one question: "Do you wish to take command?"[^pd-ic] **Communication cadence.** Agree audiences and timings in advance. PagerDuty's guideline, a starting point: a first post within 5 minutes of the call starting, a scoping update (impact, components, regions) about 5 minutes later, updates at least every 20 minutes for the first 2 hours, and a resolution post stating any data loss.[^pd-ext] Statuspage's states follow the same arc: *Investigating*, *Identified*, *Monitoring*, *Resolved*.[^sp-incident] PagerDuty advises against "outage" unless it was a full outage.[^pd-pm] Prepare two or three templates so nobody drafts under stress.[^wb-ir] **Running the post-incident review**, after PagerDuty's process and Etsy's facilitation guide: 1. **Owner.** The IC names a review owner as the call ends; the owner books the meeting first (PagerDuty: within 3 calendar days for SEV-1, 5 business days for SEV-2) and invites everyone in the timeline.[^pd-pm][^etsy-debrief] 2. **Prepare.** Build the timeline from the record, each entry linked to its graph or message;[^pd-pm] talk first to the people closest to the event, especially those who may expect blame.[^etsy-debrief] 3. **Facilitator.** Etsy: the facilitator "should not be someone who was directly involved in the event"; PagerDuty has the IC run the meeting.[^etsy-debrief][^pd-pm] This page prefers a neutral facilitator and a note-taker for SEV-1. 4. **Status:** Draft, In Review (circulated a day before the meeting), Reviewed, Closed.[^pd-pm] 5. **Close out.** A few actions, each with one owner and a due date in the owning team's tracker; an internal summary; an external message, reviewed in the meeting, if customers were affected.[^pd-pm] ### Methods, standards and frameworks | Source | Kind | Take from it | Caveat | |---|---|---|---| | ICS / NIMS (FEMA)[^nims] | Emergency-services doctrine | Single command, span of control, handovers | Built for physical incidents; adapt | | Google IMAG[^sre-managing][^wb-ir] | Published company practice | 3Cs (coordinate, communicate, control), roles, declaration criteria | Google's scale and culture | | PagerDuty response docs[^pd-about] | Open documentation, Apache 2.0 | Roles, severity table, IC phrases, comms cadence, reviews | One company's process | | Atlassian handbook[^atl-pm] | Vendor practice | Priority-action SLOs | Teaches "Five Whys" root cause, which researchers dispute | | ITIL 4 incident management | Licensed framework | Definitions, ITSM integration ([ITOps](../itops/)) | Broad definition of incident[^anatomy] | | NIS2, EU DORA[^nis2][^eu-dora] | **Law** for in-scope EU entities | Reporting clocks | Mandatory only within scope | Everything except the law is optional; pick one vocabulary and stick to it. ### Tooling landscape (as of October 2026) | Job | Examples opened on 2026-10-16 | Choose for | Watch out for | |---|---|---|---| | Alert routing | Prometheus Alertmanager (v0.34.1)[^am-config] | Grouping, inhibition, routing as code | Alert *design* is [ObservabilityOps](../observabilityops/) | | Paging and escalation | PagerDuty;[^pd-events] Jira Service Management, absorbing Opsgenie;[^opsgenie] Grafana Cloud IRM[^grafana-oncall] | Multi-channel delivery, escalation as code, independence from your stack | Opsgenie shuts down on 5 April 2027, deleting unmigrated data;[^opsgenie] Grafana OnCall OSS lost its cloud services on 24 March 2026 and is archived[^grafana-oncall] | | Coordination in chat | incident.io,[^iio-ai] PagerDuty, small apps such as Incident/Ops[^incidentopsio] | Channel, roles, timeline capture, record export | Lock-in of incident history | | Status page | Atlassian Statuspage[^sp-incident] | Independent hosting, subscriber notices, templates | Shared fate with production[^aws-s3] | | Review and actions | Your issue tracker | Links from actions to incidents; age reports | Tickets nobody follows up[^pd-pm] | Also check that a tool works when your cloud does not, exports its configuration and records (both exits above force migrations), and limits who reads channels holding customer data. ### Automation and AI Automate the mechanical work: opening channel and document, pulling recent deployments into the record, timing updates, drafting the timeline. The bigger prize is automating *mitigation*: Google found that "once a human is involved, the outage will last at least 20 to 30 minutes".[^anatomy] Self-healing belongs to [AutomationOps](../automationops/) and [SRE](../sre/). incident.io markets an "AI SRE" that "starts debugging the moment you declare an incident",[^iio-ai] and PagerDuty markets agents "ready to autonomously resolve well-known issues";[^pd-ai] these are vendor claims as of October 2026, not independent evaluations. STELLA's partners found their experiments with automation that works through chat "mixed": it is easier to imagine than to build as a genuine "team player".[^stella] Treat AI output as an SME's hypothesis, with a time-box and a check, and check what a model retains, because channels contain customer data. ML evaluation is taught in [AIOps](../aiops/). ## IncidentOps in practice ### Reference implementation *Illustrative design:* about 40 engineers in seven teams across two time zones, with Prometheus, PagerDuty, chat and a hosted status page. Four artefacts live in code. _Example status: YAML parsed by the Atlas validator; illustrative policy values, not executed._ ```yaml # severity-policy.yaml: the contract between on-call, ICs, comms and support # (values are illustrative) version: 3 declare_if_any: - "another team is needed to fix it" - "customers can see it" - "unsolved after 60 minutes of focused work" - "data loss or corruption is suspected" - "a security cause is suspected (also page the security on-call)" when_unsure: "use the higher severity; correct it in the review" # Severity sets the response; action priority sets the deadline. Atlassian sets 4 or 8 # weeks per service; this policy maps the two values to action priority instead. action_due_weeks: {P1: 4, P2: 8} # P1 = a "priority action" in Atlassian's sense levels: SEV1: criteria: "login, checkout or data integrity broken for >25% of users, or data loss" roles: [ic, deputy, ops_lead, comms_lead, scribe, customer_liaison] first_public_update_minutes: 5 update_every_minutes: 20 executive_summary_every_minutes: 60 review_meeting_within: "3 calendar days" SEV2: criteria: "a core feature degraded for many users, or a SEV1 is likely without action" roles: [ic, ops_lead, comms_lead, scribe] first_public_update_minutes: 10 update_every_minutes: 30 review_meeting_within: "5 business days" SEV3: criteria: "partial loss for a minority of users, or lost redundancy" roles: [on_call] # one person holds every role; upgrade if it spreads public_update: "only if customers report it" review: "optional; weekly incident review" SEV4: criteria: "no customer impact; needs action this week" response: "ticket" ``` _Example status: YAML parsed by the Atlas validator; field names checked against the Alertmanager configuration reference (2026-10-16); not loaded into Alertmanager or checked with `amtool`._ ```yaml # alertmanager.yml — who gets paged. Alerts labelled severity="page" page a team; the rest # become tickets (the Atlas label convention, shared with the AIOps and ChatOps pages). route: receiver: ticket-queue group_by: [alertname, service] routes: - receiver: heartbeat # Watchdog pattern; rule and design: ObservabilityOps matchers: ['alertname="Watchdog"'] group_interval: 1m # repeat_interval is rounded up to a multiple of this repeat_interval: 1m - receiver: pagerduty-checkout matchers: ['severity="page"', 'team="checkout"'] - receiver: pagerduty-platform matchers: ['severity="page"'] # Grouping and inhibition rules are taught on the AIOps page. receivers: - name: ticket-queue webhook_configs: - url: https://tickets.example.com/hooks/alertmanager - name: heartbeat webhook_configs: - url: https://heartbeat.example.com/ping/CHECK_ID - name: pagerduty-checkout pagerduty_configs: # key file: mode 0400, owned by the alertmanager user - routing_key_file: /etc/alertmanager/secrets/pd-checkout severity: critical # PagerDuty's event severity, not the alert label - name: pagerduty-platform pagerduty_configs: - routing_key_file: /etc/alertmanager/secrets/pd-platform severity: critical ``` The heartbeat route sends the "always firing" Watchdog alert to a service that notifies you "when this alert is not firing";[^watchdog] the rule belongs to [ObservabilityOps](../observabilityops/), grouping and inhibition to [AIOps](../aiops/). This file owns the receivers; keys stay out of the repository in files (`routing_key_file`) that only the Alertmanager user can read.[^am-config] Test the routing tree in CI ([runbook](#worked-runbook-pages-are-not-reaching-on-call)). _Example status: argument names checked against the PagerDuty Terraform provider documentation (v3.36.0, 2026-08-25); not planned or applied. The provider accepts an API token (`PAGERDUTY_TOKEN`), a user-level token (`PAGERDUTY_USER_TOKEN`) or an App OAuth scoped token;[^tf-pagerduty] use the one with the least access your account offers, in a test account first._ ```hcl terraform { required_providers { pagerduty = { source = "PagerDuty/pagerduty", version = "~> 3.36" } } } variable "rota" { description = "E-mail addresses of the checkout on-call rota, in rotation order" type = list(string) } variable "layer_start" { description = "When the rota starts (RFC 3339). For a drill, a time a few minutes ago" type = string default = "2026-11-02T10:00:00+01:00" } locals { # Next week's primary is this week's secondary, so the secondary already has context. secondary_order = concat(slice(var.rota, 1, length(var.rota)), [var.rota[0]]) } data "pagerduty_user" "rota" { for_each = toset(var.rota) email = each.value } data "pagerduty_user" "manager" { email = "checkout-manager@example.com" } resource "pagerduty_schedule" "primary" { name = "checkout-primary" time_zone = "Europe/Berlin" layer { name = "weekly" start = var.layer_start rotation_virtual_start = var.layer_start rotation_turn_length_seconds = 604800 users = [for e in var.rota : data.pagerduty_user.rota[e].id] } } resource "pagerduty_schedule" "secondary" { name = "checkout-secondary" time_zone = "Europe/Berlin" layer { name = "weekly" start = var.layer_start rotation_virtual_start = var.layer_start rotation_turn_length_seconds = 604800 users = [ for e in local.secondary_order : data.pagerduty_user.rota[e].id ] } } resource "pagerduty_escalation_policy" "checkout" { name = "checkout" num_loops = 2 rule { escalation_delay_in_minutes = 5 target { type = "schedule_reference" id = pagerduty_schedule.primary.id } } rule { escalation_delay_in_minutes = 10 target { type = "schedule_reference" id = pagerduty_schedule.secondary.id } } rule { escalation_delay_in_minutes = 15 target { type = "user_reference" id = data.pagerduty_user.manager.id } } } resource "pagerduty_service" "checkout" { name = "checkout-api" escalation_policy = pagerduty_escalation_policy.checkout.id acknowledgement_timeout = 1800 # an acknowledged incident re-triggers after 30 minutes auto_resolve_timeout = "null" # never auto-resolve: a human closes incidents } resource "pagerduty_service_integration" "alertmanager" { name = "alertmanager" service = pagerduty_service.checkout.id type = "events_api_v2_inbound_integration" } output "checkout_routing_key" { value = pagerduty_service_integration.alertmanager.integration_key sensitive = true } ``` The 5-minute first step follows PagerDuty's recommended escalation timeout.[^pd-oncall] `acknowledgement_timeout` re-pages a forgotten acknowledged incident; `"null"` disables auto-resolution.[^tf-pagerduty] Layers begin at `layer_start` (past, for a drill). Anyone holding the routing key can open incidents on the service, and `sensitive = true` only hides it from plan output: Terraform stores it in state and `terraform output -raw` prints it in plain text,[^tf-sensitive] so keep state in an encrypted remote backend few people can read. _Example status: JSON parsed by the Atlas validator; illustrative record matching INC-107 in Lab 1._ ```json { "id": "INC-107", "title": "Payments API errors in eu-west", "severity": "SEV1", "severity_history": [ {"at": "2026-08-09T01:45:00Z", "severity": "SEV2"}, {"at": "2026-08-09T02:05:00Z", "severity": "SEV1"} ], "roles": {"ic": "user-104", "ops_lead": "user-221", "comms_lead": "user-318", "scribe": "user-140"}, "timestamps": { "impact_start": "2026-08-09T01:12:00Z", "detected": "2026-08-09T01:40:00Z", "acknowledged": "2026-08-09T01:43:00Z", "declared": "2026-08-09T01:45:00Z", "first_public_update": "2026-08-09T01:52:00Z", "mitigated": "2026-08-09T10:05:00Z", "resolved": "2026-08-09T13:30:00Z" }, "impact": {"services": ["payments-api"], "regions": ["eu-west"], "data_loss": false}, "security_suspected": false, "dr_invoked": false, "review": {"owner": "user-221", "meeting_by": "2026-08-12", "status": "In Review"}, "actions": [ {"id": "ACT-311", "kind": "prevent", "owner": "team-payments", "priority": "P1", "due": "2026-09-06"}, {"id": "ACT-312", "kind": "detect", "owner": "team-platform", "priority": "P2", "due": "2026-10-04"} ] } ``` Timestamps kept as data feed metrics, the review timeline and regulatory clocks; the action due dates follow the policy's `action_due_weeks` (P1 four weeks, P2 eight weeks after the incident). ### Hands-on labs #### Lab 1 (beginner): measure incidents honestly - **Objective:** compute incident timings, see one long incident distort the mean, and simulate why MTTR trends mislead. - **Prerequisites:** Python 3.8 or later, standard library only; no account, cost or risk. - **Data:** invented (UTC). INC-110 has a deliberate error: detected before impact started. ```text id,sev,impact_start,detected,acknowledged,mitigated,resolved INC-101,SEV3,2026-07-02T09:10,2026-07-02T09:14,2026-07-02T09:16,2026-07-02T09:41,2026-07-02T10:05 INC-102,SEV2,2026-07-05T22:03,2026-07-05T22:05,2026-07-05T22:12,2026-07-05T22:38,2026-07-05T23:20 INC-103,SEV3,2026-07-11T14:20,2026-07-11T14:31,2026-07-11T14:33,2026-07-11T14:52,2026-07-11T15:10 INC-104,SEV3,2026-07-19T03:40,2026-07-19T03:42,2026-07-19T03:58,2026-07-19T04:20,2026-07-19T05:00 INC-105,SEV2,2026-07-28T11:00,2026-07-28T11:01,2026-07-28T11:03,2026-07-28T11:19,2026-07-28T12:30 INC-106,SEV3,2026-08-03T16:45,2026-08-03T16:47,2026-08-03T16:49,2026-08-03T17:30,2026-08-03T18:10 INC-107,SEV1,2026-08-09T01:12,2026-08-09T01:40,2026-08-09T01:43,2026-08-09T10:05,2026-08-09T13:30 INC-108,SEV3,2026-08-15T10:05,2026-08-15T10:06,2026-08-15T10:07,2026-08-15T10:25,2026-08-15T10:50 INC-109,SEV3,2026-08-22T19:30,2026-08-22T19:33,2026-08-22T19:41,2026-08-22T20:02,2026-08-22T20:30 INC-110,SEV2,2026-09-01T08:15,2026-08-31T23:59,2026-09-01T08:17,2026-09-01T08:49,2026-09-01T09:40 INC-111,SEV3,2026-09-14T13:00,2026-09-14T13:02,2026-09-14T13:04,2026-09-14T13:30,2026-09-14T14:00 INC-112,SEV3,2026-09-26T05:50,2026-09-26T05:55,2026-09-26T06:09,2026-09-26T06:35,2026-09-26T07:15 ``` _Example status: verified by execution on 2026-10-16 with Python 3.8.10 (`python3 -I incident_metrics.py incidents.csv`); the output below is copied from that run. Re-run after each revision, last on 2026-10-17 after re-wrapping the script: output identical._ ```python """Incident timing metrics with honest statistics (Python 3.8+, standard library only). Usage: python3 incident_metrics.py incidents.csv """ import csv import math import random import statistics import sys from datetime import datetime ORDER = ["impact_start", "detected", "acknowledged", "mitigated", "resolved"] INTERVALS = { # name: (from, to) "TTD": ("impact_start", "detected"), # time to detect "TTA": ("detected", "acknowledged"), # time to acknowledge "TTM": ("impact_start", "mitigated"), # time to mitigate (impact ends) "TTR": ("impact_start", "resolved"), # time to resolve (incident closed) } def minutes(a, b): return (b - a).total_seconds() / 60 def load(path): good, rejected = [], [] with open(path, newline="") as f: for row in csv.DictReader(f): ts = {k: datetime.fromisoformat(row[k]) for k in ORDER} if any(ts[x] > ts[y] for x, y in zip(ORDER, ORDER[1:])): rejected.append(row["id"]) # timestamps out of order: fix the record continue spans = {n: minutes(ts[a], ts[b]) for n, (a, b) in INTERVALS.items()} good.append({"id": row["id"], "sev": row["sev"], **spans}) return good, rejected def p90(values): return statistics.quantiles(values, n=10, method="inclusive")[-1] def summarise(rows): print(f"{'metric':<6}{'n':>3}{'mean':>8}{'median':>8}{'p90':>8}{'max':>8} (minutes)") for name in INTERVALS: v = [r[name] for r in rows] mean, median = statistics.mean(v), statistics.median(v) print(f"{name:<6}{len(v):>3}{mean:>8.1f}{median:>8.1f}" f"{p90(v):>8.1f}{max(v):>8.1f}") def simulate(trials=10_000, n=15, median_min=45.0, sigma=1.2, true_change=0.0, seed=42): """Two quarters of n incidents from the same lognormal distribution; quarter 2 is shortened by true_change (0.10 = 10 % shorter). Returns the share of trials in which the measured mean moved by 20 % or more, and the share in which it got worse.""" rnd = random.Random(seed) mu = math.log(median_min) big, worse = 0, 0 for _ in range(trials): q1 = statistics.mean(rnd.lognormvariate(mu, sigma) for _ in range(n)) q2 = statistics.mean(rnd.lognormvariate(mu, sigma) * (1 - true_change) for _ in range(n)) change = (q2 - q1) / q1 big += abs(change) >= 0.20 worse += change > 0 return big / trials, worse / trials if __name__ == "__main__": rows, rejected = load(sys.argv[1]) print(f"loaded {len(rows)} incidents; rejected (timestamps out of order): " f"{', '.join(rejected) or 'none'}") summarise(rows) worst = max(rows, key=lambda r: r["TTM"]) all_ttm = statistics.mean(r["TTM"] for r in rows) rest = statistics.mean(r["TTM"] for r in rows if r is not worst) print(f"TTM mean without {worst['id']}: {rest:.1f} min " f"(one incident moves the mean by {all_ttm - rest:.1f} min)") big, worse = simulate(true_change=0.0) print(f"simulation, nothing changed: mean moved >=20% in {big:.0%} of quarters") big, worse = simulate(true_change=0.10) print(f"simulation, real 10% improvement: measured mean got worse " f"in {worse:.0%} of quarters") ``` **Steps:** save the CSV as `incidents.csv` and the script as `incident_metrics.py`; run `python3 incident_metrics.py incidents.csv`. **Expected output:** ```text loaded 11 incidents; rejected (timestamps out of order): INC-110 metric n mean median p90 max (minutes) TTD 11 5.5 2.0 11.0 28.0 TTA 11 5.4 2.0 14.0 16.0 TTM 11 78.4 32.0 45.0 533.0 TTR 11 129.5 77.0 90.0 738.0 TTM mean without INC-107: 32.9 min (one incident moves the mean by 45.5 min) simulation, nothing changed: mean moved >=20% in 71% of quarters simulation, real 10% improvement: measured mean got worse in 42% of quarters ``` - **Validation:** INC-110 is rejected; mean TTM (78.4) is over twice the median (32.0); the simulation lines show on synthetic data what Davidovič found in real data ([Metrics and KPIs](#metrics-and-kpis) interprets all of it). - **Troubleshooting:** `AttributeError` on `statistics.quantiles` means Python older than 3.8; `ValueError` means a malformed timestamp. - **Teardown:** delete the two files. #### Lab 2 (intermediate): prove the paging path end to end _Status: not executed (needs a PagerDuty account and phones); follows the Events API v2 documentation as of 2026-10-16._ - **Objective:** prove that an alert reaches a human, escalates when ignored and closes cleanly. - **Prerequisites:** a PagerDuty trial account; the reference Terraform applied to a **test** service whose rota is you and a volunteer, with `-var layer_start=` a few minutes ago; `curl`; the mobile app. It rings real phones: warn the volunteer. Optional step 5 needs a test Prometheus with a Watchdog rule ([ObservabilityOps](../observabilityops/) Lab 3), Alertmanager with the heartbeat route, and a heartbeat service. **Steps** 1. Load the key into a variable, never onto the screen (`-raw` prints it in plain text):[^tf-sensitive] `PD_ROUTING_KEY="$(terraform output -raw checkout_routing_key)"`. 2. Send a trigger event, with the body on standard input (`--data-binary @-`): other local users can see command-line arguments in the process list, and curl's manual says credentials "should be retrieved from a file instead or similar".[^curl-man] _Example status: syntax-checked by the Atlas validator (bash -n); not executed._ ```bash curl -sS -o response.json -w '%{http_code}\n' \ -X POST https://events.pagerduty.com/v2/enqueue \ -H 'Content-Type: application/json' --data-binary @- <|"TTD"| T1["Detected"] T1 -->|"TTA"| T2["Acknowledged by a human"] T2 --> T3["Declared: IC, severity"] T1 -.->|"to declare"| T3 T3 -->|"to first update"| T4["First public update"] T4 --> T5["Mitigated: impact ends, TTM stops"] T5 --> T6["Resolved: closed, TTR stops"] T6 --> T7["Review, actions tracked"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class T0 risk class T1,T2,T3,T4,T5,T6,T7 core ``` Text equivalent: time to detect (TTD) runs from impact start to the first alert or report; time to acknowledge (TTA) from detection to a human acknowledging; time to declare from detection to declaration (dashed); time to first update from declaration to the first public post. Time to mitigate (TTM) and to resolve (TTR) start at impact and stop at mitigation and resolution; the review follows. ::: **Which "R"?** Atlassian notes that MTTR "potentially represents four different measurements. The R can stand for repair, recovery, respond, or resolve".[^atl-kpi] Google's "time to repair" runs from alerting to mitigation;[^anatomy] DORA's *failed deployment recovery time* (renamed in 2023 from time to restore service) covers only change-caused failures ([DevOps](../devops/)).[^dora-history] State start, end and population: "median time to mitigate, impact start to mitigation, SEV1–SEV2, last 90 days". | KPI | Formula | Pitfall | Cadence | |---|---|---|---| | Time to detect (TTD) | detected − impact start | Impact start is estimated afterwards | Per incident | | Time to acknowledge (TTA) | acknowledged − detected | The mean hides night-time outliers | Monthly, by hour of day | | Time to declare | declared − detected | Rewarding low values discourages declaring | Monthly | | Time to first public update | first update − declared | Ignores incidents never posted | Per SEV1–SEV2 | | Time to mitigate (TTM) | mitigated − impact start | Skewed and variable | Distribution only | | Customer impact | error-budget minutes consumed ([SRE](../sre/)) | Needs good SLIs | Monthly | | Pager load | paging incidents per shift, 21-day trailing average[^wb-oncall] | Pages are not incidents | Weekly | | Action closure | actions closed by due date ÷ actions due | Closing by downgrading | Monthly | **Why MTTx misleads.** Štěpán Davidovič's Google report simulated improvements on real incident data, whose durations "follow a positively skewed distribution". Splitting a full year of incidents (103 to 609 per company) into two samples and truly shortening every incident by 10 %, "38% of the simulations had the MTTR difference fall below zero for Company A, 40% for Company B, and 20% for Company C"; with no change at all there was "a 19% chance" of a half-hour improvement for Company A. His conclusion: "The default position should be to reject MTTx metrics" for judging trends or changes, except for large homogeneous populations or dramatic shifts; medians and 95th percentiles fared no better.[^davidovic] The 2022 VOID report, as reported by InfoQ, found most incidents resolve "in under two hours" and no correlation between duration and severity.[^infoq-void] **Worked calculation (Lab 1 data, illustrative).** - Mean TTM = 862 ÷ 11 = **78.4 minutes**; median **32**. INC-107 alone adds 45.5 minutes; "MTTM = 78" describes no real incident. - Mean TTA = 59 ÷ 11 = **5.4 minutes**; median 2. The slow acknowledgements (16 and 14 minutes) are INC-104 at 03:42 and INC-112 at 05:55: both at night, both beyond the 5-minute escalation step. Only case-level data points at the night rota. - Workload: 12 incidents happened (only INC-110's record is bad). Google's average of about 6 hours per incident covers root-cause analysis, remediation and follow-up such as postmortems,[^sre-oncall] so up to about **72 hours** a quarter; less where SEV3 reviews are skipped. **Report instead:** a dot plot of TTM naming long incidents, SLO impact, and conformance (declaration, updates and action closure on time). Counts by severity are context, not targets: Nash lists "count of severity types, total incident count" with MTTR as shallow data, while Huerta Granda sees such numbers as starting points for deeper study.[^infoq-void] Target only conformance; bonuses on incident counts or MTTR reward not declaring. ### Quality, security, privacy and compliance - **Access.** Logged, time-limited elevation, not shared admin accounts ([AccessOps](../accessops/)). - **Privacy.** Minimise customer data in records and chat; set retention; check what AI assistants store ([PrivacyOps](../privacyops/)). - **Regulatory clocks (EU, as of October 2026).** An outage can be reportable without an attacker. NIS2 defines an incident as "an event compromising the availability, authenticity, integrity or confidentiality" of data or services, counts one as significant if it causes or can cause "severe operational disruption", and requires an early warning within 24 hours and a notification within 72 hours of becoming aware, then a final report within one month of that notification.[^nis2] For financial entities, the Digital Operational Resilience Act (EU DORA, unrelated to the DORA delivery research) covers incidents with an "adverse impact on the availability" of data or services;[^eu-dora] its standards require an initial notification within four hours of classifying an incident as major and no later than 24 hours after becoming aware, an intermediate report within 72 hours of that, and a final report within one month of the latest intermediate report.[^dora-rts] So record **when you became aware** and **when you classified**, and agree with compliance who classifies. Scope depends on sector and national law; take legal advice ([ComplianceOps](../complianceops/)). Other jurisdictions are not covered here. ### Cost, capacity and sustainability People are the main cost. On-call load limits (Google's 25 % cap, minimum rota sizes, incidents per shift, capped pay) are taught in [SRE](../sre/#practice-7-sustainable-on-call); mastery question 5 applies them. Big incidents are expensive meetings: 15 people on a 3-hour call is 45 person-hours, so crowd control is cost control. Sustainability here means avoiding burnout; the environmental footprint of incident tooling is not material. ### Assurance of the operating model - **Drills:** role-play,[^sre-accel] monthly paging drills (Lab 2) and, at scale, simulations like those Atlassian adopted.[^atl-2022] - **Record audits:** sample records monthly for out-of-order timestamps (like INC-110), missing roles and late reviews. - **Leading indicators:** pager load over budget, overdue actions, missed drills, on-call survey results. - **Near misses:** make them easy to note; Nash warns that formally classifying them repeats the "shallow data" trap.[^infoq-void] ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Remediation | Prevention | |---|---|---|---| | Customers report before any page | Detection gap or broken paging path | Declare; watch dashboards; run the runbook below | Heartbeat, SLO alerts, paging drills | | Page not acknowledged | Schedule gap, override, contact settings | Page the secondary or manager directly | Schedule audits; acknowledgement timeout | | Two people change production | No IC, or freelancing | IC freezes changes except the ops lead's | Role cards, drills | | Customers hear nothing | No comms owner | Assign one; post now | Cadence timer in the incident bot | | Crowded call, slow decisions | Span of control exceeded | Sub-teams; updates channel | Large-incident playbook | | Same incident returns | Actions not done | Escalate in the monthly review | Action SLOs, single owners | | Exhausted responders | Long incident, no rotation | Explicit IC and ops handovers | Shift limits, deputy role | | Status page down too | Shared fate | Fallback channel | Independent hosting | ### Worked runbook: pages are not reaching on-call **Trigger:** customers report impact that nothing paged, or the heartbeat service reports the Watchdog alert has stopped. 1. **Declare.** A blind detection path is an incident; someone watches key dashboards until paging is proven. 2. **Is the alert firing at the source?** Prometheus exposes active alerts as the `ALERTS` series and via its HTTP API.[^prom-alerts] 3. **Did Alertmanager receive and route it?** Query its alerts; test the routing tree with the alert's labels.[^amtool] 4. **Are notifications failing?** The standard Alertmanager alert compares `alertmanager_notifications_failed_total` with `alertmanager_notifications_total`.[^am-mixin] 5. **Did the paging service accept it, and was anyone on call?** Send a Lab 2 test event and expect `202`; check schedule gaps, overrides and contact methods. _Example status: syntax-checked by the Atlas validator (bash -n); commands follow the Alertmanager and Prometheus documentation; not executed. Host names are placeholders._ ```bash # 2. Firing alerts at the source (Prometheus HTTP API) curl -s http://prometheus.internal.example:9090/api/v1/alerts | head -c 2000 # 3. What Alertmanager holds, and where a page-worthy checkout alert would go amtool --alertmanager.url=http://alertmanager.internal.example:9093 \ alert query severity=page amtool config routes test --config.file=alertmanager.yml \ --verify.receivers=pagerduty-checkout severity=page team=checkout # 4. Notification failures per integration (PromQL, paste into the Prometheus UI): # rate(alertmanager_notifications_failed_total{integration="pagerduty"}[15m]) # Remediation example: a rotated PagerDuty routing key was not updated in the secret file sudo install -m 0400 -o alertmanager /secure/new-pd-checkout-key \ /etc/alertmanager/secrets/pd-checkout curl -s -X POST http://alertmanager.internal.example:9093/-/reload ``` **Remediation:** fix the broken link (route, receiver, credential, schedule or contact method), reload Alertmanager (`SIGHUP` or a `POST` to `/-/reload`),[^am-config] and prove the path with a test event before ending the manual watch. **Prevention:** keep the Watchdog heartbeat;[^watchdog] run `amtool config routes test --verify.receivers` in CI, which "returns error code 1 on mismatch";[^amtool] repeat the paging drill monthly and after rota or tool changes. ### Anti-patterns and myths - **"The most senior engineer should lead."** PagerDuty makes the IC "the highest ranking individual on any major incident call, regardless of their day-to-day rank";[^pd-ic] train several and rotate the role, as Google advises.[^sre-managing] - **"Declaring is admitting failure."** Punish incident counts and people stop declaring. - **"Find the root cause before acting."** Mitigation needs only the fault's location.[^wb-ir] - **"Root cause: human error."** Cook: "Post-accident attribution to a 'root cause' is fundamentally wrong", and "Hindsight biases post-accident assessments of human performance".[^cook] If you use Atlassian's "Five Whys",[^atl-pm] follow several branches and stop at system conditions, not at a person. - **"More action items, better review."** The Workbook criticises actions that are "mostly mitigative", vague and equally prioritised;[^wb-pm] PagerDuty: "Generally we only want to create things that are P0/P1's".[^pd-pm] Etsy goes further: "The goal of a debriefing is not to produce recommendations or remediation items".[^etsy-debrief] - **"Everyone should join the call."** Observers raise coordination costs;[^maguire] give them the updates channel. ## Deciding and adopting ### When to use and when not to use Any team running a service for others needs *some* incident process; the question is how much ceremony. **Use a full process** when customers depend on the service in real time and failures often involve several teams; reportable incidents add reporting duties to any process, and a 24/7 promise adds a night rota. **Keep it minimal** (tier 0 or 1 below) for internal tools run in business hours with rare, single-team incidents; Google notes that the full Incident Command System "may be more than you need".[^wb-ir] **Do not build it yourself** when a managed service provider or an ITIL service desk already owns the response ([ITOps](../itops/)); design the interface instead. **Hidden responsibilities:** paying for coverage, training commanders, keeping templates approved, holding contact lists outside production, and follow-up work that competes with features. ### Decision framework ::: figure How much incident process do you need? ```mermaid %%{init: {"flowchart": {"wrappingWidth": 100, "nodeSpacing": 8, "rankSpacing": 28, "padding": 6}}}%% flowchart TB Q1{{"Real-time dependants?"}} Q1 -- "No" --> T0["Tier 0: tickets, checklist"] Q1 -- "Yes" --> Q2{{"Cross-team, or 2+ SEV1-2 a month?"}} Q2 -- "No" --> T1["Tier 1: one rota, severity, template"] Q2 -- "Yes" --> Q3{{"Often over 5 responders?"}} Q3 -- "No" --> T2["Tier 2: IC, comms lead, status page"] Q3 -- "Yes" --> T3["Tier 3: adds deputy, liaisons, sub-teams"] T1 --> A1{{"24/7 promise?"}} T2 --> A1 T3 --> A1 A1 -- "Yes" --> R1["Add night rota, IC cover"] A1 -- "No" --> A2{{"Reportable or regulated?"}} R1 --> A2 T0 --> A2 A2 -- "Yes" --> R2["Add regulatory clocks"] A2 -- "No" --> RQ["Review the tier quarterly"] R2 --> RQ classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class Q1,Q2,Q3,A1,A2 decision class T1,T2,T3,R1,R2 core class T0,RQ neutral ``` Text equivalent: no real-time dependants means tier 0. Otherwise tier 1, unless incidents span teams or SEV1–SEV2 reach two a month: then tier 2, or tier 3 when incidents often need more than about five responders (contents in the table below). From tier 1, a 24/7 promise adds night rotas and IC cover; at any tier, including tier 0, reportable or regulated services add regulatory clocks. Review the tier quarterly. ::: The thresholds are this Atlas's assumptions, not industry standards. Rota sizing is in [SRE](../sre/#practice-7-sustainable-on-call); the clocks are in [Quality, security, privacy and compliance](#quality-security-privacy-and-compliance). | Tier | Roles | Communication | Review | |---|---|---|---| | 0 | Owner fixes from the queue | None external | None | | 1 | On-call holds every role; manager as backup | Template status message | Short review for customer-visible incidents | | 2 | Trained IC rota, ops lead, comms lead, scribe | Status page on cadence; updates channel | Review within days; action SLOs | | 3 | Adds deputy, liaisons, sub-teams | Pre-approved templates; legal and PR path | Large-incident playbook | ### Maturity model *The Atlas's proposed model, not an official standard.* Move up when the exit criterion is met. | Level | Observable behaviour | Evidence | Exit criterion | |---|---|---|---| | 1. Heroic | Whoever notices fixes; customers often report first | Incidents found via support tickets | Severity policy; rota with escalation | | 2. Defined | Rota, severity levels, a status template | Policies, paging logs | Three trained ICs; one-step declaration | | 3. Coordinated | IC, comms and scribe on every SEV1–SEV2; updates on cadence | Records with roles and timestamps | Cadence kept in 90 % of SEV1–SEV2; reviews on time | | 4. Learning | Actions tracked against SLOs; drills; metrics as distributions | Action age reports, drill notes | 80 % of priority actions on time for two quarters | | 5. Adaptive | Large and cross-company incidents rehearsed; reviews steer investment | Simulation reports; decisions citing reviews | Maintained, not finished | ### Adoption roadmap: 30/60/90 days | Period | Actions | Milestone | |---|---|---| | Days 1–30 | Severity policy; rotas and escalation in code for the top three services; independently hosted status page; IC and comms role cards; one tabletop | Anyone can declare with one command | | Days 31–60 | Train four to six ICs by shadowing; approved templates; review template and action SLOs; Watchdog heartbeat; first paging drill | Every SEV1–SEV2 has an IC and a comms lead | | Days 61–90 | Reviews on time; monthly action review; metrics as distributions; drill the ResponseOps and DR handoffs; agree classification with compliance | First monthly report; both handoffs rehearsed | Dependencies: service ownership ([ServiceOps](../serviceops/)), alerts worth paging on ([ObservabilityOps](../observabilityops/)), chat tooling ([ChatOps](../chatops/)). An engineering manager owns month one; an incident-programme owner the rest. ## Toolkit ### Checklists **On-call preflight (start of shift)** - [ ] A test page reaches me on two channels (app and phone call) - [ ] I have read the handover note - [ ] I know the secondary, the IC on call and the escalation path - [ ] Laptop, network fallback and break-glass access work **IC: the first ten minutes** - [ ] Announce "I am the IC"; name ops lead, comms lead and scribe - [ ] Set severity (if unsure, the higher one) and the next update time - [ ] Size up: impact, start time, recent changes, regions - [ ] Check for an attacker or site loss before choosing a mitigation; hand off or escalate if so - [ ] Ask for a generic mitigation and its side effects - [ ] Give every task one owner and a time-box - [ ] Verify each action worked; if not, switch to the backup option and size up again **Closing** - [ ] Recovery confirmed by data; final update states any data loss - [ ] Review owner named and meeting booked - [ ] Temporary changes listed for reversal - [ ] Timestamps complete and in order **Quarterly governance audit** - [ ] Pager load within budget for every rota - [ ] Priority actions closed on time; oldest open action named - [ ] Drills held, including one handoff drill - [ ] Severity policy, contact lists and templates reviewed ### Templates The templates below are this Atlas's own; PagerDuty's Apache 2.0 documentation can also be adapted with its notices retained.[^pd-about] **Declaration message** ```text INCIDENT DECLARED — {SEV} — {short title} IC: {name} | Ops lead: {name} | Comms: {name} | Scribe: {name} Impact (current best estimate): {who / what / where} Working channel: #inc-{id} (responders) · Updates: #inc-{id}-updates (everyone) Call: {bridge link} · Record: {document link} Next update: {time, UTC} ``` **External status updates** ```text Investigating: We are investigating {symptom} affecting {product/feature} in {region}. Next update by {time UTC}. Identified: We have identified the cause of {symptom} and are applying a fix. {Workaround, if any.} Next update by {time UTC}. Monitoring: A fix has been applied and {feature} is recovering. We are monitoring. Next update by {time UTC}. Resolved: {Feature} has been fully restored since {time UTC}. {No data was lost | Data affected: ...}. A summary will follow. ``` **IC handover script** ```text Outgoing IC: "Status: {SEV}, impact {…}, current hypothesis {…}, actions in flight {who/what/until when}, next update due {time}. Open risks: {…}. You're now the incident commander, okay?" Incoming IC: "I'm now the incident commander." (Scribe records the time; IC posts it in both channels.) ``` **Post-incident review skeleton** ```text Title / ID / severity / review owner / meeting date Summary (5 lines, plain language) Impact: who, what, how long, data affected, SLO budget consumed Timeline: timestamped facts with links to graphs and messages Contributing factors: technical, process, organisational (no names as causes) What went well / where we were lucky Response review: declaration, roles, communication cadence, handovers Actions: ID | kind (prevent/mitigate/detect/process) | one owner | priority | due | link Open questions and follow-up learning ``` **Generic-mitigation runbook (one per mitigation per service)** ```text Mitigation: {rollback | drain | feature kill | failover | load shedding} - {service} Owner: {team} | Last tested: {date; drill or real use} | Approver: {role, if any} Use when: {symptoms; fault located in ...} Do not use when: {schema or data change in release; attacker suspected: ask ResponseOps} Preconditions: {previous version available; spare capacity >= 100 % of peak elsewhere} Side effects: {features lost; data written since ...; what customers notice} Steps: 1. {command or console path, expected output} 2. {...} Verify: {graph or query showing the impact stopped, within N minutes} Undo: {how to reverse the mitigation} Record: {post start, end and result in the incident channel} ``` **Drill scorecard (Lab 3)** | Criterion | Met? | Evidence | |---|---|---| | Declared within 5 minutes with IC and severity | | | | First update within policy; cadence kept | | | | Every task owned and time-boxed | | | | Executive interruption redirected | | | | Security handoff in a restricted channel | | | | IC handover acknowledged in the record | | | ### Deep comparisons | Practice | Trigger | Goal | Leader | Clock | |---|---|---|---|---| | IncidentOps (incident management) | Impact meets declaration criteria | Stop impact, inform, learn | IC | Minutes to hours | | ITIL incident and problem management ([ITOps](../itops/)) | Any unplanned interruption, even for one user | Restore service; remove causes | Service desk; problem manager | Ticket SLAs; weeks | | Security incident response ([ResponseOps](../responseops/)) | Suspected attacker | Contain, preserve evidence, notify | IR lead | Legal deadlines | | Disaster recovery ([DisasterRecoveryOps](../disasterrecoveryops/)) | Site or region loss | Restore within RTO/RPO | Invocation authority decides; DR coordinator runs it | Pre-agreed RTO | | Reliability engineering ([SRE](../sre/)) | SLO burn, review findings | Change the system | Service team | Quarters | ### FAQ **Is "IncidentOps" different from incident management?** Not in any documented way: the uses we found relabel it or name a product.[^opsio][^incidentopsio] When a vendor or job advert uses it, ask which practices it means. **We are six engineers. Do we need an incident commander?** You need the *role*, not an extra person: the on-call holds every role until a second person joins, then hands one away, usually communications. **What is the difference between severity and priority?** Severity describes impact and sets the response (who is mobilised, how often you communicate); priority orders work such as follow-up actions. A minor incident can create an urgent action. **Should we stop measuring MTTR?** Stop using it to judge trends, tools or teams;[^davidovic] keep the timestamps and report as in [Metrics and KPIs](#metrics-and-kpis). Dramatic shifts remain visible, small improvements do not. **Should post-incident reviews be published?** Internally, widely: sharing is how other teams learn.[^sre-postmortem] Externally, a customer summary for significant incidents, as AWS and Atlassian did,[^aws-s3][^atl-2022] without security-sensitive detail or personal data. ## Going further ### Advanced topics and emerging direction - **Incidents across company boundaries** (established problem, immature practice). STELLA notes that SaaS dependence makes coordination "across organization (and company) boundaries" more often necessary;[^stella] agree in advance how suppliers join your incidents (supplier path: [End-to-end workflow](#end-to-end-workflow)). - **Learning from incidents** (active research): in-depth study of selected incidents over shallow metrics.[^stella][^infoq-void] - **Regulatory reporting** (established law, maturing practice): timestamps become evidence.[^nis2][^dora-rts] ### Learning path | Stage | What to do | Result | |---|---|---| | Prerequisites | Know your service and its alerts; read [SRE](../sre/) | You can say what users would notice | | 1-day primer | Google's "Managing Incidents" and "Incident Response";[^sre-managing][^wb-ir] PagerDuty's IC training;[^pd-ic] Lab 1 | You can name the roles and explain why MTTx misleads | | 1 week | Shadow an IC; draft a severity policy; run Lab 2 | A policy draft and a proven paging path | | 30 days | Command a SEV-3, then a SEV-2 with a deputy; facilitate Lab 3; write a review | A real timeline, review and scorecard | | Advanced | Run a large simulation; read Cook, STELLA, Davidovič;[^cook][^stella][^davidovic] map regulatory clocks | You can design an incident programme | | Portfolio | Policy, rotas as code and routing tests in a repository; a drill report | Evidence of practice | ## Wrap-up ### Executive recap and cheat sheet | Area | Remember | |---|---| | Decisions | Process tier; IC model; who approves public statements; review deadlines; action SLO | | Practices | Declare early; an IC who does not fix; size up, then mitigate; updates on a timer; one record; explicit handovers | | Paging | Google: "Every page should be actionable";[^sre-monitoring] escalate after about 5 minutes; heartbeat the path | | Metrics | Timestamps, distributions, named long incidents, SLO impact, conformance, action closure; not MTTR trends | | Handoffs | Attacker → [ResponseOps](../responseops/); site loss → escalate to the DR authority ([DisasterRecoveryOps](../disasterrecoveryops/)); causes → [ITOps](../itops/), [SRE](../sre/) | | Watch-outs | Hero ICs, punished declarations, status pages sharing fate, actions nobody closes | | Next actions | Severity policy, three trained ICs, a paging drill and a tabletop within 30 days | ### A judgement call: roll back now, or wait for the fix? ::: example Illustrative scenario At 14:05 UTC, 30 % of checkout attempts fail, 20 minutes after version 42 of the payment-routing rules reached every region. The likely cause is a malformed rule; its author says: "I can patch it in 20 to 60 minutes." Rolling back to version 41 takes about 12 minutes but removes a fraud-screening rule that compliance wants live in another market. ::: **Assumptions (invented):** failures stay at 30 % until mitigation; a rollback has a 5 % chance of failing and costing 30 more minutes; the patch lands uniformly between 20 and 60 minutes (mean 40) and has a 25 % chance of failing and costing 30 more. | Option | Expected impact (minutes × failure share) | Side effects | |---|---|---| | A. Roll back now | 12 × 0.30 + 0.05 × 30 × 0.30 = 3.6 + 0.45 ≈ **4.1** | Fraud rule missing in one market for hours | | B. Wait for the patch | 40 × 0.30 + 0.25 × 30 × 0.30 = 12.0 + 2.25 ≈ **14.3** | None if the patch works first time | **Why B is attractive and wrong here.** It is precise, keeps the fraud rule and its author is confident. But the estimate comes from the person under most pressure, its range is wide, and delay costs every customer every minute: B carries about 3.5 times the expected impact. Hence the Workbook's "prioritize mitigation above all else".[^wb-ir] **Decision.** "We're proceeding with the rollback. Are there any strong objections?"[^pd-ic] Compliance agrees a temporary control (manual review of high-value orders in that market); the patch ships later as version 43. **When B would be right:** the rollback is unsafe (it reverses a data or schema change), or the fix is built, verified and minutes away. **Uncertainty:** the numbers are guesses; the value is an explicit, fast comparison. ### Learning objectives and mastery check You should now be able to **explain** what IncidentOps means and where it ends; **apply** declaration criteria and a severity policy; **run** the command loop and a communication cadence; **design** rotas, escalation and a paging heartbeat; **evaluate** incident metrics; and **decide** how much process a team needs. 1. An on-call engineer has worked alone for 50 minutes on a partial checkout failure; customers complain publicly. What now? ::: answer Show answer for question 1 Declare now: customers can see it, so Google's criteria are met.[^sre-managing] Name an IC, assign a comms lead and post a status update, freeing the engineer to work on the system. ::: 2. After a new tool arrived, mean time to resolve fell from 95 to 70 minutes in a quarter. Does that prove the tool works? ::: answer Show answer for question 2 No. Durations are skewed and highly variable: in one company's data Davidovič found a 19 % chance of a half-hour improvement when nothing changed.[^davidovic] Check the phase the tool should shorten, or study individual incidents. ::: 3. During a SEV-1 a senior executive starts directing engineers. You are the IC. What do you say? ::: answer Show answer for question 3 "Do you wish to take command?" If yes, hand over explicitly and announce it; if not, say you are in command and offer the updates channel and the next update time.[^pd-ic] ::: 4. A log suggests an unknown administrator logged in an hour before the outage. Your next three actions (Atlas proposal)? ::: answer Show answer for question 4 Restricted channel; page the security on-call; hand the security workstream to the ResponseOps lead while you keep overall coordination. Restores wait for evidence decisions; statements need legal review. PagerDuty also keeps an IC in charge of security incidents.[^pd-security] ::: 5. Six engineers want a 24/7 rota with primary and secondary on week-long shifts. Is that within Google's 25 % guideline? ::: answer Show answer for question 5 No: 2 slots × 52 weeks = 104 on-call weeks; ÷ 6 = about 17.3 weeks each, 33 % of the year. Google's guideline implies at least eight people.[^sre-oncall] Options: primary only (about 8.7 weeks, 17 %), a follow-the-sun partner team, or fewer night pages. ::: 6. Checkout errors reach 15 % during your cloud provider's regional problem, and its public status page is still green. What now? ::: answer Show answer for question 6 Declare: customers can see it. Check the provider's account-specific health feed and open a support case, but do not wait for either: fail over or degrade the dependent feature, post your own update on your customers' impact, and record the provider's case number and times.[^aws-health][^aws-support] ::: ### Conclusion "IncidentOps" is a new label on an old discipline, so use the discipline's name and evidence. That evidence is unusually consistent: Google, PagerDuty and Atlassian converge on the moves in the [cheat sheet](#executive-recap-and-cheat-sheet). Research on complex failures adds two warnings: coordination becomes expensive under pressure, and the comfortable numbers, MTTR and "root cause", say less than they seem to. The test of an incident programme: does a tired engineer at 3 a.m. know within five minutes who is in charge, where to talk and what to tell customers, and does the review a week later change something? Build for that moment and rehearse it. ## Sources and further reading Each footnote gives the date its source was last opened; claim-level notes are in the evidence ledger (`research/incidentops-ledger.md`). [^sre-managing]: [Managing Incidents (ch. 14)](https://sre.google/sre-book/managing-incidents/) — Andrew Stribblehill, *Site Reliability Engineering*, Google / O'Reilly, 2016. Checked 2026-10-16. [^sre-intro]: [Introduction](https://sre.google/sre-book/introduction/) — Benjamin Treynor Sloss, *Site Reliability Engineering*, 2016. Checked 2026-10-16. [^sre-oncall]: [Being On-Call (ch. 11)](https://sre.google/sre-book/being-on-call/) — Andrea Spadaccini, *Site Reliability Engineering*, 2016. Checked 2026-10-16. [^sre-monitoring]: [Monitoring Distributed Systems (ch. 6)](https://sre.google/sre-book/monitoring-distributed-systems/) — Rob Ewaschuk, *Site Reliability Engineering*, 2016. Checked 2026-10-16. [^sre-postmortem]: [Postmortem Culture (ch. 15)](https://sre.google/sre-book/postmortem-culture/) — John Lunney, Sue Lueder, *Site Reliability Engineering*, 2016. Checked 2026-10-16. [^sre-accel]: [Accelerating SREs to On-Call and Beyond (ch. 28)](https://sre.google/sre-book/accelerating-sre-on-call/) — Andrew Widdowson, *Site Reliability Engineering*, 2016. Checked 2026-10-16. [^wb-ir]: [Incident Response (ch. 9)](https://sre.google/workbook/incident-response/) — Jennifer Mace, Jelena Oertel, Stephen Thorne, Arup Chakrabarti et al., *The Site Reliability Workbook*, Google / O'Reilly, 2018. Checked 2026-10-16. [^wb-oncall]: [On-Call (ch. 8)](https://sre.google/workbook/on-call/) — Ollie Cook et al., *The Site Reliability Workbook*, 2018. Checked 2026-10-16. [^wb-pm]: [Postmortem Culture: Learning from Failure (ch. 10)](https://sre.google/workbook/postmortem-culture/) — Daniel Rogers et al., *The Site Reliability Workbook*, 2018. Checked 2026-10-16. [^anatomy]: [Anatomy of an Incident](https://sre.google/resources/practices-and-processes/anatomy-of-an-incident/) — Ayelet Sachto, Adrienne Walcer, with Jessie Yang, Google / O'Reilly, 2022-01. Checked 2026-10-16. [^davidovic]: [Incident Metrics in SRE: Critically Evaluating MTTR and Friends](https://sre.google/resources/practices-and-processes/incident-metrics-in-sre/) — Štěpán Davidovič, Google / O'Reilly, 2021-03. Checked 2026-10-16. [^pd-sev]: [Severity Levels](https://response.pagerduty.com/before/severity_levels/) — PagerDuty IR docs. Checked 2026-10-16. [^pd-what]: [What is an Incident?](https://response.pagerduty.com/before/what_is_an_incident/) — PagerDuty IR docs. Checked 2026-10-16. [^pd-roles]: [Different Roles](https://response.pagerduty.com/before/different_roles/) — PagerDuty IR docs. Checked 2026-10-16. [^pd-during]: [During an Incident](https://response.pagerduty.com/during/during_an_incident/) — PagerDuty IR docs. Checked 2026-10-16. [^pd-ext]: [External Communication Guidelines](https://response.pagerduty.com/during/external_communication_guidelines/) — PagerDuty IR docs. Checked 2026-10-16. [^pd-security]: [Security Incident](https://response.pagerduty.com/during/security_incident_response/) — PagerDuty IR docs. Checked 2026-10-16. [^pd-pm]: [Postmortem Process](https://response.pagerduty.com/after/post_mortem_process/) — PagerDuty IR docs. Checked 2026-10-16. [^pd-ic]: [Incident Commander training](https://response.pagerduty.com/training/incident_commander/) — PagerDuty IR docs. Checked 2026-10-16. [^pd-oncall]: [Being On-Call](https://response.pagerduty.com/oncall/being_oncall/) — PagerDuty IR docs. Checked 2026-10-16. [^pd-about]: [About (Apache License 2.0)](https://response.pagerduty.com/about/) — PagerDuty IR docs. Checked 2026-10-16. [^atl-pm]: [Incident postmortems](https://www.atlassian.com/incident-management/handbook/postmortems) — Atlassian incident management handbook, undated. Checked 2026-10-16. [^atl-kpi]: [MTBF, MTTR, MTTA, and MTTF](https://www.atlassian.com/incident-management/kpis/common-metrics) — Atlassian, undated. Checked 2026-10-16. [^nims]: [National Incident Management System, Third Edition](https://www.fema.gov/sites/default/files/2020-07/fema_nims_doctrine-2017.pdf) — FEMA, 2017-10. Checked 2026-10-16. [^cook]: [How Complex Systems Fail](https://how.complexsystems.fail/) — Richard I. Cook, University of Chicago, 1998–2000. Checked 2026-10-17. [^stella]: [STELLA: Report from the SNAFUcatchers Workshop on Coping With Complexity](https://snafucatchers.github.io/) — SNAFUcatchers, 2017. Checked 2026-10-17. [^maguire]: [Controlling the cognitive costs of coordination during incident response in critical digital services](https://open.lnu.se/index.php/rea/article/view/2032) — Laura Maguire, REA Symposium abstract, 2019. Checked 2026-10-16. [^etsy-debrief]: [Debriefing Facilitation Guide](https://extfiles.etsy.com/DebriefingFacilitationGuide.pdf) — John Allspaw, Morgan Evans, Daniel Schauenberg, Etsy, 2016. Checked 2026-10-16. [^infoq-void]: [Moving Past Simple Incident Metrics: Courtney Nash on the VOID](https://www.infoq.com/articles/incident-metrics-void/) — Courtney Nash and Matt Campbell, InfoQ, 2023-02-14. Checked 2026-10-16. [^dora-history]: [A history of DORA's software delivery metrics](https://dora.dev/insights/dora-metrics-history/) — Nathen Harvey, DORA, last updated 2026-01-02. Checked 2026-10-16. [^opsio]: [IncidentOps Explained for IT Teams](https://opsiocloud.com/knowledge-base/what-is-incidentops/) — Opsio, 2025-09-09, updated 2025-10-20. Checked 2026-10-16. [^incidentopsio]: [Incident/Ops — Incident Management for Slack](https://www.incidentops.io/) — Incident/Ops product page, undated. Checked 2026-10-16. [^nist61]: [NIST SP 800-61 Rev. 3](https://csrc.nist.gov/pubs/sp/800/61/r3/final) — NIST, 2025-04. Checked 2026-10-16. [^nist34]: [Contingency Planning Guide for Federal Information Systems, NIST SP 800-34 Rev. 1 (PDF)](https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-34r1.pdf) — M. Swanson et al., NIST, May 2010. Checked 2026-10-16. [^nis2]: [Directive (EU) 2022/2555 (NIS2), Articles 6 and 23](https://eur-lex.europa.eu/eli/dir/2022/2555/oj/eng) — European Union, 2022-12-14. Checked 2026-10-16. [^eu-dora]: [Regulation (EU) 2022/2554 (Digital Operational Resilience Act), Article 3](https://eur-lex.europa.eu/eli/reg/2022/2554/oj/eng) — European Union, 2022-12-14. Checked 2026-10-16. [^dora-rts]: [Commission Delegated Regulation (EU) 2025/301, Article 5](https://eur-lex.europa.eu/eli/reg_del/2025/301/oj/eng) — European Commission, adopted 2024-10-23. Checked 2026-10-16. [^aws-s3]: [Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/41926/) — AWS, 2017. Checked 2026-10-16. [^atl-2022]: [Post-Incident Review on the Atlassian April 2022 outage](https://www.atlassian.com/blog/how-we-build/post-incident-review-april-2022-outage) — Atlassian, 2022-04-30. Checked 2026-10-16. [^sp-incident]: [Create an incident](https://support.atlassian.com/statuspage/docs/create-an-incident/) — Atlassian Statuspage documentation. Checked 2026-10-16. [^sp-third-party]: [Add a third-party component](https://support.atlassian.com/statuspage/docs/add-a-third-party-component/) — Atlassian Statuspage documentation, modified 2025-09-25. Checked 2026-10-17. [^aws-health]: [Concepts for AWS Health](https://docs.aws.amazon.com/health/latest/ug/aws-health-concepts-and-terms.html) — AWS Health User Guide. Checked 2026-10-17. [^aws-support]: [Case management: choosing an initial severity level](https://docs.aws.amazon.com/awssupport/latest/user/case-management.html) — AWS Support User Guide. Checked 2026-10-17. [^gcp-psh]: [Personalized Service Health overview](https://cloud.google.com/service-health/docs/overview) — Google Cloud documentation, last updated 2026-10-14. Checked 2026-10-17. [^am-config]: [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/) — Prometheus documentation (Alertmanager v0.34.1, 2026-09-17). Checked 2026-10-16. [^am-mixin]: [Alertmanager mixin alerts](https://github.com/prometheus/alertmanager/blob/main/doc/alertmanager-mixin/alerts.libsonnet) — Prometheus, main branch. Checked 2026-10-16. [^amtool]: [Alertmanager README: amtool](https://github.com/prometheus/alertmanager#amtool) — Prometheus, main branch. Checked 2026-10-16. [^prom-alerts]: [Prometheus HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/) and [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) — Prometheus documentation. Checked 2026-10-16. [^watchdog]: [Watchdog runbook](https://runbooks.prometheus-operator.dev/runbooks/general/watchdog/) — kube-prometheus runbooks. Checked 2026-10-16. [^pd-events]: [Sending an Alert Event](https://docs.pagerduty.com/developer/send-alert-event) — PagerDuty docs. Checked 2026-10-17. [^pd-events-ov]: [Events API v2 Overview](https://docs.pagerduty.com/developer/events-api-v2-overview) — PagerDuty docs. Checked 2026-10-17. [^tf-pagerduty]: [PagerDuty Terraform provider docs](https://github.com/PagerDuty/terraform-provider-pagerduty/tree/master/website/docs) — PagerDuty (release v3.36.0, 2026-08-25). Checked 2026-10-16. [^tf-sensitive]: [Manage sensitive data in your configuration](https://developer.hashicorp.com/terraform/language/manage-sensitive-data) — HashiCorp Terraform documentation. Checked 2026-10-16. [^curl-man]: [curl man page](https://curl.se/docs/manpage.html) (`--data`, `--data-binary`, `--user`) — curl project. Checked 2026-10-16. [^opsgenie]: [Opsgenie migration FAQ](https://www.atlassian.com/software/opsgenie/migration) — Atlassian. Checked 2026-10-16. [^grafana-oncall]: [Grafana OnCall OSS in maintenance mode: your questions answered](https://grafana.com/blog/grafana-oncall-maintenance-mode/) — Richard Hartmann, Grafana Labs, 2025-03-12; [repository](https://github.com/grafana-cold-storage/oncall) shown as archived on GitHub. Checked 2026-10-16. [^grafana-survey]: [Observability Survey Report 2026](https://grafana.com/observability-survey/2026/) — Grafana Labs (fieldwork via Censuswide), 2026. Checked 2026-10-16. [^iio-ai]: [AI SRE](https://incident.io/solution/ai-sre) — incident.io product page. Checked 2026-10-16. [^pd-ai]: [AI Agents](https://www.pagerduty.com/platform/ai-agents/) — PagerDuty product page. Checked 2026-10-16. **Changelog.** 2026-10-16 — first draft (6 diagrams, 3 labs, Lab 1 executed); term status: overlapping synonym. 2026-10-16 — r1 fact check (`reviews/incidentops-r1.md`): 0 critical, 2 major, 21 minor, all addressed. 2026-10-17 — r2 after the independent score (`reviews/incidentops-score.md`, 89): tooling table repaired; supplier-incident path, tool criteria, mastery question 6 and four sources added; action deadlines tied to action priority; all figures redrawn to fit phones; Lab 1 re-wrapped and re-run (`reviews/incidentops-r2-response.md`). 2026-10-17, post-score polish (`reviews/incidentops-polish.md`): Lab 2's `400` cited to the Events API overview; origins chronology corrected; STELLA's "mixed" scoped to chat automation; figure 6 links tier 0 to the regulatory add-on; two recap repeats cut.