IncidentOps

IncidentOps is an informal name for incident management: how an organisation recognises that a live service has gone wrong, organises people to restore it, keeps everyone affected informed, and learns afterwards. The label adds nothing to a mature practice documented by Google, PagerDuty, Atlassian and ITIL.

In one line: IncidentOps is coordinated decision-making under time pressure, with explicit roles, a communication rhythm and a follow-up loop. Incidents rarely go badly for lack of skill; they go badly when nobody is in charge, two people change the same system, customers hear nothing, or follow-up actions are never done. Google names three hazards of an unmanaged incident: a sharp focus on the technical problem, poor communication, and “freelancing” (changing production without coordination).^[1]

Without a process, the first engineer to notice becomes investigator, fixer, spokesperson and historian at once; helpers change things in parallel; customers learn from social media; and nobody can reconstruct the timeline. In Grafana Labs’ 2026 survey (1,363 self-selected respondents; vendor-run), 30 % named alert fatigue as their biggest obstacle to faster incident response and 16 % named painful coordination across teams.^[2]

Incident management replaces improvisation with prepared decisions: what counts as an incident, who leads, who may change production, where people gather, how often updates go out, and what happens afterwards. Google’s SRE book claims a written playbook “produces roughly a 3x improvement in MTTR as compared to the strategy of ‘winging it’” (MTTR: mean time to repair),^[3] without publishing a method (Metrics and KPIs).

• Not a separate discipline. No standards body, book or foundation defines “IncidentOps”; our searches on 2026-10-16 found one consultancy article, a Slack app sold as Incident/Ops^[4] and unrelated hobby repositories. Opsio presents IncidentOps as a framework it “developed” that “emphasizes proactive prevention alongside reactive response”, extended to workplace safety.^[5] Neither shows a wider movement.

• Not security incident response. An adversary brings evidence, legal duties and containment: that is ResponseOps, which teaches NIST SP 800-61 Rev. 3.^[6] This page covers the shared coordination and handoff.

• Not problem management or SRE. Removing causes over weeks is problem management (ITOps); reliability targets are SRE.

• Not every alert. Google’s incidents are escalated and need an immediate, organised response; a planned, announced outage is not one.^[7]

Google’s incident management “is based on the Incident Command System” (ICS), which US emergency services use for large multi-agency responses.^[1] Sources date ICS differently: the SRE Workbook says it “was established in 1968 by firefighters”,^[8] while FEMA’s NIMS doctrine places the work in the 1970s, in California’s FIRESCOPE programme.^[9] Software borrowed the portable parts: one commander, a few named roles, a manageable span of control, common vocabulary and explicit handovers. Researchers of complex failures had challenged “root cause” thinking by 2000;^[10] Google published its model in 2016 and 2018, and PagerDuty opened its process (Apache License 2.0).^[11]

Mental model: an incident is a temporary organisation, with a leader, a structure, a meeting place, a record and a communication plan, dissolved deliberately. As STELLA observed, “during high tempo times the usually trivial costs of coordination become significant”;^[15] good incident management keeps those costs low so responders can think about the system.

%%{init: {"flowchart": {"wrappingWidth": 156, "nodeSpacing": 16, "rankSpacing": 30, "padding": 8}}}%%
flowchart TB
  IN["Signals in: alerts (ObservabilityOps), correlated events (AIOps), customer reports (SupportOps)"] --> INC
  CHT["ChatOps: channels"] --- INC
  INC["IncidentOps: command, comms, follow-up"]
  INC --> SRE["SRE: SLOs, culture"]
  INC --> ITO["ITOps: problem, change"]
  SRE ~~~ RSP
  ITO ~~~ DRO
  INC -.-> RSP["ResponseOps: if attacker"]
  INC -.-> DRO["DisasterRecoveryOps: if site lost"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class INC core
  class IN,SRE,ITO,CHT neutral
  class RSP,DRO risk

No official IncidentOps manifesto exists; this charter is this Atlas’s proposal, distilled from the practices cited here. Mission: when a live service fails, restore it quickly and safely, keep everyone affected accurately informed, and change the system so the failure is less likely or less harmful next time.

1. Declare early, on written criteria. “It is better to declare an incident early and then find a simple fix and close out the incident than to have to spin up the incident management framework hours into a burgeoning problem.”^[1]
2. One commander, many hands. The IC decides and coordinates; the IC is “NOT a resolver”.^[16]
3. Mitigate before you explain.^[8]
4. Communicate on a clock. Silence reads as inaction.^[8]
5. Keep one record: one channel, one live document, one timeline.
6. Hand over explicitly, with spoken confirmation.^[1]
7. Learn, then act: contributing factors without blame; owners and deadlines for the few actions that matter.
8. Keep on-call sustainable: load, pay and rest are part of reliability.^[17]

Non-goals: blame; root cause during the incident; replacing problem management or security response.

Roles are hats, not job titles: the IC “assumes all roles that have not been delegated yet”.^[8] PagerDuty adds a deputy (a “hot standby” IC), a scribe, subject-matter experts who report in CAN form (Condition, Actions, Needs), and customer and internal liaisons.^[16]

A = accountable, R = responsible, C = consulted, I = informed. The DR invocation authority, outside this table, is accountable for invoking DR (DisasterRecoveryOps). Follow-up moves to the service owner because incident roles end.

Span of control. FEMA’s doctrine gives “one supervisor to five subordinates” as the optimal span, as a guideline.^[9] Beyond about five direct reports, form sub-teams with their own leads; PagerDuty pre-names them Alpha, Bravo and Charlie.^[18]

%%{init: {"flowchart": {"wrappingWidth": 76, "nodeSpacing": 8, "rankSpacing": 34, "padding": 5}, "themeVariables": {"fontSize": "14px"}}}%%
flowchart TB
  IC["Incident commander"] --- DEP["Deputy"]
  IC --> OPS["Ops lead"]
  IC --> CL["Comms lead"]
  IC --> SCR["Scribe"]
  OPS --> A["Alpha team: lead, SMEs"]
  OPS --> B["Bravo team: lead, SMEs"]
  CL --> CUS["Customer liaison"]
  CL --> INT["Internal liaison"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class IC core
  class DEP,OPS,CL,SCR,A,B,CUS,INT neutral

Google describes a continuous lifecycle of preparedness, response, and mitigation and recovery, with the postmortem inside recovery.^[7] One incident, with the exception paths that most often go wrong:

%%{init: {"flowchart": {"wrappingWidth": 100, "nodeSpacing": 12, "rankSpacing": 28, "padding": 8}}}%%
flowchart TB
  P["Prepare: rotas, policies, drills"] --> D["Detect: alert or report"]
  D --> T{{"Incident criteria met?"}}
  T -- "Yes or unsure" --> DC["Declare: severity, IC, channel"]
  T -- "No" --> Q["Ticket or monitor"]
  DC --> M["Mobilise roles; first update"]
  M --> A{{"Size up: attacker?"}}
  A -- "No" --> L{{"Site or region lost?"}}
  A -- "Yes" --> RS["Security stream to ResponseOps (Atlas proposal)"]
  L -- "No" --> MI["Mitigate: stop the impact"]
  L -- "Yes" --> DR["Escalate to DR authority"]
  RS -->|"evidence rules"| MI
  DR -->|"failover if invoked"| MI
  MI --> V{{"Impact stopped?"}}
  V -- "Yes" --> R["Resolve and close"]
  V -- "No" --> A
  R --> RV["Review: timeline, factors"]
  RV --> FA["Follow-up actions tracked"]
  FA --> P
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class P,D,DC,M,MI,R,RV,FA core
  class T,A,L,V decision
  class RS,DR,Q risk

Governance is a few written policies: severity, on-call, communication (who approves public statements), reviews and an action SLO. Atlassian gives “priority actions” an SLO “which is either 4 or 8 weeks, depending on the service”, and engineering managers review the open list.^[20]

The architecture is mostly organisational, but its tools form a chain that must work when production does not. Two rules matter most: the paging path and the status page must not depend on the systems they report on, and the incident record is written once and reused for updates, metrics and the review.

%%{init: {"flowchart": {"wrappingWidth": 100, "nodeSpacing": 12, "rankSpacing": 30, "padding": 8}}}%%
flowchart TB
  SIG["Alerts"] --> RT["Alert routing and grouping"]
  RT --> PG["Paging: schedules and escalation"]
  HUM["People and tickets: manual page or chat command"] --> CO
  PG --> CO["Coordination: channel, call, roles"]
  CO --> REC["Incident record and timeline"]
  CO --> COM["Comms: status page, internal updates"]
  REC --> REV["Review and action tracker"]
  REV --> ENG["Engineering backlog and alert fixes"]
  ENG --> SIG
  HB["Paging-path heartbeat"] -.-> PG
  IND["Independent hosting"] -.-> COM
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class SIG,HUM,RT,PG,CO,REC,COM,REV,ENG core
  class HB,IND risk

The Amazon S3 case in Use cases and scenarios shows what fate-sharing costs.

Happy path (illustrative timings for a SEV-2):

1. T+0 detect. An SLO-based alert pages the primary; PagerDuty escalates after 5 minutes without acknowledgement.^[21]
2. T+3 triage against Google’s criteria: second team needed, visible to customers, unsolved after an hour’s concentrated analysis?^[1] If in doubt, declare.
3. T+5 declare. A chat command opens a channel and a live document, sets severity and names the IC.
4. T+8 mobilise. Ops and comms assigned; first status post (cadence).^[22]
5. T+10 onwards: the command loop (size up, stabilise, update, verify);^[18] updates on the cadence.
6. T+35 mitigate, for example by rollback.
7. T+60 resolve. Recovery and any data loss confirmed, final update posted, review owner named.^[22][23]
8. Within days: review, with actions in the owning teams’ backlogs.

Exception paths:

• Severity changes. “If you are unsure which level an incident is …, treat it as the higher one”; argue later, not on the call.^[13]
• Long incidents. Hand over command at shift boundaries with Google’s “You’re now the incident commander, okay?” and wait for acknowledgement.^[1]
• A security cause appears, at size-up or later. Atlas proposal: the IC keeps overall coordination, moves security discussion to a restricted channel, pages the security on-call and hands the security workstream to the ResponseOps lead; evidence preservation then constrains restores, and legal review gates statements. PagerDuty also keeps an IC for security incidents and says to trigger the process “anyway” when unsure.^[24]
• A site or region is lost. The IC checks the activation criteria in the DR policy and escalates, with a recommendation, to the invocation authority it names (DisasterRecoveryOps); NIST gives “the ultimate authority to activate the plan” to “A senior management official, such as the CIO”.^[25] The incident stays the coordination and communication layer.
• A supplier or cloud provider is the cause. Your customers’ impact is still your incident: declare on your own criteria, without waiting for the provider. Watch its account-specific feed (AWS Health names your affected resources; Google Cloud calls Personalized Service Health its primary channel for incidents relevant to your projects)^[26][27] and open a support case; AWS advises the highest severity your plan allows only for cases that “can’t be worked around or that directly affect production applications”.^[28] Mitigate what you control (fail over, degrade the dependent feature, queue work) and keep your own updates on cadence. Provider pages can lag: Statuspage, which can mirror a vendor’s status as a “third-party component”, lets you override it “if the external service has not yet updated their status page during an outage”.^[29] Record the supplier’s case number and times for the review.
• False alarm. Close it with a note; that is the process working.

The command loop. PagerDuty’s IC training runs command as a loop: size up, stabilise, update and verify, which asks “Have you finished?”, switches to a backup plan if an action failed and, “If problems persist, begin again from the size-up step”.^[18] Its phrases remove ambiguity: “We’re proceeding with…”; “Are there any strong objections?” rather than “Does everyone agree?”; “A, please perform B, I’ll come back to you in X minutes. Understood?”; and “Clear is better than concise”.

Mitigation first. The Workbook says “first responders must prioritize mitigation above all else”, and that “you only need to know the location of the root cause”.^[8] Keep a catalogue of generic mitigations per service (rollback, drain, feature-flag kill, failover, load shedding), each with a tested procedure, an owner and known side effects (Templates): they are “blunt instruments”.^[8]

Crowd control. Laura Maguire’s field research found incident channels that “can host several hundred participants”: useful for anticipating who is needed, but “a distraction” that can force “the core team into private channels”.^[30] PagerDuty lets anyone observe the call or chat, asks non-experts to route discussion through their service’s primary SME, and keeps a channel “For executive summary updates only”.^[31] This page proposes generalising it: a working channel for responders and an updates channel where the comms lead posts and everyone else reads. An executive who starts directing gets one question: “Do you wish to take command?”^[18]

Communication cadence. Agree audiences and timings in advance. PagerDuty’s guideline, a starting point: a first post within 5 minutes of the call starting, a scoping update (impact, components, regions) about 5 minutes later, updates at least every 20 minutes for the first 2 hours, and a resolution post stating any data loss.^[22] Statuspage’s states follow the same arc: Investigating, Identified, Monitoring, Resolved.^[32] PagerDuty advises against “outage” unless it was a full outage.^[23] Prepare two or three templates so nobody drafts under stress.^[8]

Running the post-incident review, after PagerDuty’s process and Etsy’s facilitation guide:

1. Owner. The IC names a review owner as the call ends; the owner books the meeting first (PagerDuty: within 3 calendar days for SEV-1, 5 business days for SEV-2) and invites everyone in the timeline.^[23][33]
2. Prepare. Build the timeline from the record, each entry linked to its graph or message;^[23] talk first to the people closest to the event, especially those who may expect blame.^[33]
3. Facilitator. Etsy: the facilitator “should not be someone who was directly involved in the event”; PagerDuty has the IC run the meeting.^[33][23] This page prefers a neutral facilitator and a note-taker for SEV-1.
4. Status: Draft, In Review (circulated a day before the meeting), Reviewed, Closed.^[23]
5. Close out. A few actions, each with one owner and a due date in the owning team’s tracker; an internal summary; an external message, reviewed in the meeting, if customers were affected.^[23]

Everything except the law is optional; pick one vocabulary and stick to it.

Also check that a tool works when your cloud does not, exports its configuration and records (both exits above force migrations), and limits who reads channels holding customer data.

Automate the mechanical work: opening channel and document, pulling recent deployments into the record, timing updates, drafting the timeline. The bigger prize is automating mitigation: Google found that “once a human is involved, the outage will last at least 20 to 30 minutes”.^[7] Self-healing belongs to AutomationOps and SRE.

incident.io markets an “AI SRE” that “starts debugging the moment you declare an incident”,^[39] and PagerDuty markets agents “ready to autonomously resolve well-known issues”;^[41] these are vendor claims as of October 2026, not independent evaluations. STELLA’s partners found their experiments with automation that works through chat “mixed”: it is easier to imagine than to build as a genuine “team player”.^[15] Treat AI output as an SME’s hypothesis, with a time-box and a check, and check what a model retains, because channels contain customer data. ML evaluation is taught in AIOps.

Illustrative design: about 40 engineers in seven teams across two time zones, with Prometheus, PagerDuty, chat and a hosted status page. Four artefacts live in code.

Example status: YAML parsed by the Atlas validator; illustrative policy values, not executed.

# severity-policy.yaml: the contract between on-call, ICs, comms and support
# (values are illustrative)
version: 3
declare_if_any:
  - "another team is needed to fix it"
  - "customers can see it"
  - "unsolved after 60 minutes of focused work"
  - "data loss or corruption is suspected"
  - "a security cause is suspected (also page the security on-call)"
when_unsure: "use the higher severity; correct it in the review"
# Severity sets the response; action priority sets the deadline. Atlassian sets 4 or 8
# weeks per service; this policy maps the two values to action priority instead.
action_due_weeks: {P1: 4, P2: 8}   # P1 = a "priority action" in Atlassian's sense
levels:
  SEV1:
    criteria: "login, checkout or data integrity broken for >25% of users, or data loss"
    roles: [ic, deputy, ops_lead, comms_lead, scribe, customer_liaison]
    first_public_update_minutes: 5
    update_every_minutes: 20
    executive_summary_every_minutes: 60
    review_meeting_within: "3 calendar days"
  SEV2:
    criteria: "a core feature degraded for many users, or a SEV1 is likely without action"
    roles: [ic, ops_lead, comms_lead, scribe]
    first_public_update_minutes: 10
    update_every_minutes: 30
    review_meeting_within: "5 business days"
  SEV3:
    criteria: "partial loss for a minority of users, or lost redundancy"
    roles: [on_call]          # one person holds every role; upgrade if it spreads
    public_update: "only if customers report it"
    review: "optional; weekly incident review"
  SEV4:
    criteria: "no customer impact; needs action this week"
    response: "ticket"

Example status: YAML parsed by the Atlas validator; field names checked against the Alertmanager configuration reference (2026-10-16); not loaded into Alertmanager or checked with amtool.

# alertmanager.yml — who gets paged. Alerts labelled severity="page" page a team; the rest
# become tickets (the Atlas label convention, shared with the AIOps and ChatOps pages).
route:
  receiver: ticket-queue
  group_by: [alertname, service]
  routes:
    - receiver: heartbeat          # Watchdog pattern; rule and design: ObservabilityOps
      matchers: ['alertname="Watchdog"']
      group_interval: 1m           # repeat_interval is rounded up to a multiple of this
      repeat_interval: 1m
    - receiver: pagerduty-checkout
      matchers: ['severity="page"', 'team="checkout"']
    - receiver: pagerduty-platform
      matchers: ['severity="page"']
# Grouping and inhibition rules are taught on the AIOps page.
receivers:
  - name: ticket-queue
    webhook_configs:
      - url: https://tickets.example.com/hooks/alertmanager
  - name: heartbeat
    webhook_configs:
      - url: https://heartbeat.example.com/ping/CHECK_ID
  - name: pagerduty-checkout
    pagerduty_configs:
      # key file: mode 0400, owned by the alertmanager user
      - routing_key_file: /etc/alertmanager/secrets/pd-checkout
        severity: critical         # PagerDuty's event severity, not the alert label
  - name: pagerduty-platform
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-platform
        severity: critical

The heartbeat route sends the “always firing” Watchdog alert to a service that notifies you “when this alert is not firing”;^[42] the rule belongs to ObservabilityOps, grouping and inhibition to AIOps. This file owns the receivers; keys stay out of the repository in files (routing_key_file) that only the Alertmanager user can read.^[36] Test the routing tree in CI (runbook).

Example status: argument names checked against the PagerDuty Terraform provider documentation (v3.36.0, 2026-08-25); not planned or applied. The provider accepts an API token (PAGERDUTY_TOKEN), a user-level token (PAGERDUTY_USER_TOKEN) or an App OAuth scoped token;^[43] use the one with the least access your account offers, in a test account first.

terraform {
  required_providers {
    pagerduty = { source = "PagerDuty/pagerduty", version = "~> 3.36" }
  }
}

variable "rota" {
  description = "E-mail addresses of the checkout on-call rota, in rotation order"
  type        = list(string)
}

variable "layer_start" {
  description = "When the rota starts (RFC 3339). For a drill, a time a few minutes ago"
  type        = string
  default     = "2026-11-02T10:00:00+01:00"
}

locals {
  # Next week's primary is this week's secondary, so the secondary already has context.
  secondary_order = concat(slice(var.rota, 1, length(var.rota)), [var.rota[0]])
}

data "pagerduty_user" "rota" {
  for_each = toset(var.rota)
  email    = each.value
}

data "pagerduty_user" "manager" {
  email = "checkout-manager@example.com"
}

resource "pagerduty_schedule" "primary" {
  name      = "checkout-primary"
  time_zone = "Europe/Berlin"
  layer {
    name                         = "weekly"
    start                        = var.layer_start
    rotation_virtual_start       = var.layer_start
    rotation_turn_length_seconds = 604800
    users                        = [for e in var.rota : data.pagerduty_user.rota[e].id]
  }
}

resource "pagerduty_schedule" "secondary" {
  name      = "checkout-secondary"
  time_zone = "Europe/Berlin"
  layer {
    name                         = "weekly"
    start                        = var.layer_start
    rotation_virtual_start       = var.layer_start
    rotation_turn_length_seconds = 604800
    users = [
      for e in local.secondary_order : data.pagerduty_user.rota[e].id
    ]
  }
}

resource "pagerduty_escalation_policy" "checkout" {
  name      = "checkout"
  num_loops = 2
  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.primary.id
    }
  }
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.secondary.id
    }
  }
  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user_reference"
      id   = data.pagerduty_user.manager.id
    }
  }
}

resource "pagerduty_service" "checkout" {
  name                    = "checkout-api"
  escalation_policy       = pagerduty_escalation_policy.checkout.id
  acknowledgement_timeout = 1800   # an acknowledged incident re-triggers after 30 minutes
  auto_resolve_timeout    = "null" # never auto-resolve: a human closes incidents
}

resource "pagerduty_service_integration" "alertmanager" {
  name    = "alertmanager"
  service = pagerduty_service.checkout.id
  type    = "events_api_v2_inbound_integration"
}

output "checkout_routing_key" {
  value     = pagerduty_service_integration.alertmanager.integration_key
  sensitive = true
}

The 5-minute first step follows PagerDuty’s recommended escalation timeout.^[21] acknowledgement_timeout re-pages a forgotten acknowledged incident; "null" disables auto-resolution.^[43] Layers begin at layer_start (past, for a drill). Anyone holding the routing key can open incidents on the service, and sensitive = true only hides it from plan output: Terraform stores it in state and terraform output -raw prints it in plain text,^[44] so keep state in an encrypted remote backend few people can read.

Example status: JSON parsed by the Atlas validator; illustrative record matching INC-107 in Lab 1.

{
  "id": "INC-107",
  "title": "Payments API errors in eu-west",
  "severity": "SEV1",
  "severity_history": [
    {"at": "2026-08-09T01:45:00Z", "severity": "SEV2"},
    {"at": "2026-08-09T02:05:00Z", "severity": "SEV1"}
  ],
  "roles": {"ic": "user-104", "ops_lead": "user-221",
            "comms_lead": "user-318", "scribe": "user-140"},
  "timestamps": {
    "impact_start": "2026-08-09T01:12:00Z",
    "detected": "2026-08-09T01:40:00Z",
    "acknowledged": "2026-08-09T01:43:00Z",
    "declared": "2026-08-09T01:45:00Z",
    "first_public_update": "2026-08-09T01:52:00Z",
    "mitigated": "2026-08-09T10:05:00Z",
    "resolved": "2026-08-09T13:30:00Z"
  },
  "impact": {"services": ["payments-api"], "regions": ["eu-west"], "data_loss": false},
  "security_suspected": false,
  "dr_invoked": false,
  "review": {"owner": "user-221", "meeting_by": "2026-08-12", "status": "In Review"},
  "actions": [
    {"id": "ACT-311", "kind": "prevent", "owner": "team-payments",
     "priority": "P1", "due": "2026-09-06"},
    {"id": "ACT-312", "kind": "detect", "owner": "team-platform",
     "priority": "P2", "due": "2026-10-04"}
  ]
}

Timestamps kept as data feed metrics, the review timeline and regulatory clocks; the action due dates follow the policy’s action_due_weeks (P1 four weeks, P2 eight weeks after the incident).

• Objective: compute incident timings, see one long incident distort the mean, and simulate why MTTR trends mislead.
• Prerequisites: Python 3.8 or later, standard library only; no account, cost or risk.
• Data: invented (UTC). INC-110 has a deliberate error: detected before impact started.

id,sev,impact_start,detected,acknowledged,mitigated,resolved
INC-101,SEV3,2026-07-02T09:10,2026-07-02T09:14,2026-07-02T09:16,2026-07-02T09:41,2026-07-02T10:05
INC-102,SEV2,2026-07-05T22:03,2026-07-05T22:05,2026-07-05T22:12,2026-07-05T22:38,2026-07-05T23:20
INC-103,SEV3,2026-07-11T14:20,2026-07-11T14:31,2026-07-11T14:33,2026-07-11T14:52,2026-07-11T15:10
INC-104,SEV3,2026-07-19T03:40,2026-07-19T03:42,2026-07-19T03:58,2026-07-19T04:20,2026-07-19T05:00
INC-105,SEV2,2026-07-28T11:00,2026-07-28T11:01,2026-07-28T11:03,2026-07-28T11:19,2026-07-28T12:30
INC-106,SEV3,2026-08-03T16:45,2026-08-03T16:47,2026-08-03T16:49,2026-08-03T17:30,2026-08-03T18:10
INC-107,SEV1,2026-08-09T01:12,2026-08-09T01:40,2026-08-09T01:43,2026-08-09T10:05,2026-08-09T13:30
INC-108,SEV3,2026-08-15T10:05,2026-08-15T10:06,2026-08-15T10:07,2026-08-15T10:25,2026-08-15T10:50
INC-109,SEV3,2026-08-22T19:30,2026-08-22T19:33,2026-08-22T19:41,2026-08-22T20:02,2026-08-22T20:30
INC-110,SEV2,2026-09-01T08:15,2026-08-31T23:59,2026-09-01T08:17,2026-09-01T08:49,2026-09-01T09:40
INC-111,SEV3,2026-09-14T13:00,2026-09-14T13:02,2026-09-14T13:04,2026-09-14T13:30,2026-09-14T14:00
INC-112,SEV3,2026-09-26T05:50,2026-09-26T05:55,2026-09-26T06:09,2026-09-26T06:35,2026-09-26T07:15

Example status: verified by execution on 2026-10-16 with Python 3.8.10 (python3 -I incident_metrics.py incidents.csv); the output below is copied from that run. Re-run after each revision, last on 2026-10-17 after re-wrapping the script: output identical.

"""Incident timing metrics with honest statistics (Python 3.8+, standard library only).

Usage: python3 incident_metrics.py incidents.csv
"""
import csv
import math
import random
import statistics
import sys
from datetime import datetime

ORDER = ["impact_start", "detected", "acknowledged", "mitigated", "resolved"]
INTERVALS = {  # name: (from, to)
    "TTD": ("impact_start", "detected"),      # time to detect
    "TTA": ("detected", "acknowledged"),      # time to acknowledge
    "TTM": ("impact_start", "mitigated"),     # time to mitigate (impact ends)
    "TTR": ("impact_start", "resolved"),      # time to resolve (incident closed)
}


def minutes(a, b):
    return (b - a).total_seconds() / 60


def load(path):
    good, rejected = [], []
    with open(path, newline="") as f:
        for row in csv.DictReader(f):
            ts = {k: datetime.fromisoformat(row[k]) for k in ORDER}
            if any(ts[x] > ts[y] for x, y in zip(ORDER, ORDER[1:])):
                rejected.append(row["id"])  # timestamps out of order: fix the record
                continue
            spans = {n: minutes(ts[a], ts[b]) for n, (a, b) in INTERVALS.items()}
            good.append({"id": row["id"], "sev": row["sev"], **spans})
    return good, rejected


def p90(values):
    return statistics.quantiles(values, n=10, method="inclusive")[-1]


def summarise(rows):
    print(f"{'metric':<6}{'n':>3}{'mean':>8}{'median':>8}{'p90':>8}{'max':>8}  (minutes)")
    for name in INTERVALS:
        v = [r[name] for r in rows]
        mean, median = statistics.mean(v), statistics.median(v)
        print(f"{name:<6}{len(v):>3}{mean:>8.1f}{median:>8.1f}"
              f"{p90(v):>8.1f}{max(v):>8.1f}")


def simulate(trials=10_000, n=15, median_min=45.0, sigma=1.2, true_change=0.0, seed=42):
    """Two quarters of n incidents from the same lognormal distribution; quarter 2
    is shortened by true_change (0.10 = 10 % shorter). Returns the share of trials
    in which the measured mean moved by 20 % or more, and the share in which it
    got worse."""
    rnd = random.Random(seed)
    mu = math.log(median_min)
    big, worse = 0, 0
    for _ in range(trials):
        q1 = statistics.mean(rnd.lognormvariate(mu, sigma) for _ in range(n))
        q2 = statistics.mean(rnd.lognormvariate(mu, sigma) * (1 - true_change)
                             for _ in range(n))
        change = (q2 - q1) / q1
        big += abs(change) >= 0.20
        worse += change > 0
    return big / trials, worse / trials


if __name__ == "__main__":
    rows, rejected = load(sys.argv[1])
    print(f"loaded {len(rows)} incidents; rejected (timestamps out of order): "
          f"{', '.join(rejected) or 'none'}")
    summarise(rows)
    worst = max(rows, key=lambda r: r["TTM"])
    all_ttm = statistics.mean(r["TTM"] for r in rows)
    rest = statistics.mean(r["TTM"] for r in rows if r is not worst)
    print(f"TTM mean without {worst['id']}: {rest:.1f} min "
          f"(one incident moves the mean by {all_ttm - rest:.1f} min)")
    big, worse = simulate(true_change=0.0)
    print(f"simulation, nothing changed:       mean moved >=20% in {big:.0%} of quarters")
    big, worse = simulate(true_change=0.10)
    print(f"simulation, real 10% improvement:  measured mean got worse "
          f"in {worse:.0%} of quarters")

Steps: save the CSV as incidents.csv and the script as incident_metrics.py; run python3 incident_metrics.py incidents.csv.

Expected output:

loaded 11 incidents; rejected (timestamps out of order): INC-110
metric  n    mean  median     p90     max  (minutes)
TTD    11     5.5     2.0    11.0    28.0
TTA    11     5.4     2.0    14.0    16.0
TTM    11    78.4    32.0    45.0   533.0
TTR    11   129.5    77.0    90.0   738.0
TTM mean without INC-107: 32.9 min (one incident moves the mean by 45.5 min)
simulation, nothing changed:       mean moved >=20% in 71% of quarters
simulation, real 10% improvement:  measured mean got worse in 42% of quarters

• Validation: INC-110 is rejected; mean TTM (78.4) is over twice the median (32.0); the simulation lines show on synthetic data what Davidovič found in real data (Metrics and KPIs interprets all of it).
• Troubleshooting: AttributeError on statistics.quantiles means Python older than 3.8; ValueError means a malformed timestamp.
• Teardown: delete the two files.

Status: not executed (needs a PagerDuty account and phones); follows the Events API v2 documentation as of 2026-10-16.

• Objective: prove that an alert reaches a human, escalates when ignored and closes cleanly.
• Prerequisites: a PagerDuty trial account; the reference Terraform applied to a test service whose rota is you and a volunteer, with -var layer_start= a few minutes ago; curl; the mobile app. It rings real phones: warn the volunteer. Optional step 5 needs a test Prometheus with a Watchdog rule (ObservabilityOps Lab 3), Alertmanager with the heartbeat route, and a heartbeat service.

Steps

1. Load the key into a variable, never onto the screen (-raw prints it in plain text):^[44] PD_ROUTING_KEY="$(terraform output -raw checkout_routing_key)".
2. Send a trigger event, with the body on standard input (--data-binary @-): other local users can see command-line arguments in the process list, and curl’s manual says credentials “should be retrieved from a file instead or similar”.^[45]

Example status: syntax-checked by the Atlas validator (bash -n); not executed.

curl -sS -o response.json -w '%{http_code}\n' \
  -X POST https://events.pagerduty.com/v2/enqueue \
  -H 'Content-Type: application/json' --data-binary @- <<EOF
{"routing_key":"${PD_ROUTING_KEY}","event_action":"trigger",
 "dedup_key":"drill-paging-001",
 "payload":{"summary":"DRILL: paging path test, no action needed",
 "source":"drill.example.com","severity":"critical"}}
EOF
cat response.json

3. Do not acknowledge. Note when your phone rings, then the volunteer’s (policy: 5 minutes later).
4. Acknowledge, then resend with "event_action":"resolve" and the same dedup_key.
5. Optional: stop the test Prometheus (Watchdog is a Prometheus alerting rule) and confirm the heartbeat service alarms within its grace period.

• Expected output: HTTP 202 for each call;^[46] escalation to the volunteer after about 5 minutes; a resolved incident; with step 5, a heartbeat alarm.
• Validation: record the times; escalation should land within a minute of the policy.
• Troubleshooting: on 400 (Bad Request), check that the JSON is valid^[46] and has every required field;^[14] 429 means retry with backoff;^[46] 202 but no call points to contact methods or notification rules; a resolve that does nothing used another routing key or dedup_key.^[14]
• Teardown: terraform destroy; remove test contact methods; restart Prometheus; unset PD_ROUTING_KEY; delete response.json; tell the team the drill is over.

Status: an exercise design; not run for this page. It adapts Google’s “Wheel of Misfortune”, where a game master describes an outage and two on-call engineers say what they would do,^[47] to incident command, following the Workbook’s advice for drills: “Use real tools as much as possible”.^[8]

• Objective: practise command, communication and two handoffs under time pressure.
• Prerequisites: 60 minutes; a facilitator; IC trainee, ops lead, comms lead, scribe; a drill channel and mock status page; “DRILL” before every message.
• Scenario: checkout errors reach 30 % in one region 15 minutes after a configuration push.

• Expected output: a timeline, three or more on-cadence updates, a decision log, a recorded security handoff, an IC handover.
• Validation: the drill scorecard in Templates.
• Troubleshooting: if the IC starts debugging, pause and ask “who is commanding?”. Teardown: archive the drill channel; file actions for gaps.

Three published incidents, each sourced from the operator’s own report:

GKE. Cluster creation failed in Europe for “6 hours and 40 minutes”. Before the London-to-Seattle shift change the on-call passed command to a more experienced IC, who appointed ops and comms leads; 41 people joined the chat and “seven IMAG task forces” formed. Rolling images back once the fault’s location was known would have mitigated it “by 10 a.m.” rather than about noon.^[8]

Amazon S3. An input to a playbook command was “entered incorrectly”, removing too much capacity and forcing a full restart. Until 11:37 AM PST AWS could not update its Service Health Dashboard “because of a dependency the SHD administration console has on Amazon S3”; it used Twitter and banner text, then moved the console to run across multiple regions.^[40]

Atlassian. The deletion also removed customer contact data, so many customers could not file tickets; “hundreds of engineers and customer support employees” worked without tooling for “sub-streams”. Support acknowledged the incident within an hour and Statuspage was updated at 09:03 UTC on 5 April, but the first broad public message, a tweet, came 41 hours after the deletion began. Atlassian’s review: “We did not have an incident communications playbook thorough enough to deal with this level of complexity”; it committed to a large-scale playbook with simulations.^[48]

This page defines incident timings for the Atlas. Each is the difference between two timestamps, and tools start clocks differently.

flowchart TB
  T0["Impact starts: TTD, TTM, TTR clocks start"] -->|"TTD"| T1["Detected"]
  T1 -->|"TTA"| T2["Acknowledged by a human"]
  T2 --> T3["Declared: IC, severity"]
  T1 -.->|"to declare"| T3
  T3 -->|"to first update"| T4["First public update"]
  T4 --> T5["Mitigated: impact ends, TTM stops"]
  T5 --> T6["Resolved: closed, TTR stops"]
  T6 --> T7["Review, actions tracked"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class T0 risk
  class T1,T2,T3,T4,T5,T6,T7 core

Which “R”? Atlassian notes that MTTR “potentially represents four different measurements. The R can stand for repair, recovery, respond, or resolve”.^[49] Google’s “time to repair” runs from alerting to mitigation;^[7] DORA’s failed deployment recovery time (renamed in 2023 from time to restore service) covers only change-caused failures (DevOps).^[50] State start, end and population: “median time to mitigate, impact start to mitigation, SEV1–SEV2, last 90 days”.

Why MTTx misleads. Štěpán Davidovič’s Google report simulated improvements on real incident data, whose durations “follow a positively skewed distribution”. Splitting a full year of incidents (103 to 609 per company) into two samples and truly shortening every incident by 10 %, “38% of the simulations had the MTTR difference fall below zero for Company A, 40% for Company B, and 20% for Company C”; with no change at all there was “a 19% chance” of a half-hour improvement for Company A. His conclusion: “The default position should be to reject MTTx metrics” for judging trends or changes, except for large homogeneous populations or dramatic shifts; medians and 95th percentiles fared no better.^[51] The 2022 VOID report, as reported by InfoQ, found most incidents resolve “in under two hours” and no correlation between duration and severity.^[52]

Worked calculation (Lab 1 data, illustrative).

• Mean TTM = 862 ÷ 11 = 78.4 minutes; median 32. INC-107 alone adds 45.5 minutes; “MTTM = 78” describes no real incident.
• Mean TTA = 59 ÷ 11 = 5.4 minutes; median 2. The slow acknowledgements (16 and 14 minutes) are INC-104 at 03:42 and INC-112 at 05:55: both at night, both beyond the 5-minute escalation step. Only case-level data points at the night rota.
• Workload: 12 incidents happened (only INC-110’s record is bad). Google’s average of about 6 hours per incident covers root-cause analysis, remediation and follow-up such as postmortems,^[17] so up to about 72 hours a quarter; less where SEV3 reviews are skipped.

Report instead: a dot plot of TTM naming long incidents, SLO impact, and conformance (declaration, updates and action closure on time). Counts by severity are context, not targets: Nash lists “count of severity types, total incident count” with MTTR as shallow data, while Huerta Granda sees such numbers as starting points for deeper study.^[52] Target only conformance; bonuses on incident counts or MTTR reward not declaring.

• Access. Logged, time-limited elevation, not shared admin accounts (AccessOps).
• Privacy. Minimise customer data in records and chat; set retention; check what AI assistants store (PrivacyOps).
• Regulatory clocks (EU, as of October 2026). An outage can be reportable without an attacker. NIS2 defines an incident as “an event compromising the availability, authenticity, integrity or confidentiality” of data or services, counts one as significant if it causes or can cause “severe operational disruption”, and requires an early warning within 24 hours and a notification within 72 hours of becoming aware, then a final report within one month of that notification.^[34] For financial entities, the Digital Operational Resilience Act (EU DORA, unrelated to the DORA delivery research) covers incidents with an “adverse impact on the availability” of data or services;^[35] its standards require an initial notification within four hours of classifying an incident as major and no later than 24 hours after becoming aware, an intermediate report within 72 hours of that, and a final report within one month of the latest intermediate report.^[53] So record when you became aware and when you classified, and agree with compliance who classifies. Scope depends on sector and national law; take legal advice (ComplianceOps). Other jurisdictions are not covered here.

People are the main cost. On-call load limits (Google’s 25 % cap, minimum rota sizes, incidents per shift, capped pay) are taught in SRE; mastery question 5 applies them. Big incidents are expensive meetings: 15 people on a 3-hour call is 45 person-hours, so crowd control is cost control. Sustainability here means avoiding burnout; the environmental footprint of incident tooling is not material.

• Drills: role-play,^[47] monthly paging drills (Lab 2) and, at scale, simulations like those Atlassian adopted.^[48]
• Record audits: sample records monthly for out-of-order timestamps (like INC-110), missing roles and late reviews.
• Leading indicators: pager load over budget, overdue actions, missed drills, on-call survey results.
• Near misses: make them easy to note; Nash warns that formally classifying them repeats the “shallow data” trap.^[52]

Trigger: customers report impact that nothing paged, or the heartbeat service reports the Watchdog alert has stopped.

1. Declare. A blind detection path is an incident; someone watches key dashboards until paging is proven.
2. Is the alert firing at the source? Prometheus exposes active alerts as the ALERTS series and via its HTTP API.^[54]
3. Did Alertmanager receive and route it? Query its alerts; test the routing tree with the alert’s labels.^[55]
4. Are notifications failing? The standard Alertmanager alert compares alertmanager_notifications_failed_total with alertmanager_notifications_total.^[56]
5. Did the paging service accept it, and was anyone on call? Send a Lab 2 test event and expect 202; check schedule gaps, overrides and contact methods.

Example status: syntax-checked by the Atlas validator (bash -n); commands follow the Alertmanager and Prometheus documentation; not executed. Host names are placeholders.

# 2. Firing alerts at the source (Prometheus HTTP API)
curl -s http://prometheus.internal.example:9090/api/v1/alerts | head -c 2000

# 3. What Alertmanager holds, and where a page-worthy checkout alert would go
amtool --alertmanager.url=http://alertmanager.internal.example:9093 \
  alert query severity=page
amtool config routes test --config.file=alertmanager.yml \
  --verify.receivers=pagerduty-checkout severity=page team=checkout

# 4. Notification failures per integration (PromQL, paste into the Prometheus UI):
#    rate(alertmanager_notifications_failed_total{integration="pagerduty"}[15m])

# Remediation example: a rotated PagerDuty routing key was not updated in the secret file
sudo install -m 0400 -o alertmanager /secure/new-pd-checkout-key \
  /etc/alertmanager/secrets/pd-checkout
curl -s -X POST http://alertmanager.internal.example:9093/-/reload

Remediation: fix the broken link (route, receiver, credential, schedule or contact method), reload Alertmanager (SIGHUP or a POST to /-/reload),^[36] and prove the path with a test event before ending the manual watch. Prevention: keep the Watchdog heartbeat;^[42] run amtool config routes test --verify.receivers in CI, which “returns error code 1 on mismatch”;^[55] repeat the paging drill monthly and after rota or tool changes.

• “The most senior engineer should lead.” PagerDuty makes the IC “the highest ranking individual on any major incident call, regardless of their day-to-day rank”;^[18] train several and rotate the role, as Google advises.^[1]
• “Declaring is admitting failure.” Punish incident counts and people stop declaring.
• “Find the root cause before acting.” Mitigation needs only the fault’s location.^[8]
• “Root cause: human error.” Cook: “Post-accident attribution to a ‘root cause’ is fundamentally wrong”, and “Hindsight biases post-accident assessments of human performance”.^[10] If you use Atlassian’s “Five Whys”,^[20] follow several branches and stop at system conditions, not at a person.
• “More action items, better review.” The Workbook criticises actions that are “mostly mitigative”, vague and equally prioritised;^[57] PagerDuty: “Generally we only want to create things that are P0/P1’s”.^[23] Etsy goes further: “The goal of a debriefing is not to produce recommendations or remediation items”.^[33]
• “Everyone should join the call.” Observers raise coordination costs;^[30] give them the updates channel.

Any team running a service for others needs some incident process; the question is how much ceremony. Use a full process when customers depend on the service in real time and failures often involve several teams; reportable incidents add reporting duties to any process, and a 24/7 promise adds a night rota. Keep it minimal (tier 0 or 1 below) for internal tools run in business hours with rare, single-team incidents; Google notes that the full Incident Command System “may be more than you need”.^[8] Do not build it yourself when a managed service provider or an ITIL service desk already owns the response (ITOps); design the interface instead. Hidden responsibilities: paying for coverage, training commanders, keeping templates approved, holding contact lists outside production, and follow-up work that competes with features.

%%{init: {"flowchart": {"wrappingWidth": 100, "nodeSpacing": 8, "rankSpacing": 28, "padding": 6}}}%%
flowchart TB
  Q1{{"Real-time dependants?"}}
  Q1 -- "No" --> T0["Tier 0: tickets, checklist"]
  Q1 -- "Yes" --> Q2{{"Cross-team, or 2+ SEV1-2 a month?"}}
  Q2 -- "No" --> T1["Tier 1: one rota, severity, template"]
  Q2 -- "Yes" --> Q3{{"Often over 5 responders?"}}
  Q3 -- "No" --> T2["Tier 2: IC, comms lead, status page"]
  Q3 -- "Yes" --> T3["Tier 3: adds deputy, liaisons, sub-teams"]
  T1 --> A1{{"24/7 promise?"}}
  T2 --> A1
  T3 --> A1
  A1 -- "Yes" --> R1["Add night rota, IC cover"]
  A1 -- "No" --> A2{{"Reportable or regulated?"}}
  R1 --> A2
  T0 --> A2
  A2 -- "Yes" --> R2["Add regulatory clocks"]
  A2 -- "No" --> RQ["Review the tier quarterly"]
  R2 --> RQ
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q1,Q2,Q3,A1,A2 decision
  class T1,T2,T3,R1,R2 core
  class T0,RQ neutral

The thresholds are this Atlas’s assumptions, not industry standards. Rota sizing is in SRE; the clocks are in Quality, security, privacy and compliance.

The Atlas’s proposed model, not an official standard. Move up when the exit criterion is met.

Dependencies: service ownership (ServiceOps), alerts worth paging on (ObservabilityOps), chat tooling (ChatOps). An engineering manager owns month one; an incident-programme owner the rest.

On-call preflight (start of shift)

• A test page reaches me on two channels (app and phone call)
• I have read the handover note
• I know the secondary, the IC on call and the escalation path
• Laptop, network fallback and break-glass access work

IC: the first ten minutes

• Announce “I am the IC”; name ops lead, comms lead and scribe
• Set severity (if unsure, the higher one) and the next update time
• Size up: impact, start time, recent changes, regions
• Check for an attacker or site loss before choosing a mitigation; hand off or escalate if so
• Ask for a generic mitigation and its side effects
• Give every task one owner and a time-box
• Verify each action worked; if not, switch to the backup option and size up again

Closing

• Recovery confirmed by data; final update states any data loss
• Review owner named and meeting booked
• Temporary changes listed for reversal
• Timestamps complete and in order

Quarterly governance audit

• Pager load within budget for every rota
• Priority actions closed on time; oldest open action named
• Drills held, including one handoff drill
• Severity policy, contact lists and templates reviewed

The templates below are this Atlas’s own; PagerDuty’s Apache 2.0 documentation can also be adapted with its notices retained.^[11]

Declaration message

INCIDENT DECLARED — {SEV} — {short title}
IC: {name} | Ops lead: {name} | Comms: {name} | Scribe: {name}
Impact (current best estimate): {who / what / where}
Working channel: #inc-{id} (responders) · Updates: #inc-{id}-updates (everyone)
Call: {bridge link} · Record: {document link}
Next update: {time, UTC}

External status updates

Investigating: We are investigating {symptom} affecting {product/feature} in {region}.
               Next update by {time UTC}.
Identified:    We have identified the cause of {symptom} and are applying a fix.
               {Workaround, if any.} Next update by {time UTC}.
Monitoring:    A fix has been applied and {feature} is recovering. We are monitoring.
               Next update by {time UTC}.
Resolved:      {Feature} has been fully restored since {time UTC}.
               {No data was lost | Data affected: ...}. A summary will follow.

IC handover script

Outgoing IC: "Status: {SEV}, impact {…}, current hypothesis {…},
  actions in flight {who/what/until when}, next update due {time}. Open risks: {…}.
  You're now the incident commander, okay?"
Incoming IC: "I'm now the incident commander."
(Scribe records the time; IC posts it in both channels.)

Post-incident review skeleton

Title / ID / severity / review owner / meeting date
Summary (5 lines, plain language)
Impact: who, what, how long, data affected, SLO budget consumed
Timeline: timestamped facts with links to graphs and messages
Contributing factors: technical, process, organisational (no names as causes)
What went well / where we were lucky
Response review: declaration, roles, communication cadence, handovers
Actions: ID | kind (prevent/mitigate/detect/process) | one owner | priority | due | link
Open questions and follow-up learning

Generic-mitigation runbook (one per mitigation per service)

Mitigation: {rollback | drain | feature kill | failover | load shedding} - {service}
Owner: {team} | Last tested: {date; drill or real use} | Approver: {role, if any}
Use when: {symptoms; fault located in ...}
Do not use when: {schema or data change in release; attacker suspected: ask ResponseOps}
Preconditions: {previous version available; spare capacity >= 100 % of peak elsewhere}
Side effects: {features lost; data written since ...; what customers notice}
Steps: 1. {command or console path, expected output}  2. {...}
Verify: {graph or query showing the impact stopped, within N minutes}
Undo: {how to reverse the mitigation}
Record: {post start, end and result in the incident channel}

Drill scorecard (Lab 3)

Is “IncidentOps” different from incident management? Not in any documented way: the uses we found relabel it or name a product.^[5][4] When a vendor or job advert uses it, ask which practices it means.

We are six engineers. Do we need an incident commander? You need the role, not an extra person: the on-call holds every role until a second person joins, then hands one away, usually communications.

What is the difference between severity and priority? Severity describes impact and sets the response (who is mobilised, how often you communicate); priority orders work such as follow-up actions. A minor incident can create an urgent action.

Should we stop measuring MTTR? Stop using it to judge trends, tools or teams;^[51] keep the timestamps and report as in Metrics and KPIs. Dramatic shifts remain visible, small improvements do not.

Should post-incident reviews be published? Internally, widely: sharing is how other teams learn.^[58] Externally, a customer summary for significant incidents, as AWS and Atlassian did,^[40][48] without security-sensitive detail or personal data.

• Incidents across company boundaries (established problem, immature practice). STELLA notes that SaaS dependence makes coordination “across organization (and company) boundaries” more often necessary;^[15] agree in advance how suppliers join your incidents (supplier path: End-to-end workflow).
• Learning from incidents (active research): in-depth study of selected incidents over shallow metrics.^[15][52]
• Regulatory reporting (established law, maturing practice): timestamps become evidence.^[34][53]

Assumptions (invented): failures stay at 30 % until mitigation; a rollback has a 5 % chance of failing and costing 30 more minutes; the patch lands uniformly between 20 and 60 minutes (mean 40) and has a 25 % chance of failing and costing 30 more.

Why B is attractive and wrong here. It is precise, keeps the fraud rule and its author is confident. But the estimate comes from the person under most pressure, its range is wide, and delay costs every customer every minute: B carries about 3.5 times the expected impact. Hence the Workbook’s “prioritize mitigation above all else”.^[8]

Decision. “We’re proceeding with the rollback. Are there any strong objections?”^[18] Compliance agrees a temporary control (manual review of high-value orders in that market); the patch ships later as version 43.

When B would be right: the rollback is unsafe (it reverses a data or schema change), or the fix is built, verified and minutes away. Uncertainty: the numbers are guesses; the value is an explicit, fast comparison.

You should now be able to explain what IncidentOps means and where it ends; apply declaration criteria and a severity policy; run the command loop and a communication cadence; design rotas, escalation and a paging heartbeat; evaluate incident metrics; and decide how much process a team needs.

1. An on-call engineer has worked alone for 50 minutes on a partial checkout failure; customers complain publicly. What now?

2. After a new tool arrived, mean time to resolve fell from 95 to 70 minutes in a quarter. Does that prove the tool works?

3. During a SEV-1 a senior executive starts directing engineers. You are the IC. What do you say?

4. A log suggests an unknown administrator logged in an hour before the outage. Your next three actions (Atlas proposal)?

5. Six engineers want a 24/7 rota with primary and secondary on week-long shifts. Is that within Google’s 25 % guideline?

6. Checkout errors reach 15 % during your cloud provider’s regional problem, and its public status page is still green. What now?

“IncidentOps” is a new label on an old discipline, so use the discipline’s name and evidence. That evidence is unusually consistent: Google, PagerDuty and Atlassian converge on the moves in the cheat sheet. Research on complex failures adds two warnings: coordination becomes expensive under pressure, and the comfortable numbers, MTTR and “root cause”, say less than they seem to.

The test of an incident programme: does a tired engineer at 3 a.m. know within five minutes who is in charge, where to talk and what to tell customers, and does the review a week later change something? Build for that moment and rehearse it.

Each footnote gives the date its source was last opened; claim-level notes are in the evidence ledger (research/incidentops-ledger.md).

Changelog. 2026-10-16 — first draft (6 diagrams, 3 labs, Lab 1 executed); term status: overlapping synonym. 2026-10-16 — r1 fact check (reviews/incidentops-r1.md): 0 critical, 2 major, 21 minor, all addressed. 2026-10-17 — r2 after the independent score (reviews/incidentops-score.md, 89): tooling table repaired; supplier-incident path, tool criteria, mastery question 6 and four sources added; action deadlines tied to action priority; all figures redrawn to fit phones; Lab 1 re-wrapped and re-run (reviews/incidentops-r2-response.md). 2026-10-17, post-score polish (reviews/incidentops-polish.md): Lab 2’s 400 cited to the Events API overview; origins chronology corrected; STELLA’s “mixed” scoped to chat automation; figure 6 links tier 0 to the regulatory add-on; two recap repeats cut.