--- title: "AIOps" slug: aiops category: data term_status: established evidence_status: verified summary: "AIOps (artificial intelligence for IT operations) applies statistics, machine learning and language models to operational events and telemetry to cut alert noise, group related alerts into incidents, flag anomalies, rank probable causes and run guarded fixes." audience: [NOC and ITOps leads, SREs and on-call engineers, Observability and platform engineers, Tool buyers and engineering leaders, Beginners] last_verified: 2026-10-17 key_takeaways: - "AIOps means AI for IT operations, not operations for AI. Gartner's glossary names event correlation, anomaly detection and causality determination; according to vendors quoting the paywalled guide, Gartner's 2025 market guide renamed its AIOps-platforms market event intelligence solutions." - "Most noise reduction comes from unglamorous steps: a label contract, deduplication, grouping, guarded inhibition, topology and change data. Add machine learning where labelled incidents show that rules fail." - "Judge correlation by precision and recall against labelled incidents, not by compression. In this page's lab a plain time window compressed alerts 5.5 times while merging unrelated incidents." - "Topology is a prerequisite, not an extra. Removing one dependency edge cut the lab's pairwise recall from 0.93 to 0.53." - "Anomaly detectors need context. A seasonal baseline caught all four injected faults but raised 34 false alerts, 32 of them on an undeclared holiday; one line of calendar data removed all 32." - "Keep people in charge of production changes. In ITBench (ICML 2025), agents built on early-2025 models resolved 11.4% of SRE scenarios; newer diagnosis-only tests score higher but measure something narrower. Automate only allow-listed, reversible, rate-limited actions." - "Detection engines come and go (Azure Anomaly Detector retired on 1 October 2026; Amazon DevOps Guru closes to new customers on 29 October 2026), so keep your rules, labels and evaluation harness portable." related: [observabilityops, incidentops, itops, aiops-for-security, sre, serviceops, mlops, automationops] --- ## What AIOps is ### In 30 seconds **AIOps** (artificial intelligence for IT operations) applies statistics, machine learning and, more recently, large language models (LLMs) to the events and telemetry that IT systems emit. The aim is that people see fewer, better-grouped alerts, reach a probable cause sooner, and let safe, repetitive fixes run on their own. Gartner's glossary keeps the definition narrow: AIOps "combines big data and machine learning to automate IT operations processes, including event correlation, anomaly detection and causality determination".[^gartner-glossary] **In one line: AIOps turns a flood of operational signals into a short list of actionable incidents, and it is only as good as the labels, topology and feedback you give it.** This page is for NOC (network operations centre) and ITOps leads facing alert floods, SREs and on-call engineers, the engineers who feed the event pipeline, and buyers of "AIOps" or "event intelligence" platforms. Beginners can read the first two parts and the [cheat sheet](#executive-recap-and-cheat-sheet). Engineers should go to the [labs](#hands-on-labs), and leaders to the [decision framework](#decision-framework) and the [judgement call](#a-judgement-call-buy-a-platform-now-or-fix-alert-hygiene-first). ::: note Term status: established, but contested The label has a decade of continuous use. Gartner research used "AIOps" in July 2016 for what its summary calls "Algorithmic IT operations platforms",[^gartner-2016] and its glossary now expands it as artificial intelligence for IT operations.[^gartner-glossary] Forrester's Wave for AIOps platforms (second quarter of 2025) covered 10 providers on 26 criteria, according to a Dynatrace press release,[^forrester-wave] and there is a peer-reviewed literature.[^dang2019][^notaro2020] The contest is about breadth. In its 2025 Market Guide for Event Intelligence Solutions, Gartner wrote, as quoted by Selector (a vendor named in that guide), that the term "has been widely adopted by vendors across multiple IT operations markets, often without a clear definition of, or consensus on, what it entails". The same quotation says that renaming the market "from AIOps platforms to EIS serves to direct focus to the intended domain and set of use cases".[^selector2026] Academic surveys also complain of inconsistent terminology.[^notaro2020][^remil2024] This page uses the narrow core: machine analysis of operational events for correlation, anomaly detection, probable cause and guarded remediation. ::: ### At a glance | Question | Answer | |---|---| | Goal | Fewer, better-grouped, actionable incidents; faster triage; automation that is measured and reversible | | Typical users | NOC and ITOps teams, SREs, on-call engineers, observability and platform teams, IT service management (ITSM) process owners | | In scope here | Ingesting and normalising events, deduplication, grouping and inhibition, topology and change correlation, anomaly detection, ranking probable causes, guarded remediation, and evaluation (precision and recall are defined here for the whole Atlas) | | Out of scope | Telemetry pipelines → [ObservabilityOps](../observabilityops/); incident roles and communications → [IncidentOps](../incidentops/); the same techniques against attackers → [AIOps for Security](../aiops-for-security/); training pipelines → [MLOps](../mlops/); ITIL processes → [ITOps](../itops/) | | Prerequisites | Working alert rules, named service owners, a list of services and their dependencies, a change log | | Key outputs | A label contract, correlation and inhibition rules, a topology file, an evaluation scorecard, a remediation allow-list | | Term status | Established label with contested breadth (see the note above) | ### The problem it solves When one shared component fails, everything that depends on it complains. Alertmanager's documentation gives the classic example: during a network partition, hundreds of service instances each alert because they cannot reach the database, although one page would do.[^am-concepts] Researchers call this an **alert storm** ([Scenario 1](#use-cases-and-scenarios) has a documented case).[^zhao2020] Noise has a human cost. Google's SRE book states it as a philosophy of paging: "I can only react with a sense of urgency a few times a day before I become fatigued."[^sremon] In Grafana Labs' 2026 survey, 30% of respondents named alert fatigue as the biggest obstacle to faster incident response. The survey is a vendor's, and its respondents chose themselves, so read it as a signal rather than a measurement.[^grafana2026] Without event intelligence: - **Duplicate paging.** Several teams are woken for one fault, and each starts its own investigation. - **Symptom chasing.** Responders work on the loudest alert, not the failing dependency. - **Hidden incidents.** A second, unrelated failure disappears inside a storm. - **Manual toil.** The same safe fix (rotate logs, restart a stuck worker) is typed by hand at 3 a.m. ::: example Before and after (illustrative scenario) Before: a database failover makes 40 alerts fire across six services in four minutes. Three teams are paged, and two spend twenty minutes proving their service is a victim. After: the 40 alerts arrive as **one** incident that names the database as the probable cause, shows a configuration change deployed eight minutes earlier, and pages the database owner. The other teams are told their symptoms are linked, and the original alerts remain queryable. ::: ### What AIOps is not - **Not operations *for* AI.** Running models and LLM applications is [MLOps](../mlops/) and [LLMOps](../llmops/). AIOps applies AI *to* IT operations. - **Not observability.** Collecting, shaping and storing metrics, logs and traces belongs to [ObservabilityOps](../observabilityops/). AIOps consumes that data. - **Not incident management.** Roles, severity, communication and post-incident reviews belong to [IncidentOps](../incidentops/). AIOps decides what reaches them and with what context. - **Not security analytics.** Detecting attackers uses the same techniques against an adversary who adapts. That is [AIOps for Security](../aiops-for-security/). - **Not autonomous operations.** "Self-healing" is a direction, not a state you can buy (see [NoOps](../noops/)). - **Not a replacement for good alert rules.** Google's SRE book (2016) says: "We avoid 'magic' systems that try to learn thresholds or automatically detect causality."[^sremon] - **Not root-cause certainty.** Tools rank *probable* causes from a limited menu (Datadog's Watchdog names four types).[^dd-rca] ## Where AIOps comes from and where it sits ### Origins and evolution **Event consolidation is older than the label.** A 2024 review describes the "traditional methods" that AIOps set out to improve as "reliant on manual tasks and rule-based approaches".[^remil2024] IBM describes its Tivoli Netcool/OMNIbus event console as consolidating information from separate network management platforms and automating the processing of alerts.[^ibm-omnibus] What changed in the 2010s was scale (cloud, microservices, many monitoring tools) and the hope that learning could replace some hand-written rules. | When | Milestone | Why it matters | |---|---|---| | July 2016 | A Gartner research note by Colin Fletcher on "AIOps Platforms" opens its summary with "Algorithmic IT operations platforms"[^gartner-2016] | An early use of the AIOps label, as a platform market | | 2019–2021 | Microsoft researchers present AIOps challenges at ICSE;[^dang2019] a mapping study finds 62% of contributions treat failure-related tasks;[^notaro2020] a white paper calls the field "emerging"[^bogatinovski2021] | A research community forms around failure management | | 2020 | Peer-reviewed industry papers on alert storms[^zhao2020] and on incident management at Microsoft[^chen2020] | Evidence from production systems, not only demos | | 2023 | Microsoft studies LLMs for root-causing more than 40,000 incidents;[^ahmed2023] Dell completes its acquisition of Moogsoft on 17 September 2023[^moogsoft-dell] | LLMs enter the field; vendors consolidate | | 2025 | Elastic agrees to acquire Keep, an open-source AIOps company (7 May);[^elastic-keep] Gartner renames its market event intelligence solutions (as quoted by Selector);[^selector2026] Forrester publishes its AIOps Wave[^forrester-wave] | The label is contested, but the category persists | | 2025–2026 | Agent benchmarks such as AIOpsLab and ITBench;[^aiopslab][^itbench] managed anomaly-detection services retire (see [tooling](#tooling-landscape-as-of-october-2026)) | Agents are evaluated; engines churn | Gartner's 2026 market guide (5 May 2026)[^pd-gartner] defines event intelligence solutions, as quoted by BigPanda, as "tools that apply artificial intelligence (AI) and data analytics to augment, accelerate, and automate responses to signals or events detected from digital services".[^bigpanda2026] ### Glossary and mental model | Term | Meaning on this page | Example | |---|---|---| | Event | Any recorded change of state that may matter to operations | "Disk at 85%" | | Alert | An event that a rule judged worth someone's attention | `HighErrorRatio` firing for `checkout-api` | | Incident | An unplanned interruption or degradation that people manage as one piece of work | "Checkout failing in eu-1" | | Fingerprint | A stable identity for an alert, usually a hash of its labels | Repeats share one | | Deduplication | Collapsing repeats of the same fingerprint into one alert | 20 re-notifications become 1 | | Enrichment | Adding owner, service, tier, runbook and recent changes to an event | `team=payments`, a runbook link | | Grouping (correlation) | Putting alerts that share a cause into one incident | Database alert plus its callers' error alerts | | Inhibition | Muting notifications for some alerts while another alert is firing[^am-concepts] | "Cluster unreachable" mutes per-service alerts | | Silence | A time-limited, matcher-based mute that a person sets[^am-concepts] | Planned maintenance | | Topology | Which services, hosts and networks depend on which | `api → db` | | Change event | A record that something was altered: a deploy, configuration change or feature flag | `checkout-api v2026.10.16-3` | | Anomaly | A value or pattern unlike the learned normal one | Night traffic at 2.5 times its usual level | | Baseline and seasonality | The learned normal, including daily and weekly cycles | Lower weekend traffic | | Probable cause | The component or change ranked most likely to explain a group | "orders-db", "deploy v3" | | Ground truth label | What responders confirm at close: which alerts belonged together, what the cause was | "Grouped correctly: yes" | | Precision and recall | The share of a system's flags or merges that were right, and of true cases it found; F1 is their harmonic mean ([Metrics](#metrics-and-kpis)) | P = 0.58, R = 1.00 | | TP, FP, FN | True positive (a right flag or merge), false positive (a wrong one), false negative (a miss) | Merging two incidents: FP | | Guarded remediation | An automated fix that is allow-listed, rate-limited, verified and reversible | Rotate logs on a full volume | | SLO and burn rate | A service level objective is a target for a user-visible measure; "Burn rate is how fast, relative to the SLO, the service consumes the error budget"[^sreslo] (see [SRE](../sre/)) | 99.9% of checkouts succeed | | CMDB | Configuration management database: the ITSM record of components and their relations | A (often stale) topology source | | MTTx (MTTA, MTTR) | Mean time to acknowledge, repair or resolve; see the caveats in [IncidentOps](../incidentops/#metrics-and-kpis) | MTTR | **Mental model: a funnel with a feedback loop.** Signals become events, events become alerts, alerts become incidents, and people confirm or correct the grouping. Every stage throws information away; the craft is to discard noise, keep signal, and know by measuring which of the two you lost. ::: figure The event funnel: each stage discards information, and feedback shows whether it was noise ```mermaid flowchart TB S["Signals: metrics, logs, traces, checks"] --> E["Events (thousands)"] E --> D["Deduplicate and normalise"] D --> A["Alerts (hundreds)"] A --> G["Group, inhibit, enrich"] G --> I["Incidents (a few)"] I --> R["People act and close"] R --> L["Labels: right group? right cause?"] L -.->|"monthly scorecard: tune rules and models"| G L -.->|"fix noisy rules"| A classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 class S,E,A,I neutral class D,G,R core class L decision ``` Text equivalent: signals become thousands of events; deduplication and normalisation leave hundreds of alerts; grouping, inhibition and enrichment leave a few incidents. People act, close them and record whether grouping and cause were right. Those labels feed a monthly scorecard that tunes grouping, and point at noisy alert rules to fix. ::: ### How AIOps relates to other Ops AIOps sits between the teams that produce telemetry and the teams that respond to incidents. Where a boundary below is the Atlas's own choice rather than an industry standard, it is marked as an **editorial distinction**. | Neighbour | Shared ground | What AIOps adds | Read it for | |---|---|---|---| | [ObservabilityOps](../observabilityops/) | Telemetry, labels, alert rules | Correlation and anomaly models over that data | Pipelines, cardinality, cost of telemetry | | [IncidentOps](../incidentops/) | Incidents, on-call, MTTx | What reaches responders, grouped and enriched | Roles, severity, communications, MTTR caveats | | [SRE](../sre/) | Alert quality, toil | ML grouping on top of SLO-based alerting | SLOs, error budgets, burn-rate alerts | | [ITOps](../itops/) | Event, incident and change processes, the CMDB | Machine analysis inside those processes | ITIL practices, change enablement | | [ServiceOps](../serviceops/) | Service catalogue and ownership | Uses ownership and tiers to route and rank | Service ownership model | | [AIOps for Security](../aiops-for-security/) | Same techniques | Editorial distinction: faults here, adversaries there | Base rates, adversarial evaluation | | [MLOps](../mlops/) | Model lifecycle | Applies models; owns no training platform | Registries, drift, retraining | | [AutomationOps](../automationops/) | Remediation runbooks | Decides when an automated action may fire | Automation safety in general | | [ChatOps](../chatops/) | Where incidents surface | Context posted into the channel | Chat interfaces and bots | ::: figure Where AIOps sits: between telemetry and response, beside security analytics ```mermaid flowchart TB OBS["Telemetry (ObservabilityOps)"] --> AIO CHG["Changes and deploys"] --> AIO TOPO["Topology and owners (ServiceOps, CMDB)"] --> AIO AIO["AIOps: correlate, detect, rank, guard"] AIO --> INC["IncidentOps: people respond"] AIO --> AUTO["AutomationOps: safe fixes"] INC -.->|"labels"| AIO AIO -.-|"editorial boundary"| SEC["AIOps for Security"] AIO -.-|"model practices"| ML["MLOps"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class AIO core class OBS,CHG,TOPO,INC,AUTO,ML neutral class SEC risk ``` Text equivalent: telemetry (ObservabilityOps), changes and deploys, and topology and owners (ServiceOps, CMDB) flow into AIOps, which correlates, detects, ranks and guards. It sends incidents to IncidentOps and allow-listed fixes to AutomationOps; IncidentOps returns labels. An editorial boundary separates AIOps for Security; MLOps supplies model-lifecycle practices. ::: ## Principles and operating model ### Charter and principles (this Atlas's proposal) No body publishes an official AIOps manifesto; this charter derives from the evidence on this page. **Mission:** reduce the attention that operational events cost people, without hiding a single incident that matters, and make every automated decision explainable and reversible. 1. **Rules before learning.** Start with deduplication, explicit grouping, inhibition and SLO-based alerting (Google's stance, quoted above). Add learning where labelled incidents show that rules fail. 2. **Never drop silently.** Every grouped, inhibited or suppressed alert stays queryable; every suppression has an owner, a reason and an expiry date. 3. **Measure both kinds of error.** Wrong merges (precision) and splits or misses (recall) both cost something; compression alone is a vanity metric. 4. **Context beats cleverness.** Topology, ownership, changes and calendars explain more alerts than any algorithm can infer from alert text. Dynatrace says of its own engine: "time correlation alone is not sufficient".[^dt-rca] 5. **People own production changes.** Machines may run allow-listed, reversible, rate-limited actions; anything else is a suggestion for a person. 6. **Explain every decision.** Each grouping and each probable cause shows *why*: shared labels, a topology path, a change, a model score. 7. **Own what is portable.** The label contract, rules, labelled incidents and evaluation harness belong to you; engines get replaced (see [tooling](#tooling-landscape-as-of-october-2026)). **Non-goals:** fully autonomous operations, replacing on-call engineers, or claiming certainty about root causes. ### Building blocks | Block | Job | Main inputs | Typical failure | |---|---|---|---| | Ingest and normalise | Map events from every monitor to one schema | Webhooks, APIs, log streams | Fields that differ between tools | | Label contract | Guarantee that every alert carries `service`, `team`, `severity`, the affected `cluster`, the evaluating server and a runbook | Alert rules, target labels, external labels | Missing labels, or one label with two meanings, break grouping and inhibition | | Deduplicate | Collapse repeats by fingerprint | Fingerprints | Fingerprints that include volatile labels, such as a pod name | | Enrich | Add owner, tier, runbook and recent changes | Service catalogue, CMDB, change log | Stale ownership data | | Context stores | Hold topology, changes and calendars | Tracing-derived dependencies, CI/CD, change records | Missing shared infrastructure (networks, DNS) | | Correlate | Group alerts into incidents | Rules, time windows, topology, learned similarity | Over-merging or over-splitting | | Detect anomalies | Flag deviations that no static rule describes | Metric history, seasonality | Business events mistaken for faults | | Rank probable causes | Order the likeliest components and changes | Topology, change log, model scores | Confident answers from a narrow menu | | Act | Route, notify, open tickets, run guarded fixes | Routing rules, runbooks, runner | Over-privileged runners; unverified actions | | Feedback and evaluation | Store labels; score precision, recall and accuracy | Responder corrections | No labels, so no measurement | ### Roles, ownership and RACI AIOps fails more often for lack of an owner than of an algorithm. Name an **event-intelligence owner**, accountable for the pipeline's precision, recall and backlog; in a small organisation this is part of an SRE's or observability engineer's job. | Activity | Event-intelligence owner | Service owners | On-call responders | Platform team | Change authority | |---|---|---|---|---|---| | Define and enforce the label contract | A | R | I | R | I | | Maintain topology and ownership data | C | A/R | I | R | I | | Change grouping or inhibition rules | A/R | C | C | C | I | | Roll out an anomaly detector to a service | A | R | C | C | I | | Label groupings and causes at incident close | C | C | R | I | I | | Approve a class of automatic remediation | C | R | C | C | A | | Monthly evaluation scorecard | A/R | I | C | C | I | | Select or replace a platform | R | C | C | C | A | R = responsible, A = accountable, C = consulted, I = informed; "platform team" includes observability engineers. With an ITIL change process, approving a remediation class is usually a standard change (see [ITOps](../itops/)). ### Lifecycle of a correlation or detection policy Treat every grouping rule, inhibition rule, anomaly detector and remediation playbook as a **policy** with a lifecycle. The states below are this Atlas's proposal; "shadow mode" means the policy runs and is scored, but changes nothing that people see. ::: figure Lifecycle of a correlation or detection policy (this Atlas's proposal) ```mermaid stateDiagram-v2 direction TB [*] --> Proposed Proposed --> Shadow: tests pass Shadow --> Canary: precision and recall meet targets Shadow --> Proposed: misses or bad merges Canary --> Active: one or two services stay healthy Canary --> Shadow: complaints or a missed incident Active --> Review: monthly score drops or topology changes Review --> Active: re-tuned Review --> Retired: no longer earns its keep Retired --> [*] ``` Text equivalent: a Proposed policy enters Shadow when its tests pass. Shadow leads to Canary if precision and recall meet targets, or back to Proposed after misses or bad merges. Canary becomes Active if one or two services stay healthy, or returns to Shadow after complaints or a missed incident. Active goes to Review when the monthly score drops or topology changes; Review re-tunes it back to Active or retires it. ::: - **Day 0 (design).** Inventory and baseline alert sources ([roadmap](#adoption-roadmap-306090-days)); write the label contract and a first topology file; choose rules first. - **Day 1 (rollout).** Run new policies in shadow mode against a few weeks of labelled incidents, then enable them service by service, starting with volunteers. - **Day 2 (operation).** Topology, traffic and rules drift. Review the scorecard monthly, re-label disputed incidents, retire dead rules, and re-evaluate after every major platform change. ### Operating model and governance - **Everything as code.** Grouping rules, inhibition rules, detector settings and playbooks live in version control and change by pull request. CI lints every alert rule, runs `promtool test rules` for important ones and `amtool config routes test` for routing ([Artefact 1](#reference-implementation)).[^promtool][^am-readme] - **Three change classes.** Routing and grouping changes need the owning team's review. Suppression and inhibition need evidence and an expiry date. A new class of automatic remediation needs the change authority's approval and a rollback plan. - **Feedback at incident close.** Ask responders two one-click questions: *were these alerts correctly grouped?* and *was the probable cause right?* Without these labels you cannot compute precision or recall. PagerDuty's documentation shows why the capture point matters: its learned grouping learns from manual merges and moves; "Only manual merges and unmerges in the PagerDuty web app influence the algorithm", merges made through the API do not, and alert titles rewritten by Event Orchestration also count.[^pd-iag] - **Cadence.** A weekly noise review (the ten noisiest rules and actions taken), a monthly scorecard, and a quarterly review of platform fit. ## Architecture and workflow ### Reference architecture The architecture is vendor-neutral: an all-in-one platform implements most layers itself, a composable stack assembles them from separate tools, and the trust boundaries stay the same. ::: figure Reference architecture for event intelligence ```mermaid flowchart TB SRC["Alert sources: metric rules, log and trace signals, probes"] SRC --> N["Normalise; check label contract"] MH["Metric history"] --> AD["Anomaly detection"] N --> DD["Deduplicate and enrich"] DD --> CO["Group, inhibit and route"] AD --> CO T["Topology and owners"] --> CO CO --> RC["Rank probable causes"] CH["Change log and calendar"] --> RC RC --> IM["Incident tool and paging"] IM --> CHAT["Chat channel"] CO -.->|"allow-listed alerts only"| RR["Remediation runner"] RR -.->|"outcome; page on failure"| IM IM -.->|"labels at close"| LB["Labelled incidents"] LB -.->|"tune"| CO classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class N,DD,CO,AD,RC core class RR risk class SRC,MH,T,CH,LB,IM,CHAT neutral ``` Text equivalent: alert sources (metric rules, log and trace signals, probes) are normalised and checked against the label contract, deduplicated and enriched, then grouped, inhibited and routed, with input from anomaly detection (which reads metric history) and from topology and owners. Groups are ranked for probable cause using the change log and calendar, then reach the incident tool and paging, which posts to chat. Routing also sends allow-listed alerts, and only those, straight to the remediation runner, the risk point, before any ranking; the runner reports its outcome and pages on failure. Labels captured at close tune grouping. ::: **Trust boundaries.** Four places deserve security review. (1) **Ingest endpoints**: anything that can post an event can create or hide an incident, so authenticate webhooks in both directions. (2) **The alert manager's own API**: Prometheus's security model says that anyone who can reach the Alertmanager HTTP endpoint can "create, modify and delete silences", and that it serves `/api/v2` with `Access-Control-Allow-Origin: *`, so any website opened in a browser that can reach it can call the API.[^prom-security] One request can silence an outage. (3) **The remediation runner** holds production credentials: give it the narrowest rights per allow-listed action. (4) **An LLM** that reads alert text sees attacker-controllable strings: treat them as untrusted input (see [LLMOps](../llmops/)). | Option | Strengths | Limits | Fits when | |---|---|---|---| | Built into your observability platform | Shares the vendor's topology and telemetry; little integration work | Sees mainly that vendor's data; lock-in | Most telemetry already lives in one platform | | Standalone event-intelligence platform | Cross-tool correlation across many monitors | Integration and data-mapping effort; another bill | Many monitoring tools; a central NOC | | Composable open source (for example Alertmanager, Keep, Event-Driven Ansible) | Control, portability, no per-event pricing | You own operations, upgrades and evaluation | Strong platform team; mostly Prometheus-style alerts | | Build your own models | Fits unusual data | Highest cost; needs ML skills and labels | Very large estates with data scientists; research settings | ### End-to-end workflow The sequence follows one incident; the `alt` block is the exception path, a wrong grouping or cause. ::: figure End-to-end workflow, with the exception path ```mermaid sequenceDiagram autonumber participant M as Monitors participant E as Event intel participant I as Incident tool participant O as On-call M->>E: 40 alerts in 4 min Note over E,I: dedup, group by topology, attach change E->>I: 1 incident, cause: orders-db I->>O: page DB owner alt grouping or cause wrong O->>I: split, merge, fix cause else grouping and cause right O->>I: confirm, work from the cause end O->>I: resolve, confirm labels I->>E: labelled incident for evaluation ``` Text equivalent: monitors send 40 alerts in four minutes to the event-intelligence layer, which deduplicates and groups them by topology and attaches the recent change, then sends one incident (probable cause: orders-db) to the incident tool, which pages the database owner. If the grouping or cause is wrong, the engineer splits, merges or corrects it; otherwise the engineer confirms and works from the cause. The engineer resolves and confirms the labels, and the labelled incident returns for evaluation. ::: A corrected wrong grouping is a training signal; an unnoticed one teaches the system to repeat the mistake. An allow-listed fault takes a third path: the alert goes straight to the runner, which pages a person if its checks or its fix fail (see the [guarded-remediation gate](#reference-implementation)). ## Practices, methods and tools ### Core practices and playbooks The practices are in the order in which they usually pay off; the first four need no machine learning. | Practice | Goal | Trigger | Owner | Output | Verification | |---|---|---|---|---|---| | 1. Label contract | Every alert can be routed, grouped and inhibited | New alert rule or monitor | Service owner | Labels `service`, `team`, `severity`, `remediation`, the affected `cluster`, the evaluating `prometheus`; annotation `runbook_url` | A linter fails CI on every rule; Alertmanager measures the compliant share | | 2. Deduplicate | One alert per fingerprint per episode | Repeat notifications | Event-intelligence owner | Stable fingerprints that exclude volatile labels | Re-notifications per alert ≈ 1 | | 3. Group and inhibit | One notification per cause | Several alerts share a service, cluster or dependency | Event-intelligence owner | `group_by`; guarded inhibition rules | Pairwise precision and recall on labelled incidents | | 4. Topology and change correlation | Link symptoms to failing dependencies and recent changes | Any group with more than one service | Service owners and platform team | Topology file; change events with service labels | Share of incidents with the right cause in the top 3 | | 5. Anomaly detection where rules cannot reach | Catch deviations no static threshold describes | Seasonal or business metrics | Service owner | Seasonal baseline, calendar, minimum duration | Precision and recall on injected or labelled anomalies | | 6. Probable-cause ranking | Point responders at the right component first | Every incident | Event-intelligence owner | Ranked candidates with the evidence for each | Top-k cause accuracy | | 7. Guarded remediation | Remove robotic pages | A known fault with a reversible fix | Automation owner, approved by the change authority | Allow-listed playbook, throttle, verification, rollback | Success, rollback and override rates | | 8. Feedback and evaluation | Know whether 1–7 work | Incident close; monthly | Event-intelligence owner | Labels; scorecard | Trend in the scorecard metrics | **Practice 1 is the foundation.** Grouping by `service` fails when half the rules call it `app`. Inhibition by `cluster` misbehaves in two ways. When the label is absent, Alertmanager treats it as an empty value, so "if all the label names listed in equal are missing from both the source and target alerts, the inhibition rule will apply".[^am-config] And when one label carries two meanings, *where the rule was evaluated* and *which cluster is affected*, two unrelated alerts can share a value. The [worked runbook](#worked-runbook-the-inhibition-rule-that-silenced-an-outage) shows both hiding an outage. Keep the meanings in separate labels, `cluster` (affected) and `prometheus` (evaluating), as [Artefact 1](#reference-implementation) does. **Practice 4 exploits a common cause of outages.** Google's SRE book (2016) says: "SRE has found that roughly 70% of outages are due to changes in a live system."[^sreintro] The figure is Google's and its method unpublished, but attaching recent changes to every incident is cheap and often decisive. Datadog's Watchdog RCA names version changes as one of its four root-cause types, alongside traffic increases, AWS instance failures and full disks.[^dd-rca] **Practice 7 starts from a paging principle:** "If a page merely merits a robotic response, it shouldn't be a page."[^sremon] Robotic responses are candidates for automation, behind the [guarded-remediation gate](#reference-implementation). ### Methods and frameworks: rules, statistics, machine learning and language models No standard defines AIOps methods; the literature organises them by task: anomaly detection, failure prediction, root-cause analysis and remediation.[^notaro2020][^remil2024] | Technique | Good at | Fails when | Documented example | |---|---|---|---| | Exact-match grouping on chosen fields | Predictable, explainable grouping | Labels are inconsistent | PagerDuty Content-Based Alert Grouping, "without the need to train an algorithm";[^pd-cbag] Alertmanager `group_by`[^am-config] | | Tags plus time window | Cross-tool grouping | Unrelated faults overlap in time | BigPanda correlation patterns combine source systems, tags, a time window and an optional filter[^bigpanda-corr] | | Inhibition | Muting known downstream symptoms | Labels are missing (see the runbook) | Alertmanager inhibition rules[^am-concepts] | | Topology-aware correlation | Linking symptoms to a failing dependency | Topology is stale or omits shared infrastructure | Dynatrace's fault-tree analysis over its Smartscape topology[^dt-rca] | | Change correlation | Pointing at the recent deploy | Changes are not recorded with service labels | Datadog Watchdog RCA (version changes)[^dd-rca] | | Statistical baselines | Seasonal metrics such as traffic | Holidays, campaigns, slow drifts | Modified z-score with median and MAD;[^nist-outliers] STL (seasonal-trend decomposition using Loess);[^fpp3] Grafana's PromQL anomaly bands[^grafana-promql] | | Forecasts (predictive alerts) | Resources that run out steadily: disks, quotas | Bursty use; clean-up jobs; counters | Prometheus `predict_linear`;[^prom-functions] node_exporter's filesystem alerts[^node-mixin] | | Unsupervised ML | Many dimensions, no labels | Nobody can say why a point is anomalous | scikit-learn's Isolation Forest, and its distinction between outlier and novelty detection[^sklearn] | | Learned grouping from feedback | Adapting to how a team actually groups alerts | Feedback is sparse or inconsistent | PagerDuty Intelligent Alert Grouping learns from manual merges and moves[^pd-iag] | | Statistical log analysis | Explaining a spike in log volume | Logs lack structure | Elastic's log rate analysis, log pattern analysis and change point detection[^elastic-aiops] | | LLMs | Summarising, classifying and drafting from incident text | No ground truth; untrusted input; hallucination | Microsoft studies on root-causing incidents[^ahmed2023][^rcacopilot] | **When rules beat machine learning.** Prefer rules when the failure has a clear symptom (an SLO burn rate, an error ratio), people know the topology, no labelled incidents exist to evaluate a model, a wrong suppression would be costly, or on-call engineers must predict the system's behaviour at 3 a.m. Google's SRE book allows one exception to its simple rules: detecting "unexpected changes in end-user request rates".[^sremon] Its workbook judges alert rules on precision, recall, detection time and reset time,[^sreslo] the same attributes this page uses for models. **When learning earns its place.** When many tools produce inconsistent alerts, metrics are strongly seasonal, topology is too large or fast-changing to maintain by hand, and enough labelled incidents exist to show in shadow mode that a model beats the rules. ::: figure Choosing a technique, including the decision not to adopt ```mermaid flowchart TB Q1{"Pages follow symptoms and SLOs?"} -->|"no"| R1["Fix alert rules first (SRE)"] Q1 -->|"yes"| Q2{"Alerts carry service, team, cluster?"} Q2 -->|"yes"| Q3{"Storms from shared dependencies?"} Q2 -->|"no"| R2["Label contract, dedup, group_by"] Q3 -->|"yes"| R3["Inhibition plus topology correlation"] Q3 -->|"no"| Q4{"Seasonal metrics hide faults?"} Q4 -->|"no"| Q5{"100+ labelled incidents, rules failing?"} Q4 -->|"yes"| R4["Seasonal baselines plus calendars"] Q5 -->|"yes"| R5["Shadow-trial learned grouping"] Q5 -->|"no"| R6["Stop: rules are enough"] classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class Q1,Q2,Q3,Q4,Q5 decision class R2,R3,R4,R5 core class R1,R6 risk ``` Text equivalent: if pages do not yet follow symptoms and SLOs, fix the alert rules first. If alerts lack service, team and cluster labels, introduce a label contract, deduplication and grouping. If storms come from shared dependencies, use inhibition plus topology correlation; otherwise, if seasonal metrics hide faults, use baselines with calendars; otherwise, with about 100 labelled incidents and failing rules, shadow-trial learned grouping, and without them stop: rules are enough. The 100 is this Atlas's rule of thumb, not a published figure. ::: #### Predictive alerting and failure prediction Notaro and colleagues divide failure management into "failure prevention, online failure prediction, failure detection, root cause analysis (RCA) and remediation"; the papers they map predict disk, hardware and software failures.[^notaro2020] In daily operations the most defensible form is a forecast of exhaustion: disk space, quotas, connection pools. Prometheus's `predict_linear` "predicts the value of time series t seconds from now, based on the range vector v, using simple linear regression" and "should only be used with gauges".[^prom-functions] The node_exporter project's filesystem alert shows the careful pattern: free space already low, a 6-hour trend reaching zero within the horizon, a writable filesystem, and the condition held for an hour.[^node-mixin] _Example status: PromQL adapted from the node_exporter 1.12.1 mixin (critical variant, checked 2026-10-17); not executed._ ```promql # Alert rule expression, with for: 1h in the rule: under 20% free AND the # 6-hour trend reaches zero within 4 hours AND the filesystem is writable. ( node_filesystem_avail_bytes{mountpoint="/var/log"} / node_filesystem_size_bytes{mountpoint="/var/log"} * 100 < 20 and predict_linear( node_filesystem_avail_bytes{mountpoint="/var/log"}[6h], 4 * 3600 ) < 0 and node_filesystem_readonly{mountpoint="/var/log"} == 0 ) ``` - **A straight line extrapolates the window.** A burst of writes predicts exhaustion; a clean-up job predicts plenty. The mixin names two causes and two remedies. Scheduled clean-up jobs "may not align with the default time window of these alerts. This can cause these alerts to start flapping"; its remedy is a *shorter* alert horizon (the 4 hours above), which leaves the job time to run. Short-lived fluctuation is the other cause: a *longer* prediction window (the 6 hours of history) can "reduce false positives caused by temporary spikes".[^node-mixin] - **Evaluate it like any detector.** On past exhaustion events, weigh the lead time gained over a static threshold against false forecasts per week; ten minutes gained with weekly flapping is noise. - **Forecast resources, not users.** For user-visible symptoms an SLO burn-rate alert already forecasts budget exhaustion ([SRE](../sre/)). Learned failure prediction (disks, for example) is a learned detector: shadow mode and labelled outcomes first. #### How the learning techniques work: logs, similarity and causal graphs **Logs become templates first.** A log line mixes a fixed message with variables, so analysis starts by mining templates: Drain3 turns "user davidoh logged in" and "user eranr logged in" into `user <:*:> logged in`, and opens a new template when too few tokens match (similarity threshold 0.4 by default).[^drain3] Detectors then count each template per time window or session and flag unusual counts (PCA, invariant mining, clustering, classifiers),[^he2016] or learn the normal order of templates and flag changed sequences.[^landauer2024] The evidence is sobering. An ICSE 2022 study found that training-data choice, grouping, class balance and noise significantly affected the evaluation of five deep-learning detectors, which "do not always work well";[^le2022] an FSE 2024 study found simple detectors competitive with advanced ones on common benchmark data sets: flagging template types never seen in training sufficed on one, comparing template counts on another.[^landauer2024] Start with new-template and template-rate alerts; score anything cleverer as in Lab 3. **Learned grouping is a similarity score plus a threshold.** Documentation shows three kinds of evidence. Shared fields: Splunk ITSI's Smart Mode weighs "category" fields (a status) against "text" fields (a description) and can split by service first.[^splunk-smart] Text: PagerDuty compares alert summaries by default and warns that a larger time window raises "the chance of overgrouping".[^pd-iag] Co-occurrence: Moogsoft's Tempus compares arrival patterns with the Jaccard index; its documentation rules it out for slow degradation.[^moog-tempus] Lab 2's `related()` functions are the hand-written version. Ask which fields, weights and window decide a merge, then score it with pairwise precision and recall. **Graph and causal root-cause analysis rank candidates.** Lab 2 picks an alerting service none of whose dependencies alert. Graph methods generalise this: take the graph from topology or learn a causal one from metrics (PC or GES in PyRCA), then score each node, for example by a random walk that assumes "a metric that is highly correlated with the anomaly metrics is more likely to be the root cause".[^pyrca] An ASE 2024 evaluation of 21 such methods for microservices found that "no method stands out in all situations" and that synthetic results "may not accurately reflect their performance in real systems".[^pham2024] Treat the output as a ranked list with evidence, scored by top-k accuracy. ### Tooling landscape (as of October 2026) Examples are documented capabilities checked on 2026-10-16, not endorsements. | Category | Job | Examples (verified) | Selection notes | |---|---|---|---| | Alert routing and grouping | Deduplicate, group, inhibit, silence, route | Prometheus Alertmanager 0.34.1 (17 September 2026)[^am-release] | Free and predictable; rule-based only | | Open-source alert management and AIOps | Unify alerts from many tools; deduplicate, correlate, run workflows | Keep 0.54.3 (9 September 2026), "The open-source AIOps and alert management platform"[^keep-repo] | Elastic announced an agreement to acquire Keep in May 2025; it would "remain open source"[^elastic-keep] | | Statistical detection in the query language | Seasonal bands as recording and alerting rules | Grafana's PromQL anomaly-detection framework[^grafana-promql] | Most strategies need 24–26 hours of data; a drop to zero needs a separate rule | | Event intelligence in incident platforms | Group alerts, pause transient alerts, orchestrate events | PagerDuty Intelligent and Content-Based Alert Grouping, Auto-Pause, Event Orchestration[^pd-iag][^pd-cbag][^pd-autopause][^pd-eo] | Learned grouping comes with the PagerDuty AIOps add-on or certain plans (packaging changes) | | Standalone event-intelligence platforms | Cross-tool correlation | BigPanda (correlation patterns);[^bigpanda-corr] Dell APEX AIOps Incident Management, formerly Moogsoft (acquired by Dell on 17 September 2023)[^moogsoft-dell][^dell-apex] | Integration effort; check the limits (BigPanda caps an incident at 300 correlated alerts) | | Built into observability and ITOM platforms | Correlation and cause analysis on the vendor's own topology | Dynatrace root-cause analysis;[^dt-rca] Datadog Watchdog RCA;[^dd-rca] Elastic AIOps Labs;[^elastic-aiops] Splunk ITSI Event Analytics (correlation searches, aggregation policies, episodes)[^splunk-itsi] | Strongest where that vendor already sees most of your telemetry | | Event-driven automation | Run playbooks on events, with throttling | Event-Driven Ansible: ansible-rulebook 1.3.2[^eda-rules] | Runner credentials are the main risk | | Libraries | Build or study algorithms | scikit-learn outlier detection;[^sklearn] Drain3 (log templates);[^drain3] Salesforce PyRCA (root-cause analysis)[^pyrca] | Salesforce's Merlion time-series library was archived on 11 March 2026[^merlion] | **Engines churn; plan for it.** Managed services retire and libraries get archived: | Service or project | Status | Source | |---|---|---| | Amazon Lookout for Metrics | AWS ended support in October 2025 | AWS blog[^aws-lookout] | | Azure AI Anomaly Detector | No new resources since 20 September 2023; retired on 1 October 2026 | Microsoft Learn[^azure-ad] | | Amazon DevOps Guru | Closes to new customers on 29 October 2026; existing customers until 30 September 2027 | AWS documentation[^aws-devopsguru] | | Salesforce Merlion | Repository archived on 11 March 2026 | GitHub[^merlion] | The lesson is principle 7, not "avoid managed services": keep what you need to replace an engine and prove the replacement no worse. **Selection criteria.** Beyond the [weighted worksheet](#decision-framework), ask whether every suppressed alert stays queryable and whether pricing penalises context. ### Automation and AI: language models and agents LLMs have found real work in incident handling, mainly reading and writing text. Microsoft researchers evaluated LLMs for recommending root causes and mitigations on more than 40,000 incidents, with incident owners judging the output.[^ahmed2023] Microsoft's RCACopilot gathers diagnostics, predicts a root-cause category with an explanation, and reports "RCA accuracy up to 0.766"; its diagnostic-collection module had run in more than 30 teams for over four years, its LLM component in one incident-management team for several months.[^rcacopilot] A 2025 paper reports that its system proposed changes to 1,174 alert rules at an anonymised company, and SREs accepted 375 (32%).[^alertguardian] The evidence on autonomy is thin. ITBench, peer-reviewed at ICML 2025, built "an initial set of 102 real-world scenarios" and found that "agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios"; the models were those of early 2025.[^itbench] Newer tests measure something narrower. Artificial Analysis's ITBench-AA gives agents an offline snapshot of 59 Kubernetes incidents and scores the diagnosis by "average precision at full recall"; the best listed model scored 56.2% (checked 16 October 2026).[^itbench-aa] Diagnosing from a snapshot is not resolving a live system, and leaderboards change monthly, so neither number licenses unsupervised action. AIOpsLab injects faults so that agents can be evaluated before anyone trusts them.[^aiopslab] Practical rules for LLMs in operations: 1. **Start read-only.** Summaries, timelines, similar-incident search and runbook retrieval carry little risk if a person reads them. 2. **Treat alert and log text as untrusted input.** Never let model output reach a shell or an API without an allow-list. 3. **Evaluate like any other policy.** Measure top-k cause accuracy and wrong suggestions on labelled incidents, in shadow mode. 4. **Mind the data.** Redact customer data and secrets before incident text leaves your boundary ([PrivacyOps](../privacyops/)); token spend belongs in [FinOps for AI](../finops-for-ai/). ## AIOps in practice ### Reference implementation **Design.** A composable, vendor-neutral start for a team with Prometheus-style alerting: a label contract that a linter enforces and Alertmanager measures, Alertmanager for grouping and guarded inhibition, a topology file in Git, a normalised event schema with change events, and a guarded remediation runner. **Assumptions:** Linux hosts, Prometheus 3.x (3.15.0 was released on 24 September 2026[^prom-release]), Alertmanager 0.34.x,[^am-release] pint 0.89.x,[^pint-lint] ansible-rulebook 1.3.x[^eda-rules] on the Alertmanager host, and a separate incident tool that accepts webhooks. **Constraints:** no machine learning yet; every suppression is visible; only one remediation class is automated. **Artefact 1: the label contract and its enforcement.** Two labels look alike and must never be merged: the cluster an alert is *about* and the server that *evaluated* it. | Label | Meaning | Set by | |---|---|---| | `service`, `team` | The service the alert is about, and its owner | Group-level `labels` in the rule file[^prom-rules] | | `severity`, `remediation` | `page` or `ticket`, the convention shared with [IncidentOps](../incidentops/) and [ChatOps](../chatops/); `auto` or `none` | Each rule | | `cluster` | The affected cluster | The monitored target's labels; on an in-cluster server, `external_labels` as a default | | `prometheus` | The server that evaluated the rule | `external_labels` on every server | | `runbook_url` (annotation) | Where the responder starts | Each rule | External labels never override a label an alert already has,[^prom-config] so a rule that watches another cluster keeps that cluster's name. _Example status: syntax-checked, not executed (requires Prometheus and `promtool`)._ ```yaml # checkout-rules.yml (in-cluster server): service, team, severity, remediation # and runbook_url are static; cluster and prometheus come from the server's # external_labels. groups: - name: checkout-api labels: service: checkout-api team: payments rules: - alert: HighErrorRatio expr: | sum(rate(http_requests_total{job="checkout-api",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="checkout-api"}[5m])) > 0.05 for: 5m keep_firing_for: 5m labels: severity: page database: orders-db remediation: none annotations: summary: "checkout-api 5xx ratio above 5% for 5 minutes" runbook_url: "https://runbooks.example.com/checkout-api/high-error-ratio" ``` ```yaml # prometheus.yml on the in-cluster server of eu-1. The meta-monitoring server sets # only prometheus: meta because its alerts are about other clusters. global: external_labels: cluster: eu-1 # a default: a cluster label already on an alert wins prometheus: eu-1 ``` ```yaml # meta-rules.yml on the meta-monitoring server. Each target of this job carries # its own cluster label (for example cluster="dev-3"), so the alert names the # affected cluster. A target without one yields an alert that, by Artefact 2, # inhibits nothing. groups: - name: meta-monitoring labels: service: monitoring team: observability rules: - alert: ClusterUnreachable expr: up{job="cluster-prometheus"} == 0 for: 5m labels: severity: page remediation: none annotations: summary: "Prometheus of cluster {{ $labels.cluster }} unreachable for 5m" runbook_url: "https://runbooks.example.com/monitoring/cluster-unreachable" ``` Enforcement needs three checks, because each sees only part of the contract. (1) A **linter over every rule file.** pint's `rule/label` check with `required = true` flags any alerting rule that lacks a label; it "uses static labels set on alerting or recording rule" (group-level labels since pint 0.72.0), so it cannot see `cluster` or `prometheus`.[^pint-label][^pint-changelog] Its findings are warnings by default and `pint lint` fails only on `bug` or worse, so raise the severity.[^pint-lint] (2) A **configuration check** in CI: every server sets `prometheus`, in-cluster servers set `cluster`, the meta-monitoring server does not. (3) A **measurement in Alertmanager**: the share of current alerts carrying the whole contract, sampled from its API. That share is the scorecard's label-contract compliance and maturity level 1's 95% exit criterion. _Example status: written to the pint documentation for `rule/label` and `alerts/annotation`; not executed (requires pint)._ ```hcl # .pint.hcl: run pint lint rules/ in CI. # Severity "bug" makes a missing label fail the run. rule { match { kind = "alerting" } label "service" { severity = "bug" required = true } label "team" { severity = "bug" required = true } label "severity" { severity = "bug" required = true values = ["page", "ticket"] } label "remediation" { severity = "bug" required = true values = ["auto", "none"] } annotation "runbook_url" { severity = "bug" required = true } } ``` _Example status: executed on 2026-10-17 (Python 3.8.10, PyYAML 5.3.1) against four sample configurations and a saved API response; not run against a live Alertmanager._ ```python """label_contract.py: check the label contract in two places. python3 label_contract.py configs prometheus/*.yml # CI: server-level labels python3 label_contract.py alerts [ALERTS_API_URL] # runtime: compliant share Standard library, plus PyYAML for the configs check. """ import json import sys import urllib.request ALERT_LABELS = ("service", "team", "severity", "remediation", "prometheus") def check_configs(paths): import yaml # PyYAML bad = 0 for path in paths: with open(path) as f: glob = (yaml.safe_load(f) or {}).get("global") or {} labels = glob.get("external_labels") or {} server = labels.get("prometheus") if not server: problem = "no prometheus label" elif server == "meta" and "cluster" in labels: problem = "meta-monitoring must not set cluster" elif server != "meta" and "cluster" not in labels: problem = "no cluster label" else: continue print(f"{path}: {problem}") bad += 1 return 1 if bad else 0 def compliant(labels): if not all(labels.get(k) for k in ALERT_LABELS): return False # Meta-monitoring alerts may concern things outside any cluster. return bool(labels.get("cluster")) or labels.get("prometheus") == "meta" def check_alerts(url): with urllib.request.urlopen(url, timeout=10) as resp: alerts = json.load(resp) ok = sum(compliant(a["labels"]) for a in alerts) share = ok / len(alerts) if alerts else 1.0 print(f"alerts={len(alerts)} compliant={ok} share={share:.2f}") return 0 if __name__ == "__main__": mode = sys.argv[1] if len(sys.argv) > 1 else "" if mode == "configs": sys.exit(check_configs(sys.argv[2:])) if mode == "alerts": default = "http://127.0.0.1:9093/api/v2/alerts" sys.exit(check_alerts(sys.argv[2] if len(sys.argv) > 2 else default)) sys.exit(__doc__) ``` On the samples, `configs` flagged a meta-monitoring file with `cluster: meta` and a file without external labels (exit code 1), and `alerts` printed `alerts=4 compliant=2 share=0.50`. A `promtool` unit test remains useful as a **regression test for one important rule**: it checks only rules that have a test case and never sees `external_labels`, but its `exp_labels` must match exactly, so a dropped label fails the build.[^promtool] _Example status: syntax-checked, not executed (requires `promtool`)._ ```yaml # checkout-rules.test.yml: run with promtool test rules checkout-rules.test.yml rule_files: - checkout-rules.yml evaluation_interval: 1m tests: - interval: 1m input_series: - series: 'http_requests_total{job="checkout-api",code="500"}' values: '0+10x20' - series: 'http_requests_total{job="checkout-api",code="200"}' values: '0+90x20' alert_rule_test: - eval_time: 15m alertname: HighErrorRatio exp_alerts: - exp_labels: severity: page service: checkout-api team: payments database: orders-db remediation: none exp_annotations: summary: "checkout-api 5xx ratio above 5% for 5 minutes" runbook_url: >- https://runbooks.example.com/checkout-api/high-error-ratio ``` **Artefact 2: grouping, routing and guarded inhibition.** The defaults shown (`group_wait` 30s, `group_interval` 5m, `repeat_interval` 4h) are Alertmanager's documented defaults, written out for reviewers.[^am-config] Three details carry the safety: each inhibit rule requires every `equal` label to be non-empty on the source, so a source without it mutes nothing; rule 1 never mutes the meta-monitoring server, which watches the alert path itself; and the gateways authenticate Alertmanager by a bearer token read from a file (`credentials_file` under `http_config` → `authorization`).[^am-config] _Example status: syntax-checked, not executed (requires Alertmanager 0.34.x; Lab 1 runs it)._ ```yaml # alertmanager.yml: grouping, routing and guarded inhibition (Alertmanager 0.34.x). global: resolve_timeout: 5m route: receiver: ticket-queue group_by: [cluster, service] # one notification per service per cluster group_wait: 30s # let related (and inhibiting) alerts arrive group_interval: 5m repeat_interval: 4h routes: - matchers: ['remediation="auto"'] # allow-listed classes go to the runner... receiver: eda-remediation continue: true # ...and still reach a ticket or page - matchers: ['severity="page"'] receiver: oncall-pager - matchers: ['severity="ticket"'] receiver: ticket-queue inhibit_rules: # 1. A cluster is unreachable: mute the per-service symptoms in that cluster. - source_matchers: - alertname="ClusterUnreachable" - cluster=~".+" # guard: no cluster, mutes nothing target_matchers: - alertname!="ClusterUnreachable" - prometheus!="meta" # never mute the meta-monitoring alerts equal: [cluster] # 2. A database is down: mute error and latency symptoms of services that use it. - source_matchers: - alertname="DatabaseDown" - cluster=~".+" # guard - database=~".+" # guard target_matchers: ['alertname=~"HighErrorRatio|HighLatency"'] equal: [cluster, database] receivers: - name: oncall-pager webhook_configs: - url: https://incident-gateway.example.com/hooks/alertmanager http_config: authorization: # Bearer by default; file mode 0400 credentials_file: /etc/alertmanager/secrets/incident-gateway - name: ticket-queue webhook_configs: - url: https://ticket-gateway.example.com/hooks/alertmanager http_config: authorization: credentials_file: /etc/alertmanager/secrets/ticket-gateway - name: eda-remediation webhook_configs: - url: http://127.0.0.1:5050/alerts # rulebook on this host (loopback) send_resolved: false ``` **The residual risk is an observer-side partition.** If the meta-monitoring server loses *its own* route to eu-1 while eu-1's Prometheus still reaches Alertmanager, `ClusterUnreachable{cluster="eu-1"}` fires and rule 1 mutes eu-1's genuine alerts; the only page is about the monitoring path. So fire the source only when two independent vantage points agree (for example `up == 0` from the meta server *and* a failed probe from a second location), and make `amtool alert query --inhibited cluster=eu-1` the first step of that alert's runbook. Protect Alertmanager's API as well ([security controls](#quality-security-privacy-and-compliance)). **Artefact 3: a topology file.** This is this Atlas's minimal format, not a standard. Generate it from tracing data where you can and review it by hand. List shared infrastructure explicitly, because faults there fan out across services that never call each other (Lab 2 shows the cost of leaving it out). _Example status: syntax-checked; a data file, nothing to execute._ ```yaml services: web: {team: storefront, tier: 1, depends_on: [checkout-api]} checkout-api: {team: payments, tier: 1, depends_on: [orders-db, cache, payments-gw]} payments-gw: {team: payments, tier: 1, depends_on: [payments-db]} worker: {team: fulfilment, tier: 2, depends_on: [queue, orders-db]} shared_infrastructure: net-zone-a: {owner: network, carries: [cache, queue, orders-db]} ``` **Artefact 4: a normalised event.** Anything that leaves Alertmanager, or arrives from another monitor, is mapped to one schema before correlation. The `correlation` block records who decided what, so that decisions can be audited and evaluated. _Example status: JSON syntax-checked; the schema name is this Atlas's own._ ```json { "schema": "atlas.ops-event/v1", "fingerprint": "3f6c2a9d1b7e4c08", "source": "prometheus/eu-1", "status": "firing", "starts_at": "2026-10-16T09:14:30Z", "alertname": "HighErrorRatio", "service": "checkout-api", "team": "payments", "environment": "production", "cluster": "eu-1", "severity": "page", "runbook_url": "https://runbooks.example.com/checkout-api/high-error-ratio", "recent_changes": ["deploy checkout-api v2026.10.16-3 at 2026-10-16T09:02:11Z", "config net-zone-a acl-118 at 2026-10-16T08:47:05Z"], "correlation": {"group_id": null, "probable_cause": null, "decided_by": null} } ``` **Change events fill `recent_changes`.** A deploy's last step records the change with the alerts' `service` value and a version (OpenTelemetry's counterpart is `service.version`).[^otel-service] For an alert or a group, the changes of the preceding hour to its services, their dependencies and the shared infrastructure under them (Artefact 3) are attached, the alerting services' own first. The hour and the order are this Atlas's starting values; tune them on change-caused incidents. _Example status: executed on 2026-10-17 (Python 3.8.10, PyYAML 5.3.1): `record` against a local test receiver, `attach` on four sample changes; not run in a CI system._ ```python """change_events.py: record deploys as change events; attach them to alert groups. python3 change_events.py record SERVICE VERSION [URL] # last step of a deploy python3 change_events.py attach topology.yml changes.jsonl group.json [MINUTES] Standard library, plus PyYAML to read the topology file (Artefact 3). """ import json import os import sys import urllib.request from datetime import datetime, timedelta, timezone FMT = "%Y-%m-%dT%H:%M:%SZ" def record(service, version, url=None): event = { "schema": "atlas.change-event/v1", "kind": os.environ.get("CHANGE_KIND", "deploy"), "service": service, # the same value as the alerts' label "version": version, "environment": os.environ.get("ENVIRONMENT", "production"), "by": os.environ.get("CHANGE_ACTOR", "unknown"), # from your CI system "at": datetime.now(timezone.utc).strftime(FMT), } body = json.dumps(event).encode() print(body.decode()) if url: # the bearer token comes from a file, never the command line with open(os.environ["CHANGE_LOG_TOKEN_FILE"]) as f: token = f.read().strip() req = urllib.request.Request(url, data=body, method="POST", headers={ "Content-Type": "application/json", "Authorization": f"Bearer {token}"}) urllib.request.urlopen(req, timeout=10).close() return 0 def nearby(topology, services): """The group's services, their dependencies, and shared infrastructure.""" near = set(services) for name in services: entry = (topology.get("services") or {}).get(name) or {} near.update(entry.get("depends_on", [])) for name, entry in (topology.get("shared_infrastructure") or {}).items(): if near & set(entry.get("carries", [])): near.add(name) return near def attach(topology_path, changes_path, group_path, minutes=60): import yaml # PyYAML with open(topology_path) as f: topology = yaml.safe_load(f) with open(changes_path) as f: changes = [json.loads(line) for line in f if line.strip()] with open(group_path) as f: group = json.load(f) services = {a["service"] for a in group["alerts"]} start = min(datetime.strptime(a["starts_at"], FMT) for a in group["alerts"]) since = start - timedelta(minutes=minutes) near = nearby(topology, services) hits = [c for c in changes if c["service"] in near and since <= datetime.strptime(c["at"], FMT) <= start] hits.sort(key=lambda c: c["at"], reverse=True) # newest first, but hits.sort(key=lambda c: c["service"] not in services) # own services first print(f"group {group['group_id']}: {len(hits)} of {len(changes)} changes " f"in the {minutes} minutes before {start.strftime(FMT)}") for c in hits: where = "alerting service" if c["service"] in services else "dependency" print(f' {c["kind"]} {c["service"]} {c["version"]} at {c["at"]} ({where})') return 0 if __name__ == "__main__": args = sys.argv[1:] if args[:1] == ["record"] and len(args) in (3, 4): sys.exit(record(*args[1:])) if args[:1] == ["attach"] and len(args) in (4, 5): sys.exit(attach(*args[1:4], *[int(m) for m in args[4:]])) sys.exit(__doc__) ``` For a group of `checkout-api` and `web` alerts from 09:14:30, `attach` left out a `worker` deploy (not a dependency) and yesterday's `checkout-api` deploy: ```text group g-0917: 2 of 4 changes in the 60 minutes before 2026-10-16T09:14:30Z deploy checkout-api v2026.10.16-3 at 2026-10-16T09:02:11Z (alerting service) config net-zone-a acl-118 at 2026-10-16T08:47:05Z (dependency) ``` **Artefact 5: guarded remediation.** One class is automated: rotate and compress logs when a log volume is nearly full. The rule sets `remediation: auto`; Artefact 2 sends such alerts to the rulebook and on to the ticket queue; the rulebook's allow-list decides whether anything runs; a second rule pages at 95% regardless. The gate figure below shows where each check lives; the per-host throttle uses ansible-rulebook's `throttle` "to counter event storms",[^eda-conditions] and runs go one at a time.[^eda-rulebooks] Rotation has nothing to undo, so failure pages a person; a class that changes configuration needs a rollback task there. [AutomationOps](../automationops/) teaches these controls in general. Three security facts shape it. The `ansible.eda.alertmanager` source (collection 2.13.0) is a plain HTTP webhook that performs no authentication, defaults to `localhost`, and copies every request header into each event (`meta.headers`);[^eda-am] keep it on loopback beside Alertmanager. A runner on another host should use `ansible.eda.webhook`, which checks a bearer token, supports TLS and mutual TLS, and drops the `Authorization` header from events; events then arrive under `event.payload`, and the `insert_hosts_to_meta` filter must supply the hosts.[^eda-webhook][^eda-hostlimit] And "You cannot limit privilege escalation permissions to certain commands",[^ansible-become] so the runner may become only the log owner, never root. _Example status: YAML syntax-checked; keys checked against the Prometheus, ansible-rulebook 1.3.0 and Ansible documentation; not executed (requires node_exporter, ansible-rulebook with the `ansible.eda` collection, and an inventory whose host names match the `instance` label without its port)._ ```yaml # node-log-rules.yml: one automatable class, and a page that needs no runner. groups: - name: node-logs labels: service: node-logs team: platform rules: - alert: LogVolumeNearlyFull # allow-listed: a ticket plus the runner expr: | 1 - node_filesystem_avail_bytes{mountpoint="/var/log"} / node_filesystem_size_bytes{mountpoint="/var/log"} > 0.85 for: 10m labels: severity: ticket remediation: auto annotations: summary: "/var/log on {{ $labels.instance }} is over 85% full" runbook_url: "https://runbooks.example.com/node/log-volume" - alert: LogVolumeCritical # pages whether or not automation ran expr: | 1 - node_filesystem_avail_bytes{mountpoint="/var/log"} / node_filesystem_size_bytes{mountpoint="/var/log"} > 0.95 for: 5m labels: severity: page remediation: none annotations: summary: "/var/log on {{ $labels.instance }} is over 95% full" runbook_url: "https://runbooks.example.com/node/log-volume" ``` ```yaml # remediation-rulebook.yml: run with # ansible-rulebook -r remediation-rulebook.yml -i inventory.yml - name: Guarded remediation for known, reversible faults hosts: all execution_strategy: sequential # one action at a time (the default) sources: - ansible.eda.alertmanager: host: 127.0.0.1 # loopback: the source has no auth port: 5050 data_alerts_path: alerts data_host_path: labels.instance # host:port; the source strips the port rules: - name: Rotate logs on a nearly full log volume (the allow-list) condition: >- event.alert.status == "firing" and event.alert.labels.alertname == "LogVolumeNearlyFull" and event.alert.labels.remediation == "auto" throttle: once_within: 30 minutes group_by_attributes: - event.meta.hosts action: run_playbook: name: playbooks/rotate_logs.yml - name: Labelled auto but not allow-listed here, so it stays with people condition: >- event.alert.status == "firing" and event.alert.labels.alertname != "LogVolumeNearlyFull" action: debug: msg: "Not on the allow-list: {{ event.alert.labels.alertname }}" ``` ```yaml # playbooks/rotate_logs.yml: the checks the rulebook cannot make. - name: Rotate logs on a nearly full log volume, within caps, then verify hosts: all # ansible-rulebook limits it to the alert's host serial: 1 max_fail_percentage: 0 vars: run_log: /var/lib/eda/rotate_logs.runs # runner host, one line per run host_daily_cap: 3 # runs per host per day fleet_hourly_cap: 5 # runs per hour, all hosts: blast radius max_used_pct: 85 # the alert threshold gateway_token_file: /etc/eda/secrets/gateway-token tasks: - name: Read earlier runs ansible.builtin.command: cat {{ run_log }} register: runs changed_when: false failed_when: false delegate_to: localhost - name: Stop unless both caps allow another run vars: today: "{{ now(utc=true, fmt='%Y-%m-%d') }}" this_hour: "{{ now(utc=true, fmt='%Y-%m-%dT%H') }}" host_runs: >- {{ runs.stdout_lines | select('match', today ~ '.* ' ~ inventory_hostname ~ '$') | list | length }} fleet_runs: >- {{ runs.stdout_lines | select('match', this_hour) | list | length }} ansible.builtin.assert: that: - host_runs | int < host_daily_cap - fleet_runs | int < fleet_hourly_cap fail_msg: "Cap reached: no automatic run; the ticket and the 95% page stay" - name: Record this run before acting ansible.builtin.lineinfile: path: "{{ run_log }}" line: >- {{ now(utc=true, fmt='%Y-%m-%dT%H:%M:%S') }} {{ inventory_hostname }} create: true mode: "0640" delegate_to: localhost - name: Act, verify, and page a person on any failure block: - name: Rotate and compress within the retention the log owner approved ansible.builtin.command: >- logrotate --force --state /var/lib/applog/logrotate.state /etc/logrotate.d/app become: true become_user: applog # the log owner, never root - name: Verify that usage fell below the alert threshold ansible.builtin.command: df --output=pcent /var/log register: usage changed_when: false failed_when: >- usage.stdout_lines[-1] | trim | replace('%', '') | int >= max_used_pct rescue: - name: Page a person (rotation has nothing to undo) ansible.builtin.uri: url: https://incident-gateway.example.com/hooks/remediation-failed method: POST headers: Authorization: >- Bearer {{ lookup('ansible.builtin.file', gateway_token_file) }} body_format: json body: host: "{{ inventory_hostname }}" action: rotate_logs result: failed delegate_to: localhost - name: Fail the run so that the rulebook log shows it ansible.builtin.fail: msg: "rotate_logs failed on {{ inventory_hostname }}; a person is paged" ``` Every automated action passes the same gate. Google's account of an automated decommissioning job that wiped the disks of its CDN machines explains the gate's checks: afterwards Google added "more sanity checks—including rate limiting" and made the workflow idempotent.[^sreauto] ::: figure The guarded-remediation gate, with where each check lives (this Atlas's proposal) ```mermaid flowchart TB T["Alert with remediation=auto"] --> A{"On the rulebook allow-list?"} A -->|"no"| H["Stays with people: ticket now, page at 95%"] A -->|"yes"| C{"Host throttle free? (rulebook)"} C -->|"no"| H C -->|"yes"| B{"Daily and fleet caps free? (playbook)"} B -->|"no"| H B -->|"yes"| D["Run as the log owner, not root"] D --> E{"Verified fixed? (playbook)"} E -->|"yes"| F["Record outcome; alert resolves"] E -->|"no"| G["Roll back if possible; page a person"] classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class A,B,C,E decision class D,F core class H,G risk ``` Text equivalent: an alert labelled for automatic remediation must be on the rulebook's allow-list, pass the rulebook's per-host throttle, and pass the playbook's daily and fleet-wide caps; failing any check leaves it with people (a ticket now, a page at 95%). Otherwise the playbook acts as the log owner, not root. A verified fix is recorded and the alert resolves; otherwise the playbook rolls back where possible and pages a person. ::: ### Hands-on labs Three labs, from configuration to measurement; Labs 2 and 3 were **executed** and their outputs are verbatim. #### Lab 1 (beginner): group and inhibit with Alertmanager - **Objective:** see grouping and inhibition work, reproduce the missing-label pitfall, then watch the guard stop it. - **Prerequisites:** Linux with `bash`, `curl`, `tar` and `sha256sum`; port 127.0.0.1:9093 free; Artefact 2 saved as `alertmanager.yml`. Run it as a script (`bash lab1.sh alertmanager.yml`); pasted into a terminal, `set -e` would close the shell at the first error. **Cost and risk:** none; everything binds to localhost. Notification errors (`example.com`, missing credential files) are expected. _Example status: syntax-checked, not executed (downloads a third-party binary)._ ```bash #!/usr/bin/env bash # lab1.sh: run with bash lab1.sh /path/to/alertmanager.yml (Artefact 2) set -euo pipefail CONFIG="$(realpath "${1:?usage: bash lab1.sh /path/to/alertmanager.yml}")" VERSION=0.34.1 TARBALL="alertmanager-${VERSION}.linux-amd64.tar.gz" BASE="https://github.com/prometheus/alertmanager/releases/download/v${VERSION}" WORK="$(mktemp -d)" && cd "$WORK" curl -fsSLO "${BASE}/${TARBALL}" curl -fsSLO "${BASE}/sha256sums.txt" grep "${TARBALL}" sha256sums.txt | sha256sum -c - tar xzf "${TARBALL}" && cd "alertmanager-${VERSION}.linux-amd64" grep -v '# guard' "$CONFIG" > unguarded.yml # Artefact 2 without its guards ./amtool check-config "$CONFIG" unguarded.yml ./amtool config routes test --config.file="$CONFIG" \ --verify.receivers=oncall-pager severity=page service=checkout-api cp unguarded.yml active.yml # Single node: an empty cluster address disables HA mode. ./alertmanager --config.file=active.yml --storage.path=./data \ --web.listen-address=127.0.0.1:9093 --cluster.listen-address="" > am.log 2>&1 & AM_PID=$! trap 'kill "$AM_PID"; rm -rf "$WORK"' EXIT # teardown when the script ends sleep 3 AM="--alertmanager.url=http://127.0.0.1:9093" # 1. Inhibition as intended: the same cluster label on source and target. ./amtool $AM alert add alertname=ClusterUnreachable cluster=eu-1 severity=page ./amtool $AM alert add alertname=HighErrorRatio cluster=eu-1 \ service=checkout-api severity=page ./amtool $AM alert add alertname=HighErrorRatio cluster=eu-2 \ service=checkout-api severity=page sleep 5 echo "== active"; ./amtool $AM alert query echo "== inhibited"; ./amtool $AM alert query --inhibited # 2. The pitfall: neither alert has a cluster label, so the unguarded rule applies. ./amtool $AM alert add alertname=ClusterUnreachable severity=page ./amtool $AM alert add alertname=EndpointDown service=storefront-probe severity=page sleep 5 echo "== inhibited, unguarded"; ./amtool $AM alert query --inhibited # 3. The fix: load the guarded file and ask again. cp "$CONFIG" active.yml curl -fsS -X POST http://127.0.0.1:9093/-/reload sleep 2 echo "== inhibited, guarded"; ./amtool $AM alert query --inhibited ``` - **Expected output:** `check-config` reports `SUCCESS` for both files, with 2 inhibit rules, 3 receivers and 0 templates (the format in the Alertmanager documentation[^am-config]); `routes test` prints `oncall-pager`. Step 1: active are `ClusterUnreachable` (eu-1) and `HighErrorRatio` (eu-2); only the eu-1 `HighErrorRatio` is inhibited. Step 2: `EndpointDown` is inhibited too, although it concerns no cluster. Step 3: it is active again. - **Validation:** the active eu-2 alert proves that `equal: [cluster]` works when the label exists; step 2 shows the documented missing-label behaviour;[^am-config] step 3 shows the guard removing it. With only `--inhibited`, `amtool` lists only inhibited alerts.[^am-cli] - **Troubleshooting:** `address already in use` means another Alertmanager is running; if nothing is inhibited, raise the `sleep` values; read `am.log` for configuration errors. - **Teardown:** automatic: the `trap` stops Alertmanager and deletes the working directory. To explore by hand, comment out the `trap` line and clean up yourself. #### Lab 2 (intermediate): measure correlation quality - **Objective:** compare three grouping strategies on labelled incidents using pairwise precision and recall, then see what a stale topology does. - **Prerequisites:** Python 3.8 or later; save the script as `correlation_lab.py`. **Cost and risk:** none. **How it is scored.** Every pair of alerts is a decision: the strategy put them in the same group or not, and the ground truth says whether they belong to the same incident. Manning and colleagues present this pair-decision view as an alternative to measures such as purity and normalised mutual information.[^irbook] | Pair of alerts | Same incident (truth) | Different incidents (truth) | |---|---|---| | **Same group** | TP: a correct merge | FP: a wrong merge | | **Different groups** | FN: a missed merge | TN: correctly kept apart | Pairwise precision = TP / (TP + FP); pairwise recall = TP / (TP + FN).[^gml] Pairs grow with the square of an incident's size, so one large incident can dominate both numbers. Report the per-incident over-merge and split counts next to them, as the lab does. _Example status: executed on 2026-10-17 (Python 3.8.10); output below is verbatim._ ```python """Lab 2: compare three alert-grouping strategies against labelled incidents. Standard library only. Usage: python3 correlation_lab.py [--drop-edge api:db] """ import itertools import random import sys from collections import defaultdict # Caller -> callees. A failing callee makes its callers alert too. TOPOLOGY = { "web": ["api"], "api": ["db", "cache", "payments"], "payments": ["payments-db"], "worker": ["queue", "db"], "reports": [], } # Ground truth: (incident id, root service, start minute, alerts per service). INCIDENTS = [ ("INC-1", "db", 0, {"db": ["DBDown"], "api": ["HighErrorRate", "HighLatency"], "web": ["HighErrorRate"], "worker": ["JobFailures"]}), ("INC-2", "reports", 2, {"reports": ["DiskFull"]}), ("INC-3", "payments-db", 40, {"payments-db": ["SlowQueries"], "payments": ["HighLatency"], "api": ["HighLatency"]}), ("INC-4", "queue", 41, {"queue": ["QueueBacklog"], "worker": ["JobLag"]}), ("INC-5", "cache", 90, {"cache": ["Evictions"]}), # A shared network fault: not in the service topology, # and the switch itself is not monitored. ("INC-6", "network", 120, {"cache": ["ConnTimeouts"], "queue": ["ConnTimeouts"]}), ] WINDOW = 5 # minutes def make_events(seed=7): """Each alert fires 1-3 times (re-notifications), with minutes of jitter.""" rng = random.Random(seed) events = [] for inc, _root, start, per_service in INCIDENTS: for svc, names in per_service.items(): for name in names: first = start + rng.randint(0, 3) for repeat in range(rng.randint(1, 3)): events.append({"t": first + repeat, "service": svc, "alertname": name, "truth": inc}) return sorted(events, key=lambda e: (e["t"], e["service"], e["alertname"])) def deduplicate(events, episode_gap=10): """Drop repeats of a fingerprint (service + alertname) within episode_gap.""" last_seen, out = {}, [] for e in events: key = (e["service"], e["alertname"]) if key not in last_seen or e["t"] - last_seen[key] > episode_gap: out.append(e) last_seen[key] = e["t"] return out def neighbours(topology): graph = defaultdict(set) for caller, callees in topology.items(): for callee in callees: graph[caller].add(callee) graph[callee].add(caller) return graph def group(alerts, related): """Join every group with a related member seen within WINDOW minutes. An alert that matches several groups merges them (it bridges them). """ groups = [] for a in alerts: hits = [g for g in groups if any(a["t"] - m["t"] <= WINDOW and related(a, m) for m in g)] if not hits: groups.append([a]) continue keep = hits[0] for other in hits[1:]: keep.extend(other) groups.remove(other) keep.append(a) return groups def pair_scores(groups): tp = fp = fn = 0 members = [(gi, a["truth"]) for gi, g in enumerate(groups) for a in g] for (g1, t1), (g2, t2) in itertools.combinations(members, 2): same_group, same_truth = g1 == g2, t1 == t2 tp += same_group and same_truth fp += same_group and not same_truth fn += (not same_group) and same_truth precision = tp / (tp + fp) if tp + fp else 1.0 recall = tp / (tp + fn) if tp + fn else 1.0 total = precision + recall f1 = 2 * precision * recall / total if total else 0.0 return precision, recall, f1 def probable_cause(g, graph_down): """Candidate root: an alerting service whose callees do not alert here.""" alerting = {a["service"] for a in g} leaves = [s for s in alerting if not (set(graph_down.get(s, [])) & alerting)] return sorted(leaves) def report(name, groups, raw_count): p, r, f1 = pair_scores(groups) truths = defaultdict(set) for gi, g in enumerate(groups): for a in g: truths[a["truth"]].add(gi) merged = sum(1 for g in groups if len({a["truth"] for a in g}) > 1) split = sum(1 for gs in truths.values() if len(gs) > 1) compr = raw_count / len(groups) print(f"{name:<20} groups={len(groups):>2} compr={compr:4.1f}x " f"P={p:.2f} R={r:.2f} F1={f1:.2f} merged={merged} split={split}") def main(): topology = {k: list(v) for k, v in TOPOLOGY.items()} if "--drop-edge" in sys.argv: caller, callee = sys.argv[sys.argv.index("--drop-edge") + 1].split(":") topology[caller].remove(callee) print(f"(topology edge {caller} -> {callee} removed: " "simulating a stale CMDB)") raw = make_events() alerts = deduplicate(raw) print(f"raw events={len(raw)} after dedup={len(alerts)} " f"true incidents={len(INCIDENTS)}") graph = neighbours(topology) strategies = { "S1 time window only": lambda a, b: True, "S2 same service": lambda a, b: a["service"] == b["service"], "S3 topology-aware": lambda a, b: (a["service"] == b["service"] or b["service"] in graph[a["service"]]), } results = {} for name, rel in strategies.items(): results[name] = group(alerts, rel) report(name, results[name], len(raw)) print("S3 probable causes:") roots = {inc: root for inc, root, _s, _a in INCIDENTS} hits = 0 for g in results["S3 topology-aware"]: truths = [a["truth"] for a in g] main_truth = max(set(truths), key=truths.count) cand = probable_cause(g, topology) hit = cand == [roots[main_truth]] hits += hit print(f" {main_truth}: candidates={cand} truth={roots[main_truth]} " f"{'OK' if hit else 'MISS'}") n_groups = len(results["S3 topology-aware"]) print(f"top-1 probable-cause accuracy={hits}/{n_groups}") if __name__ == "__main__": main() ``` **Steps.** (1) Run `python3 correlation_lab.py`. (2) Run `python3 correlation_lab.py --drop-edge api:db`: a CMDB that has lost the fact that the API calls the database. (3) Set `WINDOW` to 2, then 15, and re-run. **Expected output** (step 1, verbatim): ```text raw events=22 after dedup=14 true incidents=6 S1 time window only groups= 4 compr= 5.5x P=0.58 R=1.00 F1=0.73 merged=2 split=0 S2 same service groups=13 compr= 1.7x P=1.00 R=0.07 F1=0.12 merged=0 split=4 S3 topology-aware groups= 7 compr= 3.1x P=1.00 R=0.93 F1=0.97 merged=0 split=1 S3 probable causes: INC-1: candidates=['db'] truth=db OK INC-2: candidates=['reports'] truth=reports OK INC-3: candidates=['payments-db'] truth=payments-db OK INC-4: candidates=['queue'] truth=queue OK INC-5: candidates=['cache'] truth=cache OK INC-6: candidates=['queue'] truth=network MISS INC-6: candidates=['cache'] truth=network MISS top-1 probable-cause accuracy=5/7 ``` Step 2 changes the S3 line to `groups= 8 compr= 2.8x P=1.00 R=0.53 F1=0.70 merged=0 split=2`, and probable-cause accuracy falls to 5/8. - **Validation and interpretation.** The time window has the best compression (5.5×) and perfect recall, but merged the database outage with an unrelated full disk and the payments slowdown with an unrelated queue backlog: precision 0.58. Grouping by service never merges wrongly but finds almost nothing (recall 0.07). Topology does best *because the data were generated from it*, which flatters it. The realistic lessons are the network incident, split because shared infrastructure is missing from the graph, and step 2, where one missing edge cuts recall from 0.93 to 0.53. - **Troubleshooting:** `ValueError: list.remove(x)` means the `--drop-edge` pair does not exist (use `caller:callee` from `TOPOLOGY`); an f-string syntax error means Python is older than 3.6. - **Teardown:** delete the script; it writes nothing. #### Lab 3 (advanced): seasonal anomaly detection against a static threshold - **Objective:** compare a static band with a seasonal robust baseline on two weeks of synthetic traffic with four labelled faults and two unlabelled business events (a weekday holiday and a marketing campaign). - **Method:** the modified z-score, 0.6745 × (x − median) / MAD, against the same 15-minute slot of the same type of day, flagging absolute scores above 3.5, the threshold Iglewicz and Hoaglin recommend according to the NIST handbook.[^nist-outliers] As with a Prometheus `for` clause, a flag must hold for five minutes before an alert fires.[^prom-rules] - **Prerequisites:** Python 3.8 or later; save the script as `anomaly_lab.py`. **Cost and risk:** none (about 0.1 s of CPU). _Example status: executed on 2026-10-17 (Python 3.8.10); output below is verbatim._ ```python """Lab 3: static thresholds versus a seasonal robust baseline. Standard library only. A synthetic request rate, two weeks at 1-minute resolution: week 1 trains, week 2 is scored. Usage: python3 anomaly_lab.py [--calendar] """ import math import random import statistics import sys DAY = 1440 rng = random.Random(42) def normal_rate(minute): """Daily cycle (low at night, peak mid-afternoon), quieter weekends, noise.""" hour = (minute % DAY) / 60 weekday = (minute // DAY) % 7 < 5 if 6 <= hour <= 22: base = 200 + 800 * max(0.0, math.sin(math.pi * (hour - 6) / 16)) else: base = 200 base *= 1.0 if weekday else 0.6 return max(0.0, rng.gauss(base, 0.05 * base + 10)) series = [normal_rate(m) for m in range(14 * DAY)] # Labelled anomalies in week 2: (name, start minute, length, transform). W2 = 7 * DAY ANOMALIES = [ ("peak-hour drop to 40%", W2 + 0 * DAY + 14 * 60, 30, lambda v, i: v * 0.4), ("night spike x2.5", W2 + 1 * DAY + 3 * 60, 15, lambda v, i: v * 2.5), ("total outage (zero)", W2 + 2 * DAY + 10 * 60, 20, lambda v, i: 0.0), ("slow 30% decline", W2 + 3 * DAY + 8 * 60, 600, lambda v, i: v * (1 - 0.3 * i / 600)), ] truth = set() for _name, start, length, f in ANOMALIES: for i in range(length): series[start + i] = f(series[start + i], i) truth.add(start + i) # Two unlabelled business events in week 2: real changes in traffic, not incidents. HOLIDAY = 11 # a public holiday on a weekday (day 11): weekend-like traffic for m in range(HOLIDAY * DAY, (HOLIDAY + 1) * DAY): series[m] *= 0.6 for m in range(W2 + 5 * DAY + 18 * 60, W2 + 5 * DAY + 20 * 60): # a campaign series[m] *= 1.6 train, test_start = series[:W2], W2 # Detector 1: one static band from the training week (0.1th and 99.9th percentiles). q = statistics.quantiles(train, n=1000) low, high = q[0], q[-1] def static_flag(m): return series[m] < low or series[m] > high # Detector 2: robust z-score against the same 15-minute slot of the same day type. # --calendar tells the detector that day 11 is a holiday (treat it like a weekend). HOLIDAYS = {HOLIDAY} if "--calendar" in sys.argv else set() def slot(m): day = m // DAY return (day % 7 < 5 and day not in HOLIDAYS, (m % DAY) // 15) buckets = {} for m, v in enumerate(train): buckets.setdefault(slot(m), []).append(v) baseline = {} for k, vs in buckets.items(): med = statistics.median(vs) mad = statistics.median(abs(v - med) for v in vs) or 1.0 baseline[k] = (med, mad) def seasonal_flag(m): med, mad = baseline[slot(m)] # Modified z-score (Iglewicz and Hoaglin): flag when |M| > 3.5. return abs(0.6745 * (series[m] - med) / mad) > 3.5 def alerts(flag, for_minutes=5): """Fire once the flag has held for for_minutes (like Prometheus 'for:').""" fired, run, out = False, 0, [] for m in range(test_start, len(series)): run = run + 1 if flag(m) else 0 if run >= for_minutes and not fired: out.append(m) fired = True if run == 0: fired = False return out def matches(m): """True if an alert at minute m belongs to a labelled anomaly (10 min grace).""" return any(s <= m < s + n + 10 for _a, s, n, _f in ANOMALIES) def evaluate(name, fired): detected, delays = [], [] for aname, start, length, _f in ANOMALIES: hits = [m for m in fired if start <= m < start + length + 10] detected.append(aname if hits else None) if hits: delays.append(hits[0] - start) false = [m for m in fired if not matches(m)] on_holiday = sum(1 for m in false if m // DAY == HOLIDAY) precision = (len(fired) - len(false)) / len(fired) if fired else 1.0 recall = sum(d is not None for d in detected) / len(ANOMALIES) print(f"{name:<18} alerts={len(fired):>3} false={len(false):>3} " f"(holiday {on_holiday:>2}) P={precision:.2f} R={recall:.2f}") print(f" delays to first alert (min): {delays}") for (aname, *_rest), d in zip(ANOMALIES, detected): print(f" {'caught' if d else 'MISSED':<6} {aname}") print(f"static band: alert below {low:.0f} or above {high:.0f} requests/min") evaluate("static band", alerts(static_flag)) evaluate("seasonal robust z", alerts(seasonal_flag)) ``` **Steps.** (1) Run `python3 anomaly_lab.py`. (2) Run `python3 anomaly_lab.py --calendar`, which tells the detector that day 11 is a holiday. **Expected output** (step 1, verbatim): ```text static band: alert below 84 or above 1129 requests/min static band alerts= 1 false= 0 (holiday 0) P=1.00 R=0.25 delays to first alert (min): [4] MISSED peak-hour drop to 40% MISSED night spike x2.5 caught total outage (zero) MISSED slow 30% decline seasonal robust z alerts= 44 false= 34 (holiday 32) P=0.23 R=1.00 delays to first alert (min): [4, 4, 4, 401] caught peak-hour drop to 40% caught night spike x2.5 caught total outage (zero) caught slow 30% decline ``` With `--calendar` the seasonal line becomes `alerts= 12 false= 2 (holiday 0) P=0.83 R=1.00`; nothing else changes. - **Validation and interpretation.** A static band wide enough for the daily cycle misses everything except the total outage. The seasonal baseline catches all four faults, the slow decline only after 401 minutes, but without context it fires 34 false alerts, 32 of them on the holiday. One line of calendar data removes all 32; the 2 left come from the marketing campaign, which only a change or event feed could explain. - **Troubleshooting:** `AttributeError: module 'statistics' has no attribute 'quantiles'` means Python is older than 3.8. - **Teardown:** delete the script; it writes nothing. ### Use cases and scenarios | Scenario | Context | Method | Trade-offs | Success metric | Pitfall | |---|---|---|---|---|---| | 1. Alert storm from a shared dependency | A database, network zone or cluster fails; dozens of services complain | Guarded inhibition plus topology-aware grouping | Wrong labels make inhibition hide real incidents | Pairwise precision and recall; alerts examined per incident | Shared infrastructure missing from the topology | | 2. Change-induced regressions | Frequent deploys; many outages follow changes | Attach recent changes to each incident; rank recent changes as candidate causes | Needs change events labelled by service | Top-3 cause accuracy on change-caused incidents | Changes made outside CI/CD are invisible | | 3. Business-traffic anomalies | Seasonal traffic where a 40% drop at peak is invisible to static thresholds | Seasonal baselines with calendars and minimum durations | False alerts on holidays and campaigns; slow drifts detected late | Precision and recall on labelled or injected anomalies | Treating every anomaly as an incident | | 4. Guarded auto-remediation | A recurring fault with a known, reversible fix | Allow-listed playbook, throttle, verification, rollback | Automation can amplify a mistake across many hosts | Success, rollback and human-override rates | Allow-lists that grow without review | | 5. LLM-assisted triage | Long incidents with many responders and noisy channels | Summaries, timelines, similar-incident search, draft causes for a person to confirm | Plausible but wrong text; data leakage | Time to engage the right team; rate of wrong suggestions | Letting model output trigger actions | **Scenario 1: a documented case.** Zhao and colleagues applied their alert-storm method at a large commercial bank (China EverBright Bank), reporting storm detection with an F1-score above 0.9 and more than 98% fewer alerts to examine.[^zhao2020] Microsoft researchers, after two years of incident management, named "incomplete service/resource dependencies" one of two critical challenges,[^chen2020] the weakness Lab 2's `--drop-edge` step exposes. **Scenarios 2 and 3: evidence for the approach, not documented cases.** Scenario 2 rests on the change evidence in [core practices](#core-practices-and-playbooks); Lab 3 shows why Scenario 3 needs calendars and change feeds. **Scenarios 4 and 5: illustrative.** Scenario 4 is Artefact 5; Scenario 5 rests on the Microsoft studies in [Automation and AI](#automation-and-ai-language-models-and-agents). ## Measure, govern, pay ### Metrics and KPIs These definitions apply across the Atlas. **Precision** is the share of a system's positive decisions that were right; **recall** is the share of the true cases it found.[^gml] For grouping, a decision is a pair of alerts placed together or apart, as in Lab 2.[^irbook] Security detection adds base rates and adversarial testing; see [AIOps for Security](../aiops-for-security/). | KPI | Definition and formula | Data source | Pitfall | Review cadence | |---|---|---|---|---| | Pairwise grouping precision | TP pairs ÷ (TP + FP pairs) | Labelled incidents | Needs labels; big incidents dominate (Lab 2) | Monthly | | Pairwise grouping recall | TP pairs ÷ (TP + FN pairs) | Labelled incidents | Same | Monthly | | Over-merge rate | Groups containing alerts from more than one incident ÷ groups | Labels | Hides the second incident | Monthly | | Split rate | Incidents whose alerts landed in more than one group ÷ incidents | Labels | Causes duplicate pages | Monthly | | Missed-incident rate | Confirmed incidents for which no person was notified in time ÷ incidents | Post-incident reviews, customer reports | Rare, so trends are noisy; review every case | Every incident | | Actionable page rate | Pages that led to a human action ÷ pages | Incident tool | Responders may "act" on noise to look busy | Weekly | | Compression ratio | Raw alerts ÷ incidents presented | Pipeline counters | **Vanity metric** without precision and recall | Context only | | Top-k probable-cause accuracy | Incidents whose confirmed cause is in the top k candidates ÷ incidents with a confirmed cause | Labels | Causes are often multiple; record the main one | Monthly | | Detector precision, recall and delay | Alert episodes that matched a labelled anomaly ÷ episodes; anomalies caught ÷ anomalies; median minutes to first alert | Labels, injected test anomalies | Unlabelled business events | Monthly | | Forecast lead time and false forecasts | Median minutes a forecast alert precedes exhaustion (or the static alert); forecasts with no exhaustion per week | Labelled exhaustion events | Clean-up jobs make forecasts flap | Quarterly | | Label-contract compliance | Alerts carrying every contract label ÷ alerts, sampled from the Alertmanager API (Artefact 1) | `label_contract.py alerts` | A label that is present but means the wrong thing still passes | Weekly | | Remediation success, rollback and override rates | Verified fixes ÷ runs; rollbacks ÷ runs; human overrides ÷ runs | Runner logs | Success that only masks a cause | Monthly | | Time to engage the owning team | First alert → acknowledgement by the team that fixed it | Incident tool | Spell out which clock you mean; see the MTTx caveats in [IncidentOps](../incidentops/#metrics-and-kpis) | Monthly | **The asymmetry that sets targets.** On a paging route a split costs a duplicate page, while a wrong merge can hide a second incident: set strict targets for precision and missed incidents (review every miss), a looser one for splits. A ticket-only route can tilt towards recall. Derive the numbers from your own shadow-mode baseline; the research for this page found no credible public benchmarks. ::: example Worked calculation (illustrative numbers) In one month the pipeline received 2,400 alerts and presented 180 groups: a compression ratio of 13.3. Responders labelled the 60 incidents in the month; scoring every alert pair in them gave TP = 1,860, FP = 240 and FN = 410 pairs. - Pairwise precision = 1,860 ÷ (1,860 + 240) = 1,860 ÷ 2,100 = **0.886** - Pairwise recall = 1,860 ÷ (1,860 + 410) = 1,860 ÷ 2,270 = **0.819** - F1 = 2 × 0.886 × 0.819 ÷ (0.886 + 0.819) = **0.851** - Over-merge rate = 9 mixed groups ÷ 180 = **5.0%**; split rate = 11 split incidents ÷ 60 = **18.3%** - Missed incidents = 1 ÷ 60 = **1.7%**: one outage was reported by customers first. It gets its own review. - Top-3 probable-cause accuracy = 38 ÷ 52 incidents with a confirmed cause = **73%** - Remediation: 140 runs, 131 verified (**93.6%**), 6 rollbacks (**4.3%**), 3 human overrides Reading: a compression of 13.3 looks excellent, but the 5% of mixed groups and the missed incident come first; the 18% split rate (duplicate pages) comes second. ::: ### Quality, security, privacy and compliance | Risk | Signal | Mitigation | Owner | |---|---|---|---| | A suppression or inhibition hides a real incident | Customer-reported incidents; the missed-incident rate | Guarded rules; expiry dates; a monthly review of sampled inhibited alerts | Event-intelligence owner | | Forged or replayed events create or hide incidents | Unknown sources; spikes from one sender | Authenticate ingest webhooks; allow-list sources; rate-limit per source | Platform team | | Anyone who reaches the Alertmanager API silences an outage | Silences created by unknown users or from unknown addresses | Network policy on port 9093; basic auth and TLS (`--web.config.file`, still "experimental"); silences only through an authenticated proxy that records who set them; monthly silence audit[^prom-security][^am-https] | Platform team | | An over-privileged remediation runner | Broad credentials; shared accounts | One narrow credential per action; separate runner identity; approval for new action classes | Automation owner | | Prompt injection through alert or log text into an LLM assistant | Odd instructions in summaries | Treat event text as untrusted; no tool access without an allow-list (see [LLMOps](../llmops/)) | Event-intelligence owner | | Personal data in events and incident text | User identifiers in log lines | Redact at ingest; set retention; check vendor data location ([PrivacyOps](../privacyops/)) | Service owners | | Unclear regulatory status of automated control | AIOps actions in critical infrastructure | Legal review (below); decision records | Risk or compliance | **Regulation, briefly and with caveats (as of October 2026).** Most AIOps uses carry no high-risk obligations, but two EU duties deserve a check. First, Article 4 of the AI Act (AI literacy, applicable since 2 February 2025) binds deployers of any AI system, whatever its risk class. As amended by the Digital Omnibus on AI (Regulation (EU) 2026/1744, in force since 27 July 2026), deployers "shall take measures to support the development of AI literacy" of their staff.[^euaiact-art4][^eu-omnibus] Whether a tool is an "AI system" in law is a legal question; a learned correlation engine is more likely to be one than a rule-based one. Second, Annex III lists as high-risk "AI systems intended to be used as safety components in the management and operation of critical digital infrastructure, road traffic, or in the supply of water, gas, heating or electricity";[^euaiact] the Omnibus makes the high-risk obligations for Annex III systems apply from 2 December 2027.[^eu-omnibus] If you operate critical digital infrastructure in the EU and an AIOps function could act as a safety component, take legal advice. Mapping AI systems to the Act belongs to [ModelOps](../modelops/), and audit evidence to [ComplianceOps](../complianceops/). Elsewhere, NIST's AI Risk Management Framework is "intended for voluntary use", on 7 April 2026 NIST released a concept note for a profile on trustworthy AI in critical infrastructure, and NIST says it is revising AI RMF 1.0.[^nist-airmf] **Auditability.** Record for every grouping, suppression and automated action which rule or model, which version, which inputs, and what a person later confirmed; Artefact 4's `correlation` block is a minimal version. ### Cost, capacity and sustainability - **Cost drivers.** Licences (pricing changes often; get written quotes), integration engineering, the owner's and service owners' time, labelling, model compute and LLM tokens ([FinOps for AI](../finops-for-ai/)). - **Unit view.** Track cost and triage people-hours per incident presented. Book "pages avoided" as savings only if you can show they were noise. - **Capacity: survive the storm you were built for.** A correlation layer is busiest when everything else is failing. Alertmanager warns that `group_interval` "also sets the context timeout for the notification pipeline", so a slow receiver can get notifications cancelled.[^am-config] BigPanda caps an incident at 300 correlated alerts,[^bigpanda-corr] and PagerDuty's learned grouping only adds alerts to incidents under 24 hours old.[^pd-iag] Load-test with a replayed storm. - **Sustainability.** The footprint is usually small next to the telemetry it reads; keep model training and LLM calls proportional to measured value ([GreenOps](../greenops/)). ### Assurance of the operating model 1. **A heartbeat that bypasses correlation.** The always-firing Watchdog alert and its external checker ([IncidentOps, Lab 2](../incidentops/#lab-2-intermediate-prove-the-paging-path-end-to-end)) catch a dead pipeline, not wrong grouping. 2. **Missed-incident review.** Every post-incident review asks: *did the event pipeline hide, delay or misroute anything?* 3. **Suppression audit.** The monthly sample of inhibited and suppressed alerts and silences ([checklists](#checklists)), including who created each silence. 4. **Drift signals.** Alert volume per service against its baseline, the age of topology entries, and label-contract compliance. 5. **Evaluation history.** Keep the monthly scorecard in version control, so that a platform change can be compared with the past. ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Remediation | Prevention | |---|---|---|---|---| | Customers report an outage, but nobody was paged | Over-broad inhibition or suppression; missing or ambiguous labels | `amtool alert query --inhibited`; `amtool silence query`[^am-cli][^am-readme] | Fix the labels or guard the rule; page manually | Guarded inhibit rules; label linting; the heartbeat; a suppression audit | | Two teams paged for one fault | Over-splitting: a topology gap or inconsistent service names | Compare group IDs for the incident's alerts | Add the edge; normalise the labels | Generate topology from traces; enforce the contract | | One incident contains unrelated faults | Window too wide; learned grouping drifted | A drop in pairwise precision | Narrow the window; add label constraints; reset or retrain the learner | Shadow-mode evaluation before every change | | Anomaly alerts on holidays and campaigns | No calendar or change context | False alerts cluster on dates | Add calendars and campaign feeds | A detector review each quarter | | Slow degradation never alerts | The baseline absorbs gradual drift | Compare with the same slot last month | Add SLO burn-rate alerts (see [SRE](../sre/)) | Keep SLO alerts outside the anomaly layer | | Auto-remediation runs every few minutes | The fix masks the cause; throttle missing | Runner logs | Disable the action class; page the owner | Throttle plus a daily cap and escalation | | The probable cause is always "the database" | A hub node in the topology, or a narrow menu of causes | Cause accuracy by service | Weight by recent changes; fix the topology | Evaluate top-k accuracy monthly | | Notifications delayed or cancelled during storms | Pipeline overload; receiver slower than `group_interval` | Alertmanager logs | Raise `group_interval`; scale receivers | Storm replay tests | ### Worked runbook: the inhibition rule that silenced an outage **Scenario (illustrative; the mechanism is documented).** A team's first label contract took `cluster` only from `external_labels`, and its meta-monitoring Prometheus had none. That server raises `EndpointDown` for a storefront synthetic probe, without a `cluster` label, and `ClusterUnreachable` for a forgotten development cluster whose scrape target has no `cluster` label either. The unguarded inhibit rule uses `equal: [cluster]`; with the label missing from both alerts, Alertmanager applies it.[^am-config] Customers report failed checkouts 25 minutes later. ::: figure Before and after: how a missing label hides an outage, and how the guard stops it ```mermaid flowchart TB S["Source: ClusterUnreachable, no cluster label"] --> R{"Unguarded rule: equal cluster"} T["Target: EndpointDown probe, no cluster label"] --> R R -->|"before: empty equals empty"| X["Inhibited: outage hidden"] S --> G{"Guarded rule: source needs cluster=~'.+'"} G -->|"after: source does not match"| N["Probe pages"] F["Wrong fix: external_labels cluster=meta"] -.->|"both alerts get meta: still equal"| X classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class R,G decision class X,F risk class N core class S,T neutral ``` Text equivalent: a ClusterUnreachable source and an EndpointDown probe both lack a cluster label. Before the fix, the unguarded rule finds the two empty values equal and inhibits the probe, hiding the outage. After it, the guarded rule needs a non-empty cluster on the source, the source does not match, and the probe pages. The tempting wrong fix, external label cluster=meta on the meta-monitoring server, gives both alerts the value meta, so the probe stays inhibited. ::: 1. **Declare and page by hand.** Do not wait for the pipeline. Follow the incident process in [IncidentOps](../incidentops/). 2. **Is the alert there at all?** `amtool --alertmanager.url=http://127.0.0.1:9093 alert query alertname=EndpointDown` returns nothing; adding `--inhibited` returns it.[^am-cli] Now you know that it is muted, not missing. 3. **What is muting it?** Silences: `amtool silence query`. Inhibitions: the v2 API returns, with `active=false`, only silenced or inhibited alerts, and each alert's `status.inhibitedBy` lists the fingerprints of the alerts that inhibit it.[^am-openapi][^am-inhibit] _Example status: syntax-checked, not executed (requires a running Alertmanager and `jq`)._ ```bash curl -s 'http://127.0.0.1:9093/api/v2/alerts?active=false' \ | jq -r '.[] | [.labels.alertname, (.labels.cluster // "-"), (.status.inhibitedBy | join(","))] | @tsv' ``` 4. **Mitigate.** Remove the stale target so that the source resolves, and add the guard (`cluster=~".+"` in `source_matchers`, as in Artefact 2); reload with `POST /-/reload` or `SIGHUP`.[^am-mgmt] Do **not** add `external_labels: {cluster: meta}` to the meta-monitoring server: both alerts would carry `cluster="meta"`, still equal, and the probe would stay muted. 5. **Verify.** `amtool alert query --inhibited` no longer lists `EndpointDown`, and a test `EndpointDown` from the meta-monitoring server pages. 6. **Prevent.** (a) Give `cluster` one meaning, the affected cluster, and record the evaluating server in `prometheus` (Artefact 1). (b) Guard every inhibit rule: each `equal` label non-empty on the source. (c) Never let meta-monitoring alerts be inhibition targets (`prometheus!="meta"`). (d) Lint every rule, check every server's external labels in CI, and track compliance (a `promtool` test never sees `external_labels`). (e) Monthly, fire a test alert without the label from the meta-monitoring server and confirm that it pages. ### Anti-patterns and myths - **"Buy AIOps to fix alert noise."** If rules page on causes, flap, or lack owners, a platform groups the noise more neatly. Fix the rules first ([SRE](../sre/)). - **Reporting compression as success.** Lab 2's 5.5-times compression merged unrelated incidents. - **Suppression without expiry.** Every permanent silence is a future missed incident. - **"The model will learn our topology."** Learned grouping learns only from the feedback its vendor counts (see [governance](#operating-model-and-governance)); PagerDuty warns that test data can make it behave unpredictably.[^pd-iag] - **Root-cause theatre.** A confident single cause from a narrow menu can send responders the wrong way. Show the evidence and the runners-up. - **Autonomy first.** Agents diagnose better than they resolve ([Automation and AI](#automation-and-ai-language-models-and-agents)). Start with suggestions; earn automation one action class at a time. - **Black-box vendor evaluation.** A demo on the vendor's data proves nothing about yours. Insist on shadow mode with your labels. ## Deciding and adopting ### When to use and when not to use | Use AIOps techniques when | Do not use them yet when | |---|---| | One failure regularly produces many alerts across services or tools | Alert rules page on causes, flap, or have no owners: fix the rules first | | Several monitoring tools page the same people with inconsistent alerts | You have one monitoring tool and a small estate whose topology people know | | Seasonal metrics hide real faults from static thresholds | Nobody will label groupings and causes, so nothing can be measured | | A recurring fault has a known, reversible fix that people apply by hand | The "fix" changes data, money or security posture: keep a person in the loop | | Responders lose time reconstructing what changed and who owns what | Ownership and change records do not exist: build those first; they pay off on their own | **Hidden responsibilities.** Someone must own the pipeline, keep topology current, run evaluations, review suppressions and approve automation classes; a managed platform hides that work rather than removing it. ### Decision framework The [technique tree](#methods-and-frameworks-rules-statistics-machine-learning-and-language-models) decides *what* to adopt; this weighted worksheet chooses *which platform or stack*, after a shadow-mode trial on your labelled incidents. The weights are this Atlas's suggestion; change them, but write down why. | Criterion | Weight | How to score 1–5 | Evidence to ask for | |---|---|---|---| | Pairwise precision on your labelled incidents | 20 | 5 = meets your target in shadow mode | Their export of groups, scored with your labels | | Pairwise recall and missed incidents | 20 | 5 = no missed incidents in the trial | Same | | Explainability of each grouping and cause | 15 | 5 = every decision shows its evidence | Screenshots or API fields per decision | | Integration with your monitors, ITSM and topology | 15 | 5 = native, two-way, no custom glue | A working connector in the trial | | Portability (export alerts, groups, labels, rules) | 10 | 5 = full export through an API | API documentation | | Safety of automation (allow-lists, throttles, approvals, audit) | 10 | 5 = all four, enforced | Configuration demonstrated live | | Data handling (residency, retention, redaction) | 5 | 5 = meets policy without exceptions | Contract and documentation | | Total cost over three years, including people | 5 | 5 = lowest of the options | Written quote plus internal effort estimate | ::: decision How to read the score A weighted total below 3.5 out of 5, or a 1 on either recall or explainability, means "not yet", whatever the demo looked like. A composable open-source stack scored honestly (high on portability and cost, lower on integration) is a legitimate winner for a team that can run it. ::: ### Maturity model (this Atlas's model) This model is this Atlas's proposal, not an industry standard. Move up only when the exit criteria hold for a full quarter. | Level | Observable behaviour | Exit criteria | Typical constraint | |---|---|---|---| | 0. Raw alerts | Every monitor pages directly; duplicates are common | Alert inventory exists; noisiest rules known | No ownership data | | 1. Hygiene | Label contract; deduplication; `group_by`; owners on every alert | 95% of alerts pass the contract (measured in Alertmanager); actionable-page rate measured | Inconsistent tools | | 2. Context | Topology and change data attached to incidents; guarded inhibition | Labels captured at close for most incidents; first precision and recall scorecard | Topology freshness | | 3. Measured intelligence | Learned or statistical grouping and anomaly detection run in shadow mode before release | Policies meet targets; missed incidents reviewed every time | Labelling effort | | 4. Guarded automation | Allow-listed remediation for several fault classes; LLM summaries with human confirmation | Success and rollback rates within targets; no automation-caused incident in a quarter | Change-authority trust | There is deliberately no level 5 "autonomous operations": the evidence on this page does not support it as a goal for most organisations in 2026. ### Adoption roadmap: 30/60/90 days | Window | Actions | Milestone | Metric | |---|---|---|---| | Days 1–30 | Name the event-intelligence owner. Export three months of alerts and incidents; measure alerts per incident, pages per week and the actionable-page rate. Write the label contract; lint every rule in CI; measure compliance weekly. Add one-click labels at close | Baseline report; contract in CI | Contract compliance; actionable-page rate | | Days 31–60 | Deduplicate and group by `cluster` and `service`. Add two guarded inhibition rules. Build a topology file for tier-1 services, including shared infrastructure. Attach recent changes to incidents (Artefact 4). Start the monthly scorecard | First scorecard | Pairwise precision and recall; split rate | | Days 61–90 | Shadow-trial one statistical or learned technique (or one vendor) against the labels. Automate one reversible remediation class through the gate. Run a storm replay test | Go or no-go decision recorded | Shadow-mode scores; remediation success and rollback rates | Quick wins usually come from the ten noisiest rules and from attaching changes to incidents; topology data is the usual blocker, so start collecting it on day 1. ## Toolkit ### Checklists **Quick start (first week)** - [ ] An event-intelligence owner is named, and the time is in their objectives - [ ] Three months of alerts and incidents exported; baseline alerts per incident and actionable-page rate - [ ] The label contract ([Artefact 1](#reference-implementation)) is written - [ ] Incident close asks "grouped correctly?" and "cause correct?" **Before enabling any suppression, inhibition or new grouping policy** - [ ] The policy is in version control, with checks (rule linting, `promtool test rules`, `amtool config routes test`) - [ ] Every label in `equal` names the affected component (not the evaluating server) and is required non-empty in `source_matchers` - [ ] Alerts from meta-monitoring are excluded as inhibition targets - [ ] Shadow-mode results on labelled incidents meet the precision, recall and missed-incident targets - [ ] Suppressed alerts stay queryable; the suppression has an owner and an expiry date - [ ] A rollback (revert and reload) has been rehearsed **Before automating a remediation class** - [ ] The fault is well understood, and the fix is reversible and idempotent - [ ] Blast-radius limit, throttle and daily cap are set - [ ] The runner's credential can do this action and nothing else; the ingest endpoint is authenticated or bound to localhost - [ ] Verification and rollback steps are automated; failure pages a person - [ ] The change authority has approved the class; the review date is set **Monthly review** - [ ] Scorecard updated (Template 3), and every missed incident reviewed - [ ] A person has inspected a random sample of inhibited and suppressed alerts, and the silences - [ ] Topology entries older than 90 days re-confirmed by their owners - [ ] The ten noisiest rules have actions assigned ### Templates **Template 1: policy record (one per grouping rule, inhibition, detector or remediation class)** ```text Policy ID / name: e.g. INH-007 "ClusterUnreachable mutes symptoms" Type: grouping | inhibition | suppression | detector | remediation Owner (team, person): Scope: services, clusters, alertnames Logic: link to the rule or config in version control (commit) Labels relied on: each label in equal or group_by: its meaning (affected or evaluating), who sets it, and the non-empty guard Evidence for release: shadow-mode period, labelled incidents used, precision, recall, missed incidents Failure mode if wrong: what could be hidden, merged or changed; e.g. INH-007: the meta server loses its route to eu-1 while eu-1 still alerts, so eu-1's real alerts are muted Expiry / review date: Rollback: revert commit + reload, expected time Approved by / date: ``` **Template 2: automation safety review** ```text Action class: e.g. rotate and compress logs on LogVolumeNearlyFull Trigger (exact labels): Allow-list entry: yes/no, and the commit Blast-radius limit: max hosts per run / per hour; share of capacity Throttle and daily cap: once_within ... ; max runs per day ... Credential: identity, permissions, where stored, rotation Verification: check that proves the fix worked, and timeout Rollback: automatic step; who is paged on failure Data, money or security impact: none / describe (if any, keep a person in the loop) Change authority decision: approved / rejected, date, review date ``` **Template 3: monthly evaluation scorecard** | Metric | Target | This month | Last month | Notes and actions | |---|---|---|---|---| | Pairwise precision (paging routes) | | | | | | Pairwise recall | | | | | | Over-merge rate | | | | | | Split rate | | | | | | Missed incidents (count, links to reviews) | 0 | | | | | Top-3 probable-cause accuracy | | | | | | Actionable page rate | | | | | | Remediation success / rollback / override | | | | | | Label-contract compliance | | | | | **Template 4: platform shadow-trial plan** ```text Candidate and version: Data supplied: N weeks of raw alerts, M labelled incidents (list), topology file, change log Mode: shadow only; no paging changes Success criteria: precision >= __, recall >= __, missed incidents = 0, an explanation shown for every group Exports required: alerts, groups, causes, decisions with reasons (API) Security and data review: residency, retention, redaction, sub-processors Decision date and owner: ``` ### Deep comparisons Four traditions handle operational events; they are layers, not rivals. | Approach | Unit of work | How it reduces noise | Strength | Weakness | Atlas page | |---|---|---|---|---|---| | ITIL event management | Informational, warning and exception events[^splunk-itil] | Filtering, categorising and correlating within a process | Ties events to incident, problem and change processes | Rules need constant upkeep | [ITOps](../itops/) | | SRE symptom-based alerting | SLOs and burn rates | Page only on user-visible symptoms | Few pages, high precision | Says what is broken, not why | [SRE](../sre/) | | AIOps and event intelligence | Groups of alerts as incidents | Deduplication, grouping, topology, learning | Works across tools and at scale | Needs labels and context; can be opaque | This page | | Security analytics | Detections and cases | Tuning, ML-assisted triage | Built for an adversary | Base rates; evasion | [AIOps for Security](../aiops-for-security/) | A sound design uses SLO alerting to decide what deserves a page, event management to connect events to processes, and AIOps to group, enrich and rank what remains. ### FAQ **Do we need data scientists?** Not to start: levels 1 and 2 of the [maturity model](#maturity-model-this-atlass-model) are engineering and ownership work. Evaluating detectors and vendors needs statistical literacy; data scientists only if you build models. **How many labelled incidents do we need?** No credible published threshold exists. A few dozen expose gross over-merging; a fair comparison of learned grouping with rules needs more, across kinds of failure. The decision tree's 100 is a rule of thumb. **Will AIOps reduce MTTR?** It can shorten the time to engage the right team and find a probable cause, but overall MTTR measures that poorly: durations are skewed and the "R" is ambiguous ([IncidentOps](../incidentops/#metrics-and-kpis)). Measure the phases AIOps affects. **Can an LLM do root-cause analysis for us?** It can suggest causes for a person to confirm: Microsoft reported root-cause *category* accuracy up to 0.766 on its own incidents,[^rcacopilot] but agents rarely resolve live faults (see [Automation and AI](#automation-and-ai-language-models-and-agents)). **Should security alerts go through the same correlation engine?** They can share plumbing, but evaluate and govern them separately: an attacker adapts to your detectors, and missed detections cost differently. See [AIOps for Security](../aiops-for-security/). **Is Kubernetes self-healing AIOps?** No. Restarting failed containers and rescheduling pods is the orchestrator's built-in control loop ([KubernetesOps](../kubernetesops/)). AIOps sits above it and correlates what such loops cannot fix. ## Going further ### Advanced topics and emerging direction | Topic | Status (October 2026) | What to watch | |---|---|---| | Graph-based and causal root-cause analysis | Active research; no method best in all situations;[^pham2024] vendor fault-tree analysis[^dt-rca] | Results on real systems, not synthetic data | | LLM summaries and diagnosis | Evaluated at Microsoft; one team's production use (2023)[^ahmed2023][^rcacopilot] | Evaluation on your incidents; prompt injection; data handling | | Agents that act ("AgentOps", in AIOpsLab's term) | Research and early products; low resolution rates in ITBench (early-2025 models); higher, differently defined diagnosis scores in 2026[^aiopslab][^itbench][^itbench-aa] | Whether gains in diagnosis carry over to live resolution; human approval gates | | Shared semantic conventions | OpenTelemetry resource conventions (1.44.0) standardise service identity[^otel-service] | Fewer label mismatches between tools | **Distinguish three kinds of claim.** *Established*: deduplication, grouping, inhibition, topology and change correlation, statistical baselines. *Active research with limited production evidence*: learned grouping, LLM summaries, LLM-assisted cause classification. *Marketing and speculation*: "autonomous operations", single-number noise-reduction promises, root cause without evidence. ### Learning path - **Prerequisites:** alerting basics and SLOs ([SRE](../sre/)), telemetry ([ObservabilityOps](../observabilityops/)), incident roles ([IncidentOps](../incidentops/)), and a little Python and statistics (median, percentiles, precision and recall). - **One-day primer:** the first three parts of this page; Google's chapter on monitoring distributed systems;[^sremon] Labs 2 and 3. - **One week:** run Lab 1; write a label contract and lint your own rules; compute alerts per incident and the actionable-page rate for a month of alerts. - **Thirty days:** label 20 incidents; score your current grouping with pairwise precision and recall; build a tier-1 topology file; attach changes to incidents. - **Advanced:** study the surveys on AIOps failure management and LLM-era methods;[^notaro2020][^remil2024][^zhang2024] trial one learned technique in shadow mode; design a guarded remediation class end to end. - **Portfolio projects:** (1) a storm-replay test harness for your alert pipeline; (2) an evaluation scorecard generated from incident labels; (3) a guarded remediation for one fault, with verification and rollback, reviewed by your change authority. ## Wrap-up ### Executive recap and cheat sheet | Topic | Remember | |---|---| | What it is | Statistics, ML and LLMs on operational events: group, detect, rank, guard | | What it is not | Observability, incident management, security analytics, or operations *for* AI | | First moves | Label contract → deduplication → grouping and inhibition → topology and changes | | When to add learning | Seasonal metrics, many inconsistent tools, enough labelled incidents to prove it in shadow mode | | Metrics | Pairwise precision and recall, over-merge and split rates, missed incidents, top-k cause accuracy, remediation rates; compression only as context | | Automation | Allow-listed, reversible, throttled, verified, least privilege; people approve new classes | | LLMs | Read-only first; treat event text as untrusted; evaluate on your incidents | | Tools | Alert routers, event-intelligence platforms, RCA built into observability suites, event-driven automation ([tooling](#tooling-landscape-as-of-october-2026)) | | Watch-outs | Missing or two-meaning labels in inhibition, an open Alertmanager API, stale topology, holidays, slow drift, engine retirements | | Next action | Measure your actionable-page rate and label 20 incidents this month | ### A judgement call: buy a platform now, or fix alert hygiene first? **Situation (illustrative).** A company runs three monitoring tools; on-call engineers receive about 1,500 pages a month, one in five actionable. Peak season starts in eight weeks. A vice-president proposes buying an event-intelligence platform now; a vendor demo claims a 90% reduction in noise. **Option A: buy now.** Fast to approve, vendor help with integration, and cross-tool correlation is the platform's strength. **Option B: an eight-week hygiene sprint**: a label contract, deduplication and grouping, retiring the noisiest rules, SLO-based paging for the top services and labels at incident close, then a shadow trial of one or two platforms after the peak. **Assumptions.** Most of the noise comes from a few rules (it usually does, but check the export). The team can spare two engineers for eight weeks. No labelled incidents exist today. **Reasoning.** Option A promises results without changing anyone's habits, and can be wrong here for three reasons: without labels nobody can tell whether its grouping is right, so a wrong merge at peak could hide an incident; integration takes weeks, and the peak arrives regardless; and a demo on the vendor's data says little about these three tools. Option B removes noise at source, creates the labels any later evaluation needs, and keeps risk low during the peak. **Decision.** Choose B, and start collecting labels on day 1 so that the post-peak shadow trial has evidence. **When A wins instead:** if the noise comes from many tools whose alerts cannot be fixed at source (for example, appliances or third-party services), and the team has no capacity to run a composable stack, buy, but in shadow mode first, scored with the [worksheet](#decision-framework). **Uncertainty.** The split between "fixable at source" and "only correlatable" noise is the crux. Measure it before deciding; a one-week analysis of the alert export settles most of the argument. ### Learning objectives and mastery check After this page you should be able to: **explain** what AIOps covers and where its neighbours begin; **select** between rules, statistics, learned models and LLMs for a given noise problem; **design** a label contract, a topology file and a guarded remediation gate; **operate** an evaluation with pairwise precision and recall; and **evaluate** a vendor's claims with your own labelled data. ::: answer 1. Three alerts are grouped together. A1 and A2 belong to incident X, and B1 to incident Y. What are the pairwise precision and recall? (A1, A2) are together and belong together: TP = 1. (A1, B1) and (A2, B1) are together but should not be: FP = 2. No pair that belongs together was split: FN = 0. Precision = 1 ÷ 3 ≈ 0.33; recall = 1 ÷ 1 = 1.0. The group found everything but mixed in an unrelated incident. ::: ::: answer 2. An inhibition rule uses `equal: [cluster]`. Neither the source alert nor a target alert has a `cluster` label. What happens, and how do you prevent it? Alertmanager treats missing labels as equal empty values, so the target is muted. A constant `external_labels` value on the server that raised both does not help: both get the same value. Prevent it by requiring the label non-empty on the source (`cluster=~".+"` in `source_matchers`), letting `cluster` mean only the affected cluster (the evaluating server goes in another label), excluding meta-monitoring alerts as targets, and firing a test alert without the label each month. ::: ::: answer 3. Your seasonal anomaly detector floods on public holidays. Name two fixes and one thing that will remain hard. A holiday calendar mapping those days to a weekend-like baseline (in Lab 3 it removed all 32 holiday false alerts, leaving 2 from a campaign), and change or campaign feeds that explain planned events. Slow drifts remain hard because the baseline absorbs them; keep SLO burn-rate alerts as an independent net. ::: ::: answer 4. Which faults qualify for automatic remediation? Well-understood faults with a reversible, idempotent fix, a limited blast radius, an automated check that proves success, and an automatic rollback or page. The runner acts with the narrowest identity, and the change authority approves the class. Anything touching data, money or security posture keeps a person in the loop. ::: ::: answer 5. A vendor demo shows a 95% reduction in noise. What do you ask for before believing it? A shadow-mode run on your alerts, scored with your labels (pairwise precision and recall, over-merge and split rates, missed incidents), an explanation of each decision, a full export, and their limits under a storm. ::: ::: answer 6. Where does this page end and AIOps for Security begin? The techniques overlap, but AIOps handles faults, which do not adapt to the detector; security analytics faces an adversary who does, which changes evaluation (base rates, evasion, missed detections) and safe autonomy. That is an editorial boundary of this Atlas. ::: ### Conclusion AIOps is best understood as a measurement discipline sold as a product category. Its techniques, from deduplication to language models, are real and useful, and each can also hide an outage. The name does not tell you which outcome you will get, and that is why this page keeps returning to labelled incidents. The durable assets are not the engine. They are the label contract, the topology, the change log, the labelled incidents and the habit of scoring every policy before and after it ships. Organisations that build those can swap engines as vendors merge, rename or retire products, and can tell whether a new technique helps. Organisations that skip them buy a cleaner-looking alert stream and learn its precision from their customers. ## Sources and further reading Claim-by-claim notes, support strength and the sources that could not be fetched are in the evidence ledger (`research/aiops-ledger.md`). Analyst reports are paywalled; only text quoted by vendors is used, attributed as such. **Re-checking fast-moving facts (all as of October 2026).** Monthly: the ITBench-AA leaderboard. Quarterly: pinned versions (Alertmanager, Prometheus, pint, ansible-rulebook, the `ansible.eda` collection, the node_exporter mixin, Keep, OpenTelemetry), vendor packaging and limits (PagerDuty's AIOps add-on, BigPanda's 300-alert cap), retirement or archive notices (DevOps Guru, Merlion) and NIST's AI RMF revision. Whenever the AI Act is amended: its dates, on EUR-Lex or the AI Act Explorer. The Gartner glossary refuses automated access, so re-check it through a new Internet Archive snapshot. Update the footnote date and ledger row at each re-check. **Term, analysts and history** [^gartner-glossary]: [Definition of AIOps (Artificial Intelligence for IT Operations)](https://web.archive.org/web/20260214203554/https://www.gartner.com/en/information-technology/glossary/aiops-artificial-intelligence-operations) — Gartner IT Glossary, Internet Archive snapshot of 2026-02-14. Checked 2026-10-16. [^gartner-2016]: [Applying AIOps Platforms to Broader Datasets Will Create Unique Business Insights](https://web.archive.org/web/20250126102219/https://www.gartner.com/en/documents/3364418) — Colin Fletcher, Gartner, published 2016-07-01 (public summary, archived 2025-01-26). Checked 2026-10-16. [^selector2026]: [Key Takeaways From the 2025 Gartner Market Guide for Event Intelligence Solutions](https://www.selector.ai/blog/key-takeaways-from-the-2025-gartner-market-guide-for-event-intelligence-solutions/) — Dallon Robinette, Selector, 2026-02-06. Checked 2026-10-16. [^bigpanda2026]: [Gartner Market Guide for Event Intelligence Solutions, 2026](https://www.bigpanda.io/ar-gartner-event-intelligence-solutions/) — BigPanda report page quoting Gartner's definition, 2026. Checked 2026-10-16. [^pd-gartner]: [Gartner Market Guide for Event Intelligence Solutions](https://www.pagerduty.com/resources/analyst-reports/analyst-report/gartner-market-guide-for-event-intelligence-solutions/) — PagerDuty report page (citation: Martin Caren, Andre Bridges, Leo Li, 5 May 2026). Checked 2026-10-16. [^forrester-wave]: [Dynatrace Named a Leader in AIOps in Report by a Leading Independent Research Firm](https://www.dynatrace.com/news/press-release/forrester-wave-aiops-platforms-q2-2025/) — Dynatrace press release on The Forrester Wave: AIOps Platforms, Q2 2025, 2025-04-15. Checked 2026-10-17. [^dang2019]: [AIOps: Real-World Challenges and Research Innovations](https://www.microsoft.com/en-us/research/publication/aiops-real-world-challenges-and-research-innovations/) — Yingnong Dang, Qingwei Lin, Peng Huang, ICSE 2019 Technical Briefings, Microsoft Research, 2019. Checked 2026-10-16. [^notaro2020]: [A Systematic Mapping Study in AIOps](https://arxiv.org/abs/2012.09108) — Paolo Notaro, Jorge Cardoso, Michael Gerndt, arXiv:2012.09108 (AIOPS 2020 workshop), 2020-12-15. Checked 2026-10-16. [^remil2024]: [AIOps Solutions for Incident Management: Technical Guidelines and A Comprehensive Literature Review](https://arxiv.org/abs/2404.01363) — Youcef Remil, Anes Bendimerad, Romain Mathonat, Mehdi Kaytoue, arXiv:2404.01363, 2024-04-01. Checked 2026-10-16. [^zhang2024]: [A Survey of AIOps for Failure Management in the Era of Large Language Models](https://arxiv.org/abs/2406.11213) — Lingzhe Zhang et al., arXiv:2406.11213 (v4), 2024-06. Checked 2026-10-16. [^bogatinovski2021]: [Artificial Intelligence for IT Operations (AIOPS) Workshop White Paper](https://arxiv.org/abs/2101.06054) — Jasmin Bogatinovski et al., arXiv:2101.06054, 2021-01. Checked 2026-10-16. [^ibm-omnibus]: [Introduction to Tivoli Netcool/OMNIbus](https://www.ibm.com/docs/en/netcoolomnibus/8.1.0?topic=overview-introduction-tivoli-netcoolomnibus) — IBM Documentation, Netcool/OMNIbus 8.1.0. Checked 2026-10-16. [^moogsoft-dell]: [Dell Technologies acquires Moogsoft](https://www.moogsoft.com/dell-technologies-acquires-moogsoft/) — Phil Tee, Moogsoft, 2023-10-19. Checked 2026-10-16. [^dell-apex]: [IT Operations Needs a New Wardrobe](https://www.dell.com/en-us/blog/it-operations-needs-a-new-wardrobe/) — Dell Technologies blog, 2024-05-20. Checked 2026-10-16. [^elastic-keep]: [Elastic and Keep join forces to help users manage alerts and automate workflows](https://www.elastic.co/blog/elastic-and-keep-join-forces) — Ken Exner, Elastic, 2025-05-07. Checked 2026-10-16. **Practice and reliability engineering** [^sremon]: [Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/) — Rob Ewaschuk, in *Site Reliability Engineering*, Google / O'Reilly, 2016. Checked 2026-10-16. [^sreintro]: [Introduction](https://sre.google/sre-book/introduction/) — Benjamin Treynor Sloss, in *Site Reliability Engineering*, Google / O'Reilly, 2016. Checked 2026-10-16. [^sreslo]: [Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/) — *The Site Reliability Workbook*, Google / O'Reilly, 2018. Checked 2026-10-16. [^sreauto]: [The Evolution of Automation at Google](https://sre.google/sre-book/automation-at-google/) — *Site Reliability Engineering*, Google / O'Reilly, 2016. Checked 2026-10-16. [^zhao2020]: [Understanding and Handling Alert Storm for Online Service Systems](https://2020.icse-conferences.org/details/icse-2020-Software-Engineering-in-Practice/24/Understanding-and-Handling-Alert-Storm-for-Online-Service-Systems) — Nengwen Zhao et al., ICSE 2020 (Software Engineering in Practice). Checked 2026-10-16. [^chen2020]: [Towards intelligent incident management: why we need it and how we make it](https://www.microsoft.com/en-us/research/publication/towards-intelligent-incident-management-why-we-need-it-and-how-we-make-it/) — Zhuangbin Chen et al., ESEC/FSE, Microsoft Research, 2020-11. Checked 2026-10-16. [^grafana2026]: [Observability Survey Report 2026](https://grafana.com/observability-survey/2026/) — Grafana Labs (1,363 self-selected responses via Censuswide), 2026. Checked 2026-10-16. **Alerting, labels and configuration** [^am-concepts]: [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) — Prometheus documentation. Checked 2026-10-16. [^am-config]: [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/) — Prometheus documentation. Checked 2026-10-16. [^am-readme]: [Alertmanager README (amtool)](https://github.com/prometheus/alertmanager/blob/v0.34.1/README.md) — Prometheus project, v0.34.1. Checked 2026-10-16. [^am-cli]: [cli/alert_query.go](https://github.com/prometheus/alertmanager/blob/v0.34.1/cli/alert_query.go) — Alertmanager source, v0.34.1 (flags of `amtool alert query`; `amtool alert add` is in `cli/alert_add.go`). Checked 2026-10-16. [^am-https]: [HTTPS and authentication](https://prometheus.io/docs/alerting/latest/https/) — Prometheus documentation for Alertmanager ("experimental"). Checked 2026-10-16. [^prom-security]: [Security model](https://prometheus.io/docs/operating/security/) — Prometheus documentation (Alertmanager section). Checked 2026-10-16. [^am-release]: [Alertmanager release 0.34.1](https://github.com/prometheus/alertmanager/releases/tag/v0.34.1) — Prometheus project on GitHub, 2026-09-17. Checked 2026-10-16. [^am-openapi]: [Alertmanager API v2 OpenAPI specification](https://raw.githubusercontent.com/prometheus/alertmanager/v0.34.1/api/v2/openapi.yaml) — Prometheus project, v0.34.1. Checked 2026-10-16. [^am-inhibit]: [inhibit.go](https://github.com/prometheus/alertmanager/blob/v0.34.1/inhibit/inhibit.go) — Alertmanager source, v0.34.1 (`inhibitedBy` holds the inhibiting alerts' fingerprints). Checked 2026-10-16. [^am-mgmt]: [Alertmanager management API](https://prometheus.io/docs/alerting/latest/management_api/) — Prometheus documentation. Checked 2026-10-16. [^prom-rules]: [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) — Prometheus documentation. Checked 2026-10-16. [^prom-config]: [Configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) — Prometheus documentation (`external_labels`). Checked 2026-10-16. [^promtool]: [Unit testing for rules](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/) — Prometheus documentation. Checked 2026-10-16. [^prom-functions]: [Query functions](https://prometheus.io/docs/prometheus/latest/querying/functions/) — Prometheus documentation (`predict_linear`). Checked 2026-10-16. [^node-mixin]: [node-mixin config.libsonnet](https://github.com/prometheus/node_exporter/blob/v1.12.1/docs/node-mixin/config.libsonnet) — Prometheus node_exporter project, v1.12.1 of 2026-07-14 (the rule is in `alerts/alerts.libsonnet`). Checked 2026-10-17. [^pint-label]: [rule/label](https://cloudflare.github.io/pint/checks/rule/label.html) — pint documentation, Cloudflare. Checked 2026-10-16. [^pint-changelog]: [Changelog](https://cloudflare.github.io/pint/changelog.html) — pint documentation (v0.72.0 entry). Checked 2026-10-16. [^pint-lint]: [cmd/pint/lint.go](https://github.com/cloudflare/pint/blob/v0.89.2/cmd/pint/lint.go) — pint v0.89.2 source (released 2026-10-08; `--fail-on` defaults to `bug`). Checked 2026-10-17. [^prom-release]: [Prometheus release 3.15.0](https://github.com/prometheus/prometheus/releases/tag/v3.15.0) — Prometheus project on GitHub, 2026-09-24. Checked 2026-10-16. [^otel-service]: [Service semantic conventions](https://opentelemetry.io/docs/specs/semconv/resource/service/) — OpenTelemetry, semantic conventions 1.44.0. Checked 2026-10-16. [^splunk-itil]: [IT Event Management Explained](https://www.splunk.com/en_us/blog/learn/it-event-management.html) — Splunk (summary of ITIL 4 event categories), undated. Checked 2026-10-16. **Products and projects (time-sensitive)** [^pd-iag]: [Intelligent Alert Grouping](https://support.pagerduty.com/main/docs/intelligent-alert-grouping) — PagerDuty Knowledge Base, updated 2026-09. Checked 2026-10-16. [^pd-cbag]: [Content-Based Alert Grouping](https://support.pagerduty.com/main/docs/content-based-alert-grouping) — PagerDuty Knowledge Base, updated 2026-09. Checked 2026-10-16. [^pd-autopause]: [Auto-Pause Incident Notifications](https://support.pagerduty.com/main/docs/auto-pause-incident-notifications) — PagerDuty Knowledge Base. Checked 2026-10-16. [^pd-eo]: [Event Orchestration](https://support.pagerduty.com/main/docs/event-orchestration) — PagerDuty Knowledge Base. Checked 2026-10-16. [^bigpanda-corr]: [Alert Correlation Logic](https://docs.bigpanda.io/en/alert-correlation-logic--adr-) — BigPanda documentation. Checked 2026-10-16. [^splunk-itsi]: [Overview of Event Analytics in ITSI](https://docs.splunk.com/Documentation/ITSI/latest/EA/AboutEA) — Splunk IT Service Intelligence documentation. Checked 2026-10-16. [^splunk-smart]: [Group similar events with Smart Mode in ITSI](https://docs.splunk.com/Documentation/ITSI/latest/EA/SmartMode) — Splunk IT Service Intelligence documentation. Checked 2026-10-17. [^moog-tempus]: [Time-based Clustering with Tempus](https://docs.moogsoft.com/v9/en/time-based-clustering-with-tempus.html) — Moogsoft Onprem/Hosted documentation, v9. Checked 2026-10-17. [^dt-rca]: [Root cause analysis concepts](https://docs.dynatrace.com/docs/dynatrace-intelligence/root-cause-analysis/concepts) — Dynatrace documentation, updated 2026-10-12. Checked 2026-10-16. [^dd-rca]: [Watchdog RCA](https://docs.datadoghq.com/watchdog/rca/) — Datadog documentation, modified 2026-08-06. Checked 2026-10-16. [^elastic-aiops]: [AIOps Labs](https://www.elastic.co/docs/explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops) — Elastic documentation. Checked 2026-10-16. [^keep-repo]: [keephq/keep](https://github.com/keephq/keep) — Keep on GitHub (release v0.54.3 of 2026-09-09). Checked 2026-10-16. [^grafana-promql]: [promql-anomaly-detection](https://github.com/grafana/promql-anomaly-detection) — Grafana Labs on GitHub, README. Checked 2026-10-16. [^eda-rules]: [Rules — Ansible Rulebook documentation](https://docs.ansible.com/projects/rulebook/en/v1.3.0/rules.html) — Red Hat / Ansible community, v1.3.0 (latest release 1.3.2: https://github.com/ansible/ansible-rulebook/releases/tag/v1.3.2). Checked 2026-10-16. [^eda-conditions]: [Conditions — Ansible Rulebook documentation](https://docs.ansible.com/projects/rulebook/en/v1.3.0/conditions.html) — Red Hat / Ansible community, v1.3.0. Checked 2026-10-16. [^eda-am]: [alertmanager.py event source plugin](https://github.com/ansible/event-driven-ansible/blob/v2.13.0/extensions/eda/plugins/event_source/alertmanager.py) — ansible/event-driven-ansible on GitHub, collection v2.13.0 of 2026-07-07. Checked 2026-10-17. [^eda-webhook]: [webhook.py event source plugin](https://github.com/ansible/event-driven-ansible/blob/v2.13.0/extensions/eda/plugins/event_source/webhook.py) — ansible/event-driven-ansible on GitHub, collection v2.13.0 of 2026-07-07. Checked 2026-10-17. [^eda-hostlimit]: [Limiting hosts](https://docs.ansible.com/projects/rulebook/en/v1.3.0/host_limit.html) — Ansible Rulebook documentation, v1.3.0. Checked 2026-10-17. [^eda-rulebooks]: [Rulebooks](https://docs.ansible.com/projects/rulebook/en/v1.3.0/rulebooks.html) — Ansible Rulebook documentation, v1.3.0 (`execution_strategy`, default sequential). Checked 2026-10-17. [^ansible-become]: [Understanding privilege escalation: become](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_privilege_escalation.html) — Ansible Community Documentation. Checked 2026-10-17. [^pyrca]: [PyRCA: A Library for Metric-based Root Cause Analysis](https://arxiv.org/abs/2306.11417) — Chenghao Liu et al., Salesforce, arXiv:2306.11417, 2023 (code: https://github.com/salesforce/PyRCA). Checked 2026-10-17. [^pham2024]: [Root Cause Analysis for Microservice System based on Causal Inference: How Far Are We?](https://arxiv.org/abs/2408.13729) — Luan Pham, Huong Ha, Hongyu Zhang, ASE 2024. Checked 2026-10-17. [^merlion]: [Merlion](https://github.com/salesforce/Merlion) — Salesforce on GitHub (archived 2026-03-11). Checked 2026-10-16. [^azure-ad]: [What is Anomaly Detector?](https://learn.microsoft.com/en-us/previous-versions/azure/ai-services/anomaly-detector/overview) — Microsoft Learn (previous versions), updated 2025-06-12. Checked 2026-10-16. [^aws-lookout]: [Transitioning off Amazon Lookout for Metrics](https://aws.amazon.com/blogs/machine-learning/transitioning-off-amazon-lookout-for-metrics/) — AWS Machine Learning Blog, 2024-10-09. Checked 2026-10-16. [^aws-devopsguru]: [What is Amazon DevOps Guru?](https://docs.aws.amazon.com/devops-guru/latest/userguide/welcome.html) — AWS documentation (end-of-support notice). Checked 2026-10-16. **Methods and statistics** [^nist-outliers]: [Detection of Outliers (1.3.5.17)](https://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm) — NIST/SEMATECH e-Handbook of Statistical Methods. Checked 2026-10-16. [^fpp3]: [STL decomposition (§3.6)](https://otexts.com/fpp3/stl.html) — Rob J. Hyndman and George Athanasopoulos, *Forecasting: Principles and Practice*, 3rd ed., OTexts. Checked 2026-10-16. [^drain3]: [Drain3](https://github.com/logpai/Drain3) — logpai on GitHub, README (online template miner after Drain, ICWS 2017). Checked 2026-10-17. [^he2016]: [Experience Report: System Log Analysis for Anomaly Detection](https://jiemingzhu.github.io/pub/slhe_issre2016.pdf) — Shilin He, Jieming Zhu, Pinjia He, Michael R. Lyu, ISSRE 2016. Checked 2026-10-17. [^le2022]: [Log-based Anomaly Detection with Deep Learning: How Far Are We?](https://arxiv.org/abs/2202.04301) — Van-Hoang Le, Hongyu Zhang, ICSE 2022. Checked 2026-10-17. [^landauer2024]: [A Critical Review of Common Log Data Sets Used for Evaluation of Sequence-based Anomaly Detection Techniques](https://arxiv.org/abs/2309.02854) — Max Landauer, Florian Skopik, Markus Wurzenberger, FSE 2024 (§IV-C). Checked 2026-10-17. [^sklearn]: [Novelty and outlier detection](https://scikit-learn.org/stable/modules/outlier_detection.html) — scikit-learn user guide. Checked 2026-10-16. [^gml]: [Classification: Accuracy, recall, precision, and related metrics](https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall) — Google Machine Learning Crash Course. Checked 2026-10-16. [^irbook]: [Evaluation of clustering](https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html) — Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze, *Introduction to Information Retrieval*, Cambridge University Press, 2008. Checked 2026-10-16. **Language models and agents** [^ahmed2023]: [Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models](https://arxiv.org/abs/2301.03797) — Toufique Ahmed et al., ICSE 2023, arXiv:2301.03797. Checked 2026-10-16. [^rcacopilot]: [Automatic Root Cause Analysis via Large Language Models for Cloud Incidents](https://arxiv.org/abs/2305.15778) — Yinfang Chen et al., arXiv:2305.15778 (v4, 2023-11-13; deployment in §1 and §5.5 of the [full text](https://arxiv.org/html/2305.15778v4)). Checked 2026-10-16. [^aiopslab]: [AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds](https://arxiv.org/abs/2501.06706) — Yinfang Chen et al., arXiv:2501.06706, 2025-01. Checked 2026-10-16. [^itbench]: [ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks](https://icml.cc/virtual/2025/poster/44303) — Saurabh Jha et al., ICML 2025 spotlight poster (preprint v1, [arXiv:2502.05352](https://arxiv.org/abs/2502.05352): 94 scenarios, 13.8%). Checked 2026-10-16. [^itbench-aa]: [ITBench-AA Benchmark Leaderboard](https://artificialanalysis.ai/evaluations/itbench-aa) — Artificial Analysis (undated page, © 2026). Checked 2026-10-16. [^alertguardian]: [AlertGuardian: Intelligent Alert Life-Cycle Management for Large-scale Cloud Systems](https://arxiv.org/abs/2601.14912) — Guangba Yu et al., ASE 2025, arXiv:2601.14912, 2026-01-21. Checked 2026-10-16. **Governance** [^euaiact]: [Annex III: High-Risk AI Systems Referred to in Article 6(2)](https://artificialintelligenceact.eu/annex/3/) — EU AI Act text as published by the Future of Life Institute's AI Act Explorer. Checked 2026-10-17. [^euaiact-art4]: [Article 4: AI literacy](https://artificialintelligenceact.eu/article/4/) — EU AI Act as amended, AI Act Explorer. Checked 2026-10-16. [^eu-omnibus]: [Digital Omnibus on AI](https://artificialintelligenceact.eu/ai-act-explorer/digital-omnibus/) — Regulation (EU) 2026/1744 of 8 July 2026, official text as reproduced in the AI Act Explorer (in force 2026-07-27). Checked 2026-10-17. [^nist-airmf]: [AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) — National Institute of Standards and Technology, living page (AI RMF 1.0 of 2023-01-26). Checked 2026-10-17. **Further reading** - Google, [Site Reliability Engineering](https://sre.google/sre-book/table-of-contents/) and [The Site Reliability Workbook](https://sre.google/workbook/table-of-contents/): the alerting and automation chapters are the best antidote to over-engineered event pipelines. **Changelog** - 2026-10-16: first version (research, ledger, labs executed, diagrams validated) by the managing author; awaiting independent fact-check review. - 2026-10-17: round-1 revision (`reviews/aiops-r1.md`, response in `reviews/aiops-r1-response.md`): affected cluster separated from evaluating server; guarded inhibition and corrected runbook fix; contract linting and measurement; remediation wired end to end; ITBench (ICML 2025) and a dated 2026 leaderboard; predictive alerting; Alertmanager API security; EU AI Act Article 4 and Digital Omnibus; labs re-run; one redundant figure removed. - 2026-10-17: round-2 revision (`reviews/aiops-score.md`, response in `reviews/aiops-r2-response.md`): how log templates, learned grouping and graph root-cause analysis work, with sceptical evidence; re-check guidance; runner after routing in the architecture figure; both mixin remedies; observer-side partition risk; code wrapped; repetition cut; labs re-run. - 2026-10-17, post-score polish (`reviews/aiops-score-r2.md`; actions in `reviews/aiops-polish.md`): change events recorded and attached (Artefact 4, executed); Landauer finding made precise; Forrester figures attributed; NIST AI RMF revision noted; moving-branch sources pinned; re-check list extended; repetition cut.