AIOps

AIOps (artificial intelligence for IT operations) applies statistics, machine learning and, more recently, large language models (LLMs) to the events and telemetry that IT systems emit. The aim is that people see fewer, better-grouped alerts, reach a probable cause sooner, and let safe, repetitive fixes run on their own. Gartner’s glossary keeps the definition narrow: AIOps “combines big data and machine learning to automate IT operations processes, including event correlation, anomaly detection and causality determination”.^[1]

In one line: AIOps turns a flood of operational signals into a short list of actionable incidents, and it is only as good as the labels, topology and feedback you give it.

This page is for NOC (network operations centre) and ITOps leads facing alert floods, SREs and on-call engineers, the engineers who feed the event pipeline, and buyers of “AIOps” or “event intelligence” platforms. Beginners can read the first two parts and the cheat sheet. Engineers should go to the labs, and leaders to the decision framework and the judgement call.

When one shared component fails, everything that depends on it complains. Alertmanager’s documentation gives the classic example: during a network partition, hundreds of service instances each alert because they cannot reach the database, although one page would do.^[8] Researchers call this an alert storm (Scenario 1 has a documented case).^[9]

Noise has a human cost. Google’s SRE book states it as a philosophy of paging: “I can only react with a sense of urgency a few times a day before I become fatigued.”^[10] In Grafana Labs’ 2026 survey, 30% of respondents named alert fatigue as the biggest obstacle to faster incident response. The survey is a vendor’s, and its respondents chose themselves, so read it as a signal rather than a measurement.^[11]

Without event intelligence:

• Duplicate paging. Several teams are woken for one fault, and each starts its own investigation.
• Symptom chasing. Responders work on the loudest alert, not the failing dependency.
• Hidden incidents. A second, unrelated failure disappears inside a storm.
• Manual toil. The same safe fix (rotate logs, restart a stuck worker) is typed by hand at 3 a.m.

• Not operations for AI. Running models and LLM applications is MLOps and LLMOps. AIOps applies AI to IT operations.
• Not observability. Collecting, shaping and storing metrics, logs and traces belongs to ObservabilityOps. AIOps consumes that data.
• Not incident management. Roles, severity, communication and post-incident reviews belong to IncidentOps. AIOps decides what reaches them and with what context.
• Not security analytics. Detecting attackers uses the same techniques against an adversary who adapts. That is AIOps for Security.
• Not autonomous operations. “Self-healing” is a direction, not a state you can buy (see NoOps).
• Not a replacement for good alert rules. Google’s SRE book (2016) says: “We avoid ‘magic’ systems that try to learn thresholds or automatically detect causality.”^[10]
• Not root-cause certainty. Tools rank probable causes from a limited menu (Datadog’s Watchdog names four types).^[12]

Event consolidation is older than the label. A 2024 review describes the “traditional methods” that AIOps set out to improve as “reliant on manual tasks and rule-based approaches”.^[7] IBM describes its Tivoli Netcool/OMNIbus event console as consolidating information from separate network management platforms and automating the processing of alerts.^[13] What changed in the 2010s was scale (cloud, microservices, many monitoring tools) and the hope that learning could replace some hand-written rules.

Gartner’s 2026 market guide (5 May 2026)^[21] defines event intelligence solutions, as quoted by BigPanda, as “tools that apply artificial intelligence (AI) and data analytics to augment, accelerate, and automate responses to signals or events detected from digital services”.^[22]

Mental model: a funnel with a feedback loop. Signals become events, events become alerts, alerts become incidents, and people confirm or correct the grouping. Every stage throws information away; the craft is to discard noise, keep signal, and know by measuring which of the two you lost.

flowchart TB
  S["Signals: metrics, logs, traces, checks"] --> E["Events (thousands)"]
  E --> D["Deduplicate and normalise"]
  D --> A["Alerts (hundreds)"]
  A --> G["Group, inhibit, enrich"]
  G --> I["Incidents (a few)"]
  I --> R["People act and close"]
  R --> L["Labels: right group? right cause?"]
  L -.->|"monthly scorecard: tune rules and models"| G
  L -.->|"fix noisy rules"| A
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class S,E,A,I neutral
  class D,G,R core
  class L decision

AIOps sits between the teams that produce telemetry and the teams that respond to incidents. Where a boundary below is the Atlas’s own choice rather than an industry standard, it is marked as an editorial distinction.

flowchart TB
  OBS["Telemetry (ObservabilityOps)"] --> AIO
  CHG["Changes and deploys"] --> AIO
  TOPO["Topology and owners (ServiceOps, CMDB)"] --> AIO
  AIO["AIOps: correlate, detect, rank, guard"]
  AIO --> INC["IncidentOps: people respond"]
  AIO --> AUTO["AutomationOps: safe fixes"]
  INC -.->|"labels"| AIO
  AIO -.-|"editorial boundary"| SEC["AIOps for Security"]
  AIO -.-|"model practices"| ML["MLOps"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class AIO core
  class OBS,CHG,TOPO,INC,AUTO,ML neutral
  class SEC risk

No body publishes an official AIOps manifesto; this charter derives from the evidence on this page.

Mission: reduce the attention that operational events cost people, without hiding a single incident that matters, and make every automated decision explainable and reversible.

1. Rules before learning. Start with deduplication, explicit grouping, inhibition and SLO-based alerting (Google’s stance, quoted above). Add learning where labelled incidents show that rules fail.
2. Never drop silently. Every grouped, inhibited or suppressed alert stays queryable; every suppression has an owner, a reason and an expiry date.
3. Measure both kinds of error. Wrong merges (precision) and splits or misses (recall) both cost something; compression alone is a vanity metric.
4. Context beats cleverness. Topology, ownership, changes and calendars explain more alerts than any algorithm can infer from alert text. Dynatrace says of its own engine: “time correlation alone is not sufficient”.^[24]
5. People own production changes. Machines may run allow-listed, reversible, rate-limited actions; anything else is a suggestion for a person.
6. Explain every decision. Each grouping and each probable cause shows why: shared labels, a topology path, a change, a model score.
7. Own what is portable. The label contract, rules, labelled incidents and evaluation harness belong to you; engines get replaced (see tooling).

Non-goals: fully autonomous operations, replacing on-call engineers, or claiming certainty about root causes.

AIOps fails more often for lack of an owner than of an algorithm. Name an event-intelligence owner, accountable for the pipeline’s precision, recall and backlog; in a small organisation this is part of an SRE’s or observability engineer’s job.

R = responsible, A = accountable, C = consulted, I = informed; “platform team” includes observability engineers. With an ITIL change process, approving a remediation class is usually a standard change (see ITOps).

Treat every grouping rule, inhibition rule, anomaly detector and remediation playbook as a policy with a lifecycle. The states below are this Atlas’s proposal; “shadow mode” means the policy runs and is scored, but changes nothing that people see.

stateDiagram-v2
  direction TB
  [*] --> Proposed
  Proposed --> Shadow: tests pass
  Shadow --> Canary: precision and recall meet targets
  Shadow --> Proposed: misses or bad merges
  Canary --> Active: one or two services stay healthy
  Canary --> Shadow: complaints or a missed incident
  Active --> Review: monthly score drops or topology changes
  Review --> Active: re-tuned
  Review --> Retired: no longer earns its keep
  Retired --> [*]

• Day 0 (design). Inventory and baseline alert sources (roadmap); write the label contract and a first topology file; choose rules first.
• Day 1 (rollout). Run new policies in shadow mode against a few weeks of labelled incidents, then enable them service by service, starting with volunteers.
• Day 2 (operation). Topology, traffic and rules drift. Review the scorecard monthly, re-label disputed incidents, retire dead rules, and re-evaluate after every major platform change.

• Everything as code. Grouping rules, inhibition rules, detector settings and playbooks live in version control and change by pull request. CI lints every alert rule, runs promtool test rules for important ones and amtool config routes test for routing (Artefact 1).^[25][26]
• Three change classes. Routing and grouping changes need the owning team’s review. Suppression and inhibition need evidence and an expiry date. A new class of automatic remediation needs the change authority’s approval and a rollback plan.
• Feedback at incident close. Ask responders two one-click questions: were these alerts correctly grouped? and was the probable cause right? Without these labels you cannot compute precision or recall. PagerDuty’s documentation shows why the capture point matters: its learned grouping learns from manual merges and moves; “Only manual merges and unmerges in the PagerDuty web app influence the algorithm”, merges made through the API do not, and alert titles rewritten by Event Orchestration also count.^[27]
• Cadence. A weekly noise review (the ten noisiest rules and actions taken), a monthly scorecard, and a quarterly review of platform fit.

The architecture is vendor-neutral: an all-in-one platform implements most layers itself, a composable stack assembles them from separate tools, and the trust boundaries stay the same.

flowchart TB
  SRC["Alert sources: metric rules, log and trace signals, probes"]
  SRC --> N["Normalise; check label contract"]
  MH["Metric history"] --> AD["Anomaly detection"]
  N --> DD["Deduplicate and enrich"]
  DD --> CO["Group, inhibit and route"]
  AD --> CO
  T["Topology and owners"] --> CO
  CO --> RC["Rank probable causes"]
  CH["Change log and calendar"] --> RC
  RC --> IM["Incident tool and paging"]
  IM --> CHAT["Chat channel"]
  CO -.->|"allow-listed alerts only"| RR["Remediation runner"]
  RR -.->|"outcome; page on failure"| IM
  IM -.->|"labels at close"| LB["Labelled incidents"]
  LB -.->|"tune"| CO
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class N,DD,CO,AD,RC core
  class RR risk
  class SRC,MH,T,CH,LB,IM,CHAT neutral

Trust boundaries. Four places deserve security review. (1) Ingest endpoints: anything that can post an event can create or hide an incident, so authenticate webhooks in both directions. (2) The alert manager’s own API: Prometheus’s security model says that anyone who can reach the Alertmanager HTTP endpoint can “create, modify and delete silences”, and that it serves /api/v2 with Access-Control-Allow-Origin: *, so any website opened in a browser that can reach it can call the API.^[28] One request can silence an outage. (3) The remediation runner holds production credentials: give it the narrowest rights per allow-listed action. (4) An LLM that reads alert text sees attacker-controllable strings: treat them as untrusted input (see LLMOps).

The sequence follows one incident; the alt block is the exception path, a wrong grouping or cause.

sequenceDiagram
  autonumber
  participant M as Monitors
  participant E as Event intel
  participant I as Incident tool
  participant O as On-call
  M->>E: 40 alerts in 4 min
  Note over E,I: dedup, group by topology, attach change
  E->>I: 1 incident, cause: orders-db
  I->>O: page DB owner
  alt grouping or cause wrong
    O->>I: split, merge, fix cause
  else grouping and cause right
    O->>I: confirm, work from the cause
  end
  O->>I: resolve, confirm labels
  I->>E: labelled incident for evaluation

A corrected wrong grouping is a training signal; an unnoticed one teaches the system to repeat the mistake. An allow-listed fault takes a third path: the alert goes straight to the runner, which pages a person if its checks or its fix fail (see the guarded-remediation gate).

The practices are in the order in which they usually pay off; the first four need no machine learning.

Practice 1 is the foundation. Grouping by service fails when half the rules call it app. Inhibition by cluster misbehaves in two ways. When the label is absent, Alertmanager treats it as an empty value, so “if all the label names listed in equal are missing from both the source and target alerts, the inhibition rule will apply”.^[29] And when one label carries two meanings, where the rule was evaluated and which cluster is affected, two unrelated alerts can share a value. The worked runbook shows both hiding an outage. Keep the meanings in separate labels, cluster (affected) and prometheus (evaluating), as Artefact 1 does.

Practice 4 exploits a common cause of outages. Google’s SRE book (2016) says: “SRE has found that roughly 70% of outages are due to changes in a live system.”^[30] The figure is Google’s and its method unpublished, but attaching recent changes to every incident is cheap and often decisive. Datadog’s Watchdog RCA names version changes as one of its four root-cause types, alongside traffic increases, AWS instance failures and full disks.^[12]

Practice 7 starts from a paging principle: “If a page merely merits a robotic response, it shouldn’t be a page.”^[10] Robotic responses are candidates for automation, behind the guarded-remediation gate.

No standard defines AIOps methods; the literature organises them by task: anomaly detection, failure prediction, root-cause analysis and remediation.^[5][7]

When rules beat machine learning. Prefer rules when the failure has a clear symptom (an SLO burn rate, an error ratio), people know the topology, no labelled incidents exist to evaluate a model, a wrong suppression would be costly, or on-call engineers must predict the system’s behaviour at 3 a.m. Google’s SRE book allows one exception to its simple rules: detecting “unexpected changes in end-user request rates”.^[10] Its workbook judges alert rules on precision, recall, detection time and reset time,^[23] the same attributes this page uses for models.

When learning earns its place. When many tools produce inconsistent alerts, metrics are strongly seasonal, topology is too large or fast-changing to maintain by hand, and enough labelled incidents exist to show in shadow mode that a model beats the rules.

flowchart TB
  Q1{"Pages follow symptoms and SLOs?"} -->|"no"| R1["Fix alert rules first (SRE)"]
  Q1 -->|"yes"| Q2{"Alerts carry service, team, cluster?"}
  Q2 -->|"yes"| Q3{"Storms from shared dependencies?"}
  Q2 -->|"no"| R2["Label contract, dedup, group_by"]
  Q3 -->|"yes"| R3["Inhibition plus topology correlation"]
  Q3 -->|"no"| Q4{"Seasonal metrics hide faults?"}
  Q4 -->|"no"| Q5{"100+ labelled incidents, rules failing?"}
  Q4 -->|"yes"| R4["Seasonal baselines plus calendars"]
  Q5 -->|"yes"| R5["Shadow-trial learned grouping"]
  Q5 -->|"no"| R6["Stop: rules are enough"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class Q1,Q2,Q3,Q4,Q5 decision
  class R2,R3,R4,R5 core
  class R1,R6 risk

Notaro and colleagues divide failure management into “failure prevention, online failure prediction, failure detection, root cause analysis (RCA) and remediation”; the papers they map predict disk, hardware and software failures.^[5] In daily operations the most defensible form is a forecast of exhaustion: disk space, quotas, connection pools. Prometheus’s predict_linear “predicts the value of time series t seconds from now, based on the range vector v, using simple linear regression” and “should only be used with gauges”.^[36] The node_exporter project’s filesystem alert shows the careful pattern: free space already low, a 6-hour trend reaching zero within the horizon, a writable filesystem, and the condition held for an hour.^[37]

Example status: PromQL adapted from the node_exporter 1.12.1 mixin (critical variant, checked 2026-10-17); not executed.

# Alert rule expression, with  for: 1h  in the rule: under 20% free AND the
# 6-hour trend reaches zero within 4 hours AND the filesystem is writable.
(
  node_filesystem_avail_bytes{mountpoint="/var/log"}
    / node_filesystem_size_bytes{mountpoint="/var/log"} * 100 < 20
and
  predict_linear(
    node_filesystem_avail_bytes{mountpoint="/var/log"}[6h], 4 * 3600
  ) < 0
and
  node_filesystem_readonly{mountpoint="/var/log"} == 0
)

• A straight line extrapolates the window. A burst of writes predicts exhaustion; a clean-up job predicts plenty. The mixin names two causes and two remedies. Scheduled clean-up jobs “may not align with the default time window of these alerts. This can cause these alerts to start flapping”; its remedy is a shorter alert horizon (the 4 hours above), which leaves the job time to run. Short-lived fluctuation is the other cause: a longer prediction window (the 6 hours of history) can “reduce false positives caused by temporary spikes”.^[37]
• Evaluate it like any detector. On past exhaustion events, weigh the lead time gained over a static threshold against false forecasts per week; ten minutes gained with weekly flapping is noise.
• Forecast resources, not users. For user-visible symptoms an SLO burn-rate alert already forecasts budget exhaustion (SRE). Learned failure prediction (disks, for example) is a learned detector: shadow mode and labelled outcomes first.

Logs become templates first. A log line mixes a fixed message with variables, so analysis starts by mining templates: Drain3 turns “user davidoh logged in” and “user eranr logged in” into user <:*:> logged in, and opens a new template when too few tokens match (similarity threshold 0.4 by default).^[41] Detectors then count each template per time window or session and flag unusual counts (PCA, invariant mining, clustering, classifiers),^[42] or learn the normal order of templates and flag changed sequences.^[43] The evidence is sobering. An ICSE 2022 study found that training-data choice, grouping, class balance and noise significantly affected the evaluation of five deep-learning detectors, which “do not always work well”;^[44] an FSE 2024 study found simple detectors competitive with advanced ones on common benchmark data sets: flagging template types never seen in training sufficed on one, comparing template counts on another.^[43] Start with new-template and template-rate alerts; score anything cleverer as in Lab 3.

Learned grouping is a similarity score plus a threshold. Documentation shows three kinds of evidence. Shared fields: Splunk ITSI’s Smart Mode weighs “category” fields (a status) against “text” fields (a description) and can split by service first.^[45] Text: PagerDuty compares alert summaries by default and warns that a larger time window raises “the chance of overgrouping”.^[27] Co-occurrence: Moogsoft’s Tempus compares arrival patterns with the Jaccard index; its documentation rules it out for slow degradation.^[46] Lab 2’s related() functions are the hand-written version. Ask which fields, weights and window decide a merge, then score it with pairwise precision and recall.

Graph and causal root-cause analysis rank candidates. Lab 2 picks an alerting service none of whose dependencies alert. Graph methods generalise this: take the graph from topology or learn a causal one from metrics (PC or GES in PyRCA), then score each node, for example by a random walk that assumes “a metric that is highly correlated with the anomaly metrics is more likely to be the root cause”.^[47] An ASE 2024 evaluation of 21 such methods for microservices found that “no method stands out in all situations” and that synthetic results “may not accurately reflect their performance in real systems”.^[48] Treat the output as a ranked list with evidence, scored by top-k accuracy.

Examples are documented capabilities checked on 2026-10-16, not endorsements.

Engines churn; plan for it. Managed services retire and libraries get archived:

The lesson is principle 7, not “avoid managed services”: keep what you need to replace an engine and prove the replacement no worse.

Selection criteria. Beyond the weighted worksheet, ask whether every suppressed alert stays queryable and whether pricing penalises context.

LLMs have found real work in incident handling, mainly reading and writing text. Microsoft researchers evaluated LLMs for recommending root causes and mitigations on more than 40,000 incidents, with incident owners judging the output.^[16] Microsoft’s RCACopilot gathers diagnostics, predicts a root-cause category with an explanation, and reports “RCA accuracy up to 0.766”; its diagnostic-collection module had run in more than 30 teams for over four years, its LLM component in one incident-management team for several months.^[40] A 2025 paper reports that its system proposed changes to 1,174 alert rules at an anonymised company, and SREs accepted 375 (32%).^[60]

The evidence on autonomy is thin. ITBench, peer-reviewed at ICML 2025, built “an initial set of 102 real-world scenarios” and found that “agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios”; the models were those of early 2025.^[20] Newer tests measure something narrower. Artificial Analysis’s ITBench-AA gives agents an offline snapshot of 59 Kubernetes incidents and scores the diagnosis by “average precision at full recall”; the best listed model scored 56.2% (checked 16 October 2026).^[61] Diagnosing from a snapshot is not resolving a live system, and leaderboards change monthly, so neither number licenses unsupervised action. AIOpsLab injects faults so that agents can be evaluated before anyone trusts them.^[19]

Practical rules for LLMs in operations:

1. Start read-only. Summaries, timelines, similar-incident search and runbook retrieval carry little risk if a person reads them.
2. Treat alert and log text as untrusted input. Never let model output reach a shell or an API without an allow-list.
3. Evaluate like any other policy. Measure top-k cause accuracy and wrong suggestions on labelled incidents, in shadow mode.
4. Mind the data. Redact customer data and secrets before incident text leaves your boundary (PrivacyOps); token spend belongs in FinOps for AI.

Design. A composable, vendor-neutral start for a team with Prometheus-style alerting: a label contract that a linter enforces and Alertmanager measures, Alertmanager for grouping and guarded inhibition, a topology file in Git, a normalised event schema with change events, and a guarded remediation runner. Assumptions: Linux hosts, Prometheus 3.x (3.15.0 was released on 24 September 2026^[62]), Alertmanager 0.34.x,^[49] pint 0.89.x,^[63] ansible-rulebook 1.3.x^[55] on the Alertmanager host, and a separate incident tool that accepts webhooks. Constraints: no machine learning yet; every suppression is visible; only one remediation class is automated.

Artefact 1: the label contract and its enforcement. Two labels look alike and must never be merged: the cluster an alert is about and the server that evaluated it.

External labels never override a label an alert already has,^[65] so a rule that watches another cluster keeps that cluster’s name.

Example status: syntax-checked, not executed (requires Prometheus and promtool).

# checkout-rules.yml (in-cluster server): service, team, severity, remediation
# and runbook_url are static; cluster and prometheus come from the server's
# external_labels.
groups:
  - name: checkout-api
    labels:
      service: checkout-api
      team: payments
    rules:
      - alert: HighErrorRatio
        expr: |
          sum(rate(http_requests_total{job="checkout-api",code=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="checkout-api"}[5m])) > 0.05
        for: 5m
        keep_firing_for: 5m
        labels:
          severity: page
          database: orders-db
          remediation: none
        annotations:
          summary: "checkout-api 5xx ratio above 5% for 5 minutes"
          runbook_url: "https://runbooks.example.com/checkout-api/high-error-ratio"

# prometheus.yml on the in-cluster server of eu-1. The meta-monitoring server sets
# only  prometheus: meta  because its alerts are about other clusters.
global:
  external_labels:
    cluster: eu-1        # a default: a cluster label already on an alert wins
    prometheus: eu-1

# meta-rules.yml on the meta-monitoring server. Each target of this job carries
# its own cluster label (for example cluster="dev-3"), so the alert names the
# affected cluster. A target without one yields an alert that, by Artefact 2,
# inhibits nothing.
groups:
  - name: meta-monitoring
    labels:
      service: monitoring
      team: observability
    rules:
      - alert: ClusterUnreachable
        expr: up{job="cluster-prometheus"} == 0
        for: 5m
        labels:
          severity: page
          remediation: none
        annotations:
          summary: "Prometheus of cluster {{ $labels.cluster }} unreachable for 5m"
          runbook_url: "https://runbooks.example.com/monitoring/cluster-unreachable"

Enforcement needs three checks, because each sees only part of the contract. (1) A linter over every rule file. pint’s rule/label check with required = true flags any alerting rule that lacks a label; it “uses static labels set on alerting or recording rule” (group-level labels since pint 0.72.0), so it cannot see cluster or prometheus.^[66][67] Its findings are warnings by default and pint lint fails only on bug or worse, so raise the severity.^[63] (2) A configuration check in CI: every server sets prometheus, in-cluster servers set cluster, the meta-monitoring server does not. (3) A measurement in Alertmanager: the share of current alerts carrying the whole contract, sampled from its API. That share is the scorecard’s label-contract compliance and maturity level 1’s 95% exit criterion.

Example status: written to the pint documentation for rule/label and alerts/annotation; not executed (requires pint).

# .pint.hcl: run  pint lint rules/  in CI.
# Severity "bug" makes a missing label fail the run.
rule {
  match {
    kind = "alerting"
  }
  label "service" {
    severity = "bug"
    required = true
  }
  label "team" {
    severity = "bug"
    required = true
  }
  label "severity" {
    severity = "bug"
    required = true
    values   = ["page", "ticket"]
  }
  label "remediation" {
    severity = "bug"
    required = true
    values   = ["auto", "none"]
  }
  annotation "runbook_url" {
    severity = "bug"
    required = true
  }
}

Example status: executed on 2026-10-17 (Python 3.8.10, PyYAML 5.3.1) against four sample configurations and a saved API response; not run against a live Alertmanager.

"""label_contract.py: check the label contract in two places.

  python3 label_contract.py configs prometheus/*.yml   # CI: server-level labels
  python3 label_contract.py alerts [ALERTS_API_URL]    # runtime: compliant share
Standard library, plus PyYAML for the configs check.
"""
import json
import sys
import urllib.request

ALERT_LABELS = ("service", "team", "severity", "remediation", "prometheus")


def check_configs(paths):
    import yaml  # PyYAML
    bad = 0
    for path in paths:
        with open(path) as f:
            glob = (yaml.safe_load(f) or {}).get("global") or {}
        labels = glob.get("external_labels") or {}
        server = labels.get("prometheus")
        if not server:
            problem = "no prometheus label"
        elif server == "meta" and "cluster" in labels:
            problem = "meta-monitoring must not set cluster"
        elif server != "meta" and "cluster" not in labels:
            problem = "no cluster label"
        else:
            continue
        print(f"{path}: {problem}")
        bad += 1
    return 1 if bad else 0


def compliant(labels):
    if not all(labels.get(k) for k in ALERT_LABELS):
        return False
    # Meta-monitoring alerts may concern things outside any cluster.
    return bool(labels.get("cluster")) or labels.get("prometheus") == "meta"


def check_alerts(url):
    with urllib.request.urlopen(url, timeout=10) as resp:
        alerts = json.load(resp)
    ok = sum(compliant(a["labels"]) for a in alerts)
    share = ok / len(alerts) if alerts else 1.0
    print(f"alerts={len(alerts)} compliant={ok} share={share:.2f}")
    return 0


if __name__ == "__main__":
    mode = sys.argv[1] if len(sys.argv) > 1 else ""
    if mode == "configs":
        sys.exit(check_configs(sys.argv[2:]))
    if mode == "alerts":
        default = "http://127.0.0.1:9093/api/v2/alerts"
        sys.exit(check_alerts(sys.argv[2] if len(sys.argv) > 2 else default))
    sys.exit(__doc__)

On the samples, configs flagged a meta-monitoring file with cluster: meta and a file without external labels (exit code 1), and alerts printed alerts=4 compliant=2 share=0.50. A promtool unit test remains useful as a regression test for one important rule: it checks only rules that have a test case and never sees external_labels, but its exp_labels must match exactly, so a dropped label fails the build.^[25]

Example status: syntax-checked, not executed (requires promtool).

# checkout-rules.test.yml: run with  promtool test rules checkout-rules.test.yml
rule_files:
  - checkout-rules.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout-api",code="500"}'
        values: '0+10x20'
      - series: 'http_requests_total{job="checkout-api",code="200"}'
        values: '0+90x20'
    alert_rule_test:
      - eval_time: 15m
        alertname: HighErrorRatio
        exp_alerts:
          - exp_labels:
              severity: page
              service: checkout-api
              team: payments
              database: orders-db
              remediation: none
            exp_annotations:
              summary: "checkout-api 5xx ratio above 5% for 5 minutes"
              runbook_url: >-
                https://runbooks.example.com/checkout-api/high-error-ratio

Artefact 2: grouping, routing and guarded inhibition. The defaults shown (group_wait 30s, group_interval 5m, repeat_interval 4h) are Alertmanager’s documented defaults, written out for reviewers.^[29] Three details carry the safety: each inhibit rule requires every equal label to be non-empty on the source, so a source without it mutes nothing; rule 1 never mutes the meta-monitoring server, which watches the alert path itself; and the gateways authenticate Alertmanager by a bearer token read from a file (credentials_file under http_config → authorization).^[29]

Example status: syntax-checked, not executed (requires Alertmanager 0.34.x; Lab 1 runs it).

# alertmanager.yml: grouping, routing and guarded inhibition (Alertmanager 0.34.x).
global:
  resolve_timeout: 5m
route:
  receiver: ticket-queue
  group_by: [cluster, service]          # one notification per service per cluster
  group_wait: 30s                       # let related (and inhibiting) alerts arrive
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['remediation="auto"']  # allow-listed classes go to the runner...
      receiver: eda-remediation
      continue: true                    # ...and still reach a ticket or page
    - matchers: ['severity="page"']
      receiver: oncall-pager
    - matchers: ['severity="ticket"']
      receiver: ticket-queue
inhibit_rules:
  # 1. A cluster is unreachable: mute the per-service symptoms in that cluster.
  - source_matchers:
      - alertname="ClusterUnreachable"
      - cluster=~".+"                   # guard: no cluster, mutes nothing
    target_matchers:
      - alertname!="ClusterUnreachable"
      - prometheus!="meta"              # never mute the meta-monitoring alerts
    equal: [cluster]
  # 2. A database is down: mute error and latency symptoms of services that use it.
  - source_matchers:
      - alertname="DatabaseDown"
      - cluster=~".+"                   # guard
      - database=~".+"                  # guard
    target_matchers: ['alertname=~"HighErrorRatio|HighLatency"']
    equal: [cluster, database]
receivers:
  - name: oncall-pager
    webhook_configs:
      - url: https://incident-gateway.example.com/hooks/alertmanager
        http_config:
          authorization:                # Bearer by default; file mode 0400
            credentials_file: /etc/alertmanager/secrets/incident-gateway
  - name: ticket-queue
    webhook_configs:
      - url: https://ticket-gateway.example.com/hooks/alertmanager
        http_config:
          authorization:
            credentials_file: /etc/alertmanager/secrets/ticket-gateway
  - name: eda-remediation
    webhook_configs:
      - url: http://127.0.0.1:5050/alerts   # rulebook on this host (loopback)
        send_resolved: false

The residual risk is an observer-side partition. If the meta-monitoring server loses its own route to eu-1 while eu-1’s Prometheus still reaches Alertmanager, ClusterUnreachable{cluster="eu-1"} fires and rule 1 mutes eu-1’s genuine alerts; the only page is about the monitoring path. So fire the source only when two independent vantage points agree (for example up == 0 from the meta server and a failed probe from a second location), and make amtool alert query --inhibited cluster=eu-1 the first step of that alert’s runbook.

Protect Alertmanager’s API as well (security controls).

Artefact 3: a topology file. This is this Atlas’s minimal format, not a standard. Generate it from tracing data where you can and review it by hand. List shared infrastructure explicitly, because faults there fan out across services that never call each other (Lab 2 shows the cost of leaving it out).

Example status: syntax-checked; a data file, nothing to execute.

services:
  web:          {team: storefront, tier: 1, depends_on: [checkout-api]}
  checkout-api: {team: payments,   tier: 1,
                 depends_on: [orders-db, cache, payments-gw]}
  payments-gw:  {team: payments,   tier: 1, depends_on: [payments-db]}
  worker:       {team: fulfilment, tier: 2, depends_on: [queue, orders-db]}
shared_infrastructure:
  net-zone-a:   {owner: network, carries: [cache, queue, orders-db]}

Artefact 4: a normalised event. Anything that leaves Alertmanager, or arrives from another monitor, is mapped to one schema before correlation. The correlation block records who decided what, so that decisions can be audited and evaluated.

Example status: JSON syntax-checked; the schema name is this Atlas’s own.

{
  "schema": "atlas.ops-event/v1",
  "fingerprint": "3f6c2a9d1b7e4c08",
  "source": "prometheus/eu-1",
  "status": "firing",
  "starts_at": "2026-10-16T09:14:30Z",
  "alertname": "HighErrorRatio",
  "service": "checkout-api",
  "team": "payments",
  "environment": "production",
  "cluster": "eu-1",
  "severity": "page",
  "runbook_url": "https://runbooks.example.com/checkout-api/high-error-ratio",
  "recent_changes": ["deploy checkout-api v2026.10.16-3 at 2026-10-16T09:02:11Z",
                     "config net-zone-a acl-118 at 2026-10-16T08:47:05Z"],
  "correlation": {"group_id": null, "probable_cause": null, "decided_by": null}
}

Change events fill recent_changes. A deploy’s last step records the change with the alerts’ service value and a version (OpenTelemetry’s counterpart is service.version).^[68] For an alert or a group, the changes of the preceding hour to its services, their dependencies and the shared infrastructure under them (Artefact 3) are attached, the alerting services’ own first. The hour and the order are this Atlas’s starting values; tune them on change-caused incidents.

Example status: executed on 2026-10-17 (Python 3.8.10, PyYAML 5.3.1): record against a local test receiver, attach on four sample changes; not run in a CI system.

"""change_events.py: record deploys as change events; attach them to alert groups.

  python3 change_events.py record SERVICE VERSION [URL]  # last step of a deploy
  python3 change_events.py attach topology.yml changes.jsonl group.json [MINUTES]
Standard library, plus PyYAML to read the topology file (Artefact 3).
"""
import json
import os
import sys
import urllib.request
from datetime import datetime, timedelta, timezone

FMT = "%Y-%m-%dT%H:%M:%SZ"


def record(service, version, url=None):
    event = {
        "schema": "atlas.change-event/v1",
        "kind": os.environ.get("CHANGE_KIND", "deploy"),
        "service": service,              # the same value as the alerts' label
        "version": version,
        "environment": os.environ.get("ENVIRONMENT", "production"),
        "by": os.environ.get("CHANGE_ACTOR", "unknown"),  # from your CI system
        "at": datetime.now(timezone.utc).strftime(FMT),
    }
    body = json.dumps(event).encode()
    print(body.decode())
    if url:  # the bearer token comes from a file, never the command line
        with open(os.environ["CHANGE_LOG_TOKEN_FILE"]) as f:
            token = f.read().strip()
        req = urllib.request.Request(url, data=body, method="POST", headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {token}"})
        urllib.request.urlopen(req, timeout=10).close()
    return 0


def nearby(topology, services):
    """The group's services, their dependencies, and shared infrastructure."""
    near = set(services)
    for name in services:
        entry = (topology.get("services") or {}).get(name) or {}
        near.update(entry.get("depends_on", []))
    for name, entry in (topology.get("shared_infrastructure") or {}).items():
        if near & set(entry.get("carries", [])):
            near.add(name)
    return near


def attach(topology_path, changes_path, group_path, minutes=60):
    import yaml  # PyYAML
    with open(topology_path) as f:
        topology = yaml.safe_load(f)
    with open(changes_path) as f:
        changes = [json.loads(line) for line in f if line.strip()]
    with open(group_path) as f:
        group = json.load(f)
    services = {a["service"] for a in group["alerts"]}
    start = min(datetime.strptime(a["starts_at"], FMT) for a in group["alerts"])
    since = start - timedelta(minutes=minutes)
    near = nearby(topology, services)
    hits = [c for c in changes if c["service"] in near
            and since <= datetime.strptime(c["at"], FMT) <= start]
    hits.sort(key=lambda c: c["at"], reverse=True)         # newest first, but
    hits.sort(key=lambda c: c["service"] not in services)  # own services first
    print(f"group {group['group_id']}: {len(hits)} of {len(changes)} changes "
          f"in the {minutes} minutes before {start.strftime(FMT)}")
    for c in hits:
        where = "alerting service" if c["service"] in services else "dependency"
        print(f'  {c["kind"]} {c["service"]} {c["version"]} at {c["at"]} ({where})')
    return 0


if __name__ == "__main__":
    args = sys.argv[1:]
    if args[:1] == ["record"] and len(args) in (3, 4):
        sys.exit(record(*args[1:]))
    if args[:1] == ["attach"] and len(args) in (4, 5):
        sys.exit(attach(*args[1:4], *[int(m) for m in args[4:]]))
    sys.exit(__doc__)

For a group of checkout-api and web alerts from 09:14:30, attach left out a worker deploy (not a dependency) and yesterday’s checkout-api deploy:

group g-0917: 2 of 4 changes in the 60 minutes before 2026-10-16T09:14:30Z
  deploy checkout-api v2026.10.16-3 at 2026-10-16T09:02:11Z (alerting service)
  config net-zone-a acl-118 at 2026-10-16T08:47:05Z (dependency)

Artefact 5: guarded remediation. One class is automated: rotate and compress logs when a log volume is nearly full. The rule sets remediation: auto; Artefact 2 sends such alerts to the rulebook and on to the ticket queue; the rulebook’s allow-list decides whether anything runs; a second rule pages at 95% regardless. The gate figure below shows where each check lives; the per-host throttle uses ansible-rulebook’s throttle “to counter event storms”,^[69] and runs go one at a time.^[70] Rotation has nothing to undo, so failure pages a person; a class that changes configuration needs a rollback task there. AutomationOps teaches these controls in general.

Three security facts shape it. The ansible.eda.alertmanager source (collection 2.13.0) is a plain HTTP webhook that performs no authentication, defaults to localhost, and copies every request header into each event (meta.headers);^[71] keep it on loopback beside Alertmanager. A runner on another host should use ansible.eda.webhook, which checks a bearer token, supports TLS and mutual TLS, and drops the Authorization header from events; events then arrive under event.payload, and the insert_hosts_to_meta filter must supply the hosts.^[72][73] And “You cannot limit privilege escalation permissions to certain commands”,^[74] so the runner may become only the log owner, never root.

Example status: YAML syntax-checked; keys checked against the Prometheus, ansible-rulebook 1.3.0 and Ansible documentation; not executed (requires node_exporter, ansible-rulebook with the ansible.eda collection, and an inventory whose host names match the instance label without its port).

# node-log-rules.yml: one automatable class, and a page that needs no runner.
groups:
  - name: node-logs
    labels:
      service: node-logs
      team: platform
    rules:
      - alert: LogVolumeNearlyFull        # allow-listed: a ticket plus the runner
        expr: |
          1 - node_filesystem_avail_bytes{mountpoint="/var/log"}
            / node_filesystem_size_bytes{mountpoint="/var/log"} > 0.85
        for: 10m
        labels:
          severity: ticket
          remediation: auto
        annotations:
          summary: "/var/log on {{ $labels.instance }} is over 85% full"
          runbook_url: "https://runbooks.example.com/node/log-volume"
      - alert: LogVolumeCritical          # pages whether or not automation ran
        expr: |
          1 - node_filesystem_avail_bytes{mountpoint="/var/log"}
            / node_filesystem_size_bytes{mountpoint="/var/log"} > 0.95
        for: 5m
        labels:
          severity: page
          remediation: none
        annotations:
          summary: "/var/log on {{ $labels.instance }} is over 95% full"
          runbook_url: "https://runbooks.example.com/node/log-volume"

# remediation-rulebook.yml: run with
#   ansible-rulebook -r remediation-rulebook.yml -i inventory.yml
- name: Guarded remediation for known, reversible faults
  hosts: all
  execution_strategy: sequential        # one action at a time (the default)
  sources:
    - ansible.eda.alertmanager:
        host: 127.0.0.1                 # loopback: the source has no auth
        port: 5050
        data_alerts_path: alerts
        data_host_path: labels.instance # host:port; the source strips the port
  rules:
    - name: Rotate logs on a nearly full log volume (the allow-list)
      condition: >-
        event.alert.status == "firing" and
        event.alert.labels.alertname == "LogVolumeNearlyFull" and
        event.alert.labels.remediation == "auto"
      throttle:
        once_within: 30 minutes
        group_by_attributes:
          - event.meta.hosts
      action:
        run_playbook:
          name: playbooks/rotate_logs.yml
    - name: Labelled auto but not allow-listed here, so it stays with people
      condition: >-
        event.alert.status == "firing" and
        event.alert.labels.alertname != "LogVolumeNearlyFull"
      action:
        debug:
          msg: "Not on the allow-list: {{ event.alert.labels.alertname }}"

# playbooks/rotate_logs.yml: the checks the rulebook cannot make.
- name: Rotate logs on a nearly full log volume, within caps, then verify
  hosts: all                    # ansible-rulebook limits it to the alert's host
  serial: 1
  max_fail_percentage: 0
  vars:
    run_log: /var/lib/eda/rotate_logs.runs  # runner host, one line per run
    host_daily_cap: 3                       # runs per host per day
    fleet_hourly_cap: 5                     # runs per hour, all hosts: blast radius
    max_used_pct: 85                        # the alert threshold
    gateway_token_file: /etc/eda/secrets/gateway-token
  tasks:
    - name: Read earlier runs
      ansible.builtin.command: cat {{ run_log }}
      register: runs
      changed_when: false
      failed_when: false
      delegate_to: localhost

    - name: Stop unless both caps allow another run
      vars:
        today: "{{ now(utc=true, fmt='%Y-%m-%d') }}"
        this_hour: "{{ now(utc=true, fmt='%Y-%m-%dT%H') }}"
        host_runs: >-
          {{ runs.stdout_lines
             | select('match', today ~ '.* ' ~ inventory_hostname ~ '$')
             | list | length }}
        fleet_runs: >-
          {{ runs.stdout_lines | select('match', this_hour) | list | length }}
      ansible.builtin.assert:
        that:
          - host_runs | int < host_daily_cap
          - fleet_runs | int < fleet_hourly_cap
        fail_msg: "Cap reached: no automatic run; the ticket and the 95% page stay"

    - name: Record this run before acting
      ansible.builtin.lineinfile:
        path: "{{ run_log }}"
        line: >-
          {{ now(utc=true, fmt='%Y-%m-%dT%H:%M:%S') }} {{ inventory_hostname }}
        create: true
        mode: "0640"
      delegate_to: localhost

    - name: Act, verify, and page a person on any failure
      block:
        - name: Rotate and compress within the retention the log owner approved
          ansible.builtin.command: >-
            logrotate --force --state /var/lib/applog/logrotate.state
            /etc/logrotate.d/app
          become: true
          become_user: applog         # the log owner, never root
        - name: Verify that usage fell below the alert threshold
          ansible.builtin.command: df --output=pcent /var/log
          register: usage
          changed_when: false
          failed_when: >-
            usage.stdout_lines[-1] | trim | replace('%', '') | int >= max_used_pct
      rescue:
        - name: Page a person (rotation has nothing to undo)
          ansible.builtin.uri:
            url: https://incident-gateway.example.com/hooks/remediation-failed
            method: POST
            headers:
              Authorization: >-
                Bearer {{ lookup('ansible.builtin.file', gateway_token_file) }}
            body_format: json
            body:
              host: "{{ inventory_hostname }}"
              action: rotate_logs
              result: failed
          delegate_to: localhost
        - name: Fail the run so that the rulebook log shows it
          ansible.builtin.fail:
            msg: "rotate_logs failed on {{ inventory_hostname }}; a person is paged"

Every automated action passes the same gate. Google’s account of an automated decommissioning job that wiped the disks of its CDN machines explains the gate’s checks: afterwards Google added “more sanity checks—including rate limiting” and made the workflow idempotent.^[75]

flowchart TB
  T["Alert with remediation=auto"] --> A{"On the rulebook allow-list?"}
  A -->|"no"| H["Stays with people: ticket now, page at 95%"]
  A -->|"yes"| C{"Host throttle free? (rulebook)"}
  C -->|"no"| H
  C -->|"yes"| B{"Daily and fleet caps free? (playbook)"}
  B -->|"no"| H
  B -->|"yes"| D["Run as the log owner, not root"]
  D --> E{"Verified fixed? (playbook)"}
  E -->|"yes"| F["Record outcome; alert resolves"]
  E -->|"no"| G["Roll back if possible; page a person"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class A,B,C,E decision
  class D,F core
  class H,G risk

Three labs, from configuration to measurement; Labs 2 and 3 were executed and their outputs are verbatim.

• Objective: see grouping and inhibition work, reproduce the missing-label pitfall, then watch the guard stop it.
• Prerequisites: Linux with bash, curl, tar and sha256sum; port 127.0.0.1:9093 free; Artefact 2 saved as alertmanager.yml. Run it as a script (bash lab1.sh alertmanager.yml); pasted into a terminal, set -e would close the shell at the first error. Cost and risk: none; everything binds to localhost. Notification errors (example.com, missing credential files) are expected.

Example status: syntax-checked, not executed (downloads a third-party binary).

#!/usr/bin/env bash
# lab1.sh: run with  bash lab1.sh /path/to/alertmanager.yml  (Artefact 2)
set -euo pipefail
CONFIG="$(realpath "${1:?usage: bash lab1.sh /path/to/alertmanager.yml}")"
VERSION=0.34.1
TARBALL="alertmanager-${VERSION}.linux-amd64.tar.gz"
BASE="https://github.com/prometheus/alertmanager/releases/download/v${VERSION}"
WORK="$(mktemp -d)" && cd "$WORK"
curl -fsSLO "${BASE}/${TARBALL}"
curl -fsSLO "${BASE}/sha256sums.txt"
grep "${TARBALL}" sha256sums.txt | sha256sum -c -
tar xzf "${TARBALL}" && cd "alertmanager-${VERSION}.linux-amd64"

grep -v '# guard' "$CONFIG" > unguarded.yml     # Artefact 2 without its guards
./amtool check-config "$CONFIG" unguarded.yml
./amtool config routes test --config.file="$CONFIG" \
  --verify.receivers=oncall-pager severity=page service=checkout-api
cp unguarded.yml active.yml

# Single node: an empty cluster address disables HA mode.
./alertmanager --config.file=active.yml --storage.path=./data \
  --web.listen-address=127.0.0.1:9093 --cluster.listen-address="" > am.log 2>&1 &
AM_PID=$!
trap 'kill "$AM_PID"; rm -rf "$WORK"' EXIT     # teardown when the script ends
sleep 3
AM="--alertmanager.url=http://127.0.0.1:9093"

# 1. Inhibition as intended: the same cluster label on source and target.
./amtool $AM alert add alertname=ClusterUnreachable cluster=eu-1 severity=page
./amtool $AM alert add alertname=HighErrorRatio cluster=eu-1 \
  service=checkout-api severity=page
./amtool $AM alert add alertname=HighErrorRatio cluster=eu-2 \
  service=checkout-api severity=page
sleep 5
echo "== active";    ./amtool $AM alert query
echo "== inhibited"; ./amtool $AM alert query --inhibited

# 2. The pitfall: neither alert has a cluster label, so the unguarded rule applies.
./amtool $AM alert add alertname=ClusterUnreachable severity=page
./amtool $AM alert add alertname=EndpointDown service=storefront-probe severity=page
sleep 5
echo "== inhibited, unguarded"; ./amtool $AM alert query --inhibited

# 3. The fix: load the guarded file and ask again.
cp "$CONFIG" active.yml
curl -fsS -X POST http://127.0.0.1:9093/-/reload
sleep 2
echo "== inhibited, guarded"; ./amtool $AM alert query --inhibited

• Expected output: check-config reports SUCCESS for both files, with 2 inhibit rules, 3 receivers and 0 templates (the format in the Alertmanager documentation^[29]); routes test prints oncall-pager. Step 1: active are ClusterUnreachable (eu-1) and HighErrorRatio (eu-2); only the eu-1 HighErrorRatio is inhibited. Step 2: EndpointDown is inhibited too, although it concerns no cluster. Step 3: it is active again.
• Validation: the active eu-2 alert proves that equal: [cluster] works when the label exists; step 2 shows the documented missing-label behaviour;^[29] step 3 shows the guard removing it. With only --inhibited, amtool lists only inhibited alerts.^[76]
• Troubleshooting: address already in use means another Alertmanager is running; if nothing is inhibited, raise the sleep values; read am.log for configuration errors.
• Teardown: automatic: the trap stops Alertmanager and deletes the working directory. To explore by hand, comment out the trap line and clean up yourself.

• Objective: compare three grouping strategies on labelled incidents using pairwise precision and recall, then see what a stale topology does.
• Prerequisites: Python 3.8 or later; save the script as correlation_lab.py. Cost and risk: none.

How it is scored. Every pair of alerts is a decision: the strategy put them in the same group or not, and the ground truth says whether they belong to the same incident. Manning and colleagues present this pair-decision view as an alternative to measures such as purity and normalised mutual information.^[77]

Pairwise precision = TP / (TP + FP); pairwise recall = TP / (TP + FN).^[78] Pairs grow with the square of an incident’s size, so one large incident can dominate both numbers. Report the per-incident over-merge and split counts next to them, as the lab does.

Example status: executed on 2026-10-17 (Python 3.8.10); output below is verbatim.

"""Lab 2: compare three alert-grouping strategies against labelled incidents.

Standard library only. Usage: python3 correlation_lab.py [--drop-edge api:db]
"""
import itertools
import random
import sys
from collections import defaultdict

# Caller -> callees. A failing callee makes its callers alert too.
TOPOLOGY = {
    "web": ["api"],
    "api": ["db", "cache", "payments"],
    "payments": ["payments-db"],
    "worker": ["queue", "db"],
    "reports": [],
}

# Ground truth: (incident id, root service, start minute, alerts per service).
INCIDENTS = [
    ("INC-1", "db", 0, {"db": ["DBDown"], "api": ["HighErrorRate", "HighLatency"],
                        "web": ["HighErrorRate"], "worker": ["JobFailures"]}),
    ("INC-2", "reports", 2, {"reports": ["DiskFull"]}),
    ("INC-3", "payments-db", 40, {"payments-db": ["SlowQueries"],
                                  "payments": ["HighLatency"],
                                  "api": ["HighLatency"]}),
    ("INC-4", "queue", 41, {"queue": ["QueueBacklog"], "worker": ["JobLag"]}),
    ("INC-5", "cache", 90, {"cache": ["Evictions"]}),
    # A shared network fault: not in the service topology,
    # and the switch itself is not monitored.
    ("INC-6", "network", 120, {"cache": ["ConnTimeouts"],
                               "queue": ["ConnTimeouts"]}),
]
WINDOW = 5  # minutes


def make_events(seed=7):
    """Each alert fires 1-3 times (re-notifications), with minutes of jitter."""
    rng = random.Random(seed)
    events = []
    for inc, _root, start, per_service in INCIDENTS:
        for svc, names in per_service.items():
            for name in names:
                first = start + rng.randint(0, 3)
                for repeat in range(rng.randint(1, 3)):
                    events.append({"t": first + repeat, "service": svc,
                                   "alertname": name, "truth": inc})
    return sorted(events, key=lambda e: (e["t"], e["service"], e["alertname"]))


def deduplicate(events, episode_gap=10):
    """Drop repeats of a fingerprint (service + alertname) within episode_gap."""
    last_seen, out = {}, []
    for e in events:
        key = (e["service"], e["alertname"])
        if key not in last_seen or e["t"] - last_seen[key] > episode_gap:
            out.append(e)
        last_seen[key] = e["t"]
    return out


def neighbours(topology):
    graph = defaultdict(set)
    for caller, callees in topology.items():
        for callee in callees:
            graph[caller].add(callee)
            graph[callee].add(caller)
    return graph


def group(alerts, related):
    """Join every group with a related member seen within WINDOW minutes.

    An alert that matches several groups merges them (it bridges them).
    """
    groups = []
    for a in alerts:
        hits = [g for g in groups
                if any(a["t"] - m["t"] <= WINDOW and related(a, m) for m in g)]
        if not hits:
            groups.append([a])
            continue
        keep = hits[0]
        for other in hits[1:]:
            keep.extend(other)
            groups.remove(other)
        keep.append(a)
    return groups


def pair_scores(groups):
    tp = fp = fn = 0
    members = [(gi, a["truth"]) for gi, g in enumerate(groups) for a in g]
    for (g1, t1), (g2, t2) in itertools.combinations(members, 2):
        same_group, same_truth = g1 == g2, t1 == t2
        tp += same_group and same_truth
        fp += same_group and not same_truth
        fn += (not same_group) and same_truth
    precision = tp / (tp + fp) if tp + fp else 1.0
    recall = tp / (tp + fn) if tp + fn else 1.0
    total = precision + recall
    f1 = 2 * precision * recall / total if total else 0.0
    return precision, recall, f1


def probable_cause(g, graph_down):
    """Candidate root: an alerting service whose callees do not alert here."""
    alerting = {a["service"] for a in g}
    leaves = [s for s in alerting if not (set(graph_down.get(s, [])) & alerting)]
    return sorted(leaves)


def report(name, groups, raw_count):
    p, r, f1 = pair_scores(groups)
    truths = defaultdict(set)
    for gi, g in enumerate(groups):
        for a in g:
            truths[a["truth"]].add(gi)
    merged = sum(1 for g in groups if len({a["truth"] for a in g}) > 1)
    split = sum(1 for gs in truths.values() if len(gs) > 1)
    compr = raw_count / len(groups)
    print(f"{name:<20} groups={len(groups):>2} compr={compr:4.1f}x "
          f"P={p:.2f} R={r:.2f} F1={f1:.2f} merged={merged} split={split}")


def main():
    topology = {k: list(v) for k, v in TOPOLOGY.items()}
    if "--drop-edge" in sys.argv:
        caller, callee = sys.argv[sys.argv.index("--drop-edge") + 1].split(":")
        topology[caller].remove(callee)
        print(f"(topology edge {caller} -> {callee} removed: "
              "simulating a stale CMDB)")
    raw = make_events()
    alerts = deduplicate(raw)
    print(f"raw events={len(raw)} after dedup={len(alerts)} "
          f"true incidents={len(INCIDENTS)}")
    graph = neighbours(topology)
    strategies = {
        "S1 time window only": lambda a, b: True,
        "S2 same service": lambda a, b: a["service"] == b["service"],
        "S3 topology-aware": lambda a, b: (a["service"] == b["service"]
                                           or b["service"] in graph[a["service"]]),
    }
    results = {}
    for name, rel in strategies.items():
        results[name] = group(alerts, rel)
        report(name, results[name], len(raw))
    print("S3 probable causes:")
    roots = {inc: root for inc, root, _s, _a in INCIDENTS}
    hits = 0
    for g in results["S3 topology-aware"]:
        truths = [a["truth"] for a in g]
        main_truth = max(set(truths), key=truths.count)
        cand = probable_cause(g, topology)
        hit = cand == [roots[main_truth]]
        hits += hit
        print(f"  {main_truth}: candidates={cand} truth={roots[main_truth]} "
              f"{'OK' if hit else 'MISS'}")
    n_groups = len(results["S3 topology-aware"])
    print(f"top-1 probable-cause accuracy={hits}/{n_groups}")


if __name__ == "__main__":
    main()

Steps. (1) Run python3 correlation_lab.py. (2) Run python3 correlation_lab.py --drop-edge api:db: a CMDB that has lost the fact that the API calls the database. (3) Set WINDOW to 2, then 15, and re-run.

Expected output (step 1, verbatim):

raw events=22 after dedup=14 true incidents=6
S1 time window only  groups= 4 compr= 5.5x P=0.58 R=1.00 F1=0.73 merged=2 split=0
S2 same service      groups=13 compr= 1.7x P=1.00 R=0.07 F1=0.12 merged=0 split=4
S3 topology-aware    groups= 7 compr= 3.1x P=1.00 R=0.93 F1=0.97 merged=0 split=1
S3 probable causes:
  INC-1: candidates=['db'] truth=db OK
  INC-2: candidates=['reports'] truth=reports OK
  INC-3: candidates=['payments-db'] truth=payments-db OK
  INC-4: candidates=['queue'] truth=queue OK
  INC-5: candidates=['cache'] truth=cache OK
  INC-6: candidates=['queue'] truth=network MISS
  INC-6: candidates=['cache'] truth=network MISS
top-1 probable-cause accuracy=5/7

Step 2 changes the S3 line to groups= 8 compr= 2.8x P=1.00 R=0.53 F1=0.70 merged=0 split=2, and probable-cause accuracy falls to 5/8.

• Validation and interpretation. The time window has the best compression (5.5×) and perfect recall, but merged the database outage with an unrelated full disk and the payments slowdown with an unrelated queue backlog: precision 0.58. Grouping by service never merges wrongly but finds almost nothing (recall 0.07). Topology does best because the data were generated from it, which flatters it. The realistic lessons are the network incident, split because shared infrastructure is missing from the graph, and step 2, where one missing edge cuts recall from 0.93 to 0.53.
• Troubleshooting: ValueError: list.remove(x) means the --drop-edge pair does not exist (use caller:callee from TOPOLOGY); an f-string syntax error means Python is older than 3.6.
• Teardown: delete the script; it writes nothing.

• Objective: compare a static band with a seasonal robust baseline on two weeks of synthetic traffic with four labelled faults and two unlabelled business events (a weekday holiday and a marketing campaign).
• Method: the modified z-score, 0.6745 × (x − median) / MAD, against the same 15-minute slot of the same type of day, flagging absolute scores above 3.5, the threshold Iglewicz and Hoaglin recommend according to the NIST handbook.^[33] As with a Prometheus for clause, a flag must hold for five minutes before an alert fires.^[64]
• Prerequisites: Python 3.8 or later; save the script as anomaly_lab.py. Cost and risk: none (about 0.1 s of CPU).

Example status: executed on 2026-10-17 (Python 3.8.10); output below is verbatim.

"""Lab 3: static thresholds versus a seasonal robust baseline.

Standard library only. A synthetic request rate, two weeks at 1-minute resolution:
week 1 trains, week 2 is scored. Usage: python3 anomaly_lab.py [--calendar]
"""
import math
import random
import statistics
import sys

DAY = 1440
rng = random.Random(42)


def normal_rate(minute):
    """Daily cycle (low at night, peak mid-afternoon), quieter weekends, noise."""
    hour = (minute % DAY) / 60
    weekday = (minute // DAY) % 7 < 5
    if 6 <= hour <= 22:
        base = 200 + 800 * max(0.0, math.sin(math.pi * (hour - 6) / 16))
    else:
        base = 200
    base *= 1.0 if weekday else 0.6
    return max(0.0, rng.gauss(base, 0.05 * base + 10))


series = [normal_rate(m) for m in range(14 * DAY)]

# Labelled anomalies in week 2: (name, start minute, length, transform).
W2 = 7 * DAY
ANOMALIES = [
    ("peak-hour drop to 40%", W2 + 0 * DAY + 14 * 60, 30, lambda v, i: v * 0.4),
    ("night spike x2.5", W2 + 1 * DAY + 3 * 60, 15, lambda v, i: v * 2.5),
    ("total outage (zero)", W2 + 2 * DAY + 10 * 60, 20, lambda v, i: 0.0),
    ("slow 30% decline", W2 + 3 * DAY + 8 * 60, 600,
     lambda v, i: v * (1 - 0.3 * i / 600)),
]
truth = set()
for _name, start, length, f in ANOMALIES:
    for i in range(length):
        series[start + i] = f(series[start + i], i)
        truth.add(start + i)

# Two unlabelled business events in week 2: real changes in traffic, not incidents.
HOLIDAY = 11  # a public holiday on a weekday (day 11): weekend-like traffic
for m in range(HOLIDAY * DAY, (HOLIDAY + 1) * DAY):
    series[m] *= 0.6
for m in range(W2 + 5 * DAY + 18 * 60, W2 + 5 * DAY + 20 * 60):  # a campaign
    series[m] *= 1.6

train, test_start = series[:W2], W2


# Detector 1: one static band from the training week (0.1th and 99.9th percentiles).
q = statistics.quantiles(train, n=1000)
low, high = q[0], q[-1]


def static_flag(m):
    return series[m] < low or series[m] > high


# Detector 2: robust z-score against the same 15-minute slot of the same day type.
# --calendar tells the detector that day 11 is a holiday (treat it like a weekend).
HOLIDAYS = {HOLIDAY} if "--calendar" in sys.argv else set()


def slot(m):
    day = m // DAY
    return (day % 7 < 5 and day not in HOLIDAYS, (m % DAY) // 15)


buckets = {}
for m, v in enumerate(train):
    buckets.setdefault(slot(m), []).append(v)
baseline = {}
for k, vs in buckets.items():
    med = statistics.median(vs)
    mad = statistics.median(abs(v - med) for v in vs) or 1.0
    baseline[k] = (med, mad)


def seasonal_flag(m):
    med, mad = baseline[slot(m)]
    # Modified z-score (Iglewicz and Hoaglin): flag when |M| > 3.5.
    return abs(0.6745 * (series[m] - med) / mad) > 3.5


def alerts(flag, for_minutes=5):
    """Fire once the flag has held for for_minutes (like Prometheus 'for:')."""
    fired, run, out = False, 0, []
    for m in range(test_start, len(series)):
        run = run + 1 if flag(m) else 0
        if run >= for_minutes and not fired:
            out.append(m)
            fired = True
        if run == 0:
            fired = False
    return out


def matches(m):
    """True if an alert at minute m belongs to a labelled anomaly (10 min grace)."""
    return any(s <= m < s + n + 10 for _a, s, n, _f in ANOMALIES)


def evaluate(name, fired):
    detected, delays = [], []
    for aname, start, length, _f in ANOMALIES:
        hits = [m for m in fired if start <= m < start + length + 10]
        detected.append(aname if hits else None)
        if hits:
            delays.append(hits[0] - start)
    false = [m for m in fired if not matches(m)]
    on_holiday = sum(1 for m in false if m // DAY == HOLIDAY)
    precision = (len(fired) - len(false)) / len(fired) if fired else 1.0
    recall = sum(d is not None for d in detected) / len(ANOMALIES)
    print(f"{name:<18} alerts={len(fired):>3} false={len(false):>3} "
          f"(holiday {on_holiday:>2}) P={precision:.2f} R={recall:.2f}")
    print(f"    delays to first alert (min): {delays}")
    for (aname, *_rest), d in zip(ANOMALIES, detected):
        print(f"    {'caught' if d else 'MISSED':<6} {aname}")


print(f"static band: alert below {low:.0f} or above {high:.0f} requests/min")
evaluate("static band", alerts(static_flag))
evaluate("seasonal robust z", alerts(seasonal_flag))

Steps. (1) Run python3 anomaly_lab.py. (2) Run python3 anomaly_lab.py --calendar, which tells the detector that day 11 is a holiday.

Expected output (step 1, verbatim):

static band: alert below 84 or above 1129 requests/min
static band        alerts=  1 false=  0 (holiday  0) P=1.00 R=0.25
    delays to first alert (min): [4]
    MISSED peak-hour drop to 40%
    MISSED night spike x2.5
    caught total outage (zero)
    MISSED slow 30% decline
seasonal robust z  alerts= 44 false= 34 (holiday 32) P=0.23 R=1.00
    delays to first alert (min): [4, 4, 4, 401]
    caught peak-hour drop to 40%
    caught night spike x2.5
    caught total outage (zero)
    caught slow 30% decline

With --calendar the seasonal line becomes alerts= 12 false= 2 (holiday 0) P=0.83 R=1.00; nothing else changes.

• Validation and interpretation. A static band wide enough for the daily cycle misses everything except the total outage. The seasonal baseline catches all four faults, the slow decline only after 401 minutes, but without context it fires 34 false alerts, 32 of them on the holiday. One line of calendar data removes all 32; the 2 left come from the marketing campaign, which only a change or event feed could explain.
• Troubleshooting: AttributeError: module 'statistics' has no attribute 'quantiles' means Python is older than 3.8.
• Teardown: delete the script; it writes nothing.

Scenario 1: a documented case. Zhao and colleagues applied their alert-storm method at a large commercial bank (China EverBright Bank), reporting storm detection with an F1-score above 0.9 and more than 98% fewer alerts to examine.^[9] Microsoft researchers, after two years of incident management, named “incomplete service/resource dependencies” one of two critical challenges,^[15] the weakness Lab 2’s --drop-edge step exposes.

Scenarios 2 and 3: evidence for the approach, not documented cases. Scenario 2 rests on the change evidence in core practices; Lab 3 shows why Scenario 3 needs calendars and change feeds.

Scenarios 4 and 5: illustrative. Scenario 4 is Artefact 5; Scenario 5 rests on the Microsoft studies in Automation and AI.

These definitions apply across the Atlas. Precision is the share of a system’s positive decisions that were right; recall is the share of the true cases it found.^[78] For grouping, a decision is a pair of alerts placed together or apart, as in Lab 2.^[77] Security detection adds base rates and adversarial testing; see AIOps for Security.

The asymmetry that sets targets. On a paging route a split costs a duplicate page, while a wrong merge can hide a second incident: set strict targets for precision and missed incidents (review every miss), a looser one for splits. A ticket-only route can tilt towards recall. Derive the numbers from your own shadow-mode baseline; the research for this page found no credible public benchmarks.

Regulation, briefly and with caveats (as of October 2026). Most AIOps uses carry no high-risk obligations, but two EU duties deserve a check. First, Article 4 of the AI Act (AI literacy, applicable since 2 February 2025) binds deployers of any AI system, whatever its risk class. As amended by the Digital Omnibus on AI (Regulation (EU) 2026/1744, in force since 27 July 2026), deployers “shall take measures to support the development of AI literacy” of their staff.^[80][81] Whether a tool is an “AI system” in law is a legal question; a learned correlation engine is more likely to be one than a rule-based one. Second, Annex III lists as high-risk “AI systems intended to be used as safety components in the management and operation of critical digital infrastructure, road traffic, or in the supply of water, gas, heating or electricity”;^[82] the Omnibus makes the high-risk obligations for Annex III systems apply from 2 December 2027.^[81] If you operate critical digital infrastructure in the EU and an AIOps function could act as a safety component, take legal advice. Mapping AI systems to the Act belongs to ModelOps, and audit evidence to ComplianceOps. Elsewhere, NIST’s AI Risk Management Framework is “intended for voluntary use”, on 7 April 2026 NIST released a concept note for a profile on trustworthy AI in critical infrastructure, and NIST says it is revising AI RMF 1.0.^[83]

Auditability. Record for every grouping, suppression and automated action which rule or model, which version, which inputs, and what a person later confirmed; Artefact 4’s correlation block is a minimal version.

• Cost drivers. Licences (pricing changes often; get written quotes), integration engineering, the owner’s and service owners’ time, labelling, model compute and LLM tokens (FinOps for AI).
• Unit view. Track cost and triage people-hours per incident presented. Book “pages avoided” as savings only if you can show they were noise.
• Capacity: survive the storm you were built for. A correlation layer is busiest when everything else is failing. Alertmanager warns that group_interval “also sets the context timeout for the notification pipeline”, so a slow receiver can get notifications cancelled.^[29] BigPanda caps an incident at 300 correlated alerts,^[32] and PagerDuty’s learned grouping only adds alerts to incidents under 24 hours old.^[27] Load-test with a replayed storm.
• Sustainability. The footprint is usually small next to the telemetry it reads; keep model training and LLM calls proportional to measured value (GreenOps).

1. A heartbeat that bypasses correlation. The always-firing Watchdog alert and its external checker (IncidentOps, Lab 2) catch a dead pipeline, not wrong grouping.
2. Missed-incident review. Every post-incident review asks: did the event pipeline hide, delay or misroute anything?
3. Suppression audit. The monthly sample of inhibited and suppressed alerts and silences (checklists), including who created each silence.
4. Drift signals. Alert volume per service against its baseline, the age of topology entries, and label-contract compliance.
5. Evaluation history. Keep the monthly scorecard in version control, so that a platform change can be compared with the past.

Scenario (illustrative; the mechanism is documented). A team’s first label contract took cluster only from external_labels, and its meta-monitoring Prometheus had none. That server raises EndpointDown for a storefront synthetic probe, without a cluster label, and ClusterUnreachable for a forgotten development cluster whose scrape target has no cluster label either. The unguarded inhibit rule uses equal: [cluster]; with the label missing from both alerts, Alertmanager applies it.^[29] Customers report failed checkouts 25 minutes later.

flowchart TB
  S["Source: ClusterUnreachable, no cluster label"] --> R{"Unguarded rule: equal cluster"}
  T["Target: EndpointDown probe, no cluster label"] --> R
  R -->|"before: empty equals empty"| X["Inhibited: outage hidden"]
  S --> G{"Guarded rule: source needs cluster=~'.+'"}
  G -->|"after: source does not match"| N["Probe pages"]
  F["Wrong fix: external_labels cluster=meta"] -.->|"both alerts get meta: still equal"| X
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class R,G decision
  class X,F risk
  class N core
  class S,T neutral

1. Declare and page by hand. Do not wait for the pipeline. Follow the incident process in IncidentOps.

2. Is the alert there at all? amtool --alertmanager.url=http://127.0.0.1:9093 alert query alertname=EndpointDown returns nothing; adding --inhibited returns it.^[76] Now you know that it is muted, not missing.

3. What is muting it? Silences: amtool silence query. Inhibitions: the v2 API returns, with active=false, only silenced or inhibited alerts, and each alert’s status.inhibitedBy lists the fingerprints of the alerts that inhibit it.^[84][85]

   Example status: syntax-checked, not executed (requires a running Alertmanager and jq).

   curl -s 'http://127.0.0.1:9093/api/v2/alerts?active=false' \
     | jq -r '.[] | [.labels.alertname, (.labels.cluster // "-"),
                     (.status.inhibitedBy | join(","))] | @tsv'
   

4. Mitigate. Remove the stale target so that the source resolves, and add the guard (cluster=~".+" in source_matchers, as in Artefact 2); reload with POST /-/reload or SIGHUP.^[86] Do not add external_labels: {cluster: meta} to the meta-monitoring server: both alerts would carry cluster="meta", still equal, and the probe would stay muted.

5. Verify. amtool alert query --inhibited no longer lists EndpointDown, and a test EndpointDown from the meta-monitoring server pages.

6. Prevent. (a) Give cluster one meaning, the affected cluster, and record the evaluating server in prometheus (Artefact 1). (b) Guard every inhibit rule: each equal label non-empty on the source. © Never let meta-monitoring alerts be inhibition targets (prometheus!="meta"). (d) Lint every rule, check every server’s external labels in CI, and track compliance (a promtool test never sees external_labels). (e) Monthly, fire a test alert without the label from the meta-monitoring server and confirm that it pages.

• “Buy AIOps to fix alert noise.” If rules page on causes, flap, or lack owners, a platform groups the noise more neatly. Fix the rules first (SRE).
• Reporting compression as success. Lab 2’s 5.5-times compression merged unrelated incidents.
• Suppression without expiry. Every permanent silence is a future missed incident.
• “The model will learn our topology.” Learned grouping learns only from the feedback its vendor counts (see governance); PagerDuty warns that test data can make it behave unpredictably.^[27]
• Root-cause theatre. A confident single cause from a narrow menu can send responders the wrong way. Show the evidence and the runners-up.
• Autonomy first. Agents diagnose better than they resolve (Automation and AI). Start with suggestions; earn automation one action class at a time.
• Black-box vendor evaluation. A demo on the vendor’s data proves nothing about yours. Insist on shadow mode with your labels.

Hidden responsibilities. Someone must own the pipeline, keep topology current, run evaluations, review suppressions and approve automation classes; a managed platform hides that work rather than removing it.

The technique tree decides what to adopt; this weighted worksheet chooses which platform or stack, after a shadow-mode trial on your labelled incidents. The weights are this Atlas’s suggestion; change them, but write down why.

This model is this Atlas’s proposal, not an industry standard. Move up only when the exit criteria hold for a full quarter.

There is deliberately no level 5 “autonomous operations”: the evidence on this page does not support it as a goal for most organisations in 2026.

Quick wins usually come from the ten noisiest rules and from attaching changes to incidents; topology data is the usual blocker, so start collecting it on day 1.

Quick start (first week)

• An event-intelligence owner is named, and the time is in their objectives
• Three months of alerts and incidents exported; baseline alerts per incident and actionable-page rate
• The label contract (Artefact 1) is written
• Incident close asks “grouped correctly?” and “cause correct?”

Before enabling any suppression, inhibition or new grouping policy

• The policy is in version control, with checks (rule linting, promtool test rules, amtool config routes test)
• Every label in equal names the affected component (not the evaluating server) and is required non-empty in source_matchers
• Alerts from meta-monitoring are excluded as inhibition targets
• Shadow-mode results on labelled incidents meet the precision, recall and missed-incident targets
• Suppressed alerts stay queryable; the suppression has an owner and an expiry date
• A rollback (revert and reload) has been rehearsed

Before automating a remediation class

• The fault is well understood, and the fix is reversible and idempotent
• Blast-radius limit, throttle and daily cap are set
• The runner’s credential can do this action and nothing else; the ingest endpoint is authenticated or bound to localhost
• Verification and rollback steps are automated; failure pages a person
• The change authority has approved the class; the review date is set

Monthly review

• Scorecard updated (Template 3), and every missed incident reviewed
• A person has inspected a random sample of inhibited and suppressed alerts, and the silences
• Topology entries older than 90 days re-confirmed by their owners
• The ten noisiest rules have actions assigned

Template 1: policy record (one per grouping rule, inhibition, detector or remediation class)

Policy ID / name:            e.g. INH-007 "ClusterUnreachable mutes symptoms"
Type:                        grouping | inhibition | suppression | detector |
                             remediation
Owner (team, person):
Scope:                       services, clusters, alertnames
Logic:                       link to the rule or config in version control (commit)
Labels relied on:            each label in equal or group_by: its meaning
                             (affected or evaluating), who sets it, and the
                             non-empty guard
Evidence for release:        shadow-mode period, labelled incidents used,
                             precision, recall, missed incidents
Failure mode if wrong:       what could be hidden, merged or changed; e.g. INH-007:
                             the meta server loses its route to eu-1 while eu-1
                             still alerts, so eu-1's real alerts are muted
Expiry / review date:
Rollback:                    revert commit + reload, expected time
Approved by / date:

Template 2: automation safety review

Action class:                e.g. rotate and compress logs on LogVolumeNearlyFull
Trigger (exact labels):
Allow-list entry:            yes/no, and the commit
Blast-radius limit:          max hosts per run / per hour; share of capacity
Throttle and daily cap:      once_within ... ; max runs per day ...
Credential:                  identity, permissions, where stored, rotation
Verification:                check that proves the fix worked, and timeout
Rollback:                    automatic step; who is paged on failure
Data, money or security impact: none / describe (if any, keep a person in the loop)
Change authority decision:   approved / rejected, date, review date

Template 3: monthly evaluation scorecard

Template 4: platform shadow-trial plan

Candidate and version:
Data supplied:               N weeks of raw alerts, M labelled incidents (list),
                             topology file, change log
Mode:                        shadow only; no paging changes
Success criteria:            precision >= __, recall >= __, missed incidents = 0,
                             an explanation shown for every group
Exports required:            alerts, groups, causes, decisions with reasons (API)
Security and data review:    residency, retention, redaction, sub-processors
Decision date and owner:

Four traditions handle operational events; they are layers, not rivals.

A sound design uses SLO alerting to decide what deserves a page, event management to connect events to processes, and AIOps to group, enrich and rank what remains.

Do we need data scientists? Not to start: levels 1 and 2 of the maturity model are engineering and ownership work. Evaluating detectors and vendors needs statistical literacy; data scientists only if you build models.

How many labelled incidents do we need? No credible published threshold exists. A few dozen expose gross over-merging; a fair comparison of learned grouping with rules needs more, across kinds of failure. The decision tree’s 100 is a rule of thumb.

Will AIOps reduce MTTR? It can shorten the time to engage the right team and find a probable cause, but overall MTTR measures that poorly: durations are skewed and the “R” is ambiguous (IncidentOps). Measure the phases AIOps affects.

Can an LLM do root-cause analysis for us? It can suggest causes for a person to confirm: Microsoft reported root-cause category accuracy up to 0.766 on its own incidents,^[40] but agents rarely resolve live faults (see Automation and AI).

Should security alerts go through the same correlation engine? They can share plumbing, but evaluate and govern them separately: an attacker adapts to your detectors, and missed detections cost differently. See AIOps for Security.

Is Kubernetes self-healing AIOps? No. Restarting failed containers and rescheduling pods is the orchestrator’s built-in control loop (KubernetesOps). AIOps sits above it and correlates what such loops cannot fix.

Distinguish three kinds of claim. Established: deduplication, grouping, inhibition, topology and change correlation, statistical baselines. Active research with limited production evidence: learned grouping, LLM summaries, LLM-assisted cause classification. Marketing and speculation: “autonomous operations”, single-number noise-reduction promises, root cause without evidence.

• Prerequisites: alerting basics and SLOs (SRE), telemetry (ObservabilityOps), incident roles (IncidentOps), and a little Python and statistics (median, percentiles, precision and recall).
• One-day primer: the first three parts of this page; Google’s chapter on monitoring distributed systems;^[10] Labs 2 and 3.
• One week: run Lab 1; write a label contract and lint your own rules; compute alerts per incident and the actionable-page rate for a month of alerts.
• Thirty days: label 20 incidents; score your current grouping with pairwise precision and recall; build a tier-1 topology file; attach changes to incidents.
• Advanced: study the surveys on AIOps failure management and LLM-era methods;^[5][7][88] trial one learned technique in shadow mode; design a guarded remediation class end to end.
• Portfolio projects: (1) a storm-replay test harness for your alert pipeline; (2) an evaluation scorecard generated from incident labels; (3) a guarded remediation for one fault, with verification and rollback, reviewed by your change authority.

Situation (illustrative). A company runs three monitoring tools; on-call engineers receive about 1,500 pages a month, one in five actionable. Peak season starts in eight weeks. A vice-president proposes buying an event-intelligence platform now; a vendor demo claims a 90% reduction in noise.

Option A: buy now. Fast to approve, vendor help with integration, and cross-tool correlation is the platform’s strength. Option B: an eight-week hygiene sprint: a label contract, deduplication and grouping, retiring the noisiest rules, SLO-based paging for the top services and labels at incident close, then a shadow trial of one or two platforms after the peak.

Assumptions. Most of the noise comes from a few rules (it usually does, but check the export). The team can spare two engineers for eight weeks. No labelled incidents exist today.

Reasoning. Option A promises results without changing anyone’s habits, and can be wrong here for three reasons: without labels nobody can tell whether its grouping is right, so a wrong merge at peak could hide an incident; integration takes weeks, and the peak arrives regardless; and a demo on the vendor’s data says little about these three tools. Option B removes noise at source, creates the labels any later evaluation needs, and keeps risk low during the peak.

Decision. Choose B, and start collecting labels on day 1 so that the post-peak shadow trial has evidence. When A wins instead: if the noise comes from many tools whose alerts cannot be fixed at source (for example, appliances or third-party services), and the team has no capacity to run a composable stack, buy, but in shadow mode first, scored with the worksheet.

Uncertainty. The split between “fixable at source” and “only correlatable” noise is the crux. Measure it before deciding; a one-week analysis of the alert export settles most of the argument.

After this page you should be able to: explain what AIOps covers and where its neighbours begin; select between rules, statistics, learned models and LLMs for a given noise problem; design a label contract, a topology file and a guarded remediation gate; operate an evaluation with pairwise precision and recall; and evaluate a vendor’s claims with your own labelled data.

AIOps is best understood as a measurement discipline sold as a product category. Its techniques, from deduplication to language models, are real and useful, and each can also hide an outage. The name does not tell you which outcome you will get, and that is why this page keeps returning to labelled incidents.

The durable assets are not the engine. They are the label contract, the topology, the change log, the labelled incidents and the habit of scoring every policy before and after it ships. Organisations that build those can swap engines as vendors merge, rename or retire products, and can tell whether a new technique helps. Organisations that skip them buy a cleaner-looking alert stream and learn its precision from their customers.

Claim-by-claim notes, support strength and the sources that could not be fetched are in the evidence ledger (research/aiops-ledger.md). Analyst reports are paywalled; only text quoted by vendors is used, attributed as such.

Re-checking fast-moving facts (all as of October 2026). Monthly: the ITBench-AA leaderboard. Quarterly: pinned versions (Alertmanager, Prometheus, pint, ansible-rulebook, the ansible.eda collection, the node_exporter mixin, Keep, OpenTelemetry), vendor packaging and limits (PagerDuty’s AIOps add-on, BigPanda’s 300-alert cap), retirement or archive notices (DevOps Guru, Merlion) and NIST’s AI RMF revision. Whenever the AI Act is amended: its dates, on EUR-Lex or the AI Act Explorer. The Gartner glossary refuses automated access, so re-check it through a new Internet Archive snapshot. Update the footnote date and ledger row at each re-check.

Term, analysts and history

Practice and reliability engineering

Alerting, labels and configuration

Products and projects (time-sensitive)

Methods and statistics

Language models and agents

Governance

Further reading

• Google, Site Reliability Engineering and The Site Reliability Workbook: the alerting and automation chapters are the best antidote to over-engineered event pipelines.

Changelog

• 2026-10-16: first version (research, ledger, labs executed, diagrams validated) by the managing author; awaiting independent fact-check review.
• 2026-10-17: round-1 revision (reviews/aiops-r1.md, response in reviews/aiops-r1-response.md): affected cluster separated from evaluating server; guarded inhibition and corrected runbook fix; contract linting and measurement; remediation wired end to end; ITBench (ICML 2025) and a dated 2026 leaderboard; predictive alerting; Alertmanager API security; EU AI Act Article 4 and Digital Omnibus; labs re-run; one redundant figure removed.
• 2026-10-17: round-2 revision (reviews/aiops-score.md, response in reviews/aiops-r2-response.md): how log templates, learned grouping and graph root-cause analysis work, with sceptical evidence; re-check guidance; runner after routing in the architecture figure; both mixin remedies; observer-side partition risk; code wrapped; repetition cut; labs re-run.
• 2026-10-17, post-score polish (reviews/aiops-score-r2.md; actions in reviews/aiops-polish.md): change events recorded and attached (Artefact 4, executed); Landauer finding made precise; Forrester figures attributed; NIST AI RMF revision noted; moving-branch sources pinned; re-check list extended; repetition cut.