--- title: "AutomationOps" slug: automationops category: delivery term_status: ambiguous evidence_status: verified summary: "AutomationOps is not a standard term. This Atlas uses it for running operational automation as a governed discipline: choosing what to automate, making it idempotent and safe to repeat, limiting how far one run can reach, and owning it until it is retired." audience: [Operations engineers and SREs, Platform teams, Automation programme leads, Engineering managers, Security and audit, Beginners] last_verified: 2026-10-17 key_takeaways: - "The label has no neutral definition. RPA vendors use it (Robocorp's AutomationOps methodology, UiPath's Automation Ops product); this page uses it for the governance and safety of IT operations automation, an editorial choice." - "Automation multiplies whatever it is given. Google's SRE book calls it \"a force multiplier, not a panacea\", and several well-documented outages were tools or scripts doing exactly what they were told, at scale." - "Five controls carry most of the safety: a dry run, strict target validation (an empty or wrong-type list stops the run), a blast-radius cap with batches, a failure budget, and a kill switch that is drilled." - "Choose candidates by net hours (saved minus residual and maintenance) and by risk tier. Consistency and speed can justify automation when the time saved alone does not." - "Every automation needs an owner, a least-privilege identity, an audit trail and a review date. Orphaned scripts with admin rights are the main governance debt." - "Keep people able to take over. Automating the easy parts leaves operators the hard ones with less practice, so rehearse manual fallbacks." related: [sre, infraops, workflowops, chatops, networkops, platformops, incidentops, noops] --- ## What AutomationOps is ### In 30 seconds **AutomationOps** is not a standard term. In this Atlas it names one discipline: **running operational automation as a production system in its own right**: deciding what to automate, making repeats harmless, limiting how far one run can reach, controlling who or what may trigger it, and keeping an owner until it is retired. Scripts, runbook jobs and auto-remediation rules hold administrator rights and act faster than anyone can intervene; this page is about governing them. Google's SRE book gives the reason: "For SRE, automation is a force multiplier, not a panacea".[^sreauto] In what this page calls the **Diskerase incident**, a decommissioning workflow restarted after a failure correctly found nothing left to erase, but "the empty set was used as a special value, interpreted to mean "everything."" Almost all of Google's colocation machines were wiped. Spare capacity limited users to "a slight increase in latency"; repair took the better part of two days, then weeks of new sanity checks.[^sreauto] ::: note Three meanings in circulation 1. **RPA delivery.** Robocorp promoted an "AutomationOps methodology" that "brings together the business users and the technical team to plan, build, maintain, and continuously improve automations";[^robocorp2023] UiPath has sold **Automation Ops** since April 2021,[^uipathga] a product for governance policies, source control and package feeds for its RPA tools.[^uipath] The Atlas has no RPA page. 2. **Governed IT operations automation** (this page): an **editorial choice**, since no standards body or major practitioner text defines the label this way. The practice itself is long established as IT automation[^redhat] or runbook automation.[^rundeck] 3. **Near-collisions.** Elastic's **AutoOps** monitors Elasticsearch clusters;[^elasticautoops] [AIOps](../aiops/) applies analytics to operational data and may trigger automation. ::: **For** engineers who write operational automation, teams that host it, managers who approve it and auditors who must trust it. Code and labs: [AutomationOps in practice](#automationops-in-practice). ### At a glance | Item | Summary | |---|---| | Term status | **Ambiguous.** Competing vendor meanings (RPA); no neutral IT-operations definition found (2026-10-16). The practice is long established. | | Goal | Automation that removes toil without causing outages: safe to run, to repeat and to stop. | | In scope | Choosing candidates; idempotency; dry runs; target validation; blast-radius limits; batches and failure budgets; approvals; kill switches; runbook platforms; Ansible as an executor; testing; identities, audit, ownership; metrics. | | Out of scope | IaC lifecycle → [InfraOps](../infraops/); durable workflow engines → [WorkflowOps](../workflowops/); chat as the interface → [ChatOps](../chatops/); network automation → [NetworkOps](../networkops/); measuring toil → [SRE](../sre/). | | Prerequisites | Linux and scripting basics ([SysOps](../sysops/)); the idea of toil ([SRE](../sre/)). | | Key outputs | A register with owners and risk tiers; guarded runners; readiness reviews; run metrics; kill-switch drill records. | ### The problem it solves Organisations fail with operational automation in two opposite ways. **Too little, done by hand.** Manual procedures drift. In 2012 a Knight Capital technician "did not copy the new code to one of the eight SMARS computer servers"; no second person reviewed the deployment, and a repurposed flag woke dormant code. Knight built up unwanted positions in about 45 minutes and ultimately lost more than $460 million.[^knight] For well-scoped procedures, consistency is "in many ways the primary value of automation".[^sreauto] **Too much power, too few brakes.** The SRE book notes that "a platform also centralizes mistakes", meaning a fix made once applies everywhere.[^sreauto] The other side, shown by public post-incident reports, is that one bad input also applies everywhere: | Year | What ran | What went wrong | Added afterwards | |---|---|---|---| | 2017 | AWS S3 capacity-removal tool, from a playbook | "one of the inputs to the command was entered incorrectly" | Slower removal; a floor at "minimum required capacity"[^s3-2017] | | 2021 | Facebook backbone maintenance command | The audit tool meant to stop it had a bug | Looking for ways to simulate backbone loss in its "storm" drills[^fb2021] | | 2022 | Atlassian deletion script | Site IDs instead of app IDs; 883 sites deleted in 23 minutes | Universal soft deletes, staged rollout[^atlassian2022] | | 2025 | AWS DynamoDB DNS automation | A race left an empty DNS record "that the automation failed to repair" | Automation disabled worldwide; an NLB velocity control announced[^aws2025] | | Operating reality | Without AutomationOps | With it | |---|---|---| | A typo in the target list | Runs against whatever matched | Refused before any change | | A bad change | Reaches every host at once | Stops after the first failing batch | | "What did the bot do last night?" | Grep shell histories | Read one audit log | ### What AutomationOps is not - **Not "automate everything".** Some work should be removed or stay manual ([when not to use](#when-to-use-and-when-not-to-use)); whether operations can disappear is for [NoOps](../noops/). - **Not infrastructure as code** ([InfraOps](../infraops/)), **not a workflow engine** ([WorkflowOps](../workflowops/)) and **not ChatOps**, which is one way to trigger automation ([ChatOps](../chatops/)). This page covers run-time actions: restarts, clean-ups, failovers, rotations, fleet-wide configuration runs. - **Not a way to remove human error.** Bainbridge (1983) observed that the designer who tries to eliminate the operator "still leaves the operator to do the tasks which the designer cannot think how to automate".[^bainbridge] Errors move to design time and are copied to every run. ## Where it comes from and where it sits ### Origins and evolution The label is recent and owned by no community; the practice is old. | Year | Milestone | What it shows | |---|---|---| | 1983 | Bainbridge, "Ironies of Automation"[^bainbridge] | Automation leaves people the hardest tasks and erodes skills | | 2013 | xkcd 1205, "Is It Worth the Time?"[^xkcd] | The time-saved calculation, later cited by the SRE book | | 2016, 2018 | Google's SRE book and Workbook[^sreauto][^wbtoil] | A hierarchy from manual to autonomous; defensive automation | | 2017–2025 | S3, Facebook, Atlassian and AWS DNS reports[^s3-2017][^fb2021][^atlassian2022][^aws2025] | Guards, floors and kill switches added after the fact | | 2021 onwards | UiPath Automation Ops (generally available April 2021);[^uipathga] Robocorp's "AutomationOps" methodology (archived 2023)[^robocorp2023] | The name appears, in another field (RPA) | | 2026 | AI operations agents with review and autonomous modes[^devopsagent][^srerunmodes] | Approval design becomes a product setting | ### Glossary and mental model | Term | Meaning on this page | |---|---| | **Toil** | Production work that is "manual, repetitive, automatable, tactical, devoid of enduring value" and grows with the service.[^sretoil] | | **Runbook automation** | A documented procedure turned into a job others can run ("self-service access", in Rundeck's words).[^rundeck] | | **Idempotent** | "the result of performing it once is exactly the same as the result of performing it repeatedly without any intervening actions".[^ansgloss] | | **Dry run / check mode** | Reports what would change without changing it; "just a simulation".[^anscheck] | | **Target set, blast radius** | What one run will touch; how much it can damage before something stops it. | | **Batch, failure budget** | A few targets at a time; halt after a set number of failures (Ansible `serial` and `max_fail_percentage`; AWS "rate controls").[^ansstrat][^ssmrate] | | **Guard** | A check that refuses or halts a run: validation, caps, floors, health checks, time limits. | | **Velocity limit** | A cap on how many actions automation may take, or how much capacity it may remove, per period.[^aws2025] | | **Supervised / autonomous** | Needs a person's approval, versus acts and reports. | | **Kill switch** | Stops automation at once; feature-flag practice uses the term for switches that "gracefully degrade" functionality.[^fowler] | | **Human-backed interface** | A structured request that people still fulfil, a step towards automation.[^wbtoil] | **Mental model: an automation is a change approved in advance, to be made many times, by someone who is not you, at a time you did not choose.** Judge it like that change. Risk is roughly **reach × speed × irreversibility**; every control here reduces one of them. ### How AutomationOps relates to other Ops ::: figure Where AutomationOps sits among its neighbours ```mermaid flowchart LR S1["SRE: measured toil"] --> AO S2["IncidentOps: recurring fixes"] --> AO S3["ITOps: routine requests"] --> AO RPA["RPA bots: no Atlas page"] -.->|"same controls"| AO AO["AutomationOps: choose, guard, own, retire"] --> N1["InfraOps: IaC lifecycle"] AO --> N2["WorkflowOps: durable engines"] AO --> N3["ChatOps: chat trigger"] AO --> N4["NetworkOps: network change"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class AO core class S1,S2,S3,N1,N2,N3,N4 neutral class RPA risk ``` Text equivalent: SRE (measured toil), IncidentOps (recurring fixes) and ITOps (routine requests) feed work into AutomationOps, which chooses, guards, owns and retires automation and passes its controls to the neighbours that own the mechanics: InfraOps, WorkflowOps, ChatOps and NetworkOps. RPA bots (no Atlas page) need the same controls. ::: | Neighbour | Shared ground | AutomationOps adds | Go there for | |---|---|---|---| | [SRE](../sre/) | Toil reduction | Safety and governance of the automation | Defining and capping toil | | [InfraOps](../infraops/) | Ansible and other tools | Run-time actions, guards, ownership | Terraform state, drift, modules | | [WorkflowOps](../workflowops/) | Multi-step jobs, retries | When a simple runner is enough | Durable execution, sagas | | [ChatOps](../chatops/) | Commands that change production | What the command may do | Bot design, chat permissions | | [IncidentOps](../incidentops/) | Auto-remediation, runbooks | Velocity limits, kill switches | Incident roles, MTTA/MTTR | | [PlatformOps](../platformops/) | Self-service | Tiers and controls for self-service actions | Building the platform product | | [NetworkOps](../networkops/) | Change automation | General automation governance | YANG, source of truth, pre-change verification | | [AIOps](../aiops/) | Event-driven triggers | Guarding the action | Correlation and anomaly detection | | [NoOps](../noops/) | Removing manual work | Where the displaced work goes | Whether operations can disappear | ## Principles and operating model ### A proposed charter and principles No AutomationOps manifesto exists. This charter is **this Atlas's proposal**, drawn from the SRE books, AWS's principles and the incidents above. **Mission.** Remove toil and speed up routine operations without letting any single run do more damage than a careful person would. 1. **Safe to repeat.** Actions are idempotent, or carry a request identifier so retries have no extra effect.[^idem] 2. **Show before doing.** A dry run precedes production runs, and someone or something reads it. 3. **Distrust inputs**, "even if that input is flowing from upstream systems".[^wbtoil] Empty never means all. 4. **Small steps.** Batches, rate limits and floors; AWS lists "rate control, error thresholds, and approvals" as guardrails.[^waoe] 5. **Stop fast.** A failure budget halts a run; a tested kill switch halts all runs. 6. **Least privilege, full record.** One identity per automation; every run logged. 7. **Owned or off.** Without a current owner, an automation is frozen. 8. **People stay able.** Automation should "default to human operators if it runs into an unsafe condition",[^wbtoil] and those people practise the manual path. **Non-goals:** automating judgement; maximising the number of automations; removing approvals that add real information. ### Building blocks | Block | Job | Without it | |---|---|---| | Register | Owner, tier, identity, triggers, review date | Nobody knows what runs or whom to call | | Executor | Runs the job (script runner, Ansible, Rundeck, cloud runbooks) | Shell histories as the record | | Guards | Validation, caps, floors, health checks, time limits | One typo reaches everything | | Approval gate | A second person where the tier requires it | Irreversible actions with no second look | | Batches, failure budget | Limit reach per step; halt on errors | A bad change reaches the whole fleet | | Kill switch | Stop one automation, or all | Operators fight the automation mid-incident | | Identity, audit, tests | Own least-privilege credentials; evidence; early failure | A shared admin key; no record; failure when it matters | ### Operating model and governance Governance scales with risk. These tiers are **this Atlas's proposal**. | Tier | Typical actions | Required controls | Who may trigger | |---|---|---|---| | 0 Read-only | Diagnostics, reports, health snapshots | Rate limits (reads can overload too[^wbtoil]); audit | Anyone authorised | | 1 Reversible, single target | Restart a service, clear a cache, renew one certificate | Idempotence, input validation, time limits, audit, kill switch | Self-service or events | | 2 Reversible, many targets | Fleet configuration, rolling restarts, patch runs | Tier 1 + dry run, target cap, one run at a time, batches, failure budget, health check between batches | On-call or schedule; events only with velocity limits | | 3 Irreversible or data-destroying | Deletions, decommissions, failovers that lose data, credential revocation | Tier 2 + approval by a second person, soft delete or backup, staged rollout, change record | Named requester plus approver | Tier-3 runs and changes to automation code map to NIST's dual-authorization controls ([compliance](#security-safety-and-compliance)). [ChatOps](../chatops/) numbers its command tiers differently (2 is any reversible write, 3 any high-impact command); when a chat command starts an automation, apply the stricter tier (editorial rule). **Cadence (proposal):** metrics weekly; register review quarterly; kill-switch drills twice a year for tiers 2–3. ### Roles, ownership and RACI - **Automation owner** maintains one automation and answers for it; the **service owner** owns what it touches and agrees its tier. - **Platform team** (sometimes an automation centre of excellence) runs executors, register, identities and templates. - **Approver** approves tier-3 runs and is never the requester; **on-call** receives guard trips and may pull the kill switch; **security and audit** review identities, approvals and logs. | Activity | Automation owner | Service owner | Platform team | Approver | On-call | Security | |---|---|---|---|---|---|---| | Propose and score a candidate | R | A | C | — | C | I | | Set risk tier and limits | R | A | C | — | I | C | | Build, test, register | A/R | C | C | — | I | C | | Promote to autonomous | R | A | C | — | C | I | | Approve a tier-3 run | I | C | — | A/R | I | I | | Pull the kill switch | I | I | I | — | A/R | I | | Quarterly register review | R | A | R | — | — | C | | Retire | R | A | R | — | I | I | R = responsible, A = accountable, C = consulted, I = informed. On-call can stop any automation without asking; restarting it needs the owner. ### Lifecycle: from candidate to retirement ::: figure Lifecycle of one automation ```mermaid stateDiagram-v2 direction TB [*] --> Candidate Candidate --> Rejected: no net value Candidate --> Built: scored and owned Built --> Tested: lint, idempotence, guards Tested --> Supervised: dry runs reviewed Supervised --> Autonomous: promoted Autonomous --> Supervised: demoted Autonomous --> Frozen: owner gone Supervised --> Frozen: owner gone Frozen --> Supervised: new owner Supervised --> Retired: cause fixed at source Autonomous --> Retired: cause fixed at source Frozen --> Retired: nobody needs it Rejected --> [*] Retired --> [*] ``` Text equivalent: a candidate is rejected (no net value) or built once scored and owned; it is tested, runs supervised while dry runs are reviewed, and is promoted to autonomous with a clean record and a low tier; incidents or repeated guard trips demote it. If its owner leaves it is frozen until re-owned or retired; a supervised or autonomous automation is also retired when the cause is fixed at the source. ::: **Promotion** needs evidence: clean supervised runs, guards that tripped correctly in tests, and a tier that allows it. **Demotion** after an incident should be automatic. The best exit is where Google's hierarchy ends: a system that "notices problems, and automatically fails over without human intervention".[^sreauto] ## Architecture and workflow ### Reference architecture This logical architecture fits a team-owned runner, a runbook platform such as Rundeck or a cloud service such as AWS Systems Manager Automation. **Guards and the kill switch sit between trigger and executor**, and only the executor holds credentials that can change targets. ::: figure Reference architecture of a governed automation path ```mermaid flowchart TB T["Trigger: schedule, alert or person"] --> G["Guards: validate, cap, dry run"] R["Register: owner, tier, limits"] --> G G -->|"refused"| H["On-call person"] G --> A{"Tier 3?"} A -->|"yes"| P["Second person approves"] P --> X["Executor with its own identity"] A -->|"no"| X K["Kill switch"] -.->|"checked per batch"| X X --> B1["Batch 1: canary"] B1 -->|"healthy"| B2["Batches 2 to n"] B1 -->|"budget exceeded"| H X --> L["Audit log and metrics"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class R,G,X,L core class A decision class K,H risk class T,P,B1,B2 neutral ``` Text equivalent: a trigger (schedule, alert or person) enters the guards, which read the register and validate, cap and dry-run the request; refusals go to on-call. Tier 3 needs a second person's approval before the executor, which has its own identity, runs it. The kill switch is checked before each batch. A healthy canary batch leads to batches 2 to n; an exceeded failure budget goes to on-call. Every run writes to the audit log. ::: **Trust boundaries.** Triggers are untrusted input, including your own monitoring. The executor's identity is the real blast radius: in AWS Systems Manager a run uses either a service role or "the context of the user who invoked the automation".[^ssmsetup] The executor may append to the audit log but not rewrite it: ship records to a store its identity cannot change (the lab runner's local file is not tamper-proof). **Alternatives.** Google's hierarchy runs from "an SRE has a failover script in his or her home directory" to systems "that don't need any automation".[^sreauto] A central runner is the middle step; build very frequent actions into the system itself. ### End-to-end workflow ::: figure One run, with the happy path and the exception paths ```mermaid sequenceDiagram participant T as Trigger and on-call participant R as Runner participant S as Targets T->>R: Target list Note over R: Guards: validate, cap, dry run Note over R: Tier 3: second person decides alt Refused by a guard R-->>T: Refused, on-call paged else Rejected by the approver R-->>T: Stopped, nothing changed else Accepted loop Each batch, if switch off and budget left R->>S: Apply batch S-->>R: Results, health end alt Budget, health or kill switch tripped R-->>T: Halted, on-call paged else All batches healthy R-->>T: Done, audit written end end ``` Text equivalent: the trigger sends a target list to the runner (guards plus executor), which validates, caps and dry-runs it; for tier 3 a second person approves or rejects. A guard refusal pages on-call; a rejection stops the run with nothing changed. Otherwise the runner loops over the batches, checking the kill switch and failure budget before each, and reads results and health. A tripped budget, health check or kill switch halts the run and pages on-call; otherwise it finishes and writes the audit record. ::: The exception path matters most: **every refusal and halt reaches a person** with the run id, targets touched and reason. ## Practices, frameworks and tools ### Core practices and playbooks | Practice | Owner | Verification | |---|---|---| | Score candidates | Owner, service owner | Net hours and tier recorded | | Make it idempotent | Owner | Second run changes nothing | | Dry run first | Owner | Plan reviewed or checked | | Validate targets | Owner | Bad-input tests pass | | Batch with a failure budget | Owner | Halt seen in a test | | Gate by tier | Platform team | Approvals in the log | | Kill switch | Platform team | Drill record | | Test automation code | Owner | Lint, unit, idempotence in CI | | Own and retire | Platform team | Quarterly review | **Choosing candidates.** Start from the toil log ([SRE](../sre/)), not from what is most annoying, and estimate net hours and risk tier ([Lab 1](#hands-on-labs)). xkcd's chart shows how long you can spend speeding up a routine task before losing time over five years: 5 minutes off a daily task justifies about 6 days, counted around the clock (some 150 hours).[^xkcd] It ignores maintenance and risk; the SRE book adds that automation "provides more than just time saving".[^sreauto] For messy requests, start with a human-backed interface.[^wbtoil] **Idempotency.** Write "ensure state", not "do action": *ensure the service is running*, not *start it*. Where an action cannot be idempotent (create, charge, send), use a caller-supplied request identifier recorded atomically with the change.[^idem] Google's idempotent fixes could run "every 15 minutes without fearing damage", but "not all fixes were naturally idempotent".[^sreauto] **Dry runs, and their limits.** In Ansible's check mode, modules without support "report nothing and do nothing", and Ansible "will not generate output for tasks that use conditionals based on registered variables (results of prior tasks)".[^anscheck] The trap is a read-only probe written as `command` or `shell`: check mode skips it unless it uses `creates` or `removes`, so a later `when:` has nothing real to test.[^anscommand] Mark such probes `check_mode: false`; modules such as `find` already run in check mode.[^ansfind] A clean dry run is evidence, not proof: list targets with before/after values (diff mode) and check the count. **Validating targets.** Diskerase, S3 and Atlassian were input failures: an empty set read as "everything", a mistyped input, IDs of the wrong type.[^sreauto][^s3-2017][^atlassian2022] Refuse empty lists; resolve every identifier and check its type; cap the count; keep a floor; show people names, not IDs. Review the data as well as the code: Atlassian's review "focused on which endpoint was being called and how".[^atlassian2022] **Batches and failure budgets.** Ansible aborts when `max_fail_percentage` is "exceeded, not equaled".[^anserr] AWS Systems Manager lets running invocations finish after its error threshold and suggests concurrency 1 if the limit must hold; its user guide and CLI reference disagree on whether 10% of 50 stops at the fifth or sixth error, so test the semantics.[^ssmrate][^ssmcli] **Kill switches, time limits and locks.** Keep one switch per automation and a global one, independent of what the automation manages: Facebook's out-of-band access failed with its backbone.[^fb2021] AWS engineers disabled automatic NLB failover mid-incident and later "disabled the DynamoDB DNS Planner and the DNS Enactor automation worldwide".[^aws2025] Safeguards "might be as simple as command timeouts":[^wbtoil] give every action a time limit, and take a lock so two runs cannot interleave. **Testing automation code.** ShellCheck for shell;[^shellcheck] ansible-lint, whose `no-changed-when` rule flags `command` and `shell` tasks that cannot report change;[^lint] Molecule, whose default sequence includes `idempotence`.[^molecule] Test each guard with the bad input it exists for ([Lab 2](#hands-on-labs), step 11): Facebook's audit tool had the bug that let the command through.[^fb2021] ### Methods, standards and frameworks | Source | Kind | Contribution | |---|---|---| | Google SRE book and Workbook[^sreauto][^wbtoil] | One company's practice | Value and hierarchy of automation; defensive automation | | AWS Well-Architected, operational excellence[^waoe] | Vendor recommendation | "Safely automate where possible" | | NIST SP 800-53 Rev. 5[^nist] | Control catalogue, binding only where adopted | Change control, dual authorization, least privilege | | Bainbridge (1983)[^bainbridge] | Human-factors research | Skills decay; takeover is hard | | DORA metrics[^dora] | Research programme | Change fail rate, adapted below | None defines "AutomationOps". ### Tooling landscape | Category | Representative options (verified, October 2026) | Strengths | Limits and risks | Choose by | |---|---|---|---|---| | Configuration management and ad hoc execution | Ansible (check mode, `serial`, `max_fail_percentage`)[^anscheck][^ansstrat]; Puppet, whose new binaries moved to a private repository under a EULA from early 2025, with the OpenVox community fork[^puppet2025][^openvox] | Idempotent modules, large ecosystems | Shell tasks escape idempotence; licence and packaging changes | Module coverage; licence terms; test tooling | | Runbook automation platforms | Rundeck (Apache 2.0) and its commercial edition Runbook Automation, part of PagerDuty[^rundeck]; StackStorm (event-driven, latest GitHub release 3.9.0, October 2025)[^stackstorm] | Self-service jobs, access control, logs | Another privileged system to run and patch; check project activity | Access model; audit export; maintenance status | | Event-driven automation | Event-Driven Ansible rulebooks with `throttle` and host limits[^eda][^edahosts]; StackStorm rules | Fast response to alerts | Remediation storms without throttles; open webhooks | Throttling, host limiting, source authentication | | Cloud runbook services | AWS Systems Manager Automation (rate controls, `aws:approve`)[^ssmrate][^ssmapprove]; Azure Automation (PowerShell and Python runbooks, Hybrid Runbook Worker)[^azauto] | Native identities, no servers | Provider lock-in; pricing changed (the Automation free tier ended in 2025)[^ssmabout] | Where your targets live | | Linters and tests | ShellCheck[^shellcheck], ansible-lint[^lint], Molecule[^molecule] | Cheap, early feedback | Do not prove production safety | Fit with CI | ### Automation and AI AI operations agents are automation whose plan is generated at run time, so these controls apply with more force. AWS describes its DevOps Agent (generally available 31 March 2026) as one of a class of "autonomous systems that work independently".[^devopsagent] Azure SRE Agent's documentation says "Review is the default mode", yet its table lists Autonomous as the default for response plans and scheduled tasks: check each plan. In Review mode only Azure infrastructure operations get approval buttons; other actions "proceed based on the agent's reasoning and your response plan instructions". Approval and permissions are separate. An agent lacking a permission "requests temporary access through Microsoft Entra OBO flow",[^srerunmodes] so one privileged person's consent widens its reach. Decide who may grant it. Rules: give an agent the narrowest identity of any automation; start read-only; route its actions through the same guards and tiers; judge it by the [metrics](#metrics-and-kpis). Correlation and anomaly detection belong to [AIOps](../aiops/). ## AutomationOps in practice ### Reference implementation **Assumptions.** A team automates a clean-up of temporary files on application servers and a few riskier actions, with the same controls whatever the executor. All names, accounts and hosts are placeholders. **1. Register entry** (one file per automation, reviewed like code). Field names are this Atlas's proposal. _Example status: YAML syntax-checked; a template, nothing to execute._ ```yaml id: cleanup-tmp purpose: Remove temporary files older than 7 days from app servers owner: team-checkout-ops # a team, plus a named person in your directory service_owner: checkout # tier: 0 read-only, 1 reversible single target, 2 reversible many, 3 irreversible tier: 2 triggers: [schedule, on-call] executor: ansible # or rundeck, ssm-automation, python-runner identity: automation-cleanup-tmp # its own identity; never a shared admin key permissions: ["delete files under /var/tmp/app on hosts in group app_servers"] limits: max_targets_per_run: 20 batches: [1, "25%"] max_fail_percentage: 0 kill_switch: /etc/automation/KILL-cleanup-tmp dry_run_required: true approval: none # tier 3 needs a second person tests: [ansible-lint, molecule-idempotence, guard-tests] fallback: page on-call; manual procedure RB-112 review_by: 2027-01-16 # status: candidate, built, tested, supervised, autonomous, frozen, retired status: supervised ``` **2. A guarded runner** with every control, in standard-library Python; its "fleet" is a directory of JSON files, so it is safe anywhere ([Lab 2](#hands-on-labs)). The lock and time limit are Unix-only, and the lock covers one machine. _Example status: verified by execution (Python 3.8.10, Linux, 2026-10-16) with the Lab 2 fleet._ ```python #!/usr/bin/env python3 """Guarded runner: one operational action with the safety controls built in. Dry run by default; one run at a time (lock file); refuses empty, unknown, wrong-kind, duplicate or oversized target lists; works in batches with a failure budget, a time limit per target and a health check after each batch; checks a kill switch before every batch; appends one JSON line per decision to an audit log. Standard library only (Python 3.8+, Linux or macOS). The "fleet" is a directory of JSON files, so nothing real changes. """ import argparse import fcntl import getpass import json import os import signal import sys import time import uuid EXIT_OK, EXIT_REFUSED, EXIT_HALTED, EXIT_KILLED = 0, 2, 3, 4 def load_fleet(fleet_dir): fleet = {} for name in sorted(os.listdir(fleet_dir)): if name.endswith(".json"): with open(os.path.join(fleet_dir, name)) as f: host = json.load(f) fleet[host["id"]] = host return fleet def check_targets(targets, fleet, kind, max_targets): """Return a list of reasons to refuse; an empty list means the targets are fine.""" if not targets: # An empty list must never be read as "everything" (the Diskerase lesson). return ["target list is empty: nothing to do, refusing rather than guessing"] problems = [] unknown = [t for t in targets if t not in fleet] if unknown: problems.append("unknown targets: " + ", ".join(unknown)) wrong = [t for t in targets if t in fleet and fleet[t]["kind"] != kind] if wrong: # Right IDs of the wrong type are the classic deletion accident. problems.append("targets are not of kind '%s': %s" % (kind, ", ".join(wrong))) if len(targets) > max_targets: problems.append("%d targets exceed the blast-radius cap of %d" % (len(targets), max_targets)) if len(set(targets)) != len(targets): problems.append("duplicate targets in the list") return problems def on_timeout(signum, frame): raise TimeoutError("time limit exceeded") def cleanup(host): """Idempotent action: remove temporary files. Returns (changed, before, after).""" time.sleep(host.get("hang", 0)) # a real action can hang; the time limit stops it if host.get("fail"): raise RuntimeError("simulated failure on " + host["id"]) before = host["tmp_files"] host["tmp_files"] = 0 return before != 0, before, 0 def healthy(host): """Health check after a batch; a real one would probe the service.""" return host.get("healthy", True) def audit(path, record): # Append mode only. A local file is not tamper-proof: in production, ship # these lines to a store the runner's identity cannot rewrite. with open(path, "a") as f: f.write(json.dumps(record, sort_keys=True) + "\n") def main(argv=None): p = argparse.ArgumentParser(description=__doc__) p.add_argument("--fleet", required=True, help="directory of .json files") p.add_argument("--targets", required=True, help="file with one target id per line") p.add_argument("--kind", default="app", help="the only kind of target this action may touch") p.add_argument("--apply", action="store_true", help="make changes (default: dry run)") p.add_argument("--max-targets", type=int, default=5, help="blast-radius cap per run") p.add_argument("--batch", type=int, default=2, help="targets per batch") p.add_argument("--max-failures", type=int, default=0, help="failures tolerated before halting") p.add_argument("--timeout", type=int, default=10, help="seconds allowed per target") p.add_argument("--kill-switch", default="KILL", help="if this file exists, stop before the next batch") p.add_argument("--lock", default="runner.lock", help="lock file that allows one run at a time") p.add_argument("--audit", default="audit.jsonl", help="JSON-lines audit log, opened in append mode") a = p.parse_args(argv) run_id = uuid.uuid4().hex[:8] mode = "apply" if a.apply else "dry-run" base = {"run": run_id, "actor": getpass.getuser(), "action": "cleanup", "mode": mode} lock = open(a.lock, "w") # held until the process exits try: fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) except OSError: print("REFUSED: another run holds the lock " + a.lock) audit(a.audit, dict(base, result="refused", reasons=["locked"], ts=time.time())) return EXIT_REFUSED fleet = load_fleet(a.fleet) with open(a.targets) as f: targets = [line.strip() for line in f if line.strip() and not line.startswith("#")] problems = check_targets(targets, fleet, a.kind, a.max_targets) if problems: for msg in problems: print("REFUSED: " + msg) audit(a.audit, dict(base, result="refused", reasons=problems, ts=time.time())) return EXIT_REFUSED signal.signal(signal.SIGALRM, on_timeout) failures = 0 batches = [targets[i:i + a.batch] for i in range(0, len(targets), a.batch)] for n, batch in enumerate(batches, 1): if os.path.exists(a.kill_switch): print("KILLED: kill switch %s present before batch %d; %d batch(es) not run" % (a.kill_switch, n, len(batches) - n + 1)) audit(a.audit, dict(base, result="killed", batch=n, ts=time.time())) return EXIT_KILLED for t in batch: host = fleet[t] rec = dict(base, target=t, batch=n, ts=time.time()) if not a.apply: would = "would change" if host["tmp_files"] else "unchanged" print("%-8s %-7s %s (tmp_files=%d)" % (mode, t, would, host["tmp_files"])) audit(a.audit, dict(rec, result=would)) continue try: signal.alarm(a.timeout) try: changed, before, after = cleanup(host) finally: signal.alarm(0) with open(os.path.join(a.fleet, t + ".json"), "w") as f: json.dump(host, f, indent=2) result = "changed" if changed else "unchanged" print("%-8s %-7s %s (tmp_files %d -> %d)" % (mode, t, result, before, after)) audit(a.audit, dict(rec, result=result, before=before, after=after)) except (RuntimeError, TimeoutError) as e: failures += 1 print("%-8s %-7s FAILED: %s" % (mode, t, e)) audit(a.audit, dict(rec, result="failed", error=str(e))) if failures > a.max_failures: print("HALTED: %d failure(s) exceed the budget of %d after batch %d" % (failures, a.max_failures, n)) audit(a.audit, dict(base, result="halted", batch=n, failures=failures, ts=time.time())) return EXIT_HALTED sick = [t for t in batch if a.apply and not healthy(fleet[t])] if sick: print("HALTED: health check failed after batch %d: %s" % (n, ", ".join(sick))) audit(a.audit, dict(base, result="halted", batch=n, unhealthy=sick, ts=time.time())) return EXIT_HALTED print("DONE: %s on %d target(s), %d failure(s), run %s" % (mode, len(targets), failures, run_id)) return EXIT_OK if __name__ == "__main__": sys.exit(main()) ``` **3. The same controls in Ansible.** With `serial`, the whole play, guards included, runs per batch; `any_errors_fatal` stops all hosts if one guard fails;[^ansstrat][^anserr] `ansible_play_hosts_all` lists "all the hosts that were targeted by the play".[^ansspecial] `timeout` limits each task action;[^anskw] with a budget of 0, the final health check stops the play on any unhealthy host. With several controllers, use a shared kill-switch flag. _Example status: YAML syntax-checked; not executed (no Ansible here). `ansible.builtin` modules only._ ```yaml - name: Clean temporary files on app servers (guarded) hosts: app_servers gather_facts: false serial: [1, "25%"] # one canary host, then quarters of the target set max_fail_percentage: 0 # any failed host in a batch aborts the rest of the play vars: max_targets: 20 kill_switch: /etc/automation/KILL-cleanup-tmp pre_tasks: - name: Guards (a failure here stops every host) any_errors_fatal: true block: - name: Refuse an oversized target set ansible.builtin.assert: that: ansible_play_hosts_all | length <= max_targets fail_msg: >- {{ ansible_play_hosts_all | length }} hosts targeted; the cap is {{ max_targets }} - name: Read the kill switch on the control node ansible.builtin.stat: path: "{{ kill_switch }}" delegate_to: localhost run_once: true register: killswitch - name: Stop before this batch if the kill switch is present ansible.builtin.assert: that: not killswitch.stat.exists fail_msg: "Kill switch {{ kill_switch }} is present" tasks: - name: Find temporary files older than 7 days ansible.builtin.find: paths: /var/tmp/app age: 7d register: old_files - name: Remove them (absent stays absent, so a second run changes nothing) ansible.builtin.file: path: "{{ item.path }}" state: absent loop: "{{ old_files.files }}" loop_control: label: "{{ item.path }}" timeout: 60 - name: Health check before the next batch (placeholder endpoint) ansible.builtin.uri: url: "http://{{ inventory_hostname }}:8080/healthz" delegate_to: localhost register: health retries: 3 delay: 10 until: health.status == 200 ``` **4. AWS Systems Manager.** `AWS-RestartEC2Instance` does **not** reboot: it stops, then starts, each instance.[^ssmrestart][^ssmsteps] A stop/start loses RAM and instance-store data. It also changes an automatically assigned public IPv4 address unless an Elastic IP is attached (as of October 2026).[^ec2stop] For a reboot, call `RebootInstances` from your own runbook. The script refuses an empty or oversized set and passes exactly the running instances it counted. Targeting the tag could also catch stopped instances, which the run would start (our inference; AWS warns only that "terminated instances might be included").[^ssmtargets] The run works one instance at a time and halts at the first error.[^ssmcli] _Example status: bash syntax-checked; run against a stub `aws` (empty, normal, oversized, malformed lists), not against AWS._ ```bash # Stop and start (not reboot) every RUNNING instance tagged app=checkout, one at a time. # The run acts on exactly the IDs that were counted, not on the tag again. set -euo pipefail CAP=10 ROLE=arn:aws:iam::111122223333:role/automation-restart-checkout raw=$(aws ec2 describe-instances \ --filters "Name=tag:app,Values=checkout" "Name=instance-state-name,Values=running" \ --query 'Reservations[].Instances[].InstanceId' --output text) mapfile -t ids < <(tr -s '[:space:]' '\n' <<< "$raw" | sed '/^$/d') if (( ${#ids[@]} == 0 || ${#ids[@]} > CAP )); then echo "REFUSED: ${#ids[@]} running instances selected; allowed 1 to $CAP" >&2 exit 2 fi for id in "${ids[@]}"; do if [[ ! $id =~ ^i-[0-9a-f]{8,17}$ ]]; then echo "REFUSED: not an instance ID: $id" >&2; exit 2 fi done echo "Stop/start, one at a time: ${ids[*]}" aws ssm start-automation-execution \ --document-name AWS-RestartEC2Instance \ --target-parameter-name InstanceId \ --targets "Key=ParameterValues,Values=$(IFS=,; echo "${ids[*]}")" \ --max-concurrency 1 \ --max-errors 0 \ --parameters "AutomationAssumeRole=$ROLE" ``` No IAM action means "restart": the role needs `ec2:StopInstances` and `ec2:StartInstances`, so it could also leave instances stopped. Scope both by resource tag, as in AWS's example policies,[^ec2policies] and test this starting-point policy outside production. Encrypted volumes may need the KMS actions AWS lists for EBS encryption, such as `kms:CreateGrant`;[^ebskms] an instance lacking "the required KMS key permissions" can return to stopped during a start.[^ssmchangestate] Whoever holds `iam:PassRole` for the role can use it.[^iampassrole] _Example status: JSON syntax-checked; not executed._ ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "StopStartCheckoutOnly", "Effect": "Allow", "Action": ["ec2:StopInstances", "ec2:StartInstances"], "Resource": "arn:aws:ec2:*:111122223333:instance/*", "Condition": {"StringEquals": {"aws:ResourceTag/app": "checkout"}} }, { "Sid": "ReadInstanceState", "Effect": "Allow", "Action": "ec2:DescribeInstances", "Resource": "*" } ] } ``` A tier-3 runbook adds a read-only plan, a second person and input patterns. `aws:approve` waits for approvers (default 7 days, maximum 30; up to 10 approvers; no multi-account or multi-Region runs); with `allowedPattern`, bad input means "the execution fails to start".[^ssmapprove][^ssmparams] An approver can be a role, so anyone who can assume `tenant-deletion-approvers` can approve: let its trust policy name approvers only. `tenantctl` is a placeholder. _Example status: YAML syntax-checked; not executed._ ```yaml description: Soft-delete one tenant after second-person approval (Atlas example) schemaVersion: '0.3' assumeRole: '{{ AutomationAssumeRole }}' parameters: AutomationAssumeRole: type: String description: Role used only by this runbook InstanceId: type: String allowedPattern: '^i-(?:[a-f0-9]{8}|[a-f0-9]{17})$' TenantId: type: String # This pattern is what makes the substitution into shell commands below safe. allowedPattern: '^tnt-[a-z0-9]{8}$' mainSteps: - name: plan action: aws:runCommand onFailure: Abort inputs: DocumentName: AWS-RunShellScript InstanceIds: - '{{ InstanceId }}' Parameters: commands: - /opt/tenantctl/bin/tenantctl show --tenant '{{ TenantId }}' # read-only - name: approve action: aws:approve timeoutSeconds: 3600 onFailure: Abort inputs: NotificationArn: arn:aws:sns:eu-west-1:111122223333:Automation-approvals Message: 'Soft-delete tenant {{ TenantId }}? Read the output of step "plan" first.' MinRequiredApprovals: 1 Approvers: - arn:aws:iam::111122223333:role/tenant-deletion-approvers - name: softDelete action: aws:runCommand onFailure: Abort inputs: DocumentName: AWS-RunShellScript InstanceIds: - '{{ InstanceId }}' Parameters: commands: - >- /opt/tenantctl/bin/tenantctl soft-delete --tenant '{{ TenantId }}' --retain-days 30 ``` **5. Event-driven remediation with a velocity limit.** Three settings keep one alert from reaching the fleet. First, a host limit: ansible-rulebook limits a playbook to a host only through `event.meta.hosts`. The webhook does not set it, so a filter copies the alerting host there.[^edahosts] Second, a throttle: it acts on the first event and suppresses repeats per host until the window expires.[^eda] Third, a fleet cap: per-host throttles do not cap the total, so the playbook refuses more than one host and counts restarts fleet-wide. Actions run one at a time by default, so runs do not race on the count.[^edarulebooks] The webhook binds to 0.0.0.0 by default, and its token and HMAC checks are optional:[^edasources] without them, anyone who can reach the port can fake an alert and restart workers. Bind to loopback behind an authenticating proxy and set a token. The builtin sources are documented "for testing and development"; check that your production source has the same settings. _Example status: YAML syntax-checked; not executed (needs ansible-rulebook and a `workers` inventory group)._ ```yaml # worker-rulebook.yml: run with ansible-rulebook -r worker-rulebook.yml -i inventory.yml - name: Restart a stuck worker on the alerting host only hosts: workers sources: - eda.builtin.webhook: host: 127.0.0.1 # loopback; an authenticating proxy sits in front port: 5000 token: "{{ webhook_token }}" # Bearer token from a secret store filters: - eda.builtin.insert_hosts_to_meta: host_path: payload.host # sets event.meta.hosts, which limits the playbook rules: - name: Worker stuck, at most once per host per 15 minutes condition: >- event.payload.alert == "worker_stuck" and event.payload.status == "firing" and event.payload.host is match("worker-[0-9]{2}$") throttle: once_within: 15 minutes group_by_attributes: - event.meta.hosts action: run_playbook: name: playbooks/restart-worker.yml ``` ```yaml # playbooks/restart-worker.yml: started by the rulebook, limited to one host - name: Restart one stuck worker (guarded) hosts: workers gather_facts: false vars: fleet_cap_per_hour: 3 marker_dir: /var/lib/automation/restart-worker # on the rulebook host stamp: "{{ now(utc=true, fmt='%Y%m%dT%H%M%S') }}" pre_tasks: - name: Guards (a failure here stops the run) any_errors_fatal: true block: - name: Exactly one host, never the whole group ansible.builtin.assert: that: ansible_play_hosts_all | length == 1 fail_msg: "{{ ansible_play_hosts_all | length }} hosts targeted; expected 1" - name: Make sure the marker directory exists ansible.builtin.file: path: "{{ marker_dir }}" state: directory mode: "0750" delegate_to: localhost - name: Count restarts anywhere in the fleet during the last hour ansible.builtin.find: paths: "{{ marker_dir }}" age: -1h delegate_to: localhost register: recent - name: Fleet-wide velocity limit; beyond it a person decides ansible.builtin.assert: that: recent.matched < fleet_cap_per_hour fail_msg: "{{ recent.matched }} restarts in the last hour: cap reached" tasks: - name: Restart the worker service ansible.builtin.service: name: worker state: restarted become: true timeout: 120 - name: Record the restart for the fleet-wide count ansible.builtin.file: path: "{{ marker_dir }}/{{ stamp }}-{{ inventory_hostname }}" state: touch mode: "0640" delegate_to: localhost ``` | Control | Guarded runner | Ansible | SSM Automation | Event-Driven Ansible | |---|---|---|---|---| | Dry run | Default | `--check --diff`[^anscheck] | None generic; an `Interactive` execution mode exists[^ssmcli] | Sample events | | Target validation | Existence, kind, empty, duplicates | `assert` | `allowedPattern`; count, then pass IDs | Condition; host limit; one-host `assert` | | Batches, failure budget | `--batch`, `--max-failures`, `--timeout`, lock | `serial`, `max_fail_percentage`, `timeout` | `--max-concurrency`, `--max-errors` | `throttle`[^eda]; fleet cap in the playbook | | Approval | CI approval | Controller approval | `aws:approve` | None built in | | Kill switch | `KILL` file per batch | Flag per batch | Remove triggers; deny the role | Disable the activation or stop `ansible-rulebook`; then `enabled: false` on the rule[^aapactivation][^edarules] | | Audit | JSON lines | Callback or job logs | Execution history | Rulebook output to your log store | ### Hands-on labs #### Lab 1 (beginner): score your automation candidates **Objective:** rank candidates by net hours and risk tier. **Prerequisites:** Python 3.8+; no accounts, cost or risk; 30 minutes. _Example status: verified by execution (Python 3.8.10, 2026-10-16; re-run 2026-10-17). The candidate figures are illustrative._ ```python #!/usr/bin/env python3 """Score automation candidates: net hours returned, break-even and a risk tier. Input: a JSON list of candidates (see candidates.json). Standard library only. The tiers and thresholds are this Atlas's proposal, not a standard. """ import json import sys def tier(c): if c["read_only"]: return 0 if not c["reversible"] or c["deletes_data"]: return 3 return 2 if c["max_targets"] > 1 else 1 def score(c): manual = c["runs_per_month"] * c["manual_minutes"] / 60 residual = c["runs_per_month"] * c["residual_minutes"] / 60 net = manual - residual - c["maintenance_hours_per_month"] t = tier(c) if net <= 0: advice = "only for consistency or safety" elif c["needs_judgement"]: advice = "human-backed interface first" elif t == 3: advice = "automate with approval and soft delete" else: advice = "automate; supervised, then promote" months = c["build_hours"] / net if net > 0 else None return manual, net, months, t, advice def main(path): with open(path) as f: candidates = json.load(f) print("%-20s %8s %6s %9s %4s %s" % ("candidate", "manual_h", "net_h", "breakeven", "tier", "advice")) for c in sorted(candidates, key=lambda c: -score(c)[1]): manual, net, months, t, advice = score(c) be = "%.1f mo" % months if months else "never" print("%-20s %8.1f %6.1f %9s %4d %s" % (c["name"], manual, net, be, t, advice)) if __name__ == "__main__": main(sys.argv[1] if len(sys.argv) > 1 else "candidates.json") ``` ```json [ {"name": "restart-stuck-worker", "runs_per_month": 40, "manual_minutes": 15, "residual_minutes": 1, "maintenance_hours_per_month": 1.5, "build_hours": 40, "read_only": false, "reversible": true, "deletes_data": false, "max_targets": 1, "needs_judgement": false}, {"name": "tls-cert-renewal", "runs_per_month": 6, "manual_minutes": 45, "residual_minutes": 5, "maintenance_hours_per_month": 1, "build_hours": 30, "read_only": false, "reversible": true, "deletes_data": false, "max_targets": 1, "needs_judgement": false}, {"name": "fleet-config-change", "runs_per_month": 2, "manual_minutes": 180, "residual_minutes": 20, "maintenance_hours_per_month": 3, "build_hours": 60, "read_only": false, "reversible": true, "deletes_data": false, "max_targets": 200, "needs_judgement": false}, {"name": "decommission-tenant", "runs_per_month": 4, "manual_minutes": 120, "residual_minutes": 30, "maintenance_hours_per_month": 2, "build_hours": 80, "read_only": false, "reversible": false, "deletes_data": true, "max_targets": 1, "needs_judgement": false}, {"name": "capacity-review", "runs_per_month": 0.33, "manual_minutes": 240, "residual_minutes": 120, "maintenance_hours_per_month": 1, "build_hours": 40, "read_only": true, "reversible": true, "deletes_data": false, "max_targets": 1, "needs_judgement": true} ] ``` 1. Save the script as `score_candidates.py` and the list as `candidates.json`; run `python3 score_candidates.py candidates.json`. 2. Replace the list with candidates from your toil log, including the minutes a person still spends per automated run. **Expected output** (step 1, copied from the verified run): ```text candidate manual_h net_h breakeven tier advice restart-stuck-worker 10.0 7.8 5.1 mo 1 automate; supervised, then promote decommission-tenant 8.0 4.0 20.0 mo 3 automate with approval and soft delete tls-cert-renewal 4.5 3.0 10.0 mo 1 automate; supervised, then promote fleet-config-change 6.0 2.3 25.7 mo 2 automate; supervised, then promote capacity-review 1.3 -0.3 never 0 only for consistency or safety ``` **Validation:** the fleet change pays back slowly (25.7 months) yet is worth automating for consistency; the capacity review never pays back. **Troubleshooting:** `KeyError` means a missing field; `JSONDecodeError` is often a trailing comma. **Teardown:** delete the files, or keep the list as your first register. #### Lab 2 (intermediate): run and break a guarded runner **Objective:** see each control work, then reproduce three real failure patterns. **Prerequisites:** Python 3.8+ and bash on Linux (step 10 uses `flock` from util-linux); the [reference](#reference-implementation) runner saved as `guarded_runner.py`. No cost; only the lab directory changes. _Example status: verified by execution (Python 3.8.10, bash, 2026-10-16; re-run 2026-10-17). Run ids and timings will differ._ ```bash #!/usr/bin/env bash # Create a fake fleet of eight "hosts" as JSON files. Nothing outside ./fleet is touched. set -euo pipefail mkdir -p fleet host() { # id kind tmp_files fail [hang_seconds] [healthy] { printf '{"id": "%s", "kind": "%s", "tmp_files": %d, ' "$1" "$2" "$3" printf '"fail": %s, "hang": %d, "healthy": %s}\n' "$4" "${5:-0}" "${6:-true}" } > "fleet/$1.json" } host app-01 app 12 false host app-02 app 0 false host app-03 app 7 false host app-04 app 30 false host app-05 app 9 true host app-06 app 3 false 30 # hangs for 30 seconds host app-07 app 4 false 0 false # unhealthy after a change host db-01 db 5 false printf 'app-01\napp-02\napp-03\napp-04\n' > targets.txt ls fleet ``` 1. Save the script as `make_fleet.sh`; run `bash make_fleet.sh`. 2. Dry run: `python3 guarded_runner.py --fleet fleet --targets targets.txt`. 3. Add `--apply`; then run it again. 4. **Empty list** (Diskerase): `: > empty.txt`; run with `--targets empty.txt --apply`. 5. **Wrong type** (Atlassian): `printf 'app-03\ndb-01\n' > wrong.txt`; run with `--targets wrong.txt --apply`. 6. **Failure budget:** `rm -rf fleet && bash make_fleet.sh`; `printf 'app-05\napp-01\napp-03\napp-04\n' > fail.txt`; use `--targets fail.txt --apply`. 7. **Kill switch:** rebuild the fleet, `touch KILL`, repeat step 3, then `rm KILL`. 8. **Time limit:** rebuild; `printf 'app-06\napp-01\n' > slow.txt`; run with `--targets slow.txt --apply --timeout 2`. 9. **Health check:** rebuild; `printf 'app-07\napp-01\napp-03\n' > sick.txt`; run with `--targets sick.txt --apply`. 10. **One run at a time:** rebuild; `flock runner.lock sleep 5 & sleep 1`; repeat step 3. 11. **Guard tests:** save the test below as `test_guards.py` beside the runner; run `python3 -m unittest test_guards`. _Example status: verified by execution (Python 3.8.10, 2026-10-17)._ ```python #!/usr/bin/env python3 """Guard tests for guarded_runner.check_targets: one case per known-bad input. Every postmortem that finds a new bad input adds a case here. """ import unittest from guarded_runner import check_targets FLEET = {"app-01": {"kind": "app"}, "app-02": {"kind": "app"}, "db-01": {"kind": "db"}} def problems(targets, max_targets=5): return check_targets(targets, FLEET, "app", max_targets) class GuardTests(unittest.TestCase): def test_empty_list_is_refused(self): # Diskerase: empty must not mean "all" self.assertTrue(problems([])) def test_unknown_target(self): self.assertIn("unknown targets: app-99", problems(["app-01", "app-99"])) def test_wrong_kind(self): # Atlassian: real IDs of the wrong type self.assertIn("targets are not of kind 'app': db-01", problems(["db-01"])) def test_duplicate_target(self): self.assertIn("duplicate targets in the list", problems(["app-01", "app-01"])) def test_over_the_cap(self): self.assertTrue(problems(["app-01", "app-02"], max_targets=1)) def test_valid_list_passes(self): self.assertEqual(problems(["app-01", "app-02"]), []) if __name__ == "__main__": unittest.main() ``` **Expected output** (copied from the verified run): ```text # step 2 (dry run) dry-run app-01 would change (tmp_files=12) dry-run app-02 unchanged (tmp_files=0) dry-run app-03 would change (tmp_files=7) dry-run app-04 would change (tmp_files=30) DONE: dry-run on 4 target(s), 0 failure(s), run c942b43f # step 3, second run (idempotence) apply app-01 unchanged (tmp_files 0 -> 0) ... # step 4 (exit code 2) REFUSED: target list is empty: nothing to do, refusing rather than guessing # step 5 (exit code 2) REFUSED: targets are not of kind 'app': db-01 # step 6 (exit code 3) apply app-05 FAILED: simulated failure on app-05 apply app-01 changed (tmp_files 12 -> 0) HALTED: 1 failure(s) exceed the budget of 0 after batch 1 # step 7 (exit code 4) KILLED: kill switch KILL present before batch 1; 2 batch(es) not run # step 8 (exit code 3, after about 2 seconds) apply app-06 FAILED: time limit exceeded apply app-01 changed (tmp_files 12 -> 0) HALTED: 1 failure(s) exceed the budget of 0 after batch 1 # step 9 (exit code 3) apply app-07 changed (tmp_files 4 -> 0) apply app-01 changed (tmp_files 12 -> 0) HALTED: health check failed after batch 1: app-07 # step 10 (exit code 2) REFUSED: another run holds the lock runner.lock # step 11 (exit code 0) ...... ---------------------------------------------------------------------- Ran 6 tests in 0.000s OK ``` **Validation:** `echo $?` gives 0, 0, 2, 2, 3, 4, 3, 3, 2 and 0 for steps 2–11; `audit.jsonl` has a JSON line per target or decision. In step 6 `app-01` still changed after `app-05` failed: the budget is checked between batches, the in-flight behaviour AWS also documents.[^ssmrate] In step 9 `app-03` never ran. Remove any one of the five target checks in `check_targets` from a copy of the runner and exactly one test fails. **Troubleshooting:** `FileNotFoundError` means step 1 was skipped; create `KILL` where you run the runner. **Teardown:** `rm -rf fleet targets.txt empty.txt wrong.txt fail.txt slow.txt sick.txt audit.jsonl runner.lock __pycache__`. #### Lab 3 (advanced): port the guards to Ansible and prove idempotence **Objective:** prove each guard of the [Ansible version](#reference-implementation). **Prerequisites:** Ansible; two to four disposable Linux containers or VMs in group `app_servers`, with old files in `/var/tmp/app` (`touch -d '10 days ago' /var/tmp/app/old.tmp`), each answering `http://:8080/healthz` (for a lab, `python3 -m http.server 8080` in a directory holding a `healthz` file). **Risk:** it deletes files; never use real servers. **Status:** not executed in this environment. 1. `ansible-playbook -i inventory.ini cleanup-tmp.yml --check --diff`: the plan. `find` runs normally in check mode, so the removal task reports each old file as changed, and nothing is removed.[^ansfind][^ansfile] 2. Run without `--check`; then run again. **Expected:** the second `PLAY RECAP` shows `changed=0` for every host. 3. Set `max_targets: 1` and run: the assert fails on the first batch and no host changes. 4. `touch /etc/automation/KILL-cleanup-tmp` on the control node and run: the play stops before the first batch; remove the file. 5. Stop the health endpoint on one host and run: the play stops after that host's batch. 6. Add ansible-lint and a Molecule scenario to CI; Molecule's default test sequence includes the `idempotence` step.[^molecule] **Expected output and validation:** as stated in each step; the log names the guard that stopped the run. **Troubleshooting:** if step 1 lists no files, check the path and file ages; a `command` probe in place of `find` would be skipped in check mode. **Teardown:** destroy the containers or VMs. ### Use cases and scenarios The first three are illustrative; the fourth draws on Atlassian's published review. | Scenario | Context | Method | Trade-offs | Success measure | |---|---|---|---|---| | Self-service restarts | Support waits hours for ops to restart a stuck service | Tier-1 runbook job; access by role; "ensure running"; audit | More people can touch production | Wait falls to minutes; no induced incidents | | Auto-remediation | A worker hangs several nights a week | Event rule limited to the alerting host; `once_within` per host; fleet cap of 3 an hour, then a person | Can hide a real defect | Pages fall; repeats tracked as a defect | | Fleet configuration | 200 servers need a monthly setting change | Tier 2: dry run, canary, `serial` 25%, failure budget 0 | Slower than one parallel run | No run exceeds one batch of damage | | Destructive clean-up | Removing a legacy component across customer sites | Tier 3: type-checked IDs, names shown, soft delete, approval, own sites first | Weeks slower; retention storage | No unrecoverable deletion | **Pitfalls.** Self-service inputs and alert webhooks are an attack surface; remediation can itself remove capacity. Atlassian's sites were deleted "via a standard workflow", so monitoring saw nothing.[^atlassian2022] ## Measure, secure and pay ### Metrics and KPIs No standard set exists; this is **this Atlas's proposal**, with one row adapted from DORA. | KPI | Definition and formula | Data source | Pitfall | Cadence | |---|---|---|---|---| | Net hours returned | runs × (manual − residual minutes) ÷ 60 − maintenance hours, per month | Run log, estimates | Maintenance left out | Monthly | | Run success rate | successful production runs ÷ production runs | Audit log | "Success" that quietly handed off | Weekly | | Human fallback rate | runs handed to a person ÷ production runs | Audit log, pages | Informal fallbacks never logged | Weekly | | Induced incident rate | runs that caused an incident or needed immediate intervention ÷ production runs (adapted from DORA's change fail rate[^dora]) | Incidents linked to run ids | Incidents not linked to runs | Monthly | | Guard trip rate | runs refused or halted ÷ runs | Audit log | Zero may mean guards test nothing | Monthly | | Time to stop | kill switch pulled → last in-flight action finished | Drill records | Unmeasured until the real incident | Each drill | | Ownership coverage | automations with a current owner and review date ÷ registered | Register | Owners that no longer exist | Quarterly | | Unregistered automation | jobs or keys acting on production but not registered | Credential and scheduler inventory | Counting only what you know | Quarterly | Toil share is defined in [SRE](../sre/). ::: example Worked calculation (illustrative figures from Lab 1) `restart-stuck-worker` runs 40 times a month: manual 40 × 15 min = 10.0 h; residual 40 × 1 min = 0.67 h; maintenance 1.5 h. **Net hours returned** = 10.0 − 0.67 − 1.5 = **7.83 h a month**; with 40 build hours, **break-even** = 40 ÷ 7.83 = **5.1 months**, and year one returns 12 × 7.83 − 40 = **54 hours**. That month 38 runs succeed and 2 hand off to a person: **success** 38 ÷ 40 = **95.0%**, **fallback** 2 ÷ 40 = **5.0%**, **induced incidents** 0 ÷ 40 = **0%**. **Worst-case reach** of the [rulebook](#reference-implementation) in an hour-long alert storm on four workers: the per-host throttle allows about four restarts per host, 16 in all, each on one host because of the host limit (without it, every trigger would restart all four: 64). The playbook's fleet cap is what bounds the total: **3 restarts an hour**, then a person decides. ::: **Leading and lagging:** fallback rate, guard trips and ownership coverage lead; induced incidents lag. **Vanity traps:** counting automations; "hours saved" without maintenance. Minutes are estimates: revisit them yearly. ### Security, safety and compliance - **Identity is the blast radius.** One least-privilege identity per automation; NIST's AC-6 covers "processes acting on behalf of users".[^nist] Check whose rights a run uses ([trust boundaries](#reference-architecture)) and harden service-role trust policies against the "confused deputy problem".[^ssmiam] - **Inputs are an attack surface.** Authenticate every trigger, webhooks included;[^edasources] constrain parameters with patterns, and for SSM command documents prefer `interpolationType: ENV_VAR` to string substitution.[^ssmparams] Secrets come from a secrets manager at run time. - **Approvals need separation and scope.** Requester and approver differ ("two-person control"[^nist]); whoever can assume an approver role can approve;[^ssmapprove] the approval must cover the action taken.[^srerunmodes] | NIST SP 800-53 control[^nist] | What it asks | AutomationOps practice | |---|---|---| | CM-3 | Decide which changes are controlled; review, approve, document, retain records | Register, tiers, audit log retention | | CM-3(2) | Test, validate and document changes before finalising them | Lint, idempotence and guard tests; dry runs | | CM-3(3) | Implement changes using automated mechanisms | The executor itself, under the controls above | | CM-5 | Restrict who can make changes | Who may edit, trigger and approve each automation | | CM-5(4) | Dual authorization for changes to system components | Two people approve changes to automation code and configuration | | AC-3(2) | Dual authorization for privileged commands | Second-person approval for tier-3 runs | | AC-6 | Least privilege, including processes | One identity per automation | These controls bind only where adopted. Mapping is covered in [ComplianceOps](../complianceops/), change approval in [ITOps](../itops/). ### Cost, capacity and sustainability **Maintenance is the lasting cost**, because automation must change with the systems under it ("bit rot"[^sreauto]). Add the runner platform (fees change: AWS ended the Systems Manager Automation free tier for new customers on 14 August 2025 and for existing ones on 31 December 2025[^ssmabout]), identities and on-call time. **Capacity:** "even read operations, naively implemented, can spike device load", and at scale "these safety checks can eventually dominate workload".[^wbtoil] **Unit view:** cost per automated run (platform share plus residual minutes) against cost per manual run. **Sustainability** is rarely material ([FinOps](../finops/) covers idle resources). ### Assurance of the operating model | Check | How | Signal of trouble | |---|---|---| | Kill-switch drill | Pull the switch during a staged run; time it | Time to stop unknown or rising | | Guard tests | CI cases for empty, unknown, wrong-type and oversized inputs | A guard change without a failing test | | Audit sampling | Monthly: did targets match the request, and did someone else approve? | Self-approvals; unexplained targets | | Dead-man alerts | Alert when a scheduled automation has *not* run | Silent stops after credential expiry | | Register reconciliation | Compare the register with credential use and schedulers | Keys and cron jobs nobody can explain | | Manual-path practice | Rehearse the manual procedure for critical automations | Nobody on call has done it by hand | The last row follows Bainbridge: "physical skills deteriorate when they are not used".[^bainbridge] Facebook credited its "storm" drills with its careful 2021 restart.[^fb2021] ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Remediation | Prevention | |---|---|---|---|---| | Far more targets changed than intended | Empty selector, mistyped input, wrong ID type[^sreauto][^s3-2017][^atlassian2022] | Run's target count and types vs the request | Kill switch; restore | Validation, caps, floors, names | | Remediation keeps firing; capacity drains | No velocity limit; flapping checks[^aws2025] | Actions per target per hour | Disable the automatic action | Throttles, velocity caps, demotion | | Two instances undo each other | Concurrent executors; check made long before the write[^aws2025] | Interleaved run ids on one object | Disable both; repair by hand | Locks or version checks at write time | | A guarded command did damage | Bug in the guard[^fb2021] | Replay the input against the guard | Fix the guard first | Known-bad input tests | | "Succeeded" but nothing changed | Check mode left on; `changed_when: false` | State before vs after | Re-run correctly | Verify outcomes, not exit codes | | Scheduled job silently stopped | Expired credential; owner left | Last successful run | Re-own; rotate | Dead-man alerts; reviews | | Rollback made it worse | Untested rollback of partial state; Knight's uninstall spread the fault[^knight] | What actually changed | Assess before reversing | Tested rollbacks | | Nobody can do it by hand | Manual path decayed[^sreauto][^bainbridge] | Drill fails | Pair with the owner | Manual-path practice | ### Worked runbook: automation is changing production unexpectedly ::: figure Runbook for an automation that is misbehaving ```mermaid flowchart TB A["Alert or report: unexpected changes"] --> B["Pull the kill switch; disable triggers"] B --> C{"All runs stopped?"} C -->|"no"| D["Revoke or deny the automation's identity"] D --> C C -->|"yes"| E["Scope damage from the audit log"] E --> F["Restore: soft delete, backup or roll forward"] F --> G["Re-enable as supervised, after fix and guard test"] G --> H["Postmortem; demote tier; update register"] classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 class A,B,D risk class C decision class E,F,G,H core ``` Text equivalent: pull the kill switch and disable triggers; if runs continue, revoke or deny the automation's identity until all have stopped. Then scope the damage from the audit log, restore (soft delete, backup or roll forward), re-enable only as supervised after a fix and a guard test, and run a postmortem that demotes the tier and updates the register. ::: 1. **Stop** (on-call, minutes): kill switch, then schedules and event rules; if anything still runs, deny its identity. 2. **Confirm:** no new audit records for the run id. 3. **Scope** (owner, on-call): every target touched, with before/after values; a log that cannot say is the first action item. 4. **Restore** (service owner): soft delete, backup or a reviewed roll-forward; never reverse a partial change blindly. 5. **Re-enable** (owner, approved by the service owner): supervised, after a test that reproduces the failing input. 6. **Learn:** blameless postmortem ([IncidentOps](../incidentops/)). Atlassian: "people don't cause incidents. Rather, systems allow for mistakes to be made."[^atlassian2022] ::: example Illustrative timeline 02:10 an event rule starts restarting workers fleet-wide after a bad configuration push; 02:14 the fallback-rate alert pages on-call; 02:16 on-call disables the rulebook activation, which stops event processing,[^aapactivation] and sets the global kill switch; 02:18 no new audit records. The log shows 37 restarts on 12 hosts. The postmortem sets `enabled: false` on the rule, adds a host limit and a fleet-wide cap, and demotes it to supervised. ::: ### Anti-patterns and myths - **The script in a home directory** (Google's second level[^sreauto]): one owner, no tests, no audit. Register it and use a runner. - **Automating the mess.** "You rarely want to literally transcribe a human workflow into a machine workflow."[^wbtoil] - **One god credential** shared by all automation: any bug has full reach. - **Approval for everything.** Approvers stop reading; keep approvals for tier 3 (editorial judgement). - **Auto-remediation as a hiding place.** Nightly restarts of a leaking service silence the alert and keep the defect. - **Myth: once automated, done.** Automation rots as systems change and is most fragile when rarely run.[^sreauto] ## Deciding and adopting ### When to use and when not to use **Use it** when operational actions repeat, are understood and touch production, especially when several people or systems trigger them or an auditor will ask who changed what. **Do not automate (yet)** when the task is rare and needs judgement (give people a checklist); when the need can be removed (fix the leak; build failover into the system[^sreauto]); when the process is not understood or agreed; when nobody will own the result; or when the triggering signal is unreliable (fix the signal first). **Hidden responsibilities:** maintenance, an identity to secure, an on-call path for its failures, drills and eventual retirement. ### Decision framework ::: figure Should this task be automated? ```mermaid flowchart TB Q1["1 Repeats, understood?"] -->|"no"| M["Manual checklist"] Q1 -->|"yes"| Q0["2 Need removable?"] Q0 -->|"yes"| FIX["Fix at source"] Q0 -->|"no"| Q2["3 Judgement each time?"] Q2 -->|"yes"| HB["Human-backed interface"] Q2 -->|"no"| Q3["4 Pays back, or must be consistent?"] Q3 -->|"no"| NO["Not now"] Q3 -->|"yes"| AUTO["Automate; tier sets controls"] M ~~~ FIX FIX ~~~ HB HB ~~~ NO classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class Q1,Q0,Q2,Q3 decision class FIX,AUTO core class M,HB,NO neutral ``` Text equivalent: four questions in order (orange boxes), each with one exit. 1, does the task repeat and is it understood? If not, keep a manual checklist. 2, can the need be removed? If so, fix it at the source. 3, does it need judgement each time? If so, build a human-backed interface. 4, does it pay back, or must it be consistent? If not, not now. Otherwise automate, with the controls its risk tier requires. ::: The tree assumes that removing work beats automating it. How much ceremony follows is decided by the [tiers](#operating-model-and-governance): irreversibility, not frequency, sets the controls. ### Maturity model **This Atlas's model**, not an industry standard; it measures governance, not the amount of automation. | Level | Observable behaviour | Evidence | Exit criteria | |---|---|---|---| | 1 Ad hoc | Scripts in home directories; shared admin credentials | Shell histories | Inventory of everything that changes production | | 2 Shared | Automation in version control and a runner | Repository, runner logs | Every production-changing automation registered with an owner | | 3 Guarded | Dry runs, validation, caps, batches, kill switches for tiers 2–3; own identities | Guard tests, drill records | Monthly metrics; tier policy enforced by the runner | | 4 Governed | Quarterly reviews; tiers drive approvals; fallback and induced-incident rates tracked; retirements happen | Register history, metrics | Frequent remediations moved into systems; automatic demotion | | 5 Self-correcting | Frequent actions built into systems; velocity limits on event automation; agents use the same guards | Fewer external scripts; drills pass | Sustain; keep manual skills | The typical stall is level 2: scripts in Git, no owners, one shared credential. The register is the way out. ### Adoption roadmap: 30/60/90 days | Phase | Actions | Milestone and metric | |---|---|---| | Days 1–30 (platform lead, service owners) | Inventory schedulers, CI jobs, runbook tools and script keys; register the 20 riskiest; guard the most dangerous script; score candidates | Register exists; ownership coverage measured | | Days 31–60 (platform team, owners) | One runner; own identities for the top 10; dry runs, batches and failure budgets for tier 2; first kill-switch drill | Time to stop measured | | Days 61–90 (service owners, platform team) | Promote first automations with evidence; dead-man alerts; first quarterly review; retire unused automations; publish the tier policy | Ownership coverage ≥ 90%; fallback and induced-incident rates reported | **Quick wins:** the empty-list guard; a kill switch for the automation that can delete data; a dead-man alert on certificate renewal. ## Toolkit ### Checklists **Quick start** - [ ] Every automation that changes production is registered, with owner and tier. - [ ] No automation uses a shared admin credential. - [ ] The riskiest automation refuses empty and oversized target lists. - [ ] Tier 2 and 3 automations have a kill switch that on-call can find. **Readiness review (preflight)** - [ ] Candidate scored; owner, service owner and tier agreed. - [ ] Idempotence shown by a second run with no changes. - [ ] Guard tests pass for empty, unknown, wrong-type, duplicate and oversized inputs. - [ ] Dry run lists targets with before/after values. - [ ] Time limits set; for tier 2, also a lock, batches, a failure budget and health checks between batches. - [ ] Identity scoped; secrets from a manager. - [ ] Manual fallback works; kill switch tested and timed. **Before a tier-3 run** - [ ] The plan shows names and the count matches the request. - [ ] Soft delete or a tested restore exists. - [ ] Approver differs from requester; on-call is told. - [ ] The first stage targets your own resources. **Quarterly governance audit** - [ ] Owners and review dates current; frozen automations re-owned or retired. - [ ] Register reconciled with credentials and schedulers. - [ ] Audit sample: targets matched requests; approvals separated. - [ ] Rising fallback rates have work scheduled; drills done. **Continuous improvement** - [ ] Every frequently firing remediation has a defect ticket; every automation postmortem adds a guard test. ### Templates _Example status: templates; nothing to execute._ ```text AUTOMATION READINESS REVIEW Automation id / version: Owner: Service owner: Purpose (one sentence): Tier (0-3) and why: Triggers (schedule / event / person): Who may trigger: Identity and permissions (exact actions, resources): Target selection: source of the list; validation rules; cap; floor: Dry run evidence (link): Idempotence evidence (link): Time limit / lock / batches / failure budget / health check between batches: Approval (tier 3): approver group; requester excluded? yes / no Kill switch: location; tested on (date); time to stop: Fallback: who is paged; manual procedure (link); last practised (date): Metrics: success, fallback, induced incidents; dashboard (link): Review date: Decision: approve / approve supervised only / reject Reviewers and date: ``` ```text KILL-SWITCH DRILL RECORD Date / automation / environment: Scenario (e.g. bad input reached batch 2): Switch pulled by / at (time): Last in-flight action finished at (time): Time to stop: Targets changed after the switch: Did on-call find the switch without help? yes / no Problems found and actions (owner, due date): ``` ```text AUTOMATION RETIREMENT NOTICE Automation id: Owner: Date: Reason (cause fixed at source / unused for N days / replaced by): Triggers removed: yes / no Identity and secrets revoked: yes / no Code archived at: Register status set to retired: yes / no Dependent runbooks and dashboards updated: yes / no ``` ### Deep comparisons | Approach | Best for | Main risk | Controls to add | |---|---|---|---| | Manual checklist | Rare, judgement-heavy work | Drift | Second-person check on risky steps | | Personal script | Prototyping | One owner, no audit | Register and runner | | Runbook platform or configuration management | Self-service jobs; fleet desired state | Privileged platform; fleet-wide bad change | Tiers, identities, check mode, batches | | Event-driven remediation | Frequent, understood faults | Storms; hidden defects | Host limits, throttles, fleet caps | | Built into the system | Very frequent actions | Hard to retrofit; less visible | Expose internals to operators[^sreauto] | | AI operations agent | Investigation, proposals | Plausible wrong actions | Review mode, narrow identity, same guards | ### FAQ **Is AutomationOps a job title or a certification?** We found no certification or standards body using the term for IT operations (searched 2026-10-16). SREs, platform and operations engineers do the work, sometimes in an automation centre of excellence. **Isn't this just SRE?** SRE defines and caps toil ([SRE](../sre/)); this page makes the automation that removes toil safe and owned. Small teams do both. **Should every automation need an approval?** No. Approvals add information only for irreversible or wide-reaching actions; elsewhere they train people to click. Use the tiers. **Our operational jobs run from CI pipelines. In scope?** Yes, if they change production. Scope the pipeline's credentials to the job; pipeline identity is covered in [DevSecOps](../devsecops/); infrastructure-as-code runs follow [InfraOps](../infraops/). **How much autonomy should auto-remediation have?** As much as its tier and record allow, always with a velocity limit; see the [judgement call](#a-judgement-call-automatic-failover-or-a-human-decision). ## Going further ### Advanced topics and emerging direction - **Fighting control loops.** In AWS's DynamoDB DNS system a one-time freshness check "was stale by this time", so an older plan overwrote a newer one.[^aws2025] Prefer one writer, versioned plans and compare-and-set at write time; durable engines are in [WorkflowOps](../workflowops/). - **Global change budgets and cells.** Cap how much capacity *all* automation may remove at once (AWS's announced NLB velocity control caps one mechanism[^aws2025]); automation that acts one cell at a time inherits the cell's limit, which is why AWS refactored parts of S3 "into smaller cells to reduce blast radius and improve recovery".[^s3-2017] - **Verification before execution** is mature in networking ([NetworkOps](../networkops/)); elsewhere, dry runs and guards dominate. - **AI agents** (emerging): approval modes and permission scopes exist;[^srerunmodes][^devopsagent] we found no independent evidence on their production error rates. ### Learning path | Stage | What to do | Outcome | |---|---|---| | Prerequisites | Shell and Python basics; the SRE book's toil chapter[^sretoil] | Can read and write small scripts | | 1-day primer | First four parts of this page; the S3, Facebook and Atlassian reports;[^s3-2017][^fb2021][^atlassian2022] Lab 1 | Can score candidates and name the five controls | | 1-week practice | Lab 2; add a floor guard (refuse if fewer than two hosts would stay healthy) with a test | Can build and test guards | | 30-day progression | Lab 3 with Molecule in CI; register your team's automations; a kill-switch drill | A governed automation in production | | Specialisation | Event-driven remediation; runbook platforms; access design with [IAMOps](../iamops/) | Can design a platform's controls | | Portfolio | A guarded runner with tests; an analysis of a public automation incident | Evidence of judgement | ## Wrap-up ### Executive recap and cheat sheet | Topic | Remember | |---|---| | What it is | Governance and safety of operational automation; an editorial reading of an ambiguous label | | Decide | Remove the need first; automate repetitive, understood work; irreversibility decides the ceremony | | Five controls | Dry run · target validation (empty never means all) · blast-radius cap with batches · failure budget · kill switch | | Ownership | Register, owner, tier, own identity, review date, retirement | | Metrics | Net hours returned, fallback rate, induced-incident rate, time to stop, ownership coverage | | Watch-outs | Shared credentials, approval fatigue, remediation storms, open webhooks, dry runs as proof, lost manual skills | | Next actions | Register; guard the riskiest script; drill a kill switch; score three candidates | ### A judgement call: automatic failover or a human decision? **Situation (illustrative).** A checkout database has a primary and a replica, usually under a second apart. Automated failover takes about 30 seconds; a human-approved one at night about 15 minutes. Last year the health check raised 12 alarms, 2 of them real primary failures. | Option | Real failures (2 a year) | False alarms (10 a year) | Total | |---|---|---|---| | A: automatic failover | 2 × 0.5 min = 1 min | 10 × 0.5 min = 5 min of avoidable disruption, each risking writes inside the lag | about 6 min; 12 risky failovers | | B: automatic diagnosis, human approves | 2 × 15 min = 30 min | 0 min, but 10 night pages | about 30 min; 12 pages | **The attractive choice and why it can be wrong.** A wins on minutes, but failover can lose acknowledged writes (tier 3), a flapping check can trigger repeated failovers, and the signal is wrong 10 times in 12. B costs half an hour of outage a year and night pages for alarms that are usually false. **Recommendation.** Fix the signal first. Then choose A *with conditions*: fail over automatically only if replication lag, checked at the moment of action, is within the agreed data-loss limit, the old primary can be fenced, and no automatic failover happened in the last few hours; otherwise fall back to B. **Uncertainty:** the figures are illustrative; engine mechanics are in [DBOps](../dbops/), invoking DR in [DisasterRecoveryOps](../disasterrecoveryops/). ### Learning objectives and mastery check After this page you should be able to **explain** the label and this page's reading of it; **score** candidates; **design** guards, batches, failure budgets and kill switches; **govern** with a register, identities and metrics; and **evaluate** when not to automate. Self-check: 1. A decommission script takes its server list from an inventory query, which returns nothing. What should the script do? ::: answer Show answer for question 1 Refuse and report: empty must never mean "everything" (the Diskerase incident).[^sreauto] Page the owner if the empty result is unexpected. ::: 2. A colleague says a playbook is safe because `--check` showed no errors. What is missing? ::: answer Show answer for question 2 Check mode is "just a simulation": unsupported modules "report nothing and do nothing", and conditions on registered results show nothing when the task that registered them was skipped, as a `command` probe is.[^anscheck][^anscommand] Add `--diff`, compare the target count with expectations, and rely on guards and batches as well. ::: 3. An SSM Automation run targets 50 instances with `--max-concurrency 10` and `--max-errors 10%`. How many could fail? ::: answer Show answer for question 3 Up to 14 or 15. New invocations stop at the fifth error (user guide) or the sixth (CLI reference), and up to nine invocations already running may finish and fail too.[^ssmrate][^ssmcli] To guarantee the limit, use concurrency 1, and test the semantics first. ::: 4. A task runs 10 times a month, takes 30 minutes by hand, would need 5 minutes of human time per automated run and 2 hours of maintenance a month, and 50 hours to build. Automate it? ::: answer Show answer for question 4 Net = 10 × (30 − 5) ÷ 60 − 2 = 2.17 hours a month; break-even about 23 months. Marginal on time alone: automate if consistency or speed matters or maintenance can shrink; otherwise improve the checklist and revisit. ::: 5. A nightly job deletes customer records past retention across all tenants. Approving every run would be tedious. How do you govern it? ::: answer Show answer for question 5 Tier 3, but put the second person on changes to the job's code and configuration, not on every run. Each run needs automatic guards: soft delete with a recovery window, a cap, and an anomaly check that refuses if far more records qualify than yesterday, in line with Atlassian's soft-delete and staged-rollout lessons.[^atlassian2022] ::: ### Conclusion AutomationOps is a borrowed name: RPA vendors use it, and this Atlas applies it to something older, keeping operational automation safe and owned. In each incident above, automation did what it was told; it was told something wrong (an empty list, a mistyped input, the wrong IDs, a stale plan) and nothing stopped it in time. The discipline is simple to state: remove what need not exist, automate what repeats and is understood, give every run the [charter's](#a-proposed-charter-and-principles) brakes, including a kill switch someone has actually pulled, and give each automation an owner, an identity and a retirement date. Keep people able to do the work by hand: the day the automation fails is the day they will need to. ## Sources and further reading Sources were opened on the date each footnote gives; dates are the source's own. Claim-level support is in `research/automationops-ledger.md`. **Re-check by 2027-04-17, then every six months:** agent modes and defaults, Systems Manager pricing, Rundeck and StackStorm releases; open the SEC and Red Hat pages in a browser (they block some automated clients). [^sreauto]: [The Evolution of Automation at Google](https://sre.google/sre-book/automation-at-google/) — Niall Murphy with John Looney and Michael Kacirek, *Site Reliability Engineering* (Google, O'Reilly, 2016). Checked 2026-10-16. [^sretoil]: [Eliminating Toil](https://sre.google/sre-book/eliminating-toil/) — Vivek Rau, *Site Reliability Engineering* (Google, O'Reilly, 2016). Checked 2026-10-16. [^wbtoil]: [Eliminating Toil](https://sre.google/workbook/eliminating-toil/) — David Challoner, Joanna Wijntjes, David Huska et al., *The Site Reliability Workbook* (Google, O'Reilly, 2018). Checked 2026-10-16. [^robocorp2023]: [AutomationOps: A Modern Approach to Automation (archived)](https://web.archive.org/web/20230923142810/https://robocorp.com/automation-ops-landing-page) — Robocorp, snapshot 2023-09-23; the live URL now redirects to sema4.ai. Checked 2026-10-16. [^uipath]: [Automation Ops user guide: Introduction](https://docs.uipath.com/automation-ops/automation-cloud/latest/user-guide/introduction) — UiPath documentation, latest. Checked 2026-10-16. [^uipathga]: [Automation Ops release notes: General Availability](https://docs.uipath.com/automation-ops/automation-cloud/latest/release-notes/release-notes-general-availability) — UiPath documentation, release date 2021-04-26. Checked 2026-10-16. [^redhat]: [What is IT automation?](https://www.redhat.com/en/topics/automation/what-is-it-automation) — Red Hat, 2024-11-20. Checked 2026-10-16. [^rundeck]: [Rundeck Introduction](https://docs.rundeck.com/docs/about/introduction.html) — Rundeck documentation (PagerDuty); latest open-source release v6.2.1, 2026-09-09. Checked 2026-10-16. [^elasticautoops]: [AutoOps](https://www.elastic.co/platform/autoops) — Elastic, undated product page. Checked 2026-10-16. [^knight]: [In the Matter of Knight Capital Americas LLC, Release No. 34-70694](https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf) — U.S. Securities and Exchange Commission, 2013-10-16 (blocks non-browser clients). Checked 2026-10-16. [^s3-2017]: [Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/41926/) — Amazon Web Services, 2017 (event of 2017-02-28). Checked 2026-10-16. [^fb2021]: [More details about the October 4 outage](https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/) — Santosh Janardhan, Engineering at Meta, 2021-10-05. Checked 2026-10-16. [^atlassian2022]: [Post-Incident Review on the Atlassian April 2022 outage](https://www.atlassian.com/blog/how-we-build/post-incident-review-april-2022-outage) — Atlassian, 2022-04-29. Checked 2026-10-16. [^aws2025]: [Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/101925/) — Amazon Web Services, October 2025. Checked 2026-10-16. [^bainbridge]: [Ironies of Automation (author's web copy, archived)](https://web.archive.org/web/20160305171115/http://www.bainbrdg.demon.co.uk/Papers/Ironies.html) — Lisanne Bainbridge; originally *Automatica* 19(6), 1983; snapshot 2016-03-05. Checked 2026-10-16. [^xkcd]: [Is It Worth the Time?](https://xkcd.com/1205/) — Randall Munroe, xkcd 1205, 2013-04-29. Checked 2026-10-16. [^devopsagent]: [Announcing General Availability of AWS DevOps Agent](https://aws.amazon.com/blogs/mt/announcing-general-availability-of-aws-devops-agent/) — Madhu Balaji, AWS Cloud Operations Blog, 2026-03-31. Checked 2026-10-16. [^srerunmodes]: [Run modes in Azure SRE Agent](https://learn.microsoft.com/en-us/azure/sre-agent/run-modes) — Microsoft Learn, updated 2026-06-11. Checked 2026-10-17. [^ansgloss]: [Glossary](https://docs.ansible.com/projects/ansible/latest/reference_appendices/glossary.html) — Ansible Community Documentation. Checked 2026-10-16. [^anscheck]: [Validating tasks: check mode and diff mode](https://docs.ansible.com/projects/ansible/latest/playbook_guide/playbooks_checkmode.html) — Ansible Community Documentation. Checked 2026-10-16. [^ansstrat]: [Controlling playbook execution: strategies and more](https://docs.ansible.com/projects/ansible/latest/playbook_guide/playbooks_strategies.html) — Ansible Community Documentation. Checked 2026-10-16. [^anserr]: [Error handling in playbooks](https://docs.ansible.com/projects/ansible/latest/playbook_guide/playbooks_error_handling.html) — Ansible Community Documentation. Checked 2026-10-16. [^ansspecial]: [Special variables](https://docs.ansible.com/projects/ansible/latest/reference_appendices/special_variables.html) — Ansible Community Documentation. Checked 2026-10-16. [^ansfind]: [ansible.builtin.find module](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html) — Ansible Community Documentation. Checked 2026-10-16. [^ansfile]: [ansible.builtin.file module](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/file_module.html) — Ansible Community Documentation. Checked 2026-10-16. [^anscommand]: [ansible.builtin.command module](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/command_module.html) — Ansible Community Documentation. Checked 2026-10-16. [^anskw]: [Playbook Keywords](https://docs.ansible.com/projects/ansible/latest/reference_appendices/playbooks_keywords.html) — Ansible Community Documentation. Checked 2026-10-16. [^molecule]: [Molecule configuration](https://docs.ansible.com/projects/molecule/configuration/) — Ansible Molecule documentation (Red Hat). Checked 2026-10-16. [^lint]: [no-changed-when](https://docs.ansible.com/projects/lint/rules/no-changed-when/) — Ansible Lint documentation. Checked 2026-10-16. [^shellcheck]: [ShellCheck, a static analysis tool for shell scripts](https://github.com/koalaman/shellcheck) — koalaman/shellcheck, GitHub. Checked 2026-10-16. [^eda]: [Conditions: throttle actions to counter event storms](https://docs.ansible.com/projects/rulebook/en/latest/conditions.html) — ansible-rulebook documentation, latest. Checked 2026-10-16. [^edarules]: [Rules](https://docs.ansible.com/projects/rulebook/en/latest/rules.html) — ansible-rulebook documentation, latest. Checked 2026-10-16. [^edarulebooks]: [Rulebooks](https://docs.ansible.com/projects/rulebook/en/latest/rulebooks.html) — ansible-rulebook documentation, latest. Checked 2026-10-16. [^edahosts]: [Limiting hosts](https://docs.ansible.com/projects/rulebook/en/latest/host_limit.html) — ansible-rulebook documentation, latest. Checked 2026-10-16. [^edasources]: [Event Sources](https://docs.ansible.com/projects/rulebook/en/latest/sources.html) — ansible-rulebook documentation, latest. Checked 2026-10-16. [^aapactivation]: [Configure rulebooks to take action in response to events or conditions](https://docs.redhat.com/en/documentation/red_hat_ansible_automation_platform/2.6/get_started-con_gs_rulebook_activations) — Red Hat Ansible Automation Platform 2.6 documentation. Checked 2026-10-16. [^stackstorm]: [StackStorm Overview](https://docs.stackstorm.com/overview.html) — StackStorm documentation; latest GitHub release 3.9.0, 2025-10-10. Checked 2026-10-16. [^ssmabout]: [AWS Systems Manager Automation](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-automation.html) — AWS Systems Manager User Guide. Checked 2026-10-16. [^ssmrate]: [Control automations at scale](https://docs.aws.amazon.com/systems-manager/latest/userguide/running-automations-scale-controls.html) — AWS Systems Manager User Guide. Checked 2026-10-16. [^ssmcli]: [start-automation-execution](https://docs.aws.amazon.com/cli/latest/reference/ssm/start-automation-execution.html) — AWS CLI 2.38.1 Command Reference. Checked 2026-10-16. [^ssmapprove]: [aws:approve – Pause an automation for manual approval](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-action-approve.html) — AWS Systems Manager User Guide. Checked 2026-10-16. [^ssmparams]: [Data elements and parameters](https://docs.aws.amazon.com/systems-manager/latest/userguide/documents-syntax-data-elements-parameters.html) — AWS Systems Manager User Guide. Checked 2026-10-16. [^ssmsetup]: [Setting up Automation](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-setup.html) — AWS Systems Manager User Guide. Checked 2026-10-16. [^ssmiam]: [Create the service roles for Automation using the console](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-setup-iam.html) — AWS Systems Manager User Guide. Checked 2026-10-16. [^ssmsteps]: [Run an automation step by step](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-working-executing-manually.html) — AWS Systems Manager User Guide. Checked 2026-10-16. [^ssmtargets]: [Mapping targets for an automation](https://docs.aws.amazon.com/systems-manager/latest/userguide/running-automations-map-targets.html) — AWS Systems Manager User Guide. Checked 2026-10-17. [^ec2stop]: [How EC2 instance stop and start works](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/how-ec2-instance-stop-start-works.html) — Amazon EC2 User Guide. Checked 2026-10-16. [^ec2policies]: [Example policies to control access the Amazon EC2 API](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ExamplePolicies_EC2.html) — Amazon EC2 User Guide. Checked 2026-10-16. [^iampassrole]: [Grant a user permissions to pass a role to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_passrole.html) — AWS Identity and Access Management User Guide. Checked 2026-10-16. [^ssmrestart]: [AWS-RestartEC2Instance](https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-aws-restartec2instance.html) — AWS Systems Manager Automation Runbook Reference. Checked 2026-10-17. [^ssmchangestate]: [aws:changeInstanceState – Change or assert instance state](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-action-changestate.html) — AWS Systems Manager User Guide. Checked 2026-10-17. [^ebskms]: [Requirements for Amazon EBS encryption](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-encryption-requirements.html) — Amazon EBS User Guide. Checked 2026-10-17. [^azauto]: [Azure Automation overview](https://learn.microsoft.com/en-us/azure/automation/overview) — Microsoft Learn. Checked 2026-10-16. [^puppet2025]: [Our Plans for Open Source Puppet in 2025](https://www.puppet.com/blog/open-source-puppet-updates-2025) — Puppet (Perforce), 2024-11-07. Checked 2026-10-16. [^openvox]: [OpenVox: The Community-Driven Fork of Puppet Has Arrived](https://thenewstack.io/openvox-the-community-driven-fork-of-puppet-has-arrived/) — Steven J. Vaughan-Nichols, The New Stack, 2025-01-23. Checked 2026-10-16. [^fowler]: [Feature Toggles (aka Feature Flags)](https://martinfowler.com/articles/feature-toggles.html) — Pete Hodgson, martinfowler.com, 2017-10-09. Checked 2026-10-16. [^waoe]: [Operational excellence design principles](https://docs.aws.amazon.com/wellarchitected/latest/framework/oe-design-principles.html) — AWS Well-Architected Framework. Checked 2026-10-16. [^nist]: [Security and Privacy Controls for Information Systems and Organizations (SP 800-53 Rev. 5)](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r5.pdf) — NIST, 2020-09 (updates as of 2020-12-10). Checked 2026-10-16. [^idem]: [Making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) — Malcolm Featonby, Amazon Builders' Library, undated. Checked 2026-10-16. [^dora]: [DORA's software delivery performance metrics](https://dora.dev/guides/dora-metrics/) — DORA. Checked 2026-10-16. **Changelog.** 2026-10-16, r0: first draft; runner and scoring labs executed; other examples syntax-checked only. 2026-10-17, r1 (answers `reviews/automationops-r1.md`): safer Event-Driven Ansible and AWS examples, corrected check-mode advice, runner with lock, time limits and health checks (Lab 2 re-run), two diagrams redrawn, dates, paraphrases and NIST mapping fixed; 14 sources added, all checked 2026-10-16. 2026-10-17, post-score polish: IAM policy caveats (2 sources added); tag-targeting inference labelled; Autonomous → Retired edge; Lab 1 and register fit the column; executed guard tests in Lab 2; sentences split; re-check cadence.