AutomationOps

AutomationOps is not a standard term. In this Atlas it names one discipline: running operational automation as a production system in its own right: deciding what to automate, making repeats harmless, limiting how far one run can reach, controlling who or what may trigger it, and keeping an owner until it is retired. Scripts, runbook jobs and auto-remediation rules hold administrator rights and act faster than anyone can intervene; this page is about governing them.

Google’s SRE book gives the reason: “For SRE, automation is a force multiplier, not a panacea”.^[1] In what this page calls the Diskerase incident, a decommissioning workflow restarted after a failure correctly found nothing left to erase, but “the empty set was used as a special value, interpreted to mean “everything.”” Almost all of Google’s colocation machines were wiped. Spare capacity limited users to “a slight increase in latency”; repair took the better part of two days, then weeks of new sanity checks.^[1]

For engineers who write operational automation, teams that host it, managers who approve it and auditors who must trust it. Code and labs: AutomationOps in practice.

Organisations fail with operational automation in two opposite ways.

Too little, done by hand. Manual procedures drift. In 2012 a Knight Capital technician “did not copy the new code to one of the eight SMARS computer servers”; no second person reviewed the deployment, and a repurposed flag woke dormant code. Knight built up unwanted positions in about 45 minutes and ultimately lost more than $460 million.^[8] For well-scoped procedures, consistency is “in many ways the primary value of automation”.^[1]

Too much power, too few brakes. The SRE book notes that “a platform also centralizes mistakes”, meaning a fix made once applies everywhere.^[1] The other side, shown by public post-incident reports, is that one bad input also applies everywhere:

• Not “automate everything”. Some work should be removed or stay manual (when not to use); whether operations can disappear is for NoOps.
• Not infrastructure as code (InfraOps), not a workflow engine (WorkflowOps) and not ChatOps, which is one way to trigger automation (ChatOps). This page covers run-time actions: restarts, clean-ups, failovers, rotations, fleet-wide configuration runs.
• Not a way to remove human error. Bainbridge (1983) observed that the designer who tries to eliminate the operator “still leaves the operator to do the tasks which the designer cannot think how to automate”.^[13] Errors move to design time and are copied to every run.

The label is recent and owned by no community; the practice is old.

Mental model: an automation is a change approved in advance, to be made many times, by someone who is not you, at a time you did not choose. Judge it like that change. Risk is roughly reach × speed × irreversibility; every control here reduces one of them.

flowchart LR
  S1["SRE: measured toil"] --> AO
  S2["IncidentOps: recurring fixes"] --> AO
  S3["ITOps: routine requests"] --> AO
  RPA["RPA bots: no Atlas page"] -.->|"same controls"| AO
  AO["AutomationOps: choose, guard, own, retire"] --> N1["InfraOps: IaC lifecycle"]
  AO --> N2["WorkflowOps: durable engines"]
  AO --> N3["ChatOps: chat trigger"]
  AO --> N4["NetworkOps: network change"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class AO core
  class S1,S2,S3,N1,N2,N3,N4 neutral
  class RPA risk

No AutomationOps manifesto exists. This charter is this Atlas’s proposal, drawn from the SRE books, AWS’s principles and the incidents above.

Mission. Remove toil and speed up routine operations without letting any single run do more damage than a careful person would.

1. Safe to repeat. Actions are idempotent, or carry a request identifier so retries have no extra effect.^[24]
2. Show before doing. A dry run precedes production runs, and someone or something reads it.
3. Distrust inputs, “even if that input is flowing from upstream systems”.^[15] Empty never means all.
4. Small steps. Batches, rate limits and floors; AWS lists “rate control, error thresholds, and approvals” as guardrails.^[25]
5. Stop fast. A failure budget halts a run; a tested kill switch halts all runs.
6. Least privilege, full record. One identity per automation; every run logged.
7. Owned or off. Without a current owner, an automation is frozen.
8. People stay able. Automation should “default to human operators if it runs into an unsafe condition”,^[15] and those people practise the manual path.

Non-goals: automating judgement; maximising the number of automations; removing approvals that add real information.

Governance scales with risk. These tiers are this Atlas’s proposal.

Tier-3 runs and changes to automation code map to NIST’s dual-authorization controls (compliance). ChatOps numbers its command tiers differently (2 is any reversible write, 3 any high-impact command); when a chat command starts an automation, apply the stricter tier (editorial rule). Cadence (proposal): metrics weekly; register review quarterly; kill-switch drills twice a year for tiers 2–3.

• Automation owner maintains one automation and answers for it; the service owner owns what it touches and agrees its tier.
• Platform team (sometimes an automation centre of excellence) runs executors, register, identities and templates.
• Approver approves tier-3 runs and is never the requester; on-call receives guard trips and may pull the kill switch; security and audit review identities, approvals and logs.

R = responsible, A = accountable, C = consulted, I = informed. On-call can stop any automation without asking; restarting it needs the owner.

stateDiagram-v2
  direction TB
  [*] --> Candidate
  Candidate --> Rejected: no net value
  Candidate --> Built: scored and owned
  Built --> Tested: lint, idempotence, guards
  Tested --> Supervised: dry runs reviewed
  Supervised --> Autonomous: promoted
  Autonomous --> Supervised: demoted
  Autonomous --> Frozen: owner gone
  Supervised --> Frozen: owner gone
  Frozen --> Supervised: new owner
  Supervised --> Retired: cause fixed at source
  Autonomous --> Retired: cause fixed at source
  Frozen --> Retired: nobody needs it
  Rejected --> [*]
  Retired --> [*]

Promotion needs evidence: clean supervised runs, guards that tripped correctly in tests, and a tier that allows it. Demotion after an incident should be automatic. The best exit is where Google’s hierarchy ends: a system that “notices problems, and automatically fails over without human intervention”.^[1]

This logical architecture fits a team-owned runner, a runbook platform such as Rundeck or a cloud service such as AWS Systems Manager Automation. Guards and the kill switch sit between trigger and executor, and only the executor holds credentials that can change targets.

flowchart TB
  T["Trigger: schedule, alert or person"] --> G["Guards: validate, cap, dry run"]
  R["Register: owner, tier, limits"] --> G
  G -->|"refused"| H["On-call person"]
  G --> A{"Tier 3?"}
  A -->|"yes"| P["Second person approves"]
  P --> X["Executor with its own identity"]
  A -->|"no"| X
  K["Kill switch"] -.->|"checked per batch"| X
  X --> B1["Batch 1: canary"]
  B1 -->|"healthy"| B2["Batches 2 to n"]
  B1 -->|"budget exceeded"| H
  X --> L["Audit log and metrics"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class R,G,X,L core
  class A decision
  class K,H risk
  class T,P,B1,B2 neutral

Trust boundaries. Triggers are untrusted input, including your own monitoring. The executor’s identity is the real blast radius: in AWS Systems Manager a run uses either a service role or “the context of the user who invoked the automation”.^[26] The executor may append to the audit log but not rewrite it: ship records to a store its identity cannot change (the lab runner’s local file is not tamper-proof).

Alternatives. Google’s hierarchy runs from “an SRE has a failover script in his or her home directory” to systems “that don’t need any automation”.^[1] A central runner is the middle step; build very frequent actions into the system itself.

sequenceDiagram
  participant T as Trigger and on-call
  participant R as Runner
  participant S as Targets
  T->>R: Target list
  Note over R: Guards: validate, cap, dry run
  Note over R: Tier 3: second person decides
  alt Refused by a guard
    R-->>T: Refused, on-call paged
  else Rejected by the approver
    R-->>T: Stopped, nothing changed
  else Accepted
    loop Each batch, if switch off and budget left
      R->>S: Apply batch
      S-->>R: Results, health
    end
    alt Budget, health or kill switch tripped
      R-->>T: Halted, on-call paged
    else All batches healthy
      R-->>T: Done, audit written
    end
  end

The exception path matters most: every refusal and halt reaches a person with the run id, targets touched and reason.

Choosing candidates. Start from the toil log (SRE), not from what is most annoying, and estimate net hours and risk tier (Lab 1). xkcd’s chart shows how long you can spend speeding up a routine task before losing time over five years: 5 minutes off a daily task justifies about 6 days, counted around the clock (some 150 hours).^[14] It ignores maintenance and risk; the SRE book adds that automation “provides more than just time saving”.^[1] For messy requests, start with a human-backed interface.^[15]

Idempotency. Write “ensure state”, not “do action”: ensure the service is running, not start it. Where an action cannot be idempotent (create, charge, send), use a caller-supplied request identifier recorded atomically with the change.^[24] Google’s idempotent fixes could run “every 15 minutes without fearing damage”, but “not all fixes were naturally idempotent”.^[1]

Dry runs, and their limits. In Ansible’s check mode, modules without support “report nothing and do nothing”, and Ansible “will not generate output for tasks that use conditionals based on registered variables (results of prior tasks)”.^[20] The trap is a read-only probe written as command or shell: check mode skips it unless it uses creates or removes, so a later when: has nothing real to test.^[27] Mark such probes check_mode: false; modules such as find already run in check mode.^[28] A clean dry run is evidence, not proof: list targets with before/after values (diff mode) and check the count.

Validating targets. Diskerase, S3 and Atlassian were input failures: an empty set read as “everything”, a mistyped input, IDs of the wrong type.^[1][9][11] Refuse empty lists; resolve every identifier and check its type; cap the count; keep a floor; show people names, not IDs. Review the data as well as the code: Atlassian’s review “focused on which endpoint was being called and how”.^[11]

Batches and failure budgets. Ansible aborts when max_fail_percentage is “exceeded, not equaled”.^[29] AWS Systems Manager lets running invocations finish after its error threshold and suggests concurrency 1 if the limit must hold; its user guide and CLI reference disagree on whether 10% of 50 stops at the fifth or sixth error, so test the semantics.^[22][30]

Kill switches, time limits and locks. Keep one switch per automation and a global one, independent of what the automation manages: Facebook’s out-of-band access failed with its backbone.^[10] AWS engineers disabled automatic NLB failover mid-incident and later “disabled the DynamoDB DNS Planner and the DNS Enactor automation worldwide”.^[12] Safeguards “might be as simple as command timeouts”:^[15] give every action a time limit, and take a lock so two runs cannot interleave.

Testing automation code. ShellCheck for shell;^[31] ansible-lint, whose no-changed-when rule flags command and shell tasks that cannot report change;^[32] Molecule, whose default sequence includes idempotence.^[33] Test each guard with the bad input it exists for (Lab 2, step 11): Facebook’s audit tool had the bug that let the command through.^[10]

None defines “AutomationOps”.

AI operations agents are automation whose plan is generated at run time, so these controls apply with more force. AWS describes its DevOps Agent (generally available 31 March 2026) as one of a class of “autonomous systems that work independently”.^[16] Azure SRE Agent’s documentation says “Review is the default mode”, yet its table lists Autonomous as the default for response plans and scheduled tasks: check each plan. In Review mode only Azure infrastructure operations get approval buttons; other actions “proceed based on the agent’s reasoning and your response plan instructions”. Approval and permissions are separate. An agent lacking a permission “requests temporary access through Microsoft Entra OBO flow”,^[17] so one privileged person’s consent widens its reach. Decide who may grant it.

Rules: give an agent the narrowest identity of any automation; start read-only; route its actions through the same guards and tiers; judge it by the metrics. Correlation and anomaly detection belong to AIOps.

Assumptions. A team automates a clean-up of temporary files on application servers and a few riskier actions, with the same controls whatever the executor. All names, accounts and hosts are placeholders.

1. Register entry (one file per automation, reviewed like code). Field names are this Atlas’s proposal.

Example status: YAML syntax-checked; a template, nothing to execute.

id: cleanup-tmp
purpose: Remove temporary files older than 7 days from app servers
owner: team-checkout-ops            # a team, plus a named person in your directory
service_owner: checkout
# tier: 0 read-only, 1 reversible single target, 2 reversible many, 3 irreversible
tier: 2
triggers: [schedule, on-call]
executor: ansible                   # or rundeck, ssm-automation, python-runner
identity: automation-cleanup-tmp    # its own identity; never a shared admin key
permissions: ["delete files under /var/tmp/app on hosts in group app_servers"]
limits:
  max_targets_per_run: 20
  batches: [1, "25%"]
  max_fail_percentage: 0
  kill_switch: /etc/automation/KILL-cleanup-tmp
dry_run_required: true
approval: none                      # tier 3 needs a second person
tests: [ansible-lint, molecule-idempotence, guard-tests]
fallback: page on-call; manual procedure RB-112
review_by: 2027-01-16
# status: candidate, built, tested, supervised, autonomous, frozen, retired
status: supervised

2. A guarded runner with every control, in standard-library Python; its “fleet” is a directory of JSON files, so it is safe anywhere (Lab 2). The lock and time limit are Unix-only, and the lock covers one machine.

Example status: verified by execution (Python 3.8.10, Linux, 2026-10-16) with the Lab 2 fleet.

#!/usr/bin/env python3
"""Guarded runner: one operational action with the safety controls built in.

Dry run by default; one run at a time (lock file); refuses empty, unknown,
wrong-kind, duplicate or oversized target lists; works in batches with a failure
budget, a time limit per target and a health check after each batch; checks a
kill switch before every batch; appends one JSON line per decision to an audit
log. Standard library only (Python 3.8+, Linux or macOS). The "fleet" is a
directory of JSON files, so nothing real changes.
"""
import argparse
import fcntl
import getpass
import json
import os
import signal
import sys
import time
import uuid

EXIT_OK, EXIT_REFUSED, EXIT_HALTED, EXIT_KILLED = 0, 2, 3, 4


def load_fleet(fleet_dir):
    fleet = {}
    for name in sorted(os.listdir(fleet_dir)):
        if name.endswith(".json"):
            with open(os.path.join(fleet_dir, name)) as f:
                host = json.load(f)
            fleet[host["id"]] = host
    return fleet


def check_targets(targets, fleet, kind, max_targets):
    """Return a list of reasons to refuse; an empty list means the targets are fine."""
    if not targets:
        # An empty list must never be read as "everything" (the Diskerase lesson).
        return ["target list is empty: nothing to do, refusing rather than guessing"]
    problems = []
    unknown = [t for t in targets if t not in fleet]
    if unknown:
        problems.append("unknown targets: " + ", ".join(unknown))
    wrong = [t for t in targets if t in fleet and fleet[t]["kind"] != kind]
    if wrong:
        # Right IDs of the wrong type are the classic deletion accident.
        problems.append("targets are not of kind '%s': %s" % (kind, ", ".join(wrong)))
    if len(targets) > max_targets:
        problems.append("%d targets exceed the blast-radius cap of %d"
                        % (len(targets), max_targets))
    if len(set(targets)) != len(targets):
        problems.append("duplicate targets in the list")
    return problems


def on_timeout(signum, frame):
    raise TimeoutError("time limit exceeded")


def cleanup(host):
    """Idempotent action: remove temporary files. Returns (changed, before, after)."""
    time.sleep(host.get("hang", 0))  # a real action can hang; the time limit stops it
    if host.get("fail"):
        raise RuntimeError("simulated failure on " + host["id"])
    before = host["tmp_files"]
    host["tmp_files"] = 0
    return before != 0, before, 0


def healthy(host):
    """Health check after a batch; a real one would probe the service."""
    return host.get("healthy", True)


def audit(path, record):
    # Append mode only. A local file is not tamper-proof: in production, ship
    # these lines to a store the runner's identity cannot rewrite.
    with open(path, "a") as f:
        f.write(json.dumps(record, sort_keys=True) + "\n")


def main(argv=None):
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--fleet", required=True, help="directory of <host>.json files")
    p.add_argument("--targets", required=True, help="file with one target id per line")
    p.add_argument("--kind", default="app",
                   help="the only kind of target this action may touch")
    p.add_argument("--apply", action="store_true", help="make changes (default: dry run)")
    p.add_argument("--max-targets", type=int, default=5, help="blast-radius cap per run")
    p.add_argument("--batch", type=int, default=2, help="targets per batch")
    p.add_argument("--max-failures", type=int, default=0,
                   help="failures tolerated before halting")
    p.add_argument("--timeout", type=int, default=10, help="seconds allowed per target")
    p.add_argument("--kill-switch", default="KILL",
                   help="if this file exists, stop before the next batch")
    p.add_argument("--lock", default="runner.lock",
                   help="lock file that allows one run at a time")
    p.add_argument("--audit", default="audit.jsonl",
                   help="JSON-lines audit log, opened in append mode")
    a = p.parse_args(argv)

    run_id = uuid.uuid4().hex[:8]
    mode = "apply" if a.apply else "dry-run"
    base = {"run": run_id, "actor": getpass.getuser(), "action": "cleanup", "mode": mode}

    lock = open(a.lock, "w")  # held until the process exits
    try:
        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except OSError:
        print("REFUSED: another run holds the lock " + a.lock)
        audit(a.audit, dict(base, result="refused", reasons=["locked"], ts=time.time()))
        return EXIT_REFUSED

    fleet = load_fleet(a.fleet)
    with open(a.targets) as f:
        targets = [line.strip() for line in f
                   if line.strip() and not line.startswith("#")]

    problems = check_targets(targets, fleet, a.kind, a.max_targets)
    if problems:
        for msg in problems:
            print("REFUSED: " + msg)
        audit(a.audit, dict(base, result="refused", reasons=problems, ts=time.time()))
        return EXIT_REFUSED

    signal.signal(signal.SIGALRM, on_timeout)
    failures = 0
    batches = [targets[i:i + a.batch] for i in range(0, len(targets), a.batch)]
    for n, batch in enumerate(batches, 1):
        if os.path.exists(a.kill_switch):
            print("KILLED: kill switch %s present before batch %d; %d batch(es) not run"
                  % (a.kill_switch, n, len(batches) - n + 1))
            audit(a.audit, dict(base, result="killed", batch=n, ts=time.time()))
            return EXIT_KILLED
        for t in batch:
            host = fleet[t]
            rec = dict(base, target=t, batch=n, ts=time.time())
            if not a.apply:
                would = "would change" if host["tmp_files"] else "unchanged"
                print("%-8s %-7s %s (tmp_files=%d)" % (mode, t, would, host["tmp_files"]))
                audit(a.audit, dict(rec, result=would))
                continue
            try:
                signal.alarm(a.timeout)
                try:
                    changed, before, after = cleanup(host)
                finally:
                    signal.alarm(0)
                with open(os.path.join(a.fleet, t + ".json"), "w") as f:
                    json.dump(host, f, indent=2)
                result = "changed" if changed else "unchanged"
                print("%-8s %-7s %s (tmp_files %d -> %d)"
                      % (mode, t, result, before, after))
                audit(a.audit, dict(rec, result=result, before=before, after=after))
            except (RuntimeError, TimeoutError) as e:
                failures += 1
                print("%-8s %-7s FAILED: %s" % (mode, t, e))
                audit(a.audit, dict(rec, result="failed", error=str(e)))
        if failures > a.max_failures:
            print("HALTED: %d failure(s) exceed the budget of %d after batch %d"
                  % (failures, a.max_failures, n))
            audit(a.audit, dict(base, result="halted", batch=n, failures=failures,
                                ts=time.time()))
            return EXIT_HALTED
        sick = [t for t in batch if a.apply and not healthy(fleet[t])]
        if sick:
            print("HALTED: health check failed after batch %d: %s" % (n, ", ".join(sick)))
            audit(a.audit, dict(base, result="halted", batch=n, unhealthy=sick,
                                ts=time.time()))
            return EXIT_HALTED
    print("DONE: %s on %d target(s), %d failure(s), run %s"
          % (mode, len(targets), failures, run_id))
    return EXIT_OK


if __name__ == "__main__":
    sys.exit(main())

3. The same controls in Ansible. With serial, the whole play, guards included, runs per batch; any_errors_fatal stops all hosts if one guard fails;^[21][29] ansible_play_hosts_all lists “all the hosts that were targeted by the play”.^[44] timeout limits each task action;^[45] with a budget of 0, the final health check stops the play on any unhealthy host. With several controllers, use a shared kill-switch flag.

Example status: YAML syntax-checked; not executed (no Ansible here). ansible.builtin modules only.

- name: Clean temporary files on app servers (guarded)
  hosts: app_servers
  gather_facts: false
  serial: [1, "25%"]            # one canary host, then quarters of the target set
  max_fail_percentage: 0        # any failed host in a batch aborts the rest of the play
  vars:
    max_targets: 20
    kill_switch: /etc/automation/KILL-cleanup-tmp
  pre_tasks:
    - name: Guards (a failure here stops every host)
      any_errors_fatal: true
      block:
        - name: Refuse an oversized target set
          ansible.builtin.assert:
            that: ansible_play_hosts_all | length <= max_targets
            fail_msg: >-
              {{ ansible_play_hosts_all | length }} hosts targeted;
              the cap is {{ max_targets }}

        - name: Read the kill switch on the control node
          ansible.builtin.stat:
            path: "{{ kill_switch }}"
          delegate_to: localhost
          run_once: true
          register: killswitch

        - name: Stop before this batch if the kill switch is present
          ansible.builtin.assert:
            that: not killswitch.stat.exists
            fail_msg: "Kill switch {{ kill_switch }} is present"
  tasks:
    - name: Find temporary files older than 7 days
      ansible.builtin.find:
        paths: /var/tmp/app
        age: 7d
      register: old_files

    - name: Remove them (absent stays absent, so a second run changes nothing)
      ansible.builtin.file:
        path: "{{ item.path }}"
        state: absent
      loop: "{{ old_files.files }}"
      loop_control:
        label: "{{ item.path }}"
      timeout: 60

    - name: Health check before the next batch (placeholder endpoint)
      ansible.builtin.uri:
        url: "http://{{ inventory_hostname }}:8080/healthz"
      delegate_to: localhost
      register: health
      retries: 3
      delay: 10
      until: health.status == 200

4. AWS Systems Manager. AWS-RestartEC2Instance does not reboot: it stops, then starts, each instance.^[46][47] A stop/start loses RAM and instance-store data. It also changes an automatically assigned public IPv4 address unless an Elastic IP is attached (as of October 2026).^[48] For a reboot, call RebootInstances from your own runbook. The script refuses an empty or oversized set and passes exactly the running instances it counted. Targeting the tag could also catch stopped instances, which the run would start (our inference; AWS warns only that “terminated instances might be included”).^[49] The run works one instance at a time and halts at the first error.^[30]

Example status: bash syntax-checked; run against a stub aws (empty, normal, oversized, malformed lists), not against AWS.

# Stop and start (not reboot) every RUNNING instance tagged app=checkout, one at a time.
# The run acts on exactly the IDs that were counted, not on the tag again.
set -euo pipefail
CAP=10
ROLE=arn:aws:iam::111122223333:role/automation-restart-checkout

raw=$(aws ec2 describe-instances \
  --filters "Name=tag:app,Values=checkout" "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].InstanceId' --output text)
mapfile -t ids < <(tr -s '[:space:]' '\n' <<< "$raw" | sed '/^$/d')

if (( ${#ids[@]} == 0 || ${#ids[@]} > CAP )); then
  echo "REFUSED: ${#ids[@]} running instances selected; allowed 1 to $CAP" >&2
  exit 2
fi
for id in "${ids[@]}"; do
  if [[ ! $id =~ ^i-[0-9a-f]{8,17}$ ]]; then
    echo "REFUSED: not an instance ID: $id" >&2; exit 2
  fi
done
echo "Stop/start, one at a time: ${ids[*]}"

aws ssm start-automation-execution \
  --document-name AWS-RestartEC2Instance \
  --target-parameter-name InstanceId \
  --targets "Key=ParameterValues,Values=$(IFS=,; echo "${ids[*]}")" \
  --max-concurrency 1 \
  --max-errors 0 \
  --parameters "AutomationAssumeRole=$ROLE"

No IAM action means “restart”: the role needs ec2:StopInstances and ec2:StartInstances, so it could also leave instances stopped. Scope both by resource tag, as in AWS’s example policies,^[50] and test this starting-point policy outside production. Encrypted volumes may need the KMS actions AWS lists for EBS encryption, such as kms:CreateGrant;^[51] an instance lacking “the required KMS key permissions” can return to stopped during a start.^[52] Whoever holds iam:PassRole for the role can use it.^[53]

Example status: JSON syntax-checked; not executed.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "StopStartCheckoutOnly",
      "Effect": "Allow",
      "Action": ["ec2:StopInstances", "ec2:StartInstances"],
      "Resource": "arn:aws:ec2:*:111122223333:instance/*",
      "Condition": {"StringEquals": {"aws:ResourceTag/app": "checkout"}}
    },
    {
      "Sid": "ReadInstanceState",
      "Effect": "Allow",
      "Action": "ec2:DescribeInstances",
      "Resource": "*"
    }
  ]
}

A tier-3 runbook adds a read-only plan, a second person and input patterns. aws:approve waits for approvers (default 7 days, maximum 30; up to 10 approvers; no multi-account or multi-Region runs); with allowedPattern, bad input means “the execution fails to start”.^[41][54] An approver can be a role, so anyone who can assume tenant-deletion-approvers can approve: let its trust policy name approvers only. tenantctl is a placeholder.

Example status: YAML syntax-checked; not executed.

description: Soft-delete one tenant after second-person approval (Atlas example)
schemaVersion: '0.3'
assumeRole: '{{ AutomationAssumeRole }}'
parameters:
  AutomationAssumeRole:
    type: String
    description: Role used only by this runbook
  InstanceId:
    type: String
    allowedPattern: '^i-(?:[a-f0-9]{8}|[a-f0-9]{17})$'
  TenantId:
    type: String
    # This pattern is what makes the substitution into shell commands below safe.
    allowedPattern: '^tnt-[a-z0-9]{8}$'
mainSteps:
  - name: plan
    action: aws:runCommand
    onFailure: Abort
    inputs:
      DocumentName: AWS-RunShellScript
      InstanceIds:
        - '{{ InstanceId }}'
      Parameters:
        commands:
          - /opt/tenantctl/bin/tenantctl show --tenant '{{ TenantId }}'   # read-only
  - name: approve
    action: aws:approve
    timeoutSeconds: 3600
    onFailure: Abort
    inputs:
      NotificationArn: arn:aws:sns:eu-west-1:111122223333:Automation-approvals
      Message: 'Soft-delete tenant {{ TenantId }}? Read the output of step "plan" first.'
      MinRequiredApprovals: 1
      Approvers:
        - arn:aws:iam::111122223333:role/tenant-deletion-approvers
  - name: softDelete
    action: aws:runCommand
    onFailure: Abort
    inputs:
      DocumentName: AWS-RunShellScript
      InstanceIds:
        - '{{ InstanceId }}'
      Parameters:
        commands:
          - >-
            /opt/tenantctl/bin/tenantctl soft-delete --tenant '{{ TenantId }}'
            --retain-days 30

5. Event-driven remediation with a velocity limit. Three settings keep one alert from reaching the fleet. First, a host limit: ansible-rulebook limits a playbook to a host only through event.meta.hosts. The webhook does not set it, so a filter copies the alerting host there.^[40] Second, a throttle: it acts on the first event and suppresses repeats per host until the window expires.^[39] Third, a fleet cap: per-host throttles do not cap the total, so the playbook refuses more than one host and counts restarts fleet-wide. Actions run one at a time by default, so runs do not race on the count.^[55]

The webhook binds to 0.0.0.0 by default, and its token and HMAC checks are optional:^[56] without them, anyone who can reach the port can fake an alert and restart workers. Bind to loopback behind an authenticating proxy and set a token. The builtin sources are documented “for testing and development”; check that your production source has the same settings.

Example status: YAML syntax-checked; not executed (needs ansible-rulebook and a workers inventory group).

# worker-rulebook.yml: run with  ansible-rulebook -r worker-rulebook.yml -i inventory.yml
- name: Restart a stuck worker on the alerting host only
  hosts: workers
  sources:
    - eda.builtin.webhook:
        host: 127.0.0.1               # loopback; an authenticating proxy sits in front
        port: 5000
        token: "{{ webhook_token }}"  # Bearer token from a secret store
      filters:
        - eda.builtin.insert_hosts_to_meta:
            host_path: payload.host   # sets event.meta.hosts, which limits the playbook
  rules:
    - name: Worker stuck, at most once per host per 15 minutes
      condition: >-
        event.payload.alert == "worker_stuck" and
        event.payload.status == "firing" and
        event.payload.host is match("worker-[0-9]{2}$")
      throttle:
        once_within: 15 minutes
        group_by_attributes:
          - event.meta.hosts
      action:
        run_playbook:
          name: playbooks/restart-worker.yml

# playbooks/restart-worker.yml: started by the rulebook, limited to one host
- name: Restart one stuck worker (guarded)
  hosts: workers
  gather_facts: false
  vars:
    fleet_cap_per_hour: 3
    marker_dir: /var/lib/automation/restart-worker   # on the rulebook host
    stamp: "{{ now(utc=true, fmt='%Y%m%dT%H%M%S') }}"
  pre_tasks:
    - name: Guards (a failure here stops the run)
      any_errors_fatal: true
      block:
        - name: Exactly one host, never the whole group
          ansible.builtin.assert:
            that: ansible_play_hosts_all | length == 1
            fail_msg: "{{ ansible_play_hosts_all | length }} hosts targeted; expected 1"

        - name: Make sure the marker directory exists
          ansible.builtin.file:
            path: "{{ marker_dir }}"
            state: directory
            mode: "0750"
          delegate_to: localhost

        - name: Count restarts anywhere in the fleet during the last hour
          ansible.builtin.find:
            paths: "{{ marker_dir }}"
            age: -1h
          delegate_to: localhost
          register: recent

        - name: Fleet-wide velocity limit; beyond it a person decides
          ansible.builtin.assert:
            that: recent.matched < fleet_cap_per_hour
            fail_msg: "{{ recent.matched }} restarts in the last hour: cap reached"
  tasks:
    - name: Restart the worker service
      ansible.builtin.service:
        name: worker
        state: restarted
      become: true
      timeout: 120

    - name: Record the restart for the fleet-wide count
      ansible.builtin.file:
        path: "{{ marker_dir }}/{{ stamp }}-{{ inventory_hostname }}"
        state: touch
        mode: "0640"
      delegate_to: localhost

Objective: rank candidates by net hours and risk tier. Prerequisites: Python 3.8+; no accounts, cost or risk; 30 minutes.

Example status: verified by execution (Python 3.8.10, 2026-10-16; re-run 2026-10-17). The candidate figures are illustrative.

#!/usr/bin/env python3
"""Score automation candidates: net hours returned, break-even and a risk tier.

Input: a JSON list of candidates (see candidates.json). Standard library only.
The tiers and thresholds are this Atlas's proposal, not a standard.
"""
import json
import sys


def tier(c):
    if c["read_only"]:
        return 0
    if not c["reversible"] or c["deletes_data"]:
        return 3
    return 2 if c["max_targets"] > 1 else 1


def score(c):
    manual = c["runs_per_month"] * c["manual_minutes"] / 60
    residual = c["runs_per_month"] * c["residual_minutes"] / 60
    net = manual - residual - c["maintenance_hours_per_month"]
    t = tier(c)
    if net <= 0:
        advice = "only for consistency or safety"
    elif c["needs_judgement"]:
        advice = "human-backed interface first"
    elif t == 3:
        advice = "automate with approval and soft delete"
    else:
        advice = "automate; supervised, then promote"
    months = c["build_hours"] / net if net > 0 else None
    return manual, net, months, t, advice


def main(path):
    with open(path) as f:
        candidates = json.load(f)
    print("%-20s %8s %6s %9s %4s %s"
          % ("candidate", "manual_h", "net_h", "breakeven", "tier", "advice"))
    for c in sorted(candidates, key=lambda c: -score(c)[1]):
        manual, net, months, t, advice = score(c)
        be = "%.1f mo" % months if months else "never"
        print("%-20s %8.1f %6.1f %9s %4d %s" % (c["name"], manual, net, be, t, advice))


if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "candidates.json")

[
  {"name": "restart-stuck-worker", "runs_per_month": 40, "manual_minutes": 15,
   "residual_minutes": 1, "maintenance_hours_per_month": 1.5, "build_hours": 40,
   "read_only": false, "reversible": true, "deletes_data": false, "max_targets": 1,
   "needs_judgement": false},
  {"name": "tls-cert-renewal", "runs_per_month": 6, "manual_minutes": 45,
   "residual_minutes": 5, "maintenance_hours_per_month": 1, "build_hours": 30,
   "read_only": false, "reversible": true, "deletes_data": false, "max_targets": 1,
   "needs_judgement": false},
  {"name": "fleet-config-change", "runs_per_month": 2, "manual_minutes": 180,
   "residual_minutes": 20, "maintenance_hours_per_month": 3, "build_hours": 60,
   "read_only": false, "reversible": true, "deletes_data": false, "max_targets": 200,
   "needs_judgement": false},
  {"name": "decommission-tenant", "runs_per_month": 4, "manual_minutes": 120,
   "residual_minutes": 30, "maintenance_hours_per_month": 2, "build_hours": 80,
   "read_only": false, "reversible": false, "deletes_data": true, "max_targets": 1,
   "needs_judgement": false},
  {"name": "capacity-review", "runs_per_month": 0.33, "manual_minutes": 240,
   "residual_minutes": 120, "maintenance_hours_per_month": 1, "build_hours": 40,
   "read_only": true, "reversible": true, "deletes_data": false, "max_targets": 1,
   "needs_judgement": true}
]

1. Save the script as score_candidates.py and the list as candidates.json; run python3 score_candidates.py candidates.json.
2. Replace the list with candidates from your toil log, including the minutes a person still spends per automated run.

Expected output (step 1, copied from the verified run):

candidate            manual_h  net_h breakeven tier advice
restart-stuck-worker     10.0    7.8    5.1 mo    1 automate; supervised, then promote
decommission-tenant       8.0    4.0   20.0 mo    3 automate with approval and soft delete
tls-cert-renewal          4.5    3.0   10.0 mo    1 automate; supervised, then promote
fleet-config-change       6.0    2.3   25.7 mo    2 automate; supervised, then promote
capacity-review           1.3   -0.3     never    0 only for consistency or safety

Validation: the fleet change pays back slowly (25.7 months) yet is worth automating for consistency; the capacity review never pays back. Troubleshooting: KeyError means a missing field; JSONDecodeError is often a trailing comma. Teardown: delete the files, or keep the list as your first register.

Objective: see each control work, then reproduce three real failure patterns. Prerequisites: Python 3.8+ and bash on Linux (step 10 uses flock from util-linux); the reference runner saved as guarded_runner.py. No cost; only the lab directory changes.

Example status: verified by execution (Python 3.8.10, bash, 2026-10-16; re-run 2026-10-17). Run ids and timings will differ.

#!/usr/bin/env bash
# Create a fake fleet of eight "hosts" as JSON files. Nothing outside ./fleet is touched.
set -euo pipefail
mkdir -p fleet
host() {  # id kind tmp_files fail [hang_seconds] [healthy]
  {
    printf '{"id": "%s", "kind": "%s", "tmp_files": %d, ' "$1" "$2" "$3"
    printf '"fail": %s, "hang": %d, "healthy": %s}\n' "$4" "${5:-0}" "${6:-true}"
  } > "fleet/$1.json"
}
host app-01 app 12 false
host app-02 app 0 false
host app-03 app 7 false
host app-04 app 30 false
host app-05 app 9 true
host app-06 app 3 false 30        # hangs for 30 seconds
host app-07 app 4 false 0 false   # unhealthy after a change
host db-01 db 5 false
printf 'app-01\napp-02\napp-03\napp-04\n' > targets.txt
ls fleet

1. Save the script as make_fleet.sh; run bash make_fleet.sh.
2. Dry run: python3 guarded_runner.py --fleet fleet --targets targets.txt.
3. Add --apply; then run it again.
4. Empty list (Diskerase): : > empty.txt; run with --targets empty.txt --apply.
5. Wrong type (Atlassian): printf 'app-03\ndb-01\n' > wrong.txt; run with --targets wrong.txt --apply.
6. Failure budget: rm -rf fleet && bash make_fleet.sh; printf 'app-05\napp-01\napp-03\napp-04\n' > fail.txt; use --targets fail.txt --apply.
7. Kill switch: rebuild the fleet, touch KILL, repeat step 3, then rm KILL.
8. Time limit: rebuild; printf 'app-06\napp-01\n' > slow.txt; run with --targets slow.txt --apply --timeout 2.
9. Health check: rebuild; printf 'app-07\napp-01\napp-03\n' > sick.txt; run with --targets sick.txt --apply.
10. One run at a time: rebuild; flock runner.lock sleep 5 & sleep 1; repeat step 3.
11. Guard tests: save the test below as test_guards.py beside the runner; run python3 -m unittest test_guards.

Example status: verified by execution (Python 3.8.10, 2026-10-17).

#!/usr/bin/env python3
"""Guard tests for guarded_runner.check_targets: one case per known-bad input.

Every postmortem that finds a new bad input adds a case here.
"""
import unittest

from guarded_runner import check_targets

FLEET = {"app-01": {"kind": "app"}, "app-02": {"kind": "app"}, "db-01": {"kind": "db"}}


def problems(targets, max_targets=5):
    return check_targets(targets, FLEET, "app", max_targets)


class GuardTests(unittest.TestCase):
    def test_empty_list_is_refused(self):  # Diskerase: empty must not mean "all"
        self.assertTrue(problems([]))

    def test_unknown_target(self):
        self.assertIn("unknown targets: app-99", problems(["app-01", "app-99"]))

    def test_wrong_kind(self):  # Atlassian: real IDs of the wrong type
        self.assertIn("targets are not of kind 'app': db-01", problems(["db-01"]))

    def test_duplicate_target(self):
        self.assertIn("duplicate targets in the list", problems(["app-01", "app-01"]))

    def test_over_the_cap(self):
        self.assertTrue(problems(["app-01", "app-02"], max_targets=1))

    def test_valid_list_passes(self):
        self.assertEqual(problems(["app-01", "app-02"]), [])


if __name__ == "__main__":
    unittest.main()

Expected output (copied from the verified run):

# step 2 (dry run)
dry-run  app-01  would change (tmp_files=12)
dry-run  app-02  unchanged (tmp_files=0)
dry-run  app-03  would change (tmp_files=7)
dry-run  app-04  would change (tmp_files=30)
DONE: dry-run on 4 target(s), 0 failure(s), run c942b43f
# step 3, second run (idempotence)
apply    app-01  unchanged (tmp_files 0 -> 0)
...
# step 4 (exit code 2)
REFUSED: target list is empty: nothing to do, refusing rather than guessing
# step 5 (exit code 2)
REFUSED: targets are not of kind 'app': db-01
# step 6 (exit code 3)
apply    app-05  FAILED: simulated failure on app-05
apply    app-01  changed (tmp_files 12 -> 0)
HALTED: 1 failure(s) exceed the budget of 0 after batch 1
# step 7 (exit code 4)
KILLED: kill switch KILL present before batch 1; 2 batch(es) not run
# step 8 (exit code 3, after about 2 seconds)
apply    app-06  FAILED: time limit exceeded
apply    app-01  changed (tmp_files 12 -> 0)
HALTED: 1 failure(s) exceed the budget of 0 after batch 1
# step 9 (exit code 3)
apply    app-07  changed (tmp_files 4 -> 0)
apply    app-01  changed (tmp_files 12 -> 0)
HALTED: health check failed after batch 1: app-07
# step 10 (exit code 2)
REFUSED: another run holds the lock runner.lock
# step 11 (exit code 0)
......
----------------------------------------------------------------------
Ran 6 tests in 0.000s

OK

Validation: echo $? gives 0, 0, 2, 2, 3, 4, 3, 3, 2 and 0 for steps 2–11; audit.jsonl has a JSON line per target or decision. In step 6 app-01 still changed after app-05 failed: the budget is checked between batches, the in-flight behaviour AWS also documents.^[22] In step 9 app-03 never ran. Remove any one of the five target checks in check_targets from a copy of the runner and exactly one test fails. Troubleshooting: FileNotFoundError means step 1 was skipped; create KILL where you run the runner. Teardown: rm -rf fleet targets.txt empty.txt wrong.txt fail.txt slow.txt sick.txt audit.jsonl runner.lock __pycache__.

Objective: prove each guard of the Ansible version. Prerequisites: Ansible; two to four disposable Linux containers or VMs in group app_servers, with old files in /var/tmp/app (touch -d '10 days ago' /var/tmp/app/old.tmp), each answering http://<host>:8080/healthz (for a lab, python3 -m http.server 8080 in a directory holding a healthz file). Risk: it deletes files; never use real servers. Status: not executed in this environment.

1. ansible-playbook -i inventory.ini cleanup-tmp.yml --check --diff: the plan. find runs normally in check mode, so the removal task reports each old file as changed, and nothing is removed.^[28][59]
2. Run without --check; then run again. Expected: the second PLAY RECAP shows changed=0 for every host.
3. Set max_targets: 1 and run: the assert fails on the first batch and no host changes.
4. touch /etc/automation/KILL-cleanup-tmp on the control node and run: the play stops before the first batch; remove the file.
5. Stop the health endpoint on one host and run: the play stops after that host’s batch.
6. Add ansible-lint and a Molecule scenario to CI; Molecule’s default test sequence includes the idempotence step.^[33]

Expected output and validation: as stated in each step; the log names the guard that stopped the run. Troubleshooting: if step 1 lists no files, check the path and file ages; a command probe in place of find would be skipped in check mode. Teardown: destroy the containers or VMs.

The first three are illustrative; the fourth draws on Atlassian’s published review.

Pitfalls. Self-service inputs and alert webhooks are an attack surface; remediation can itself remove capacity. Atlassian’s sites were deleted “via a standard workflow”, so monitoring saw nothing.^[11]

No standard set exists; this is this Atlas’s proposal, with one row adapted from DORA.

Toil share is defined in SRE.

Leading and lagging: fallback rate, guard trips and ownership coverage lead; induced incidents lag. Vanity traps: counting automations; “hours saved” without maintenance. Minutes are estimates: revisit them yearly.

• Identity is the blast radius. One least-privilege identity per automation; NIST’s AC-6 covers “processes acting on behalf of users”.^[34] Check whose rights a run uses (trust boundaries) and harden service-role trust policies against the “confused deputy problem”.^[60]
• Inputs are an attack surface. Authenticate every trigger, webhooks included;^[56] constrain parameters with patterns, and for SSM command documents prefer interpolationType: ENV_VAR to string substitution.^[54] Secrets come from a secrets manager at run time.
• Approvals need separation and scope. Requester and approver differ (“two-person control”^[34]); whoever can assume an approver role can approve;^[41] the approval must cover the action taken.^[17]

These controls bind only where adopted. Mapping is covered in ComplianceOps, change approval in ITOps.

Maintenance is the lasting cost, because automation must change with the systems under it (“bit rot”^[1]). Add the runner platform (fees change: AWS ended the Systems Manager Automation free tier for new customers on 14 August 2025 and for existing ones on 31 December 2025^[43]), identities and on-call time. Capacity: “even read operations, naively implemented, can spike device load”, and at scale “these safety checks can eventually dominate workload”.^[15] Unit view: cost per automated run (platform share plus residual minutes) against cost per manual run. Sustainability is rarely material (FinOps covers idle resources).

The last row follows Bainbridge: “physical skills deteriorate when they are not used”.^[13] Facebook credited its “storm” drills with its careful 2021 restart.^[10]

flowchart TB
  A["Alert or report: unexpected changes"] --> B["Pull the kill switch; disable triggers"]
  B --> C{"All runs stopped?"}
  C -->|"no"| D["Revoke or deny the automation's identity"]
  D --> C
  C -->|"yes"| E["Scope damage from the audit log"]
  E --> F["Restore: soft delete, backup or roll forward"]
  F --> G["Re-enable as supervised, after fix and guard test"]
  G --> H["Postmortem; demote tier; update register"]
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class A,B,D risk
  class C decision
  class E,F,G,H core

1. Stop (on-call, minutes): kill switch, then schedules and event rules; if anything still runs, deny its identity.
2. Confirm: no new audit records for the run id.
3. Scope (owner, on-call): every target touched, with before/after values; a log that cannot say is the first action item.
4. Restore (service owner): soft delete, backup or a reviewed roll-forward; never reverse a partial change blindly.
5. Re-enable (owner, approved by the service owner): supervised, after a test that reproduces the failing input.
6. Learn: blameless postmortem (IncidentOps). Atlassian: “people don’t cause incidents. Rather, systems allow for mistakes to be made.”^[11]

• The script in a home directory (Google’s second level^[1]): one owner, no tests, no audit. Register it and use a runner.
• Automating the mess. “You rarely want to literally transcribe a human workflow into a machine workflow.”^[15]
• One god credential shared by all automation: any bug has full reach.
• Approval for everything. Approvers stop reading; keep approvals for tier 3 (editorial judgement).
• Auto-remediation as a hiding place. Nightly restarts of a leaking service silence the alert and keep the defect.
• Myth: once automated, done. Automation rots as systems change and is most fragile when rarely run.^[1]

Use it when operational actions repeat, are understood and touch production, especially when several people or systems trigger them or an auditor will ask who changed what.

Do not automate (yet) when the task is rare and needs judgement (give people a checklist); when the need can be removed (fix the leak; build failover into the system^[1]); when the process is not understood or agreed; when nobody will own the result; or when the triggering signal is unreliable (fix the signal first).

Hidden responsibilities: maintenance, an identity to secure, an on-call path for its failures, drills and eventual retirement.

flowchart TB
  Q1["1 Repeats, understood?"] -->|"no"| M["Manual checklist"]
  Q1 -->|"yes"| Q0["2 Need removable?"]
  Q0 -->|"yes"| FIX["Fix at source"]
  Q0 -->|"no"| Q2["3 Judgement each time?"]
  Q2 -->|"yes"| HB["Human-backed interface"]
  Q2 -->|"no"| Q3["4 Pays back, or must be consistent?"]
  Q3 -->|"no"| NO["Not now"]
  Q3 -->|"yes"| AUTO["Automate; tier sets controls"]
  M ~~~ FIX
  FIX ~~~ HB
  HB ~~~ NO
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q1,Q0,Q2,Q3 decision
  class FIX,AUTO core
  class M,HB,NO neutral

The tree assumes that removing work beats automating it. How much ceremony follows is decided by the tiers: irreversibility, not frequency, sets the controls.

This Atlas’s model, not an industry standard; it measures governance, not the amount of automation.

The typical stall is level 2: scripts in Git, no owners, one shared credential. The register is the way out.

Quick wins: the empty-list guard; a kill switch for the automation that can delete data; a dead-man alert on certificate renewal.

Quick start

• Every automation that changes production is registered, with owner and tier.
• No automation uses a shared admin credential.
• The riskiest automation refuses empty and oversized target lists.
• Tier 2 and 3 automations have a kill switch that on-call can find.

Readiness review (preflight)

• Candidate scored; owner, service owner and tier agreed.
• Idempotence shown by a second run with no changes.
• Guard tests pass for empty, unknown, wrong-type, duplicate and oversized inputs.
• Dry run lists targets with before/after values.
• Time limits set; for tier 2, also a lock, batches, a failure budget and health checks between batches.
• Identity scoped; secrets from a manager.
• Manual fallback works; kill switch tested and timed.

Before a tier-3 run

• The plan shows names and the count matches the request.
• Soft delete or a tested restore exists.
• Approver differs from requester; on-call is told.
• The first stage targets your own resources.

Quarterly governance audit

• Owners and review dates current; frozen automations re-owned or retired.
• Register reconciled with credentials and schedulers.
• Audit sample: targets matched requests; approvals separated.
• Rising fallback rates have work scheduled; drills done.

Continuous improvement

• Every frequently firing remediation has a defect ticket; every automation postmortem adds a guard test.

Example status: templates; nothing to execute.

AUTOMATION READINESS REVIEW
Automation id / version:            Owner:              Service owner:
Purpose (one sentence):
Tier (0-3) and why:
Triggers (schedule / event / person):        Who may trigger:
Identity and permissions (exact actions, resources):
Target selection: source of the list; validation rules; cap; floor:
Dry run evidence (link):                     Idempotence evidence (link):
Time limit / lock / batches / failure budget / health check between batches:
Approval (tier 3): approver group; requester excluded?  yes / no
Kill switch: location; tested on (date); time to stop:
Fallback: who is paged; manual procedure (link); last practised (date):
Metrics: success, fallback, induced incidents; dashboard (link):
Review date:                Decision: approve / approve supervised only / reject
Reviewers and date:

KILL-SWITCH DRILL RECORD
Date / automation / environment:
Scenario (e.g. bad input reached batch 2):
Switch pulled by / at (time):        Last in-flight action finished at (time):
Time to stop:             Targets changed after the switch:
Did on-call find the switch without help?  yes / no
Problems found and actions (owner, due date):

AUTOMATION RETIREMENT NOTICE
Automation id:               Owner:               Date:
Reason (cause fixed at source / unused for N days / replaced by):
Triggers removed:  yes / no      Identity and secrets revoked:  yes / no
Code archived at:            Register status set to retired:  yes / no
Dependent runbooks and dashboards updated:  yes / no

Is AutomationOps a job title or a certification? We found no certification or standards body using the term for IT operations (searched 2026-10-16). SREs, platform and operations engineers do the work, sometimes in an automation centre of excellence.

Isn’t this just SRE? SRE defines and caps toil (SRE); this page makes the automation that removes toil safe and owned. Small teams do both.

Should every automation need an approval? No. Approvals add information only for irreversible or wide-reaching actions; elsewhere they train people to click. Use the tiers.

Our operational jobs run from CI pipelines. In scope? Yes, if they change production. Scope the pipeline’s credentials to the job; pipeline identity is covered in DevSecOps; infrastructure-as-code runs follow InfraOps.

How much autonomy should auto-remediation have? As much as its tier and record allow, always with a velocity limit; see the judgement call.

• Fighting control loops. In AWS’s DynamoDB DNS system a one-time freshness check “was stale by this time”, so an older plan overwrote a newer one.^[12] Prefer one writer, versioned plans and compare-and-set at write time; durable engines are in WorkflowOps.
• Global change budgets and cells. Cap how much capacity all automation may remove at once (AWS’s announced NLB velocity control caps one mechanism^[12]); automation that acts one cell at a time inherits the cell’s limit, which is why AWS refactored parts of S3 “into smaller cells to reduce blast radius and improve recovery”.^[9]
• Verification before execution is mature in networking (NetworkOps); elsewhere, dry runs and guards dominate.
• AI agents (emerging): approval modes and permission scopes exist;^[17][16] we found no independent evidence on their production error rates.

Situation (illustrative). A checkout database has a primary and a replica, usually under a second apart. Automated failover takes about 30 seconds; a human-approved one at night about 15 minutes. Last year the health check raised 12 alarms, 2 of them real primary failures.

The attractive choice and why it can be wrong. A wins on minutes, but failover can lose acknowledged writes (tier 3), a flapping check can trigger repeated failovers, and the signal is wrong 10 times in 12. B costs half an hour of outage a year and night pages for alarms that are usually false.

Recommendation. Fix the signal first. Then choose A with conditions: fail over automatically only if replication lag, checked at the moment of action, is within the agreed data-loss limit, the old primary can be fenced, and no automatic failover happened in the last few hours; otherwise fall back to B. Uncertainty: the figures are illustrative; engine mechanics are in DBOps, invoking DR in DisasterRecoveryOps.

After this page you should be able to explain the label and this page’s reading of it; score candidates; design guards, batches, failure budgets and kill switches; govern with a register, identities and metrics; and evaluate when not to automate. Self-check:

1. A decommission script takes its server list from an inventory query, which returns nothing. What should the script do?

2. A colleague says a playbook is safe because --check showed no errors. What is missing?

3. An SSM Automation run targets 50 instances with --max-concurrency 10 and --max-errors 10%. How many could fail?

4. A task runs 10 times a month, takes 30 minutes by hand, would need 5 minutes of human time per automated run and 2 hours of maintenance a month, and 50 hours to build. Automate it?

5. A nightly job deletes customer records past retention across all tenants. Approving every run would be tedious. How do you govern it?

AutomationOps is a borrowed name: RPA vendors use it, and this Atlas applies it to something older, keeping operational automation safe and owned. In each incident above, automation did what it was told; it was told something wrong (an empty list, a mistyped input, the wrong IDs, a stale plan) and nothing stopped it in time.

The discipline is simple to state: remove what need not exist, automate what repeats and is understood, give every run the charter’s brakes, including a kill switch someone has actually pulled, and give each automation an owner, an identity and a retirement date. Keep people able to do the work by hand: the day the automation fails is the day they will need to.

Sources were opened on the date each footnote gives; dates are the source’s own. Claim-level support is in research/automationops-ledger.md. Re-check by 2027-04-17, then every six months: agent modes and defaults, Systems Manager pricing, Rundeck and StackStorm releases; open the SEC and Red Hat pages in a browser (they block some automated clients).

Changelog. 2026-10-16, r0: first draft; runner and scoring labs executed; other examples syntax-checked only. 2026-10-17, r1 (answers reviews/automationops-r1.md): safer Event-Driven Ansible and AWS examples, corrected check-mode advice, runner with lock, time limits and health checks (Lab 2 re-run), two diagrams redrawn, dates, paraphrases and NIST mapping fixed; 14 sources added, all checked 2026-10-16. 2026-10-17, post-score polish: IAM policy caveats (2 sources added); tag-targeting inference labelled; Autonomous → Retired edge; Lab 1 and register fit the column; executed guard tests in Lab 2; sentences split; re-check cadence.