ChatOps

ChatOps is running operational work through a shared, persistent chat room. Bots post the events a team needs to see (a deploy started, an alert fired, a pull request is stale) and run a short list of approved commands when someone types them (/ops status checkout, /deploy api v2.3.1 to production). Because the request, the result and the discussion all happen in one place, everyone in the room can see what was done, by whom, and what happened next.

The idea is simple and old. The hard part is that a chat command is a remote control for production held inside a messaging product that many people, integrations and now AI assistants can write to. Good ChatOps is therefore mostly about two things this page concentrates on: the security model of bots and commands, and the usability of the channel (signal versus noise).

Where to start. Beginners: the glossary and Lab 1. Engineers: the reference architecture, identity and trust and Lab 2. Leaders: the decision framework and the judgement call. Product facts are dated October 2026.

Without ChatOps, operational work is invisible until it breaks. People ask the same questions all day; a slide in GitHub’s 2013 talk lists questions its author had stopped asking once the work moved into chat, such as “how’s that deploy going?”, “are you deploying that or should i?” and “is anyone responding to that nagios alert?” ^[1]. Knowledge stays in the heads of the people who ran the commands, and new staff learn by asking.

ChatOps addresses four failures:

• Invisible change. At GitHub in 2015, “All deployments happen in chat via Hubot commands, which ensures that everyone in the company (from development to operations to support) has visibility into changes that are being pushed into production.” ^[7]
• Learning by asking. Newland’s abstract credits chat commands with increasing “the speed at which new team members learn common practices” ^[1]; a slide quotes @rtomayko on “teaching by doing by making things visible”.
• Inconsistent manual steps. A command encodes guards that a person forgets: GitHub’s deploy command refused to run until CI passed, merged the latest master first, and only ran from designated rooms ^[7].
• Scattered incident context. An incident channel with bots that page people, post status changes and record key actions keeps one shared timeline ^[4][8].

It also creates two new problems, which the rest of this page treats as first-class: noise (chat becomes the new place where alerts nobody reads pile up) and privilege concentration (one bot token can do what dozens of engineers can do).

The cost of noise (illustrative). A team channel receives 400 bot messages a week and each one costs ten seconds of attention for each of 12 members who glance at it: 400 × 12 × 10 s ≈ 13 hours of attention a week, before anyone acts on anything.

• Not “a bot in Slack”. A bot that posts jokes or lunch menus is chat automation, not operations. Jason Hand also notes that “ChatOps doesn’t necessarily require a bot” ^[2]: a team that coordinates deploys in a shared channel is already doing part of it.
• Not a replacement for pipelines, runbook automation or IT service management (ITSM). The command is a front door; the work is done by CI/CD (continuous integration and delivery), a runbook engine or a cloud API behind it (AutomationOps).
• Not incident management. Incident roles, severity and communications belong to IncidentOps; ChatOps provides the channel tooling.
• Not the same as “conversational AI for IT”. Vendors market AI chat assistants for operations. They can be one part of ChatOps, under the same rules as any other bot (Automation and AI).
• Misconception: chat history is an audit trail. On Slack, for example, the Audit Logs API exists only on Enterprise plans and records actions, not message content ^[9]; messages can be edited, deleted or expire under retention rules. Keep a separate audit record.

• 2011. GitHub open-sourced a rewrite of Hubot, its Campfire chat bot, which “ships our code” and “is the interface to our CI server”; it had adapters for Campfire and IRC ^[10].
• 2013. Jesse Newland’s talk “ChatOps at GitHub” (slides dated 7 February 2013): “Hubot, GitHub’s open source chat bot, is completely revolutionizing how we do ops at GitHub.” One slide reads “By placing tools directly in the middle of the conversation” ^[1]. We did not find an earlier documented use of the word.
• 2015–2016. GitHub described deploy queues, guards and locks run from chat ^[7]; Atlassian, then a chat vendor, defined ChatOps as “a collaboration model that connects people, tools, process, and automation into a transparent workflow” ^[11]; O’Reilly published Jason Hand’s report ^[2][12].
• 2018–2026: platform churn. Atlassian discontinued Hipchat and Stride after a deal with Slack ^[13]; Slack stopped legacy custom bots on 31 March 2025 ^[14]; Microsoft scheduled the shutdown of Office 365 Connectors in Teams for 18–22 May 2026 ^[15]; Atlassian will shut down its incident tool Opsgenie on 5 April 2027 and delete customer data that has not been moved ^[16]. Each change broke, or will break, integrations built on the retired layer.
• 2021. GitHub moved deploy state out of chat into a state machine with a UI, while still starting deploys from Slack ^[17].
• 2024–2026: AI in the channel. Chat platforms and cloud providers added assistants built on large language models (LLMs), and researchers published prompt-injection attacks against them, such as PromptArmor’s against Slack AI in 2024 ^[18]; in February 2026 Slack announced a Model Context Protocol (MCP) server for AI agents ^[19].

The lasting lesson: keep logic out of the chat adapter. GitHub went from Hubot scripts, to a shell bridge where “Security contexts were often mixed up with each other”, to small RPC endpoints mapped one-to-one to chat commands ^[20].

Mental model: three surfaces, one trust chain. ChatOps has three surfaces: notifications (machines talk to people), commands (people ask machines to act) and conversation (people coordinate around both). The trust chain runs from a chat message to a production API: who typed it → is the request genuine → may this person do this, here, now → who else must agree → what exactly ran → where is it recorded.

ChatOps is an interface layer: it sits between people and the systems other pages describe. The boundaries below are editorial distinctions chosen by this Atlas; no standard defines them.

%%{init: {"flowchart": {"wrappingWidth": 170, "nodeSpacing": 26, "rankSpacing": 42}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  IN1["ObservabilityOps: which alerts exist and fire"]:::neutral
  IN2["IdentityOps, IAMOps, AccessOps: who people are, what they may do, elevation"]:::neutral
  C["ChatOps: notifications, commands and channels as an operational interface"]:::core
  OUT1["AutomationOps: the runbooks and scripts a command triggers"]:::neutral
  OUT2["IncidentOps: roles and process inside incident channels"]:::neutral
  OUT3["KnowledgeOps: turning chat answers into maintained runbooks"]:::neutral
  IN1 -->|"alerts to deliver"| C
  IN2 -->|"identity and policy"| C
  C -->|"invokes"| OUT1
  C -->|"hosts"| OUT2
  C -->|"feeds"| OUT3

No official ChatOps manifesto exists. The charter below is this Atlas’s proposal, distilled from the sources cited in each principle.

Mission. Make routine operational work visible, safe and teachable by doing it where the team already talks, without making chat the place where control or records live.

1. Visible by default. Commands and their results go to shared channels. The first anti-pattern Jason Hand names is that people “treat chat like email”: direct-message conversations are “essentially black-boxed” ^[2]. Keep direct messages for help text and refusals.
2. Decide outside the adapter. The chat adapter only translates; a gateway or the service behind it decides. GitHub’s protocol states that “CRPC servers are responsible for ensuring that the given user has the proper authorization to perform an operation.” ^[24]
3. Identity comes from the platform, never from text. Use the immutable user ID in a verified request; Slack deprecated user_name and tells developers to rely on IDs ^[25].
4. Allow-list typed commands. Every command has a name, typed arguments and a risk tier. Never pass free text to a shell.
5. Friction in proportion to risk. Reads are instant; reversible writes need the right role in the right channel; high-impact writes need a second person (dual authorization); emergency overrides exist but are loud, as GitHub’s /deploy! was ^[7].
6. Least privilege twice. Limit the bot’s own scopes and limit the credentials its executors use.
7. Audit outside chat. Write a structured record of every decision, allowed or refused, to a store the chat platform cannot edit.
8. Every notification is a contract. It has an owner, an audience and an expected action, or it goes to a dashboard instead.
9. Degrade gracefully. Plan for the chat platform or the bot being down during the incident you need it for.

Non-goals: replacing deployment UIs or ITSM records; giving everyone in a channel production rights; automating decisions that need judgement.

Most ChatOps design decisions follow from risk tiers: they decide which controls a command needs.

AutomationOps numbers its automation tiers differently; when a command starts an automation, apply the stricter of the two tiers (an editorial rule both pages share).

The building blocks: a chat platform (Slack, Microsoft Teams, Mattermost and others), an adapter or software development kit (SDK) (Hubot, Errbot, Bolt, Teams SDK), a command gateway with a catalogue and identity map, an approval mechanism, executors (CI/CD, runbook engines, cloud APIs), a notification router, an audit sink and a secret store. They are arranged in the reference architecture.

ChatOps has no job title of its own. The work is shared between the team that runs the bot as a product and the teams that own commands.

R = responsible, A = accountable, C = consulted, I = informed. Approvers are a named group (for example on-call leads), never “anyone in the channel”.

• Day 0 (design). A command proposal (see templates) names the owner, the user problem, arguments and their validation, risk tier, executor, rollback and the channels where it may run. Tier 3 proposals get a security review.
• Day 1 (rollout). The command ships behind the gateway in a test channel, often in dry-run mode first, with help text. Its first production use is announced.
• Day 2 (operate). The owner watches usage, failures and refusals; refusals are as informative as errors (a role is missing, or people use the wrong channel). Unused commands are deprecated, because every command is attack surface. Tokens are rotated on a schedule.

• Policy as code. The catalogue, identity mapping and notification routes live in a repository; changes go through review, with tier-3 commands owned by named reviewers.
• Channel taxonomy. Separate channels by purpose: #deploys-<service> for change, #alerts-<team> for notifications, #inc-<date>-<slug> for incidents, #ops-sandbox for testing. Commands are allowed per channel ID, not per name (where?).
• Cadence. Weekly glance at refusals and failures; monthly noise review of each alert channel (metrics); quarterly review of app scopes, identities and approver groups.
• Escalation. If the bot or chat platform is down, responders fall back to documented phone numbers, the pipeline UI and the runbook engine directly; PagerDuty’s paging bot shows the pattern (Case 2).

The design below separates translation (adapter), decision (gateway) and action (executors), as GitHub’s Chatops RPC does ^[20]. Notifications take a separate path, because they need routing and grouping rather than authorisation.

%%{init: {"flowchart": {"wrappingWidth": 170, "nodeSpacing": 26, "rankSpacing": 40}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  subgraph SaaS["Chat platform (vendor)"]
    U["People in channels"]:::neutral
    P["Platform API and events"]:::neutral
  end
  subgraph Yours["Your trust boundary"]
    M["Monitoring and CI events"]:::neutral
    R["Notification router: group, route, update"]:::core
    A["Adapter: Socket Mode or verified HTTPS; tokens from a secret store"]:::core
    G["Gateway: identity map, policy, approvals"]:::core
    L["Append-only audit log"]:::core
  end
  X["Executors, a separate privilege zone: CI/CD API, runbook engine, cloud API with short-lived credentials"]:::neutral
  U --> P
  P -->|"signed command"| A
  A --> G
  G --> L
  G -->|"scoped call"| X
  M --> R
  R -->|"messages"| P

Design choices and trade-offs

• Socket Mode or HTTPS (compared in deep comparisons). Either way, Slack expects a command to be acknowledged within 3,000 ms ^[25], so verify the request, acknowledge, and do the work afterwards.
• Build or buy the gateway. Runbook engines with chat front ends (StackStorm, for example) give aliases, role-based access control (RBAC) and approvals out of the box ^[28][29][30]; cloud chat integrations give command access to one provider, bounded by identity and access management (IAM) roles and guardrails ^[31]. A small custom gateway, like Lab 2, is reasonable when you have a handful of commands.
• Where replies go. Slack’s response_url can be used five times within 30 minutes and “will bypass any channel posting permissions” ^[32]; use it for replies to the invoker, and the normal API for announcements (Lab 3 does this).

This is the hardest part of ChatOps to get right, and the source of most controls on this page. Answer six questions for every command.

1. Is the request genuine? Verify the platform’s own proof of origin before doing anything else, and refuse requests that could be replays. The mechanism differs by platform (table below); on Slack it is an HMAC signature over the raw body plus a timestamp, and Slack’s example rejects requests more than five minutes from local time because “It could be a replay attack” ^[22]. Lab 1 does this.
2. Who is it? Map the platform’s immutable user ID (Slack user_id; on Teams, the Microsoft Entra object ID) to a corporate identity: from a provisioning feed, a maintained table, or the user’s verified email (on Slack, reading email needs the users:read.email scope ^[33]). Never trust display names. In channels shared with other organisations, check the workspace, organisation or tenant ID as well; Slack payloads carry team_id and enterprise_id ^[25].
3. Whose authority runs the action? Bots usually act with one powerful identity, which creates a confused deputy. In StackStorm, ChatOps executions run as the user configured in hubot (by default chatops_bot), so RBAC applies to the bot unless you pass and check the chat user ^[29][28]. Amazon Q Developer in chat applications instead combines a channel or per-user IAM role with channel guardrails: “What your users can do is the intersection of your guardrail policies and what is allowed by their roles.” ^[31]
4. Where? Restrict commands to channels whose members are the right audience, and match channels by ID, never by name: Slack says to rely on IDs “as the names might change arbitrarily” ^[25], and where members can create channels, anyone can create one whose name fits a pattern such as #inc-*. Let the declare command register each incident channel’s ID in the gateway, with an expiry. Read restrictions carefully: Errbot’s allowrooms means “Allow command only in these rooms (and direct messages)” unless you also turn off private use ^[34].
5. With which arguments? Validate each argument against a type or pattern and reject the rest. StackStorm passes extra key=value pairs not in the alias format (“it will still be extracted and supplied for execution”); pin what users must not change with immutable_parameters ^[28].
6. Who else agreed? For tier 3, a different person from the approver group, in the same channel, within a time limit, once, having seen exactly what will run. If you use buttons, check the clicker’s identity on the server; never treat an emoji reaction as an approval.

The gateway can be the same on every platform; the adapter’s trust checks cannot. Status as of October 2026, from each vendor’s documentation; check the current pages before you build.

Consequences: retries, and paths without a signed freshness check, mean the gateway must refuse duplicates itself (an idempotency key per request). Where the token does not cover the body, as on Mattermost, anyone holding it can forge any user_id (our reading of the documented scheme), so keep it in the secret store and never log raw request headers.

%%{init: {"sequence": {"actorMargin": 24, "width": 120, "messageMargin": 30, "boxMargin": 6, "noteMargin": 6, "diagramMarginX": 8}}}%%
sequenceDiagram
  participant P as People in channel
  participant C as Chat platform
  participant G as Gateway
  participant X as Executor
  P->>C: 1. Alice: /ops deploy … prod
  C->>G: 2. Signed request
  Note over G: 3. Verify origin and age
  G-->>C: 4. Ack (or error)
  Note over G: 5. Map ID, check policy
  alt Refused
    G-->>C: 6a. Refusal, audited
    C-->>P: Shown to Alice only
  else Tier 3
    G->>C: 6b. PENDING: what, where, who
    C->>P: Visible to the channel
    P->>C: 7. Bob: approve req-0001
    C->>G: 8. Signed approval, verified
    G->>X: 9. Scoped call
    X-->>G: 10. Result
    G->>C: 11. Result in thread, audited
  end

Exception paths to design for: the executor times out after the acknowledgement (post “still running” and a link); the approval expires (the request is cancelled and audited); the chat platform is down (fall back to the pipeline UI and record what was done outside the gateway, so the audit trail can be completed later); the command partially succeeded (the reply says exactly what changed and how to roll back).

Notifications are where ChatOps most often degrades. The SRE book’s monitoring chapter notes that email alerts are sometimes called “alert spam” because people seldom act on them ^[42]. Editorial interpretation: an alert channel nobody reads is today’s version of that email alias. Which alerts should exist, and the evidence on alert fatigue, belong to ObservabilityOps; deduplication, grouping and inhibition to AIOps. This section covers only what happens once a notification is on its way into chat.

%%{init: {"flowchart": {"wrappingWidth": 160, "nodeSpacing": 24, "rankSpacing": 38}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  EV["Alert or event, with owner and runbook labels"]:::neutral
  GR["Group and deduplicate; suppress symptoms of a known cause"]:::core
  SEV{"Needs a human now?"}:::decision
  PG["Page on-call, and one feed message (bot token)"]:::risk
  ACT{"Needs action this week?"}:::decision
  TM["Team channel via webhook, no mentions, no resolved post"]:::core
  DB["Dashboard or daily digest, not chat"]:::neutral
  RS["Resolved: bot edits the feed message"]:::core
  EV --> GR --> SEV
  SEV -->|"yes"| PG
  SEV -->|"no"| ACT
  ACT -->|"yes"| TM
  ACT -->|"no"| DB
  PG --> RS

Rules that follow from it:

• One channel per audience, not per tool. A team should not have to watch five channels to know its service is healthy.
• Group before you post. Slack accepts about one message per second per channel, and over-limit bursts may not be displayed ^[27], so an ungrouped storm loses messages exactly when they matter. Grouping and inhibition are taught in AIOps.
• Update, do not append. Since version 0.32.0 Alertmanager can edit an existing Slack message when an alert changes state ^[43]; this needs a bot token, because “Webhook URLs do not support updates” ^[44]. Set the token in http_config (as below) or, since 0.30.0, in app_token_file with app_url ^[43]. Choose per receiver: a webhook is simpler and bound to one channel, a bot token can edit but is a broader credential.
• Mentions are for pages only. @here on an informational alert teaches people to mute the channel.
• Every message carries an owner and a next step (runbook link, dashboard, command), and never secrets or personal data. Paging and escalation policy belong to IncidentOps.

Playbook: designing a command.

1. Start from a repeated manual task, or a time-critical incident action, with an API behind it (decision framework); name it verb-first (deploy, status, silence).
2. Define each argument’s type or pattern and a safe default (staging, not production).
3. Echo the parsed arguments before acting on anything above tier 1 (“Deploying checkout v1.4.2 to production”).
4. Make it idempotent or refuse duplicates; give it a --dry-run where the executor supports one.
5. Reply in a thread; link to the pipeline or dashboard for rich output instead of pasting it.
6. Write help text and an example; list the command in the catalogue with its tier, channels and roles.
7. Acceptance: an unauthorised user, a wrong channel and malformed arguments are each refused with a useful message, and all three appear in the audit log. Rollback: a documented inverse command or pipeline action.

Playbook: coordinating deploys in chat (pattern from GitHub’s 2015 practice ^[7]). Queue (queue me), guard (CI green, branch up to date, right room), lock the environment while a change is verified, unlock on merge and tell the next person, and allow a loud emergency override.

Playbook: incident channel tooling (process owned by IncidentOps). A declare command creates the channel, registers its ID with the gateway, invites on-call, posts links and starts a timeline (Case 2); a scribe or bot records key actions ^[8]. Pair chat with a call for intense incidents: “Disjointed discussions in the chat room are ultimately distracting.” ^[8]

There is no ChatOps standard. What exists, and how binding it is:

Categories first, examples second; versions and statuses are as of October 2026.

Lock-in advice. Keep the catalogue, identity map, policy and audit in your gateway, and treat platform adapters as replaceable; the platform churn since 2018 shows why.

Chat platforms and cloud providers now offer AI assistants (Slack AI is one ^[18]). Useful jobs: summarising a long incident channel, drafting a status update, explaining an error, proposing the next command, answering questions over read-only data. The risk is acute in chat: an assistant reads everything in its reach, and any of that text can carry instructions.

• Indirect prompt injection is a chat-native threat. OWASP’s LLM01 covers instructions arriving through external content such as websites or files ^[46]. In August 2024 PromptArmor showed that a message planted in a public Slack channel could make Slack AI render a link carrying data from a private channel the attacker was not in ^[18]; Salesforce told The Register “We’ve deployed a patch to address the issue” ^[56]. Any channel member, integration or uploaded file is a possible author of the next instruction.
• Excessive agency. OWASP’s LLM06 traces damage to excessive functionality, permissions and autonomy, and recommends human approval for high-impact actions ^[47]. In ChatOps terms: the assistant is a bot like any other, behind the same gateway, tiers and approvals; it may propose a command, a human issues it.
• Guardrails you do not control are a floor, not your policy. Amazon Q Developer in chat applications never runs iam:*, kms:*, secretsmanager:* or sts:* operations from chat, but its block list does not cover log queries or table scans ^[31] (reference implementation).
• Practical rules. Give the assistant its own identity and minimal read scopes; exclude channels that may contain secrets from retrieval; never let it approve; log its prompts, retrieved sources and tool calls; and disable link rendering or unfurling where it could carry data out.

The judgement call works through a concrete case.

Assumptions (illustrative). About 40 engineers, six services, Slack as the chat platform, Prometheus and Alertmanager for alerts, a CD system with an API, and AWS. On Teams the catalogue, gateway and audit stay the same; only the adapter changes, and it must check Bot Framework tokens and map Entra object IDs instead (proof of origin). Constraints: no command may reach production without a second person; the bot runs inside the company network; nothing the bot posts may contain secrets or personal data. The design has four artefacts plus the gateway from Lab 2.

1. The command catalogue, reviewed like code. It is the single place that says what can run, where, by whom, and how to undo it.

Example status: YAML syntax-checked, and each command’s full pattern built and compiled as in Lab 2, with Python 3.8; not executed against real executors.

# commands.yaml: the command catalogue, reviewed like code.
# Tier-3 entries need a security reviewer.
# Each argument has its own pattern; the gateway joins them with the literal
# words of 'syntax' and full-matches the whole message (Lab 2 does the same).
# Channels are matched by ID, because names can change. 'incident-registry'
# means any channel ID that the declare command registered and that is not
# yet archived.
version: 1
defaults:
  approval_ttl_minutes: 15
  reply: thread
commands:
  - name: status
    tier: 1
    owner: team-checkout
    syntax: 'status {service}'
    args: {service: '[a-z0-9-]{1,40}'}
    roles: [viewer, oncall, deployer]
    channels: ['*']
    executor: status-api
  - name: silence
    tier: 2
    owner: team-observability
    syntax: 'silence {alert} for {minutes}m'
    args: {alert: '[A-Za-z0-9_]{1,64}', minutes: '[1-9][0-9]?'}
    roles: [oncall]
    # C0ALERTSCHK is #alerts-checkout
    channels: [C0ALERTSCHK, incident-registry]
    executor: alertmanager-silences
    rollback: 'unsilence <alert>'
  - name: deploy
    tier: 2
    tier_3_when: {env: production}
    owner: team-checkout
    syntax: 'deploy {service} {version} to {env}'
    args:
      service: '[a-z0-9-]{1,40}'
      version: 'v[0-9]+[.][0-9]+[.][0-9]+'
      env: 'staging|production'
    roles: [deployer]
    channels: [C0DEPLOYS]  # #deploys-checkout
    approvers: [oncall-leads]
    executor: cd-pipeline-api
    rollback: 'deploy <service> <previous-version> to <env>'
  - name: failover
    tier: 3
    owner: team-platform
    syntax: 'failover {db} to {region}'
    args: {db: '[a-z0-9-]{1,40}', region: 'eu-west-1|eu-central-1'}
    roles: [oncall-leads]
    channels: [incident-registry]
    approvers: [platform-leads]
    executor: runbook-engine
    # refuse to run if the runbook has not been rehearsed recently
    rehearsed_within_days: 90
    rollback: 'runbook RB-DB-07, section "fail back"'
denied_by_design:
  - free-form shell, kubectl or SQL passthrough
  - any command that prints a secret, token or connection string

2. Notification routing into chat. Pages go to the pager and to one feed message that is edited as the alert changes; tickets go to the team channel without mentions; informational alerts stay out of chat; a known outage suppresses its own symptoms.

Example status: YAML syntax-checked against the Alertmanager configuration reference (version 0.34 docs); not loaded into Alertmanager (check it with amtool check-config alertmanager.yml before use).

# alertmanager.yml (excerpt): delivery into chat.
# The alert rules themselves live with ObservabilityOps.
route:
  receiver: team-checkout-chat
  group_by: [alertname, service, env]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="page"']
      receiver: oncall-pager
      continue: true
    - matchers: ['severity="page"']
      receiver: checkout-feed
    - matchers: ['severity="ticket"']
      receiver: team-checkout-chat
      repeat_interval: 24h
    - matchers: ['severity="info"']
      receiver: no-chat
# How grouping and inhibition work is taught on the AIOps page.
inhibit_rules:
  - source_matchers: ['alertname="CheckoutDown"']
    target_matchers: ['service="checkout"', 'severity=~"ticket|info"']
    equal: [env]
receivers:
  - name: oncall-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-checkout
  - name: checkout-feed
    slack_configs:
      - api_url: https://slack.com/api/chat.postMessage
        http_config:
          authorization:
            credentials_file: /etc/alertmanager/secrets/slack-bot-token
        # Channel ID of #alerts-checkout: a rename must not stop the feed.
        channel: C0ALERTSCHK
        send_resolved: true
        update_message: true
        title: >-
          {{ .Status | toUpper }}: {{ .CommonLabels.alertname }}
          ({{ .Alerts.Firing | len }} firing)
        text: >-
          {{ .CommonAnnotations.summary }}
          Runbook: {{ .CommonAnnotations.runbook_url }}
  - name: team-checkout-chat
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook-checkout
        # Informational only: a Slack webhook always posts to the channel
        # chosen when the webhook was created.
        channel: '#checkout-team'
        send_resolved: false
  - name: no-chat

3. A diagnostics-only cloud channel. For AWS diagnostics in chat, a channel configuration with per-user roles and a guardrail that allows only the calls responders need. The guardrail is the ceiling: users can do only what both the guardrail and their own role allow ^[31], and leaving GuardrailPolicies unset applies AdministratorAccess ^[54]. Two tempting managed policies are wrong here:

• ReadOnlyAccess reads data, not just configuration. AWS warns that it gives “access to read data in storage services like Amazon S3 buckets and Amazon DynamoDB tables” ^[57], and it includes logs:FilterLogEvents, logs:Get*, dynamodb:Scan and dynamodb:Query ^[58]. The chat service’s own block list stops s3:GetObject and dynamodb:GetItem, not log reads, scans or queries ^[31], so log lines and table items could be posted into a shared channel.
• ViewOnlyAccess cannot read resource content ^[57], but its current version (v47, edited 2026-10-14) allows cloudwatch:Get* and List*, not cloudwatch:Describe*, so a responder could not read alarm states ^[59].

The template therefore defines its own guardrail. Side effects to accept knowingly: every reply is visible to everyone in the channel; the service always writes an audit event for each executed command to CloudWatch Logs ^[60], and LoggingLevel (ERROR, INFO or NONE) controls which other entries are pushed there ^[54], so restrict who can read those log groups; and alarm notifications offer a “Show logs” button that runs a CloudWatch Logs Insights query and displays the first 30 log entries ^[61]. This guardrail grants no logs: actions, so we expect that button to be refused (not tested).

Example status: YAML syntax-checked against the AWS::Chatbot::SlackChannelConfiguration and AWS::IAM::ManagedPolicy references; not deployed. The channel role must already exist; its trust policy is not shown because we did not verify it against an opened AWS page.

AWSTemplateFormatVersion: "2010-09-09"
Description: Diagnostics-only Amazon Q Developer chat channel (illustrative).
Parameters:
  SlackWorkspaceId:
    Type: String
  SlackChannelId:
    Type: String
  ChannelRoleArn:
    Type: String
    Description: Existing IAM role the service assumes for this channel
  AlertsTopicArn:
    Type: String
Resources:
  DiagnosticsGuardrail:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: Ceiling for chat commands - status and metadata only, no writes
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - cloudwatch:DescribeAlarms
              - cloudwatch:DescribeAlarmHistory
              - cloudwatch:GetMetricData
              - ecs:DescribeServices
              - ecs:ListServices
              - rds:DescribeDBInstances
              - elasticloadbalancing:DescribeTargetHealth
            Resource: "*"
  CheckoutOpsChannel:
    Type: AWS::Chatbot::SlackChannelConfiguration
    Properties:
      ConfigurationName: checkout-ops-diagnostics
      SlackWorkspaceId:
        Ref: SlackWorkspaceId
      SlackChannelId:
        Ref: SlackChannelId
      IamRoleArn:
        Ref: ChannelRoleArn
      GuardrailPolicies:
        - Ref: DiagnosticsGuardrail  # Ref returns the managed policy's ARN
      UserRoleRequired: true
      LoggingLevel: INFO
      SnsTopicArns:
        - Ref: AlertsTopicArn

4. The audit record. One record per decision, with the NIST AU-3 elements (what, when, where, source, outcome, identity) ^[45], written by the gateway to append-only storage outside the chat platform. The file holds one record per line (JSON Lines); the record below is real output from Lab 2, pretty-printed exactly as its simulate.py prints it.

{
  "event_type": "chatops.command",
  "time": "2026-10-03T04:09:00+00:00",
  "where": "C0DEPLOYS",
  "source": "slack:T0EXAMPLE",
  "outcome": "executed",
  "identity": {
    "chat_user": "U02BOB",
    "principal": "bob@example.com"
  },
  "text": "approve req-0001",
  "command": "deploy",
  "args": {
    "service": "checkout",
    "version": "v1.4.2",
    "env": "production"
  },
  "request_id": "req-0001",
  "requested_by": "alice@example.com",
  "approved_by": "bob@example.com"
}

Three labs from beginner to advanced. Labs 1 and 2 need only the Python standard library and were executed with Python 3.8.10, most recently on 2026-10-17; the outputs shown are copied from those runs. Lab 3 needs a Slack workspace and was not executed. Use a supported Python release (3.10 or later); the code needs no newer features than 3.8.

Objective. Verify a Slack request signature, and see why the raw body and the timestamp matter ^[22]. Prerequisites. Python 3 (see above); no network or Slack account. Cost and risk. None; the secret is Slack’s published test value.

1. Save the script as verify_slack.py.
2. Run python3 verify_slack.py.

Example status: executed with Python 3.8.10 on 2026-10-17.

"""Lab 1: verify a Slack request signature (stdlib only)."""
import hashlib
import hmac
import time

MAX_SKEW_SECONDS = 300  # Slack's example rejects requests over five minutes old


def verify(secret: str, timestamp: str, raw_body: str, signature: str, now=None):
    """Return (ok, reason).

    raw_body must be the exact bytes Slack sent, before any parsing.
    """
    if not timestamp.isdigit() or not signature.startswith("v0="):
        return False, "malformed headers"
    now = time.time() if now is None else now
    if abs(now - int(timestamp)) > MAX_SKEW_SECONDS:
        return False, "stale timestamp (possible replay)"
    base = f"v0:{timestamp}:{raw_body}".encode()
    expected = "v0=" + hmac.new(secret.encode(), base, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, signature):
        return False, "signature mismatch"
    return True, "ok"


if __name__ == "__main__":
    # Test vector from Slack's request-verification guide (not a live secret).
    secret = "8f742231b10e8888abcd99yyyzzz85a5"
    ts = "1531420618"
    body = ("token=xyzz0WbapA4vBCDEFasx0q6G&team_id=T1DC2JH3J"
            "&team_domain=testteamnow&channel_id=G8PSS9T3V&channel_name=foobar"
            "&user_id=U2CERLKJA&user_name=roadrunner&command=%2Fwebhook-collect"
            "&text=&response_url=https%3A%2F%2Fhooks.slack.com%2Fcommands"
            "%2FT1DC2JH3J%2F397700885554%2F96rGlfmibIGlgcZRskXaIFfN"
            "&trigger_id=398738663015.47445629121.803a0bc887a14d10d2c447fce8b6703c")
    sig = "v0=a2114d57b48eac39b9ad189dd8316235a7b4a8d21a10bd27519666489c69b503"
    fresh = int(ts) + 10
    cases = [
        ("documented request", body, sig, fresh),
        ("body changed", body.replace("text=", "text=deploy"), sig, fresh),
        ("replayed 10 min later", body, sig, int(ts) + 600),
        ("wrong scheme", body, sig.replace("v0=", "v1="), fresh),
    ]
    for name, b, s, now in cases:
        ok, reason = verify(secret, ts, b, s, now=now)
        print(f"{name:24} -> {'ACCEPT' if ok else 'REJECT'} ({reason})")

Expected output.

documented request       -> ACCEPT (ok)
body changed             -> REJECT (signature mismatch)
replayed 10 min later    -> REJECT (stale timestamp (possible replay))
wrong scheme             -> REJECT (malformed headers)

Validation. Change one character of secret: all four cases must now reject. Troubleshooting. A real request that fails verification usually means the framework parsed and re-encoded the body before you hashed it (hash the raw bytes), or the server clock is wrong (check NTP). Teardown. Delete the file. On Teams the equivalent check is different: a bot validates a signed JWT, and an outgoing webhook checks an HMAC over the body with no documented freshness check (proof of origin).

Objective. Implement the identity and trust checks: workspace allow-list, identity mapping by immutable ID, typed arguments, channel and role checks, a two-person rule and AU-3 audit records. Prerequisites. Python 3 (see above). Cost and risk. None; the executor is a stub that only prints what it would do.

1. Create a directory with the three files below and run python3 simulate.py from inside it.
2. Read each refusal and find the line of gateway.py that produced it. Note that each argument in policy.json has its own pattern, and compile_syntax joins them into one regular expression that must match the whole message.

Example status: all three files executed with Python 3.8.10 on 2026-10-17.

{
  "allowed_teams": ["T0EXAMPLE"],
  "approval_ttl_seconds": 900,
  "approver_groups": ["approvers"],
  "identities": {
    "U01ALICE": {"principal": "alice@example.com",
                 "groups": ["oncall", "deployers"]},
    "U02BOB": {"principal": "bob@example.com",
               "groups": ["deployers", "approvers"]},
    "U03CAROL": {"principal": "carol@example.com", "groups": ["viewers"]}
  },
  "commands": {
    "status": {
      "syntax": "status {service}",
      "args": {"service": "[a-z0-9-]{1,40}"},
      "risk": "read", "groups": ["viewers", "oncall", "deployers"], "channels": "*"
    },
    "deploy": {
      "syntax": "deploy {service} {version} to {env}",
      "args": {
        "service": "[a-z0-9-]{1,40}",
        "version": "v[0-9]+[.][0-9]+[.][0-9]+",
        "env": "staging|production"
      },
      "risk": "write", "groups": ["deployers"], "channels": ["C0DEPLOYS"],
      "approval_if": {"env": "production"}
    }
  }
}

"""Lab 2: a chat-command gateway that decides before it acts (stdlib only)."""
import json
import re
from datetime import datetime, timezone

ARG = re.compile(r"\{(\w+)\}")


def compile_syntax(cmd):
    """'deploy {service} to {env}' plus one pattern per argument -> one regex."""
    parts = ARG.split(cmd["syntax"])  # literal, name, literal, name, ..., literal
    return re.compile("".join(
        re.escape(part) if i % 2 == 0 else f"(?P<{part}>{cmd['args'][part]})"
        for i, part in enumerate(parts)))


class Gateway:
    def __init__(self, policy, audit_path, executor):
        self.p, self.audit_path, self.execute = policy, audit_path, executor
        self.rx = {n: compile_syntax(c) for n, c in policy["commands"].items()}
        self.pending, self.counter = {}, 0

    def _audit(self, now, ev, who, outcome, **extra):
        # NIST SP 800-53 AU-3: what, when, where, source, outcome, identity
        rec = {"event_type": "chatops.command",
               "time": datetime.fromtimestamp(now, timezone.utc).isoformat(),
               "where": ev["channel_id"], "source": f"slack:{ev['team_id']}",
               "outcome": outcome,
               "identity": {"chat_user": ev["user_id"], "principal": who},
               "text": ev["text"], **extra}
        with open(self.audit_path, "a", encoding="utf-8") as f:
            f.write(json.dumps(rec) + "\n")  # JSON Lines: one record per line

    def handle(self, ev, now):
        text = ev["text"].strip()
        if ev["team_id"] not in self.p["allowed_teams"]:
            self._audit(now, ev, None, "denied:external-workspace")
            return "DENIED: commands are accepted only from our own workspace"
        # the immutable chat user ID, never a display name
        ident = self.p["identities"].get(ev["user_id"])
        if ident is None:
            self._audit(now, ev, None, "denied:unknown-identity")
            return "DENIED: your chat account is not linked to a corporate identity"
        who, groups = ident["principal"], set(ident["groups"])
        if text == "help":
            mine = [n for n, c in self.p["commands"].items()
                    if groups & set(c["groups"])]
            return ("You may run: " + ", ".join(mine)
                    + " (and 'approve <id>' if you are an approver)")
        if text.startswith("approve "):
            return self._approve(ev, now, who, groups, text.split(" ", 1)[1])
        for name, cmd in self.p["commands"].items():
            m = self.rx[name].fullmatch(text)  # fullmatch: no trailing "; rm -rf /"
            if m:
                break
        else:
            self._audit(now, ev, who, "rejected:no-matching-command")
            return "REJECTED: unknown command or invalid arguments (try: help)"
        args = m.groupdict()
        if cmd["channels"] != "*" and ev["channel_id"] not in cmd["channels"]:
            self._audit(now, ev, who, "denied:channel", command=name)
            return f"DENIED: '{name}' may only run in {', '.join(cmd['channels'])}"
        if not groups & set(cmd["groups"]):
            self._audit(now, ev, who, "denied:role", command=name)
            return f"DENIED: '{name}' needs one of {sorted(cmd['groups'])}"
        cond = cmd.get("approval_if", {})
        if cond and all(args.get(k) == v for k, v in cond.items()):
            self.counter += 1
            rid = f"req-{self.counter:04d}"
            self.pending[rid] = {"name": name, "args": args, "by": who,
                                 "channel": ev["channel_id"],
                                 "expires": now + self.p["approval_ttl_seconds"]}
            self._audit(now, ev, who, "pending-approval", command=name, args=args,
                        request_id=rid)
            # approvers see exactly what will run
            what = " ".join(f"{k}={v}" for k, v in args.items())
            return (f"PENDING {rid}: {name} {what}; "
                    f"another approver must run 'approve {rid}'")
        result = self.execute(name, args, who)
        self._audit(now, ev, who, "executed", command=name, args=args)
        return f"OK: {result}"

    def _approve(self, ev, now, who, groups, rid):
        req = self.pending.get(rid)
        if req is None or now > req["expires"]:
            self.pending.pop(rid, None)
            self._audit(now, ev, who, "denied:no-such-or-expired-request",
                        request_id=rid)
            return f"DENIED: {rid} does not exist or has expired"
        if ev["channel_id"] != req["channel"]:  # approve where everyone can see it
            self._audit(now, ev, who, "denied:approval-outside-channel",
                        request_id=rid)
            return (f"DENIED: approve {rid} in {req['channel']}, "
                    "where it was requested")
        if who == req["by"]:
            self._audit(now, ev, who, "denied:self-approval", request_id=rid)
            return "DENIED: the requester cannot approve their own request"
        if not groups & set(self.p["approver_groups"]):
            self._audit(now, ev, who, "denied:not-an-approver", request_id=rid)
            return "DENIED: you are not an approver"
        del self.pending[rid]  # single use
        result = self.execute(req["name"], req["args"], req["by"])
        self._audit(now, ev, who, "executed", command=req["name"], args=req["args"],
                    request_id=rid, requested_by=req["by"], approved_by=who)
        return f"OK ({rid}, approved by {who}): {result}"

"""Replays a short chat conversation through the gateway."""
import json
import os
import textwrap

from gateway import Gateway


def fake_executor(name, args, requested_by):
    return f"dry run - would call the {name} API with {args} for {requested_by}"


policy = json.load(open("policy.json", encoding="utf-8"))
if os.path.exists("audit.jsonl"):
    os.remove("audit.jsonl")
gw = Gateway(policy, "audit.jsonl", fake_executor)
T0 = 1_791_000_000  # fixed clock so the run is reproducible
T = "T0EXAMPLE"  # our own workspace
conversation = [
    (T, "U03CAROL", "C0RANDOM", "help"),
    (T, "U03CAROL", "C0RANDOM", "status checkout"),
    (T, "U03CAROL", "C0DEPLOYS", "deploy checkout v1.4.2 to staging"),
    (T, "U01ALICE", "C0RANDOM", "deploy checkout v1.4.2 to staging"),
    (T, "U01ALICE", "C0DEPLOYS", "deploy checkout v1.4.2 to staging; rm -rf /"),
    (T, "U01ALICE", "C0DEPLOYS", "deploy checkout v1.4.2 to staging"),
    (T, "U01ALICE", "C0DEPLOYS", "deploy checkout v1.4.2 to production"),
    (T, "U01ALICE", "C0DEPLOYS", "approve req-0001"),
    (T, "U02BOB", "D0BOBDM", "approve req-0001"),
    (T, "U02BOB", "C0DEPLOYS", "approve req-0001"),
    (T, "U02BOB", "C0DEPLOYS", "approve req-0001"),
    ("T0PARTNER", "U09EXT", "C0SHARED", "status checkout"),
    (T, "U07MALLORY", "C0DEPLOYS", "status checkout"),
]
for i, (team, user, channel, text) in enumerate(conversation):
    ev = {"team_id": team, "user_id": user, "channel_id": channel, "text": text}
    reply = gw.handle(ev, T0 + 60 * i)
    print(f"{user}: {text}")
    print(textwrap.fill(reply, width=80, initial_indent="  -> ",
                        subsequent_indent="     ", break_on_hyphens=False))
lines = open("audit.jsonl", encoding="utf-8").read().splitlines()
print(f"\n{len(lines)} audit records; the approved deploy:")
print(json.dumps(json.loads(lines[8]), indent=2))

Expected output (the audit record printed at the end is the one shown in the reference implementation):

U03CAROL: help
  -> You may run: status (and 'approve <id>' if you are an approver)
U03CAROL: status checkout
  -> OK: dry run - would call the status API with {'service': 'checkout'} for
     carol@example.com
U03CAROL: deploy checkout v1.4.2 to staging
  -> DENIED: 'deploy' needs one of ['deployers']
U01ALICE: deploy checkout v1.4.2 to staging
  -> DENIED: 'deploy' may only run in C0DEPLOYS
U01ALICE: deploy checkout v1.4.2 to staging; rm -rf /
  -> REJECTED: unknown command or invalid arguments (try: help)
U01ALICE: deploy checkout v1.4.2 to staging
  -> OK: dry run - would call the deploy API with {'service': 'checkout',
     'version': 'v1.4.2', 'env': 'staging'} for alice@example.com
U01ALICE: deploy checkout v1.4.2 to production
  -> PENDING req-0001: deploy service=checkout version=v1.4.2 env=production;
     another approver must run 'approve req-0001'
U01ALICE: approve req-0001
  -> DENIED: the requester cannot approve their own request
U02BOB: approve req-0001
  -> DENIED: approve req-0001 in C0DEPLOYS, where it was requested
U02BOB: approve req-0001
  -> OK (req-0001, approved by bob@example.com): dry run - would call the deploy
     API with {'service': 'checkout', 'version': 'v1.4.2', 'env': 'production'}
     for alice@example.com
U02BOB: approve req-0001
  -> DENIED: req-0001 does not exist or has expired
U09EXT: status checkout
  -> DENIED: commands are accepted only from our own workspace
U07MALLORY: status checkout
  -> DENIED: your chat account is not linked to a corporate identity

12 audit records; the approved deploy:

Validation. audit.jsonl has 12 lines for 13 events (help is deliberately not audited); grep -c denied audit.jsonl prints 7. Extensions. Add the silence command from the catalogue; make approvals expire by advancing the clock past approval_ttl_seconds; make deploy refuse a version older than the one running. Troubleshooting. ModuleNotFoundError: gateway means you ran the script from another directory. Teardown. Delete the directory.

Objective. Expose Lab 2 as /ops through Socket Mode, so the bot needs no public endpoint ^[23][50]. Prerequisites. A Slack workspace where you may install apps (Slack offers free developer sandboxes ^[9]), Python 3 (see above) and slack-bolt 1.30 (current on PyPI in October 2026), a secret store or at least environment variables. Cost and risk. No cost in a sandbox; do not point the executor at real systems until the gateway’s tests pass.

1. In the Slack app settings, create an app from a manifest and paste the manifest below (fields as in Slack’s manifest reference ^[62]); install it to the workspace.
2. Create an app-level token with the connections:write scope ^[23], and copy the bot token (xoxb-). App-level tokens do not expire by default ^[21]; store both in your secret store.
3. Copy policy.json and gateway.py from Lab 2 next to app.py, and replace the example IDs with your real team, user and channel IDs (T…, U…, C…).
4. export SLACK_BOT_TOKEN=… SLACK_APP_TOKEN=… from the secret store, then pip install slack-bolt==1.30.0 and python3 app.py.
5. Add the bot to a sandbox channel: a new app can post with chat.postMessage only where it is a member, unless it has the chat:write.public scope ^[63]. Run /ops help, /ops status checkout, then the production deploy and approval flow with a colleague.

As the reference architecture advises, help text and refusals go only to the caller through respond() (the response_url), and results and pending approvals are posted with chat.postMessage through Bolt’s client ^[50], so channel posting permissions apply.

The manifest sets token_rotation_enabled: false to keep the sandbox simple; do not copy that to production without deciding on rotation first (the bot is a privileged system).

Example status: Python parsed (AST) and YAML parsed; reply routing checked with stub Slack objects; not run against Slack (needs a workspace and tokens).

display_information:
  name: ops-gateway
  description: Runs allow-listed operations commands with approval and audit.
features:
  bot_user:
    display_name: ops-gateway
    always_online: false
  slash_commands:
    - command: /ops
      description: Run an allow-listed operations command (try /ops help)
      usage_hint: deploy checkout v1.4.2 to staging
      should_escape: false
oauth_config:
  scopes:
    bot:
      - commands
      - chat:write
settings:
  socket_mode_enabled: true
  token_rotation_enabled: false
  org_deploy_enabled: false

"""Lab 3: expose the Lab 2 gateway as /ops in Slack, over Socket Mode."""
import json
import os
import time

from slack_bolt import App
from slack_bolt.adapter.socket_mode import SocketModeHandler

from gateway import Gateway


def executor(name, args, requested_by):
    # Replace with calls to your status/deploy APIs, using short-lived credentials.
    return f"dry run - would call the {name} API with {args} for {requested_by}"


with open("policy.json", encoding="utf-8") as f:
    policy = json.load(f)
gateway = Gateway(policy, os.environ.get("AUDIT_PATH", "audit.jsonl"), executor)
# xoxb- bot token from the secret store, never from code
app = App(token=os.environ["SLACK_BOT_TOKEN"])


@app.command("/ops")
def ops(ack, respond, command, client):
    ack()  # Slack needs this within 3,000 ms; the work happens afterwards
    event = {"team_id": command["team_id"], "user_id": command["user_id"],
             "channel_id": command["channel_id"], "text": command.get("text", "")}
    reply = gateway.handle(event, time.time())
    if reply.startswith(("OK", "PENDING")):
        # Announce via the Web API; response_url would bypass posting permissions
        client.chat_postMessage(channel=command["channel_id"], text=reply)
    else:
        # Help text and refusals go only to the caller
        respond(text=reply, response_type="ephemeral")


if __name__ == "__main__":
    SocketModeHandler(app, os.environ["SLACK_APP_TOKEN"]).start()  # xapp- token

Expected output. /ops help returns a reply only you can see; after /ops status checkout the bot posts the dry-run result in the channel; audit.jsonl gains a line with your real U… ID. Validation. Run /ops deploy checkout v1.4.2 to staging from an unlisted channel: it must be refused and audited. Troubleshooting. If Slack shows operation_timeout, the listener did not call ack() within 3,000 ms ^[25]; if nothing arrives, check that Socket Mode is enabled and the app-level token has the right scope; if refusals arrive but results do not, the bot is not a member of the channel. Teardown. Revoke the bot token (Slack documents auth.revoke ^[41]), uninstall and delete the app, remove both tokens from the secret store, and delete audit.jsonl.

The sourced cases describe each company’s own practice at the date of the source, not today’s.

Case 1: GitHub, from chat-only deploys to “one command” (2015 → 2021). In 2015 GitHub engineers deployed “several hundred times every week” from chat, with the queue, guards, locks and loud override of the deploy playbook, and Hubot posted elevated exceptions after a deploy ^[7]. By 2021 the number of developers contributing to GitHub.com had roughly doubled in a year. A deploy now took about a dozen steps “spread across hundreds of messages” in #dotcom-ops, and developers were running chatops commands “multiple times in every deployment, and often getting them wrong” ^[17]. GitHub’s answer was not to leave chat: one chatops command starts a state-machine-backed deploy with automatic 2 % and 20 % canary stages and 5-minute timers, a UI shows its state, and developers typically start it from Slack and follow it in the UI ^[17]. Lesson: chat is an excellent trigger and summary, and a poor place to keep state.

Case 2: PagerDuty’s incident commands. PagerDuty’s public incident-response guide documents bots that react to ! commands in Slack. !ic page pages the primary, backup and shadowing Incident Commanders; the commands cache on-call schedules so they work if the PagerDuty API fails, and if paging fails the bot says so and gives phone numbers; !status stalk polls every 30 seconds and reports only when the status changes ^[4]. Two design principles stand out: fail loudly with a fallback and report changes, not states. The channel’s lifecycle around such commands looks like this (this Atlas’s model; incident roles and severity rules are taught in IncidentOps):

stateDiagram-v2
  [*] --> Declared: declare command
  Declared --> Assembling: bot creates channel, registers ID, pages
  Assembling --> Active: bot posts commander, roles, links
  Active --> Mitigated: bot posts status change
  Mitigated --> Resolved: bot posts resolution
  Resolved --> Archived: bot exports timeline, archives
  Active --> Handoff: attacker suspected
  Handoff --> Preserved: bot exports timeline, narrows access
  Archived --> [*]
  Preserved --> [*]

Suspicion of an attacker can arise in any state; the figure shows it from Active only to stay readable. Keeping the record after a security handoff matters because the channel may be evidence (ResponseOps).

Case 3: GOV.UK’s team bots. GOV.UK’s developer documentation lists Slack integrations that nudge teams instead of running production commands: Seal posts each team’s open pull requests, Dependapanda its open Dependabot pull requests, Release App Badger pull requests not yet deployed, and CI Bot repositories missing CodeQL or Dependency Review scans ^[64]. “The Seal and Dependapanda can only post to public channels.” ^[64] Low-risk, high-value ChatOps: no write access to production, and each message names an owner and an action. (The page, last updated in January 2026, is flagged as possibly out of date.)

No standard ChatOps metrics exist; the set below is this Atlas’s proposal. Measure the interface (does it make work visible and safe?) and the noise (does it waste attention?), not activity. Delivery outcomes such as deployment frequency belong to DevOps, and incident timings such as MTTA to IncidentOps.

The actionable notification ratio counts chat messages of every kind, including deploy notices and nudges. It differs from page actionability in ObservabilityOps and the actionable page rate in AIOps, which count pages: a channel can score badly here while every page is justified.

Leading and lagging. Refusal mix, notification load and credential hygiene are leading indicators; audited change coverage and the actionable ratio are lagging. Vanity metrics: number of commands, number of bot messages, “bot usage” totals.

Why chat content itself is sensitive. In the Disney case, SecurityWeek reported in 2025, citing the Justice Department, that an employee downloaded a fake AI art tool to a personal computer; credentials stored there gave the attacker the employee’s work Slack account and data from thousands of Slack channels ^[66]. A ChatOps programme adds to what such an account can see. Keep secrets, tokens and connection strings out of chat entirely, and treat ops channels as confidential.

Privacy and compliance. Chat is a data store: minimise personal data in notifications, set retention deliberately, and agree with your privacy team which channels hold what (PrivacyOps). For compliance, the gateway’s audit log, not the chat transcript, is the record of who did what. Whether a regulator also expects you to retain the chat itself depends on your sector and jurisdiction; ask your compliance team (ComplianceOps), do not assume.

• Direct costs are modest and plan-dependent: hosting a gateway is cheap, but some controls sit in premium tiers, such as Slack’s Audit Logs API (above). Prices change; check your contract.
• The largest cost is attention: about 13 hours a week for one noisy channel in the illustrative calculation. The actionable ratio is the cost-control metric.
• Maintenance cost is driven by platform churn: connectors, legacy bots and whole products retired, or scheduled for retirement, between 2018 and 2027 (origins). A thin adapter keeps rewrites small.
• Capacity limits are the platform’s, not yours: per-channel posting limits with HTTP 429 and Retry-After ^[27], acknowledgement deadlines and response_url limits. Design for alert storms and long-running commands.
• Sustainability. The compute footprint of bots is negligible next to the systems they operate; we found no evidence that would justify a carbon claim either way.

• Synthetic checks. A scheduled canary command in a test channel proves the full path (platform, adapter, gateway, executor in dry-run) and alerts the bot owner when it fails; PagerDuty’s bots also report their own failures in the room ^[4].
• Reconciliation. Monthly: gateway audit records versus executor logs (every executed command appears in both); identity map versus leavers; approver groups versus the on-call roster.
• Scope and guardrail audits. Quarterly: installed apps and their scopes against the scope policy (Slack suggests scripting such audits ^[41]); cloud chat configurations against the guardrail baseline ^[54].
• Exception review. Every break-glass use and every executed tier-3 command is reviewed in the next team meeting.
• Drills. Twice a year, run an incident drill with the chat platform declared unavailable, and one with a revoked bot token.

Illustrative scenario. At 10:12 a secret-scanning alert reports that a xoxb- token for ops-gateway was committed to a public fork of an internal tool. The bot can run tier-2 commands and post in 40 channels.

%%{init: {"flowchart": {"wrappingWidth": 160, "nodeSpacing": 24, "rankSpacing": 36}}}%%
flowchart TB
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  D["Alert: credential exposed"]:::risk
  T{"Which credential, which scopes, still valid?"}:::decision
  K["Turn off write commands (kill switch)"]:::risk
  R["Revoke and replace; restart adapter; run canary"]:::core
  I["Review use since exposure: platform, gateway, executor logs"]:::core
  Q{"Misuse found?"}:::decision
  S["Security incident: hand off to ResponseOps"]:::risk
  C["Re-enable writes; review and prevention actions"]:::core
  D --> T --> K --> R --> I --> Q
  Q -->|"yes"| S
  Q -->|"no"| C

1. Triage (T+0 to T+5). Identify the credential type (bot token, app-level token, webhook URL or signing secret) and what it can do: scopes from the app configuration, commands from the catalogue. Severity is high if it can trigger writes. Run Slack’s auth.test with the token ^[67]: for a public repository, GitHub’s partner program reports Slack tokens and webhook URLs to Slack ^[65], and Slack says it “actively searches out and revokes leaked secrets” ^[26]. If the token is already dead, the bot is silent: skip to Restore, then still investigate the exposure window.
2. Contain (by T+10). Flip the gateway’s kill switch for tier-2 and tier-3 commands. Revoke the token (Slack’s auth.revoke “Revokes a token.” ^[68]); for a webhook, delete it and create a new one; for a signing secret, regenerate it.
3. Restore (by T+20). Put the new credential in the secret store, restart the adapter, run the canary command, re-enable reads.
4. Investigate (by T+60). Establish the exposure window (commit time to revocation). Check the platform’s logs (Slack’s Audit Logs API, where your plan has it ^[9]), the gateway audit log and executor logs for activity in the window, and search channels for messages posted by the bot that nobody triggered.
5. Decide. Signs of misuse (unexplained executions, posts, new IP sources) make it a security incident: hand off to ResponseOps. Otherwise re-enable writes.
6. Prevent. Enable push protection for the repository (GitHub detects Slack tokens and webhook URLs ^[65]); restrict token use to the gateway’s public egress addresses ^[41]; plan token rotation (it cannot be turned off later ^[21]); use separate credentials per environment.

1. Confirm with the canary command or help in a test channel; check the chat vendor’s status page.
2. Fall back at once to the documented paths: pipeline UI, runbook engine, phone numbers for paging. Announce the fallback in the incident channel (or on the call if chat is down). Do not debug the bot in the middle of someone else’s incident.
3. Diagnose in order: adapter process and WebSocket connection; token validity (auth.test ^[67]); 429 responses; gateway errors; executor health.
4. Restore and run the canary; post a summary of any actions taken outside the gateway so the audit trail can be completed afterwards.
5. Review in the incident’s follow-up: the bot is part of the incident response system and has its own action items.

• A shell with a chat UI. StackStorm’s own example alias run {{cmd}} on {{hosts}} wraps a remote command action ^[28]; it teaches aliases, but deployed as is it gives a remote shell to everyone allowed to use it (GitHub’s own history in Origins). Instead: typed, allow-listed commands.
• The god-mode bot. One token with admin scopes and production credentials. Instead: least privilege for the bot, separate credentials per executor and tier.
• Chat as the database. Deploy state, approvals or incident state reconstructed by scrolling. Instead: keep state in a system with a UI and post summaries (Case 1).
• Direct messages by default, emoji approvals, logic in the adapter. Each breaks a principle: visibility, typed approvals checked by the gateway, a thin and replaceable adapter.
• The firehose channel. Every alert, build and comment in one channel. Instead: one channel per audience, grouped, updated in place, with an actionable-ratio target.
• Myth: “everything is visible, so it is controlled.” Visibility helps detection; it does not stop a forged, mistaken or injected command.
• Myth: “an AI assistant makes the catalogue unnecessary.” Natural language widens what users can ask for; the catalogue and tiers are what keep the answer safe.

Use ChatOps when operational tasks repeat often or must be coordinated fast, and have an API behind them; the team already coordinates in chat; shared visibility is valuable (deploys, incidents, on-call handovers); and the team is distributed. Newland’s abstract credits it with having “enabled and enhanced our culture of remote work” ^[1].

Do not use it, or limit it to notifications, when:

• Inputs are rich or the review is visual. Database migrations, capacity plans and infrastructure diffs need a form, a plan view or a pull request. Chat can link to them and announce results.
• Nobody will own the bot. An unowned bot with production credentials is a liability; start only with a named owner and a budget for maintenance.
• You do not control identity in the workspace. Many guests, external organisations in shared channels, or chat accounts not tied to your identity provider make command authorisation guesswork.
• Policy forbids it. Some regulated or classified environments do not allow production control through a SaaS chat platform; self-hosted chat or no chat path may be the answer.
• The chat platform is part of what you would be fixing, with no fallback path.
• Outputs contain secrets or personal data that cannot be reliably redacted.

Hidden responsibilities: running the gateway as a product, security reviews of new commands, token rotation, noise reviews and adapter rewrites when vendors change their platforms.

The everyday decision is not “should we do ChatOps?” but “should this action be a chat command, and at which tier?”

%%{init: {"flowchart": {"wrappingWidth": 150, "nodeSpacing": 22, "rankSpacing": 30}}}%%
flowchart TB
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  Q1{"Output free of secrets and personal data?"}:::decision
  Q2{"Short, typeable inputs and an API behind it?"}:::decision
  Q3{"Read-only?"}:::decision
  Q4{"Does the team gain from seeing it?"}:::decision
  Q5{"Weekly, or part of a time-critical response?"}:::decision
  Q6{"Reversible in minutes, small blast radius?"}:::decision
  N1["Not a chat command; at most a notice that it ran"]:::risk
  N2["Form, plan view or pull request; chat links to it"]:::neutral
  T1["Tier 1: verified identity, audit"]:::core
  N3["Pipeline only; message chat only if someone must act"]:::neutral
  N4["Runbook or CLI; chat announces the result"]:::neutral
  T2["Tier 2: role, channel, audit"]:::core
  T3["Tier 3: plus second person, rehearsed executor"]:::core
  Q1 -->|"yes"| Q2
  Q1 -->|"no"| N1
  Q2 -->|"yes"| Q3
  Q2 -->|"no"| N2
  Q3 -->|"no"| Q4
  Q3 -->|"yes"| T1
  Q4 -->|"yes"| Q5
  Q4 -->|"no"| N3
  Q5 -->|"yes"| Q6
  Q5 -->|"no"| N4
  Q6 -->|"yes"| T2
  Q6 -->|"no"| T3
  N1 ~~~ N2 ~~~ T1 ~~~ N3 ~~~ N4

The order matters. Output safety comes first because it applies to reads as well: a “read-only” log search can still post personal data. Time-critical incident actions such as paging or failover pass the frequency question even if they are rare, because chat is where the responders already are.

Weighted check for borderline cases (this Atlas’s proposal). It answers one question, “is this a good chat command?”; the tree’s last question then sets the tier. Score each criterion from 1 (poor fit) to 5 (good fit), multiply by the weight, and treat a total under 3.0 as “not a chat command”.

Silencing is tier 2 because it expires on its own. Failover is tier 3: hard to undo, large blast radius. Its weakest score, executor maturity, is a precondition for tier 3 rather than a weight: if the runbook has not been rehearsed, do not ship the command whatever the total (the catalogue entry refuses to run without a rehearsal in the last 90 days). Certificate rotation scores 2.15 and belongs in a pipeline with a chat notification.

A proposal by this Atlas, not an industry standard. Levels describe observable behaviour; skip none, and stop at the level your risk and team size justify.

%%{init: {"flowchart": {"wrappingWidth": 135, "nodeSpacing": 20, "rankSpacing": 30}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  L1["1 Notify. Exit: actionable ratio measured in every alert channel"]:::core
  L2["2 Read. Exit: every invocation audited with a person"]:::core
  L3["3 Guarded write. Exit: most routine changes audited; refusals reviewed weekly"]:::core
  L4["4 Governed. Exit: chat-outage and revoked-token drills passed"]:::core
  L5["5 Assisted: AI reads and proposes; humans issue and approve"]:::core
  S1["Stop at 1 if chat only needs to inform"]:::neutral
  S2["Stop at 2 if changes need not start in chat"]:::neutral
  S3["Stop at 3 if high-impact actions stay out of chat"]:::neutral
  S4["Stop at 4 if no assistant reads ops channels"]:::neutral
  L1 --> L2 --> L3 --> L4 --> L5
  L1 -.- S1
  L2 -.- S2
  L3 -.- S3
  L4 -.- S4
  S1 ~~~ S2 ~~~ S3 ~~~ S4

Quick wins: route informational alerts out of chat; add owner and runbook links to every alert message; make every refusal message say what to do instead. Dependencies: an identity provider feed (IdentityOps), executors with APIs (AutomationOps), and agreed incident roles (IncidentOps).

Quick start (before the first command)

• A named bot owner and a backup
• Inventory of existing bots, webhooks and tokens, with scopes and owners
• Channel taxonomy agreed (#deploys-…, #alerts-…, #inc-…, #ops-sandbox)
• Secret store holds every token; nothing in code, chat or tickets
• Gateway writes an audit record for every decision, outside the chat platform

Launch readiness for a new command

• Proposal approved, tier assigned, owner named
• Arguments typed and fully matched; unknown arguments refused
• Roles, channel IDs and (tier 3) approver group set in the catalogue
• Refusals tested: wrong user, wrong channel, malformed input, self-approval, expired approval
• help text, example, rollback command or runbook link
• Executor credential limited to this command’s needs

Quarterly security and governance audit

• App scopes match the scope policy; unused apps removed
• Identity map reconciled with leavers and contractors; approver groups current
• Credentials rotated or within policy; IP allowlists current
• Cloud chat configurations have explicit guardrails that exclude data-content reads
• Break-glass uses reviewed; audit log and executor logs reconciled

Monthly notification review (per alert channel)

• Actionable ratio sampled and compared with target
• Top ten noisiest alerts sent back to their owners (ObservabilityOps)
• No @here or @channel on non-paging alerts
• Every message has an owner and a runbook or dashboard link

Incident readiness

• Declare command creates the channel, pages on-call and posts links (tested this quarter)
• Fallback paths documented for a chat or bot outage, including phone numbers
• Canary command alerts the bot owner when the path breaks

Command proposal (intake form).

Command and syntax:    deploy <service> <version> to <staging|production>
Owner (team, person):  team-checkout, A. Example
Problem it solves:     manual pipeline clicks; deploys invisible to support
Frequency:             ~40 per week
Arguments, validation: service  [a-z0-9-]{1,40}
                       version  vMAJOR.MINOR.PATCH
                       env      staging|production
Risk tier and reason:  2 for staging; 3 for production
                       (customer impact; rollback takes ~10 min)
Channels and roles:    #deploys-checkout; deployers; approvers: oncall-leads
Executor, credential:  CD pipeline API; token limited to the checkout pipelines
Output shown in chat:  parsed arguments, pipeline link, final status;
                       never logs or secrets
Rollback:              deploy <service> <previous-version> to <env>
Tests:                 refusal cases in CI; dry run in #ops-sandbox
Review:                bot owner; security (tier 3 only)

Notification contract (one per service).

service: checkout
owner: team-checkout
channels:
  page_feed: '#alerts-checkout'     # one message per alert group, updated in place
  team: '#checkout-team'            # tickets, no mentions
  digest: daily-email               # informational alerts never go to chat
required_labels: [owner, runbook_url, severity]
mentions: page-only
forbidden_content: [secrets, tokens, personal data, raw log dumps]
target_actionable_ratio: 0.5
review: monthly

Architecture decision record: adopting a ChatOps gateway.

Title:        ADR-012 Route all chat commands through ops-gateway
Status:       accepted (2026-10-16)
Context:      three bots with admin tokens; no audit outside Slack;
              deploys need approvals
Decision:     one gateway owns catalogue, identity map, approvals and audit;
              adapters stay thin; executors receive requester identity;
              tier 3 needs a second person in the same channel
Alternatives: StackStorm with aliases and RBAC; vendor incident tool
              commands only; no write commands in chat
Consequences: + one place to review and audit
              + platform change limited to the adapter
              - gateway is a critical service (needs canary, on-call, drills)
Review:       after 90 days, using audited change coverage and refusal mix

Which interface for an operational action?

Socket Mode or a public HTTPS endpoint (Slack)?

The comparison with neighbouring Ops pages is in How ChatOps relates to other Ops.

Is deploying to production from chat safe? It can be, with this page’s controls: verified requests, mapped identities, typed arguments, channel and role checks, a second person and an audit log. Without them it is a remote shell with a friendly face.

Slack or Microsoft Teams? The platform your organisation already runs and whose identity it controls; both have SDKs for bots ^[50][51]. Compare audit, retention and data residency on your plan. Share the gateway; only the adapter’s trust checks differ (proof of origin).

Should the bot act with users’ own tokens? Usually not. On Slack, user tokens act “as if by the user themselves” with the user’s access ^[21]; storing many such tokens creates many high-value secrets. Prefer a bot token for chat, plus a gateway that passes the requester’s identity to executors.

Who approves at 3 a.m.? A named approver group that includes the on-call lead, with a documented break-glass path for when nobody is reachable. Break-glass is announced in the channel and reviewed afterwards.

Where does ChatOps end and IncidentOps begin? ChatOps owns the tooling (channel creation, paging commands, timeline capture); IncidentOps owns roles, severity, communication and reviews.

Will AI assistants replace slash commands? They change how people find and phrase commands. They do not remove the need for a catalogue, tiers and approvals; OWASP recommends human approval for high-impact actions ^[47].

• Established: typed command catalogues, gateways separate from adapters, two-person approval, notification grouping and in-place updates ^[24][44].
• Maturing: passing the requester’s identity end to end, so executors authorise the person rather than the bot; cloud chat integrations with per-user roles and guardrails are one implementation ^[31]. Policy engines for command authorisation belong with IAMOps.
• Moving fast (re-check before relying on it): AI assistants in chat, and protocols that let agents read and act through chat platforms. In February 2026 Slack announced an MCP server that “enables AI agents to interact with Slack content”, together with separate search scopes for public channels, private channels and direct messages ^[19]; grant the narrowest. The OWASP risks above apply unchanged ^[46][47].
• Speculative: “conversational operations” where the assistant runs remediation unattended. No public evidence we found shows this is safe at production scale; treat vendor claims as marketing until measured in your own environment.

• Prerequisites. Basic HTTP and JSON, one scripting language, how your CI/CD and alerting work (DevOps, ObservabilityOps).
• One-day primer. Read this page’s first four parts; run Labs 1 and 2; read Slack’s request-verification guide ^[22] and GitHub’s 2015 and 2021 deploy posts ^[7][17].
• One week. Run Lab 3 in a sandbox; add the silence command; route one service’s alerts with grouping and in-place updates; measure its actionable ratio.
• Thirty days. Run the 30-day roadmap slice for one team; write the notification contract and two command proposals; run a revoked-token drill.
• Specialisation. Identity propagation and authorisation (IAMOps, AccessOps); incident tooling (IncidentOps); safe automation behind commands (AutomationOps); AI assistant security (LLMOps).
• Portfolio ideas. A gateway with a policy file and tests for every refusal path; a notification-noise report for a real channel (anonymised); a threat model of an existing bot.

Situation (illustrative). A 200-engineer SaaS company has an AI assistant that can read incident channels and run cloud commands through a channel role. The SRE lead proposes (A) letting it restart unhealthy workloads and scale services automatically when it is confident, to cut response time. The security lead proposes (B) letting it read, summarise and propose commands, which a human then issues through the gateway. Assumptions: incident channels sometimes include a vendor’s engineers; alert messages quote log lines that can contain user-supplied text; restarts and scaling are reversible, but a restart loop can deepen an outage; nobody has measured the assistant’s proposal accuracy here.

Reasoning.

1. Worst credible failure. Under A, text the assistant reads (a quoted log line, a guest’s message, an uploaded file) can carry instructions. That is indirect prompt injection ^[46], and it has been demonstrated against a chat assistant ^[18]. The assistant would then act with the channel role’s permissions. Under B, the worst case is a bad proposal that a human must still type and the gateway must still allow.
2. Where time goes. In most incidents the slow part is understanding, not typing a restart. B keeps most of the speed benefit: summaries, correlations and a ready-made command.
3. Guidance. OWASP attributes damage to excessive functionality, permissions and autonomy and recommends human approval for high-impact actions ^[47]. Restarts look low-impact one at a time; in a loop they are not.
4. What would change the answer. A narrow executor that can restart only one named workload, at most once per ten minutes, with no external participants in the channel, plus a quarter of data showing that most of the assistant’s proposals were accepted unchanged. Then auto-executing that single action could be justified.

Decision: B now, with a review after a quarter. Measure accepted, edited and rejected proposals. Why A is attractive and still wrong today: it optimises the visible minute of typing and pays for it with an invisible attack surface: every author of every message in the channel. Uncertainty: we found no public data on assistant accuracy for remediation in comparable settings, so the measurement is the point of the trial.

After this page you should be able to explain what ChatOps is and is not; design a command with tiers, typed arguments and approvals; implement request verification and a gateway decision path; operate notification routing with measurable noise; respond to a leaked bot token; and evaluate whether an action, a tool or an AI assistant belongs in chat.

ChatOps began as a good idea that is easy to state: put the tools in the middle of the conversation so that everyone learns by watching. Thirteen years on, the idea holds, and the open questions are about trust and attention. A chat command is a production control surface inside a product that many people, integrations and now AI assistants can write to; a notification spends the attention of everyone who sees it. Teams that do ChatOps well put every command through a gateway that knows who is asking and keeps its own record, and measure whether their messages lead to action. Teams that do it badly hand a god-mode token to a chatty bot and call the scrollback an audit trail. The difference is not the chat platform; it is whether the team treats its bot as a privileged system with an owner.

Vendor documentation pages are live pages without a publication date unless one is given; product facts are as of October 2026. Ledger with claim-by-claim notes: research/chatops-ledger.md.

Further reading

• PagerDuty’s open Incident Response documentation shows chat commands inside a full incident process.

Changelog

• 2026-10-16: first version (research, ledger, draft, diagrams, labs 1–2 executed).
• 2026-10-16 — r1 independent fact check (reviews/chatops-r1.md): 0 critical, 3 major, 15 minor. Revision (reviews/chatops-r1-response.md): decision tree and weighted check rebuilt to match the catalogue; per-platform proof-of-origin table (Slack, Socket Mode, Teams bot, Teams outgoing webhook, Mattermost); AWS template now uses a custom diagnostics guardrail instead of ReadOnlyAccess; NIST mapping to AC-3(2) and CM-5(4); verify-before-ack sequence; channels authorised by ID; Lab 2 shows what is pending (re-run with Python 3.8.10); literature, Disney, Opsgenie and footnote metadata corrected; repetition and overlap with AIOps, ObservabilityOps and IncidentOps trimmed.
• 2026-10-17 — r2 after the independent score (reviews/chatops-score.md, 91/100; reviews/chatops-r2-response.md): Teams outgoing-webhook replay cell rewritten from the Bot Framework Activity specification; Lab 3 posts results with chat.postMessage; Figures 2, 3 and 8 redrawn; code blocks wrapped to 85 characters (Labs 1–2 re-run, audit file byte-identical); sample margin corrected; repetition trimmed.
• 2026-10-17, post-score polish (reviews/chatops-score-r2.md, 94/100; reviews/chatops-polish.md): Figure 2 labels the executors a separate privilege zone; the Alertmanager feed posts to a channel ID and notes app_token_file; three code lines shortened; repetition trimmed.