--- title: "ChatOps" slug: chatops category: delivery term_status: established evidence_status: verified summary: "ChatOps is the practice of running operational work through shared chat: bots post the events people need and run allow-listed commands on request, so actions, results and the discussion around them are visible, permissioned and auditable." audience: [Beginners, SREs and DevOps engineers, Incident responders, Platform and security engineers, Engineering leaders] last_verified: 2026-10-17 key_takeaways: - "ChatOps is an established practice pattern, named publicly at GitHub in 2013; it is not a standard, a job title or a product." - "A chat command is a remote control for production: verify every request, map the immutable chat user ID to a real identity, and let the service behind the bot decide authorisation." - "Allow-list typed commands and never wrap a shell; gate risky commands by channel, role and a second person; keep your own audit record, because chat history is not an audit log." - "Chat is a good front door and a poor database: GitHub kept starting deploys in Slack but moved deploy state into a state machine and a UI once hundreds of messages buried it." - "Notifications cause most chat noise: group them, update them in place, route them by severity, and measure how many lead to action." - "AI assistants in chat can be steered by any message they read; let them read and propose, and keep high-impact actions behind human approval." related: [incidentops, automationops, observabilityops, accessops, iamops, identityops, devops, knowledgeops] --- ## What ChatOps is ### ChatOps in 30 seconds **ChatOps is running operational work through a shared, persistent chat room.** Bots post the events a team needs to see (a deploy started, an alert fired, a pull request is stale) and run a short list of approved commands when someone types them (`/ops status checkout`, `/deploy api v2.3.1 to production`). Because the request, the result and the discussion all happen in one place, everyone in the room can see what was done, by whom, and what happened next. The idea is simple and old. The hard part is that a chat command is a **remote control for production** held inside a messaging product that many people, integrations and now AI assistants can write to. Good ChatOps is therefore mostly about two things this page concentrates on: the **security model** of bots and commands, and the **usability** of the channel (signal versus noise). ::: note Honest term status ChatOps is an **established practice pattern**: its meaning has been stable since GitHub engineers named it publicly in 2013 [^newland], and it has its own literature (an O'Reilly report by Jason Hand in 2016 [^hand-podcast]) and vendor documentation [^st2-chatops][^pd-chatops]. It is **not** a discipline with a standards body, a certification or a job family. The academic literature is small and mostly describes tools and designs, such as a 2024 journal article on a low-code ChatOps approach for microservices that uses large language models [^fgcs-2024], alongside broader studies of developer bots [^erlenhov]; we found no empirical study of ChatOps practice or its outcomes. Products reuse the word, which is a naming collision, not a different meaning. ::: **Where to start.** Beginners: the [glossary](#glossary-and-mental-model) and [Lab 1](#hands-on-labs). Engineers: the [reference architecture](#reference-architecture), [identity and trust](#identity-and-trust-who-is-really-asking) and [Lab 2](#hands-on-labs). Leaders: the [decision framework](#decision-framework) and the [judgement call](#a-judgement-call-should-the-ai-assistant-in-the-incident-channel-act). Product facts are dated **October 2026**. ### At a glance | Aspect | Summary | |---|---| | Goal | Shared situational awareness, and operational actions that are permissioned, attributable and auditable | | People | SREs and DevOps engineers, on-call and incident responders, platform and security engineers who own the bot | | In scope | Notifications into chat, command design, identity mapping, authorisation and approval *in the chat path*, audit trails, incident-channel tooling, bot and token security, AI assistants in chat | | Out of scope | Incident roles and process ([IncidentOps](../incidentops/)); the automation behind commands ([AutomationOps](../automationops/)); alert rules ([ObservabilityOps](../observabilityops/)); identity, permission design and elevation ([IdentityOps](../identityops/), [IAMOps](../iamops/), [AccessOps](../accessops/)) | | Prerequisites | A team chat platform, some automation with an API (CI/CD, runbooks, cloud CLI), and an owner for the bot | | Key outputs | A command catalogue, a bot with a least-privilege identity, a notification routing contract, an audit log | | Term status | Established practice pattern; no standard | ### The problem it solves Without ChatOps, operational work is invisible until it breaks. People ask the same questions all day; a slide in GitHub's 2013 talk lists questions its author had stopped asking once the work moved into chat, such as "how's that deploy going?", "are you deploying that or should i?" and "is anyone responding to that nagios alert?" [^newland]. Knowledge stays in the heads of the people who ran the commands, and new staff learn by asking. ChatOps addresses four failures: - **Invisible change.** At GitHub in 2015, "All deployments happen in chat via Hubot commands, which ensures that everyone in the company (from development to operations to support) has visibility into changes that are being pushed into production." [^gh-2015] - **Learning by asking.** Newland's abstract credits chat commands with increasing "the speed at which new team members learn common practices" [^newland]; a slide quotes @rtomayko on "teaching by doing by making things visible". - **Inconsistent manual steps.** A command encodes guards that a person forgets: GitHub's deploy command refused to run until CI passed, merged the latest `master` first, and only ran from designated rooms [^gh-2015]. - **Scattered incident context.** An incident channel with bots that page people, post status changes and record key actions keeps one shared timeline [^pd-chatops][^pd-during]. It also creates two new problems, which the rest of this page treats as first-class: **noise** (chat becomes the new place where alerts nobody reads pile up) and **privilege concentration** (one bot token can do what dozens of engineers can do). **The cost of noise (illustrative).** A team channel receives 400 bot messages a week and each one costs ten seconds of attention for each of 12 members who glance at it: 400 × 12 × 10 s ≈ 13 hours of attention a week, before anyone acts on anything. ### What ChatOps is not: boundaries and naming - **Not "a bot in Slack".** A bot that posts jokes or lunch menus is chat automation, not operations. Jason Hand also notes that "ChatOps doesn't necessarily require a bot" [^hand-podcast]: a team that coordinates deploys in a shared channel is already doing part of it. - **Not a replacement for pipelines, runbook automation or IT service management (ITSM).** The command is a front door; the work is done by CI/CD (continuous integration and delivery), a runbook engine or a cloud API behind it ([AutomationOps](../automationops/)). - **Not incident management.** Incident roles, severity and communications belong to [IncidentOps](../incidentops/); ChatOps provides the channel tooling. - **Not the same as "conversational AI for IT".** Vendors market AI chat assistants for operations. They can be one part of ChatOps, under the same rules as any other bot ([Automation and AI](#automation-and-ai)). - **Misconception: chat history is an audit trail.** On Slack, for example, the Audit Logs API exists only on Enterprise plans and records actions, not message content [^slack-audit]; messages can be edited, deleted or expire under retention rules. Keep a separate audit record. ## Where it comes from and where it sits ### Origins and evolution - **2011.** GitHub open-sourced a rewrite of **Hubot**, its Campfire chat bot, which "ships our code" and "is the interface to our CI server"; it had adapters for Campfire and IRC [^hubot-2011]. - **2013.** Jesse Newland's talk **"ChatOps at GitHub"** (slides dated 7 February 2013): "Hubot, GitHub's open source chat bot, is completely revolutionizing how we do ops at GitHub." One slide reads "By placing tools directly in the middle of the conversation" [^newland]. We did not find an earlier documented use of the word. - **2015–2016.** GitHub described deploy queues, guards and locks run from chat [^gh-2015]; Atlassian, then a chat vendor, defined ChatOps as "a collaboration model that connects people, tools, process, and automation into a transparent workflow" [^atl-2016]; O'Reilly published Jason Hand's report [^hand-podcast][^hand-report]. - **2018–2026: platform churn.** Atlassian discontinued Hipchat and Stride after a deal with Slack [^hipchat]; Slack stopped legacy custom bots on 31 March 2025 [^slack-legacy]; Microsoft scheduled the shutdown of Office 365 Connectors in Teams for 18–22 May 2026 [^ms-connectors]; Atlassian will shut down its incident tool Opsgenie on 5 April 2027 and delete customer data that has not been moved [^opsgenie]. Each change broke, or will break, integrations built on the retired layer. - **2021.** GitHub moved deploy state out of chat into a state machine with a UI, while still starting deploys from Slack [^gh-2021]. - **2024–2026: AI in the channel.** Chat platforms and cloud providers added assistants built on large language models (LLMs), and researchers published prompt-injection attacks against them, such as PromptArmor's against Slack AI in 2024 [^promptarmor]; in February 2026 Slack announced a Model Context Protocol (MCP) server for AI agents [^slack-mcp]. The lasting lesson: **keep logic out of the chat adapter**. GitHub went from Hubot scripts, to a shell bridge where "Security contexts were often mixed up with each other", to small RPC endpoints mapped one-to-one to chat commands [^crpc-why]. ### Glossary and mental model **Mental model: three surfaces, one trust chain.** ChatOps has three surfaces: **notifications** (machines talk to people), **commands** (people ask machines to act) and **conversation** (people coordinate around both). The trust chain runs from a chat message to a production API: *who typed it → is the request genuine → may this person do this, here, now → who else must agree → what exactly ran → where is it recorded*. | Term | Meaning | Example | |---|---|---| | Bot / app | A program with its own chat identity that reads events and posts messages | A Slack app with a bot user | | Adapter | The code that connects a bot framework to one chat platform | Hubot's Slack adapter | | Slash command | A registered command the platform routes to your app, with the platform's proof of origin | `/ops status checkout` | | Mention or prefix command | A bot reacting to `@bot …` or `!…` in a message | `!ic page` at PagerDuty [^pd-chatops] | | Command catalogue | The allow-list of commands, their arguments, risk tier and who may run them where | `policy.json` in [Lab 2](#hands-on-labs) | | Command gateway | The service that decides (identity, authorisation, approval) before anything acts | This page's reference design | | Executor | The API or engine that does the work | CI/CD API, runbook engine, cloud API | | Bot token, app-level token, signing secret | Credentials of the app; on Slack `xoxb-`, `xapp-` and a secret for HMAC (hash-based message authentication code) signatures [^slack-tokens][^slack-verify] | Stored in a secret manager | | Proof of origin | What shows a request really came from the chat platform: an HMAC signature on Slack, a signed JSON Web Token (JWT) for Teams bots ([table](#proof-of-origin-differs-by-platform)) | Checked before anything else | | Socket Mode | Slack delivers events over a WebSocket the app opens, so no public endpoint is needed [^slack-socket] | Internal bots behind a firewall | | Ephemeral message | A reply only the invoking user sees | A refusal or help text | | RBAC | Role-based access control: permissions granted to roles, and roles to people | `deployers` may run `deploy` | | Confused deputy | A privileged program (the bot) tricked or misused into acting for someone who lacks the right themselves | Executor logs show only `chatops_bot` | | Two-person rule | A second, different authorised person must approve before a risky action runs (dual authorization) | Production deploy approval | | Incident channel | A channel created for one incident, with bots that page, record and update | `#inc-2026-10-16-checkout` | | MCP | Model Context Protocol: a protocol through which AI agents call tools and read data, including chat content | Slack's MCP server [^slack-mcp] | ### How ChatOps relates to other Ops ChatOps is an **interface layer**: it sits between people and the systems other pages describe. The boundaries below are **editorial distinctions** chosen by this Atlas; no standard defines them. ::: figure Where ChatOps sits among neighbouring Ops pages ```mermaid %%{init: {"flowchart": {"wrappingWidth": 170, "nodeSpacing": 26, "rankSpacing": 42}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 IN1["ObservabilityOps: which alerts exist and fire"]:::neutral IN2["IdentityOps, IAMOps, AccessOps: who people are, what they may do, elevation"]:::neutral C["ChatOps: notifications, commands and channels as an operational interface"]:::core OUT1["AutomationOps: the runbooks and scripts a command triggers"]:::neutral OUT2["IncidentOps: roles and process inside incident channels"]:::neutral OUT3["KnowledgeOps: turning chat answers into maintained runbooks"]:::neutral IN1 -->|"alerts to deliver"| C IN2 -->|"identity and policy"| C C -->|"invokes"| OUT1 C -->|"hosts"| OUT2 C -->|"feeds"| OUT3 ``` Text equivalent: ObservabilityOps supplies the alerts to deliver, and IdentityOps, IAMOps and AccessOps supply identity, permissions and elevation. ChatOps, in the middle, invokes AutomationOps runbooks, hosts IncidentOps processes in incident channels, and feeds KnowledgeOps, which turns answers given in chat into maintained runbooks. ::: | Neighbour | Shared ground | ChatOps focus | Their focus | |---|---|---|---| | [IncidentOps](../incidentops/) | Incident channels | Bots, commands, channel automation | Roles, severity, communication, reviews | | [ObservabilityOps](../observabilityops/) | Alerts | Where and how a notification lands in chat | Whether an alert should exist and fire | | [AutomationOps](../automationops/) | Runbooks, approvals, kill switches | The conversational front door and who may invoke it | Safety and quality of the automation itself | | [IAMOps](../iamops/) · [AccessOps](../accessops/) · [IdentityOps](../identityops/) | Permissions for commands | Enforcing them in the chat path | Designing permissions, elevation, identity lifecycle | | [DevOps](../devops/) | Visible, shared delivery | One practice for visibility | The whole socio-technical model and DORA metrics | | [KnowledgeOps](../knowledgeops/) | Answers given in chat | Capturing them at the source | Keeping them as owned, current knowledge | | [AIOps](../aiops/) · [LLMOps](../llmops/) | AI in operations | AI assistants as bots in the channel | Event correlation; operating LLM applications | ## Principles and operating model ### Charter and principles No official ChatOps manifesto exists. The charter below is **this Atlas's proposal**, distilled from the sources cited in each principle. **Mission.** Make routine operational work visible, safe and teachable by doing it where the team already talks, without making chat the place where control or records live. 1. **Visible by default.** Commands and their results go to shared channels. The first anti-pattern Jason Hand names is that people "treat chat like email": direct-message conversations are "essentially black-boxed" [^hand-podcast]. Keep direct messages for help text and refusals. 2. **Decide outside the adapter.** The chat adapter only translates; a gateway or the service behind it decides. GitHub's protocol states that "CRPC servers are responsible for ensuring that the given `user` has the proper authorization to perform an operation." [^crpc-protocol] 3. **Identity comes from the platform, never from text.** Use the immutable user ID in a verified request; Slack deprecated `user_name` and tells developers to rely on IDs [^slack-slash]. 4. **Allow-list typed commands.** Every command has a name, typed arguments and a risk tier. Never pass free text to a shell. 5. **Friction in proportion to risk.** Reads are instant; reversible writes need the right role in the right channel; high-impact writes need a second person ([dual authorization](#methods-standards-and-frameworks)); emergency overrides exist but are loud, as GitHub's `/deploy!` was [^gh-2015]. 6. **Least privilege twice.** Limit the bot's own scopes and limit the credentials its executors use. 7. **Audit outside chat.** Write a structured record of every decision, allowed or refused, to a store the chat platform cannot edit. 8. **Every notification is a contract.** It has an owner, an audience and an expected action, or it goes to a dashboard instead. 9. **Degrade gracefully.** Plan for the chat platform or the bot being down during the incident you need it for. **Non-goals:** replacing deployment UIs or ITSM records; giving everyone in a channel production rights; automating decisions that need judgement. ### Fundamentals and building blocks Most ChatOps design decisions follow from **risk tiers**: they decide which controls a command needs. | Tier | Examples | Default controls | |---|---|---| | 0 Inform | Deploy started, alert fired, PR stale | Routed by owner and severity; no command rights | | 1 Read | `status`, `graph`, `who is on call`, logs summary | Verified identity; any channel member with the viewer role | | 2 Reversible write | Deploy to staging, scale within limits, silence an alert for 1 hour | Role + allowed channel + typed arguments + audit | | 3 High impact | Production deploy, failover, data repair, feature kill switch | Tier 2 + a second authorised person + time-boxed request | | Break-glass | Emergency override when guards block a fix | Tier 3 waived only by a named role, announced in channel, reviewed afterwards | [AutomationOps](../automationops/#operating-model-and-governance) numbers its automation tiers differently; when a command starts an automation, apply the stricter of the two tiers (an editorial rule both pages share). The building blocks: a **chat platform** (Slack, Microsoft Teams, Mattermost and others), an **adapter or software development kit (SDK)** (Hubot, Errbot, Bolt, Teams SDK), a **command gateway** with a **catalogue** and **identity map**, an **approval** mechanism, **executors** (CI/CD, runbook engines, cloud APIs), a **notification router**, an **audit sink** and a **secret store**. They are arranged in the [reference architecture](#reference-architecture). ### Roles, ownership and RACI ChatOps has no job title of its own. The work is shared between the team that runs the bot as a product and the teams that own commands. | Activity | Bot owner (platform team) | Command owner (service team) | Security | Chat workspace admin | On-call users | |---|---|---|---|---|---| | Add or change a command | C | **A/R** | C (tier 3) | I | I | | Install the app, approve scopes | R | I | C | **A** | I | | Store and rotate tokens and secrets | **A/R** | I | C | I | – | | Define notification routes for a service | C | **A/R** | – | – | C | | Operate the gateway (uptime, upgrades) | **A/R** | I | I | I | I | | Quarterly review of identities, roles, scopes | R | C | **A** | C | – | | Use commands, approve tier-3 requests | – | **A** | – | – | **R** | R = responsible, A = accountable, C = consulted, I = informed. Approvers are a named group (for example on-call leads), never "anyone in the channel". ### Lifecycle of a command: Day 0, Day 1, Day 2 - **Day 0 (design).** A command proposal (see [templates](#templates)) names the owner, the user problem, arguments and their validation, risk tier, executor, rollback and the channels where it may run. Tier 3 proposals get a security review. - **Day 1 (rollout).** The command ships behind the gateway in a test channel, often in dry-run mode first, with `help` text. Its first production use is announced. - **Day 2 (operate).** The owner watches usage, failures and refusals; refusals are as informative as errors (a role is missing, or people use the wrong channel). Unused commands are deprecated, because every command is attack surface. Tokens are rotated on a schedule. ### Operating model and governance - **Policy as code.** The catalogue, identity mapping and notification routes live in a repository; changes go through review, with tier-3 commands owned by named reviewers. - **Channel taxonomy.** Separate channels by purpose: `#deploys-` for change, `#alerts-` for notifications, `#inc--` for incidents, `#ops-sandbox` for testing. Commands are allowed per channel **ID**, not per name ([where?](#identity-and-trust-who-is-really-asking)). - **Cadence.** Weekly glance at refusals and failures; monthly noise review of each alert channel ([metrics](#metrics-and-kpis)); quarterly review of app scopes, identities and approver groups. - **Escalation.** If the bot or chat platform is down, responders fall back to documented phone numbers, the pipeline UI and the runbook engine directly; PagerDuty's paging bot shows the pattern ([Case 2](#use-cases-and-scenarios)). ## Architecture and workflow ### Reference architecture The design below separates **translation** (adapter), **decision** (gateway) and **action** (executors), as GitHub's Chatops RPC does [^crpc-why]. Notifications take a separate path, because they need routing and grouping rather than authorisation. ::: figure Reference architecture: adapter, gateway, executors and the notification path ```mermaid %%{init: {"flowchart": {"wrappingWidth": 170, "nodeSpacing": 26, "rankSpacing": 40}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 subgraph SaaS["Chat platform (vendor)"] U["People in channels"]:::neutral P["Platform API and events"]:::neutral end subgraph Yours["Your trust boundary"] M["Monitoring and CI events"]:::neutral R["Notification router: group, route, update"]:::core A["Adapter: Socket Mode or verified HTTPS; tokens from a secret store"]:::core G["Gateway: identity map, policy, approvals"]:::core L["Append-only audit log"]:::core end X["Executors, a separate privilege zone: CI/CD API, runbook engine, cloud API with short-lived credentials"]:::neutral U --> P P -->|"signed command"| A A --> G G --> L G -->|"scoped call"| X M --> R R -->|"messages"| P ``` Text equivalent: people type in channels on the vendor's chat platform, which sends a signed command to your adapter (over Socket Mode or a verified HTTPS endpoint, with tokens taken from a secret store). The adapter passes it to the gateway, which maps identity, applies policy and approvals, writes to an append-only audit log, and makes a scoped call to the executors: a CI/CD API, a runbook engine or a cloud API with short-lived credentials. The executors are drawn outside your boundary as a separate privilege zone (the cloud API is a provider's). Separately, and also inside your boundary, monitoring and CI events go to a notification router, which groups, routes and updates messages posted back to the platform. ::: | Boundary | What crosses it | Control | |---|---|---| | Platform → adapter | Commands and interaction events | The platform's proof of origin, which differs by platform ([table](#proof-of-origin-differs-by-platform)); on Slack, Socket Mode removes the public endpoint but makes the `xapp-` token critical [^slack-socket] | | Adapter → gateway | User ID, channel, text | In-process call, or a signed RPC; GitHub's protocol signs with RS256 plus nonce and timestamp so replays can be refused [^crpc-protocol] | | Gateway → executors | Typed arguments, requester identity | One credential per executor and tier; read and write paths separated; short-lived credentials where possible | | Router → platform | Messages | Bot token or webhook URL (a webhook URL "contains a secret" [^slack-webhooks]); about one message per second per channel [^slack-rate] | | Platform → AI assistant | Every message it can read | Treat as untrusted input ([Automation and AI](#automation-and-ai)) | **Design choices and trade-offs** - **Socket Mode or HTTPS** (compared in [deep comparisons](#deep-comparisons)). Either way, Slack expects a command to be acknowledged within 3,000 ms [^slack-slash], so verify the request, acknowledge, and do the work afterwards. - **Build or buy the gateway.** Runbook engines with chat front ends (StackStorm, for example) give aliases, role-based access control (RBAC) and approvals out of the box [^st2-aliases][^st2-rbac][^st2-inquiries]; cloud chat integrations give command access to one provider, bounded by identity and access management (IAM) roles and guardrails [^awsq-perms]. A small custom gateway, like [Lab 2](#hands-on-labs), is reasonable when you have a handful of commands. - **Where replies go.** Slack's `response_url` can be used five times within 30 minutes and "will bypass any channel posting permissions" [^slack-interact]; use it for replies to the invoker, and the normal API for announcements (Lab 3 does this). ### Identity and trust: who is really asking? This is the hardest part of ChatOps to get right, and the source of most controls on this page. Answer six questions for every command. 1. **Is the request genuine?** Verify the platform's own proof of origin before doing anything else, and refuse requests that could be replays. The mechanism differs by platform ([table below](#proof-of-origin-differs-by-platform)); on Slack it is an HMAC signature over the raw body plus a timestamp, and Slack's example rejects requests more than five minutes from local time because "It could be a replay attack" [^slack-verify]. [Lab 1](#hands-on-labs) does this. 2. **Who is it?** Map the platform's immutable user ID (Slack `user_id`; on Teams, the Microsoft Entra object ID) to a corporate identity: from a provisioning feed, a maintained table, or the user's verified email (on Slack, reading email needs the `users:read.email` scope [^slack-usersinfo]). Never trust display names. In channels shared with other organisations, check the workspace, organisation or tenant ID as well; Slack payloads carry `team_id` and `enterprise_id` [^slack-slash]. 3. **Whose authority runs the action?** Bots usually act with one powerful identity, which creates a *confused deputy*. In StackStorm, ChatOps executions run as the user configured in hubot (by default `chatops_bot`), so RBAC applies to the bot unless you pass and check the chat user [^st2-rbac][^st2-aliases]. Amazon Q Developer in chat applications instead combines a channel or per-user IAM role with channel guardrails: "What your users can do is the intersection of your guardrail policies and what is allowed by their roles." [^awsq-perms] 4. **Where?** Restrict commands to channels whose members are the right audience, and match channels by **ID**, never by name: Slack says to rely on IDs "as the names might change arbitrarily" [^slack-slash], and where members can create channels, anyone can create one whose name fits a pattern such as `#inc-*`. Let the declare command register each incident channel's ID in the gateway, with an expiry. Read restrictions carefully: Errbot's `allowrooms` means "Allow command only in these rooms (and direct messages)" unless you also turn off private use [^errbot]. 5. **With which arguments?** Validate each argument against a type or pattern and reject the rest. StackStorm passes extra `key=value` pairs not in the alias format ("it will still be extracted and supplied for execution"); pin what users must not change with `immutable_parameters` [^st2-aliases]. 6. **Who else agreed?** For tier 3, a different person from the approver group, in the same channel, within a time limit, once, having seen exactly what will run. If you use buttons, check the clicker's identity on the server; never treat an emoji reaction as an approval. #### Proof of origin differs by platform The gateway can be the same on every platform; the adapter's trust checks cannot. Status as of October 2026, from each vendor's documentation; check the current pages before you build. | Inbound path | Proof of origin (secret to protect) | Replay protection | Response deadline | Immutable user ID | |---|---|---|---|---| | Slack app over HTTPS | HMAC-SHA256 over `v0:timestamp:body` [^slack-verify] (signing secret; the bot token too) | Signed timestamp; reject if more than five minutes off [^slack-verify] | Ack within 3,000 ms [^slack-slash] | `user_id`, with `team_id` and `enterprise_id` [^slack-slash] | | Slack Socket Mode | None needed per event: events arrive "over a pre-authenticated WebSocket" the app opened [^slack-socket] (app-level `xapp-` token) | No inbound endpoint; acknowledge each event "so that Slack knows whether to retry" [^slack-socket] | Not stated on the Socket Mode page; acknowledge at once, as Lab 3 does | As above | | Teams bot (Bot Framework, Teams SDK) | Signed JWT in the `Authorization` header: check issuer, audience (your app ID), validity, signature against keys from the published OpenID metadata, and the `serviceUrl` claim; the Bot Framework SDK does this for you [^ms-botauth] (bot app password, a client secret) | Token validity only (Microsoft cites a 5-minute clock skew) [^ms-botauth]; add your own duplicate check | For invoke activities such as message-extension submits, five seconds, after which Teams retries twice [^ms-invoke] | `from.aadObjectId` (Entra object ID); tenant ID in `channelData` [^ms-conv] | | Teams outgoing webhook | HMAC-SHA256 over the body with a security token that "doesn't expire and is unique for each configuration" [^ms-outgoing] (that token) | No freshness check documented. The body follows the Bot Framework schema [^ms-outgoing], so the HMAC covers its `timestamp` and `id` when present; that spec says bots SHOULD NOT reject by `timestamp` and MAY de-duplicate by `id` [^bf-activity] | Five seconds [^ms-outgoing] | The sender is in `from` [^ms-outgoing]; inspect a test request for the ID fields | | Mattermost custom slash command | A static token in the `Authorization` header, compared with the command's token [^mm-slash] (that token) | None: the token does not cover the body | Use `response_url` for work over three seconds [^mm-slash] | `user_id` [^mm-slash] | Consequences: retries, and paths without a signed freshness check, mean the gateway must refuse duplicates itself (an idempotency key per request). Where the token does not cover the body, as on Mattermost, anyone holding it can forge any `user_id` (our reading of the documented scheme), so keep it in the secret store and never log raw request headers. ::: caution The bot is a privileged system A bot token that can deploy is a production credential held by a chat integration. Store it in a secret manager, never in code or chat; restrict where it can be used from (Slack can limit Web API token use to listed IP ranges, but "Local" addresses cannot be listed and the limit does not apply to incoming webhooks [^slack-security]); rotate it; and know how to revoke it in minutes ([runbook](#runbook-a-leaked-bot-token)). On Slack, "Token rotation may not be turned off once it's turned on." [^slack-tokens], so plan the refresh flow first. ::: ### End-to-end workflow: one command, happy and exception paths ::: figure A tier-3 command from request to audited result ```mermaid %%{init: {"sequence": {"actorMargin": 24, "width": 120, "messageMargin": 30, "boxMargin": 6, "noteMargin": 6, "diagramMarginX": 8}}}%% sequenceDiagram participant P as People in channel participant C as Chat platform participant G as Gateway participant X as Executor P->>C: 1. Alice: /ops deploy … prod C->>G: 2. Signed request Note over G: 3. Verify origin and age G-->>C: 4. Ack (or error) Note over G: 5. Map ID, check policy alt Refused G-->>C: 6a. Refusal, audited C-->>P: Shown to Alice only else Tier 3 G->>C: 6b. PENDING: what, where, who C->>P: Visible to the channel P->>C: 7. Bob: approve req-0001 C->>G: 8. Signed approval, verified G->>X: 9. Scoped call X-->>G: 10. Result G->>C: 11. Result in thread, audited end ``` Text equivalent: Alice, one of the people in the channel, types a production deploy command. The chat platform sends a signed request to the gateway, which first verifies its origin and age, then acknowledges it (or returns an error if it cannot be verified), and only then maps the user ID to an identity and checks policy. If refused, the gateway sends the refusal through the platform as an ephemeral message that only Alice sees, and audits it. For a tier-3 command the gateway posts a pending request that states exactly what will run, where and for whom; Bob, a different approver in the same channel, types the approval; the platform sends a signed approval, which the gateway verifies; the gateway calls the executor with a scoped credential, receives the result and posts it in the thread, writing an audit record. ::: **Exception paths to design for:** the executor times out after the acknowledgement (post "still running" and a link); the approval expires (the request is cancelled and audited); the chat platform is down (fall back to the pipeline UI and record what was done outside the gateway, so the audit trail can be completed later); the command partially succeeded (the reply says exactly what changed and how to roll back). ### Notification flow design Notifications are where ChatOps most often degrades. The SRE book's monitoring chapter notes that email alerts are sometimes called "alert spam" because people seldom act on them [^sre-monitoring]. **Editorial interpretation:** an alert channel nobody reads is today's version of that email alias. Which alerts should exist, and the evidence on alert fatigue, belong to [ObservabilityOps](../observabilityops/); deduplication, grouping and inhibition to [AIOps](../aiops/). This section covers only what happens once a notification is on its way into chat. ::: figure Routing an event into chat by severity and audience ```mermaid %%{init: {"flowchart": {"wrappingWidth": 160, "nodeSpacing": 24, "rankSpacing": 38}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 EV["Alert or event, with owner and runbook labels"]:::neutral GR["Group and deduplicate; suppress symptoms of a known cause"]:::core SEV{"Needs a human now?"}:::decision PG["Page on-call, and one feed message (bot token)"]:::risk ACT{"Needs action this week?"}:::decision TM["Team channel via webhook, no mentions, no resolved post"]:::core DB["Dashboard or daily digest, not chat"]:::neutral RS["Resolved: bot edits the feed message"]:::core EV --> GR --> SEV SEV -->|"yes"| PG SEV -->|"no"| ACT ACT -->|"yes"| TM ACT -->|"no"| DB PG --> RS ``` Text equivalent: every alert or event carries owner and runbook labels. It is grouped and deduplicated, and symptoms of a known cause are suppressed. If it needs a human now, on-call is paged and one feed message is posted with a bot token. If it needs action this week, it goes to the team channel through a webhook, without mentions and without a "resolved" post. Otherwise it goes to a dashboard or daily digest, not to chat. When a paged alert resolves, the bot edits the original feed message rather than posting a new one; the team channel cannot do this, because a webhook cannot edit messages. ::: Rules that follow from it: - **One channel per audience, not per tool.** A team should not have to watch five channels to know its service is healthy. - **Group before you post.** Slack accepts about one message per second per channel, and over-limit bursts may not be displayed [^slack-rate], so an ungrouped storm loses messages exactly when they matter. Grouping and inhibition are taught in [AIOps](../aiops/#lab-1-beginner-group-and-inhibit-with-alertmanager). - **Update, do not append.** Since version 0.32.0 Alertmanager can edit an existing Slack message when an alert changes state [^am-changelog]; this needs a bot token, because "Webhook URLs do not support updates" [^am-config]. Set the token in `http_config` (as below) or, since 0.30.0, in `app_token_file` with `app_url` [^am-changelog]. Choose per receiver: a webhook is simpler and bound to one channel, a bot token can edit but is a broader credential. - **Mentions are for pages only.** `@here` on an informational alert teaches people to mute the channel. - **Every message carries an owner and a next step** (runbook link, dashboard, command), and **never secrets or personal data**. Paging and escalation policy belong to [IncidentOps](../incidentops/). ## Practices, frameworks and tools ### Core practices and playbooks | Practice | Goal | Owner | Verification | |---|---|---|---| | Typed command catalogue | Only known actions with validated arguments can run | Bot owner, command owners | Every command has a syntax with typed arguments, a tier, channels and roles; CI rejects an entry without them | | Deploy coordination | One change at a time, visible to all | Service team | Queue, lock and guard behaviour tested in a sandbox channel | | Two-person approval for tier 3 | No single person (or token) can make a high-impact change | Command owner, security | Self-approval, cross-channel and replayed approvals are refused (Lab 2) | | Notification contract | Every message has an owner and a next step | Service team | Monthly actionable-ratio review per channel | | Incident channel automation | Fast assembly and a shared record | IncidentOps owner, bot owner | Game-day: declare a test incident, check channel, paging and timeline | | Token and scope hygiene | A leaked credential is worth little and short-lived | Bot owner, security | Quarterly scope audit; rotation date per credential; revocation drill | | Knowledge capture | Answers given in chat become runbooks | Team lead | Recurring questions tracked and turned into runbooks ([KnowledgeOps](../knowledgeops/)) | **Playbook: designing a command.** 1. Start from a repeated manual task, or a time-critical incident action, with an API behind it ([decision framework](#decision-framework)); name it verb-first (`deploy`, `status`, `silence`). 2. Define each argument's type or pattern and a safe default (staging, not production). 3. Echo the parsed arguments before acting on anything above tier 1 ("Deploying checkout v1.4.2 to **production**"). 4. Make it idempotent or refuse duplicates; give it a `--dry-run` where the executor supports one. 5. Reply in a thread; link to the pipeline or dashboard for rich output instead of pasting it. 6. Write `help` text and an example; list the command in the catalogue with its tier, channels and roles. 7. **Acceptance:** an unauthorised user, a wrong channel and malformed arguments are each refused with a useful message, and all three appear in the audit log. **Rollback:** a documented inverse command or pipeline action. **Playbook: coordinating deploys in chat** (pattern from GitHub's 2015 practice [^gh-2015]). Queue (`queue me`), guard (CI green, branch up to date, right room), lock the environment while a change is verified, unlock on merge and tell the next person, and allow a loud emergency override. **Playbook: incident channel tooling** (process owned by [IncidentOps](../incidentops/)). A declare command creates the channel, registers its ID with the gateway, invites on-call, posts links and starts a timeline ([Case 2](#use-cases-and-scenarios)); a scribe or bot records key actions [^pd-during]. Pair chat with a call for intense incidents: "Disjointed discussions in the chat room are ultimately distracting." [^pd-during] ### Methods, standards and frameworks There is **no ChatOps standard**. What exists, and how binding it is: | Source | Status | What to take from it | |---|---|---| | NIST SP 800-53 Rev. 5, AU-3 *Content of Audit Records* [^nist] | Mandatory for US federal systems; a widely used reference elsewhere | Each audit record should establish what type of event occurred, when, where, the source, the outcome, and the identity of those involved: use it as the schema for command audit records | | NIST SP 800-53, AC-3(2) *Dual Authorization* ("also known as two-person control") and CM-5(4) *Dual Authorization* for changes [^nist] | As above | The basis for the two-person rule on tier-3 commands. AC-3(2)'s guidance notes the risk of dual authorization "when immediate responses are necessary to ensure public and environmental safety"; a reviewed break-glass path is this page's answer to the same tension during outages | | NIST SP 800-53, AC-5 *Separation of Duties* and AC-6 *Least Privilege* [^nist] | As above | Requester and approver are different people; minimal bot and executor scopes | | OWASP Top 10 for LLM Applications 2025: LLM01 prompt injection, LLM06 excessive agency [^owasp-llm01][^owasp-llm06] | Industry guidance | Treat everything an AI assistant reads as untrusted; limit its functions, permissions and autonomy; require human approval for high-impact actions | | Slack security best practices [^slack-security]; equivalent vendor guidance for other platforms | Vendor recommendation | Scope policy with risk classes, secret managers, IP allowlisting, token rotation and revocation | | GitHub's Chatops RPC protocol [^crpc-protocol] | One company's published design | Thin adapter, signed requests with nonce and timestamp, authorisation in the service | ### Tooling landscape Categories first, examples second; versions and statuses are **as of October 2026**. | Category | Verified examples | Strengths | Limits and risks | Choose on | |---|---|---|---|---| | Chat platform | Slack, Microsoft Teams; self-hosted platforms such as Mattermost and Rocket.Chat | Where people already are | Vendor changes break integrations ([origins](#origins-and-evolution)); Slack's classic-app retirement was paused in December 2025 [^slack-classic-pause] | Identity integration, audit and retention, data residency, API stability | | Bot framework or SDK | Hubot 14.1 (Node) [^hubot-repo]; Errbot 6.2 (Python) [^errbot]; Slack Bolt (Python 1.30, JavaScript 5.1) [^bolt]; Teams SDK [^teams-sdk] | Adapters, routing, middleware | Lita (Ruby) has had no release since 4.8.0 in 2020 [^lita]; maintenance health varies | Language your team maintains; adapter health | | Automation with a chat front end | StackStorm 3.9: aliases, RBAC, Inquiries (approvals) [^st2-chatops][^st2-aliases][^st2-rbac][^st2-inquiries] | Approvals, audit and workflows built in | st2chatops 3.9 officially supports Slack only [^st2-chatops]; executions run as the bot user [^st2-rbac] | Existing runbook engine; RBAC model | | Cloud provider in chat | Amazon Q Developer in chat applications (AWS Chatbot until 2025-02-19) [^awsq-rename] | Notifications and CLI-style commands with IAM roles and guardrails [^awsq-perms] | One provider; guardrails default to AdministratorAccess if unset via API or CloudFormation [^awsq-api] | Whether reads suffice; guardrail policy | | Notification router | Prometheus Alertmanager 0.34 [^am-config][^am-changelog] | Grouping, inhibition, routing, message updates | Needs labels and ownership discipline | Label hygiene; on-call integration | | Incident tools with chat | PagerDuty, incident.io and others ([IncidentOps](../incidentops/)); Opsgenie shuts down on 5 April 2027 [^opsgenie] | Channel creation, paging, timelines | Incident history locked in a vendor | Export of timelines; identity mapping | | Kubernetes chat tools | Botkube, which now presents itself as a Kubernetes AI assistant [^botkube] | Cluster events and commands in chat | Cluster credentials behind a chat bot | Read-only first; namespace scoping | **Lock-in advice.** Keep the catalogue, identity map, policy and audit in your gateway, and treat platform adapters as replaceable; the [platform churn since 2018](#origins-and-evolution) shows why. ### Automation and AI Chat platforms and cloud providers now offer AI assistants (Slack AI is one [^promptarmor]). Useful jobs: summarising a long incident channel, drafting a status update, explaining an error, proposing the next command, answering questions over read-only data. The risk is acute in chat: an assistant reads **everything** in its reach, and any of that text can carry instructions. - **Indirect prompt injection is a chat-native threat.** OWASP's LLM01 covers instructions arriving through external content such as websites or files [^owasp-llm01]. In August 2024 PromptArmor showed that a message planted in a public Slack channel could make Slack AI render a link carrying data from a private channel the attacker was not in [^promptarmor]; Salesforce told The Register "We've deployed a patch to address the issue" [^register]. Any channel member, integration or uploaded file is a possible author of the next instruction. - **Excessive agency.** OWASP's LLM06 traces damage to excessive functionality, permissions and autonomy, and recommends human approval for high-impact actions [^owasp-llm06]. In ChatOps terms: the assistant is a bot like any other, behind the same gateway, tiers and approvals; it may **propose** a command, a human **issues** it. - **Guardrails you do not control** are a floor, not your policy. Amazon Q Developer in chat applications never runs `iam:*`, `kms:*`, `secretsmanager:*` or `sts:*` operations from chat, but its block list does not cover log queries or table scans [^awsq-perms] ([reference implementation](#reference-implementation)). - **Practical rules.** Give the assistant its own identity and minimal read scopes; exclude channels that may contain secrets from retrieval; never let it approve; log its prompts, retrieved sources and tool calls; and disable link rendering or unfurling where it could carry data out. The [judgement call](#a-judgement-call-should-the-ai-assistant-in-the-incident-channel-act) works through a concrete case. ## ChatOps in practice ### Reference implementation **Assumptions (illustrative).** About 40 engineers, six services, Slack as the chat platform, Prometheus and Alertmanager for alerts, a CD system with an API, and AWS. On Teams the catalogue, gateway and audit stay the same; only the adapter changes, and it must check Bot Framework tokens and map Entra object IDs instead ([proof of origin](#proof-of-origin-differs-by-platform)). **Constraints:** no command may reach production without a second person; the bot runs inside the company network; nothing the bot posts may contain secrets or personal data. The design has four artefacts plus the gateway from [Lab 2](#hands-on-labs). **1. The command catalogue, reviewed like code.** It is the single place that says what can run, where, by whom, and how to undo it. _Example status: YAML syntax-checked, and each command's full pattern built and compiled as in Lab 2, with Python 3.8; not executed against real executors._ ```yaml # commands.yaml: the command catalogue, reviewed like code. # Tier-3 entries need a security reviewer. # Each argument has its own pattern; the gateway joins them with the literal # words of 'syntax' and full-matches the whole message (Lab 2 does the same). # Channels are matched by ID, because names can change. 'incident-registry' # means any channel ID that the declare command registered and that is not # yet archived. version: 1 defaults: approval_ttl_minutes: 15 reply: thread commands: - name: status tier: 1 owner: team-checkout syntax: 'status {service}' args: {service: '[a-z0-9-]{1,40}'} roles: [viewer, oncall, deployer] channels: ['*'] executor: status-api - name: silence tier: 2 owner: team-observability syntax: 'silence {alert} for {minutes}m' args: {alert: '[A-Za-z0-9_]{1,64}', minutes: '[1-9][0-9]?'} roles: [oncall] # C0ALERTSCHK is #alerts-checkout channels: [C0ALERTSCHK, incident-registry] executor: alertmanager-silences rollback: 'unsilence ' - name: deploy tier: 2 tier_3_when: {env: production} owner: team-checkout syntax: 'deploy {service} {version} to {env}' args: service: '[a-z0-9-]{1,40}' version: 'v[0-9]+[.][0-9]+[.][0-9]+' env: 'staging|production' roles: [deployer] channels: [C0DEPLOYS] # #deploys-checkout approvers: [oncall-leads] executor: cd-pipeline-api rollback: 'deploy to ' - name: failover tier: 3 owner: team-platform syntax: 'failover {db} to {region}' args: {db: '[a-z0-9-]{1,40}', region: 'eu-west-1|eu-central-1'} roles: [oncall-leads] channels: [incident-registry] approvers: [platform-leads] executor: runbook-engine # refuse to run if the runbook has not been rehearsed recently rehearsed_within_days: 90 rollback: 'runbook RB-DB-07, section "fail back"' denied_by_design: - free-form shell, kubectl or SQL passthrough - any command that prints a secret, token or connection string ``` **2. Notification routing into chat.** Pages go to the pager and to one feed message that is edited as the alert changes; tickets go to the team channel without mentions; informational alerts stay out of chat; a known outage suppresses its own symptoms. _Example status: YAML syntax-checked against the Alertmanager configuration reference (version 0.34 docs); not loaded into Alertmanager (check it with `amtool check-config alertmanager.yml` before use)._ ```yaml # alertmanager.yml (excerpt): delivery into chat. # The alert rules themselves live with ObservabilityOps. route: receiver: team-checkout-chat group_by: [alertname, service, env] group_wait: 30s group_interval: 5m repeat_interval: 4h routes: - matchers: ['severity="page"'] receiver: oncall-pager continue: true - matchers: ['severity="page"'] receiver: checkout-feed - matchers: ['severity="ticket"'] receiver: team-checkout-chat repeat_interval: 24h - matchers: ['severity="info"'] receiver: no-chat # How grouping and inhibition work is taught on the AIOps page. inhibit_rules: - source_matchers: ['alertname="CheckoutDown"'] target_matchers: ['service="checkout"', 'severity=~"ticket|info"'] equal: [env] receivers: - name: oncall-pager pagerduty_configs: - routing_key_file: /etc/alertmanager/secrets/pagerduty-checkout - name: checkout-feed slack_configs: - api_url: https://slack.com/api/chat.postMessage http_config: authorization: credentials_file: /etc/alertmanager/secrets/slack-bot-token # Channel ID of #alerts-checkout: a rename must not stop the feed. channel: C0ALERTSCHK send_resolved: true update_message: true title: >- {{ .Status | toUpper }}: {{ .CommonLabels.alertname }} ({{ .Alerts.Firing | len }} firing) text: >- {{ .CommonAnnotations.summary }} Runbook: {{ .CommonAnnotations.runbook_url }} - name: team-checkout-chat slack_configs: - api_url_file: /etc/alertmanager/secrets/slack-webhook-checkout # Informational only: a Slack webhook always posts to the channel # chosen when the webhook was created. channel: '#checkout-team' send_resolved: false - name: no-chat ``` **3. A diagnostics-only cloud channel.** For AWS diagnostics in chat, a channel configuration with per-user roles and a guardrail that allows only the calls responders need. The guardrail is the ceiling: users can do only what both the guardrail and their own role allow [^awsq-perms], and leaving `GuardrailPolicies` unset applies AdministratorAccess [^awsq-api]. Two tempting managed policies are wrong here: - **`ReadOnlyAccess` reads data, not just configuration.** AWS warns that it gives "access to read data in storage services like Amazon S3 buckets and Amazon DynamoDB tables" [^aws-jobfn], and it includes `logs:FilterLogEvents`, `logs:Get*`, `dynamodb:Scan` and `dynamodb:Query` [^aws-readonly]. The chat service's own block list stops `s3:GetObject` and `dynamodb:GetItem`, not log reads, scans or queries [^awsq-perms], so log lines and table items could be posted into a shared channel. - **`ViewOnlyAccess` cannot read resource content** [^aws-jobfn], but its current version (v47, edited 2026-10-14) allows `cloudwatch:Get*` and `List*`, not `cloudwatch:Describe*`, so a responder could not read alarm states [^aws-viewonly]. The template therefore defines its own guardrail. Side effects to accept knowingly: every reply is visible to everyone in the channel; the service always writes an audit event for each executed command to CloudWatch Logs [^awsq-logs], and `LoggingLevel` (ERROR, INFO or NONE) controls which other entries are pushed there [^awsq-api], so restrict who can read those log groups; and alarm notifications offer a "Show logs" button that runs a CloudWatch Logs Insights query and displays the first 30 log entries [^awsq-cli]. This guardrail grants no `logs:` actions, so we expect that button to be refused (not tested). _Example status: YAML syntax-checked against the `AWS::Chatbot::SlackChannelConfiguration` and `AWS::IAM::ManagedPolicy` references; not deployed. The channel role must already exist; its trust policy is not shown because we did not verify it against an opened AWS page._ ```yaml AWSTemplateFormatVersion: "2010-09-09" Description: Diagnostics-only Amazon Q Developer chat channel (illustrative). Parameters: SlackWorkspaceId: Type: String SlackChannelId: Type: String ChannelRoleArn: Type: String Description: Existing IAM role the service assumes for this channel AlertsTopicArn: Type: String Resources: DiagnosticsGuardrail: Type: AWS::IAM::ManagedPolicy Properties: Description: Ceiling for chat commands - status and metadata only, no writes PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - cloudwatch:DescribeAlarms - cloudwatch:DescribeAlarmHistory - cloudwatch:GetMetricData - ecs:DescribeServices - ecs:ListServices - rds:DescribeDBInstances - elasticloadbalancing:DescribeTargetHealth Resource: "*" CheckoutOpsChannel: Type: AWS::Chatbot::SlackChannelConfiguration Properties: ConfigurationName: checkout-ops-diagnostics SlackWorkspaceId: Ref: SlackWorkspaceId SlackChannelId: Ref: SlackChannelId IamRoleArn: Ref: ChannelRoleArn GuardrailPolicies: - Ref: DiagnosticsGuardrail # Ref returns the managed policy's ARN UserRoleRequired: true LoggingLevel: INFO SnsTopicArns: - Ref: AlertsTopicArn ``` **4. The audit record.** One record per decision, with the NIST AU-3 elements (what, when, where, source, outcome, identity) [^nist], written by the gateway to append-only storage outside the chat platform. The file holds one record per line (JSON Lines); the record below is real output from Lab 2, pretty-printed exactly as its `simulate.py` prints it. ```json { "event_type": "chatops.command", "time": "2026-10-03T04:09:00+00:00", "where": "C0DEPLOYS", "source": "slack:T0EXAMPLE", "outcome": "executed", "identity": { "chat_user": "U02BOB", "principal": "bob@example.com" }, "text": "approve req-0001", "command": "deploy", "args": { "service": "checkout", "version": "v1.4.2", "env": "production" }, "request_id": "req-0001", "requested_by": "alice@example.com", "approved_by": "bob@example.com" } ``` ### Hands-on labs Three labs from beginner to advanced. Labs 1 and 2 need only the Python standard library and **were executed** with Python 3.8.10, most recently on 2026-10-17; the outputs shown are copied from those runs. Lab 3 needs a Slack workspace and was **not executed**. Use a supported Python release (3.10 or later); the code needs no newer features than 3.8. #### Lab 1 (beginner): reject forged and replayed requests **Objective.** Verify a Slack request signature, and see why the raw body and the timestamp matter [^slack-verify]. **Prerequisites.** Python 3 (see above); no network or Slack account. **Cost and risk.** None; the secret is Slack's published test value. 1. Save the script as `verify_slack.py`. 2. Run `python3 verify_slack.py`. _Example status: executed with Python 3.8.10 on 2026-10-17._ ```python """Lab 1: verify a Slack request signature (stdlib only).""" import hashlib import hmac import time MAX_SKEW_SECONDS = 300 # Slack's example rejects requests over five minutes old def verify(secret: str, timestamp: str, raw_body: str, signature: str, now=None): """Return (ok, reason). raw_body must be the exact bytes Slack sent, before any parsing. """ if not timestamp.isdigit() or not signature.startswith("v0="): return False, "malformed headers" now = time.time() if now is None else now if abs(now - int(timestamp)) > MAX_SKEW_SECONDS: return False, "stale timestamp (possible replay)" base = f"v0:{timestamp}:{raw_body}".encode() expected = "v0=" + hmac.new(secret.encode(), base, hashlib.sha256).hexdigest() if not hmac.compare_digest(expected, signature): return False, "signature mismatch" return True, "ok" if __name__ == "__main__": # Test vector from Slack's request-verification guide (not a live secret). secret = "8f742231b10e8888abcd99yyyzzz85a5" ts = "1531420618" body = ("token=xyzz0WbapA4vBCDEFasx0q6G&team_id=T1DC2JH3J" "&team_domain=testteamnow&channel_id=G8PSS9T3V&channel_name=foobar" "&user_id=U2CERLKJA&user_name=roadrunner&command=%2Fwebhook-collect" "&text=&response_url=https%3A%2F%2Fhooks.slack.com%2Fcommands" "%2FT1DC2JH3J%2F397700885554%2F96rGlfmibIGlgcZRskXaIFfN" "&trigger_id=398738663015.47445629121.803a0bc887a14d10d2c447fce8b6703c") sig = "v0=a2114d57b48eac39b9ad189dd8316235a7b4a8d21a10bd27519666489c69b503" fresh = int(ts) + 10 cases = [ ("documented request", body, sig, fresh), ("body changed", body.replace("text=", "text=deploy"), sig, fresh), ("replayed 10 min later", body, sig, int(ts) + 600), ("wrong scheme", body, sig.replace("v0=", "v1="), fresh), ] for name, b, s, now in cases: ok, reason = verify(secret, ts, b, s, now=now) print(f"{name:24} -> {'ACCEPT' if ok else 'REJECT'} ({reason})") ``` **Expected output.** ```text documented request -> ACCEPT (ok) body changed -> REJECT (signature mismatch) replayed 10 min later -> REJECT (stale timestamp (possible replay)) wrong scheme -> REJECT (malformed headers) ``` **Validation.** Change one character of `secret`: all four cases must now reject. **Troubleshooting.** A real request that fails verification usually means the framework parsed and re-encoded the body before you hashed it (hash the raw bytes), or the server clock is wrong (check NTP). **Teardown.** Delete the file. **On Teams** the equivalent check is different: a bot validates a signed JWT, and an outgoing webhook checks an HMAC over the body with no documented freshness check ([proof of origin](#proof-of-origin-differs-by-platform)). #### Lab 2 (intermediate): a command gateway that decides before it acts **Objective.** Implement the [identity and trust](#identity-and-trust-who-is-really-asking) checks: workspace allow-list, identity mapping by immutable ID, typed arguments, channel and role checks, a two-person rule and AU-3 audit records. **Prerequisites.** Python 3 (see above). **Cost and risk.** None; the executor is a stub that only prints what it would do. 1. Create a directory with the three files below and run `python3 simulate.py` from inside it. 2. Read each refusal and find the line of `gateway.py` that produced it. Note that each argument in `policy.json` has its own pattern, and `compile_syntax` joins them into one regular expression that must match the whole message. _Example status: all three files executed with Python 3.8.10 on 2026-10-17._ ```json { "allowed_teams": ["T0EXAMPLE"], "approval_ttl_seconds": 900, "approver_groups": ["approvers"], "identities": { "U01ALICE": {"principal": "alice@example.com", "groups": ["oncall", "deployers"]}, "U02BOB": {"principal": "bob@example.com", "groups": ["deployers", "approvers"]}, "U03CAROL": {"principal": "carol@example.com", "groups": ["viewers"]} }, "commands": { "status": { "syntax": "status {service}", "args": {"service": "[a-z0-9-]{1,40}"}, "risk": "read", "groups": ["viewers", "oncall", "deployers"], "channels": "*" }, "deploy": { "syntax": "deploy {service} {version} to {env}", "args": { "service": "[a-z0-9-]{1,40}", "version": "v[0-9]+[.][0-9]+[.][0-9]+", "env": "staging|production" }, "risk": "write", "groups": ["deployers"], "channels": ["C0DEPLOYS"], "approval_if": {"env": "production"} } } } ``` ```python """Lab 2: a chat-command gateway that decides before it acts (stdlib only).""" import json import re from datetime import datetime, timezone ARG = re.compile(r"\{(\w+)\}") def compile_syntax(cmd): """'deploy {service} to {env}' plus one pattern per argument -> one regex.""" parts = ARG.split(cmd["syntax"]) # literal, name, literal, name, ..., literal return re.compile("".join( re.escape(part) if i % 2 == 0 else f"(?P<{part}>{cmd['args'][part]})" for i, part in enumerate(parts))) class Gateway: def __init__(self, policy, audit_path, executor): self.p, self.audit_path, self.execute = policy, audit_path, executor self.rx = {n: compile_syntax(c) for n, c in policy["commands"].items()} self.pending, self.counter = {}, 0 def _audit(self, now, ev, who, outcome, **extra): # NIST SP 800-53 AU-3: what, when, where, source, outcome, identity rec = {"event_type": "chatops.command", "time": datetime.fromtimestamp(now, timezone.utc).isoformat(), "where": ev["channel_id"], "source": f"slack:{ev['team_id']}", "outcome": outcome, "identity": {"chat_user": ev["user_id"], "principal": who}, "text": ev["text"], **extra} with open(self.audit_path, "a", encoding="utf-8") as f: f.write(json.dumps(rec) + "\n") # JSON Lines: one record per line def handle(self, ev, now): text = ev["text"].strip() if ev["team_id"] not in self.p["allowed_teams"]: self._audit(now, ev, None, "denied:external-workspace") return "DENIED: commands are accepted only from our own workspace" # the immutable chat user ID, never a display name ident = self.p["identities"].get(ev["user_id"]) if ident is None: self._audit(now, ev, None, "denied:unknown-identity") return "DENIED: your chat account is not linked to a corporate identity" who, groups = ident["principal"], set(ident["groups"]) if text == "help": mine = [n for n, c in self.p["commands"].items() if groups & set(c["groups"])] return ("You may run: " + ", ".join(mine) + " (and 'approve ' if you are an approver)") if text.startswith("approve "): return self._approve(ev, now, who, groups, text.split(" ", 1)[1]) for name, cmd in self.p["commands"].items(): m = self.rx[name].fullmatch(text) # fullmatch: no trailing "; rm -rf /" if m: break else: self._audit(now, ev, who, "rejected:no-matching-command") return "REJECTED: unknown command or invalid arguments (try: help)" args = m.groupdict() if cmd["channels"] != "*" and ev["channel_id"] not in cmd["channels"]: self._audit(now, ev, who, "denied:channel", command=name) return f"DENIED: '{name}' may only run in {', '.join(cmd['channels'])}" if not groups & set(cmd["groups"]): self._audit(now, ev, who, "denied:role", command=name) return f"DENIED: '{name}' needs one of {sorted(cmd['groups'])}" cond = cmd.get("approval_if", {}) if cond and all(args.get(k) == v for k, v in cond.items()): self.counter += 1 rid = f"req-{self.counter:04d}" self.pending[rid] = {"name": name, "args": args, "by": who, "channel": ev["channel_id"], "expires": now + self.p["approval_ttl_seconds"]} self._audit(now, ev, who, "pending-approval", command=name, args=args, request_id=rid) # approvers see exactly what will run what = " ".join(f"{k}={v}" for k, v in args.items()) return (f"PENDING {rid}: {name} {what}; " f"another approver must run 'approve {rid}'") result = self.execute(name, args, who) self._audit(now, ev, who, "executed", command=name, args=args) return f"OK: {result}" def _approve(self, ev, now, who, groups, rid): req = self.pending.get(rid) if req is None or now > req["expires"]: self.pending.pop(rid, None) self._audit(now, ev, who, "denied:no-such-or-expired-request", request_id=rid) return f"DENIED: {rid} does not exist or has expired" if ev["channel_id"] != req["channel"]: # approve where everyone can see it self._audit(now, ev, who, "denied:approval-outside-channel", request_id=rid) return (f"DENIED: approve {rid} in {req['channel']}, " "where it was requested") if who == req["by"]: self._audit(now, ev, who, "denied:self-approval", request_id=rid) return "DENIED: the requester cannot approve their own request" if not groups & set(self.p["approver_groups"]): self._audit(now, ev, who, "denied:not-an-approver", request_id=rid) return "DENIED: you are not an approver" del self.pending[rid] # single use result = self.execute(req["name"], req["args"], req["by"]) self._audit(now, ev, who, "executed", command=req["name"], args=req["args"], request_id=rid, requested_by=req["by"], approved_by=who) return f"OK ({rid}, approved by {who}): {result}" ``` ```python """Replays a short chat conversation through the gateway.""" import json import os import textwrap from gateway import Gateway def fake_executor(name, args, requested_by): return f"dry run - would call the {name} API with {args} for {requested_by}" policy = json.load(open("policy.json", encoding="utf-8")) if os.path.exists("audit.jsonl"): os.remove("audit.jsonl") gw = Gateway(policy, "audit.jsonl", fake_executor) T0 = 1_791_000_000 # fixed clock so the run is reproducible T = "T0EXAMPLE" # our own workspace conversation = [ (T, "U03CAROL", "C0RANDOM", "help"), (T, "U03CAROL", "C0RANDOM", "status checkout"), (T, "U03CAROL", "C0DEPLOYS", "deploy checkout v1.4.2 to staging"), (T, "U01ALICE", "C0RANDOM", "deploy checkout v1.4.2 to staging"), (T, "U01ALICE", "C0DEPLOYS", "deploy checkout v1.4.2 to staging; rm -rf /"), (T, "U01ALICE", "C0DEPLOYS", "deploy checkout v1.4.2 to staging"), (T, "U01ALICE", "C0DEPLOYS", "deploy checkout v1.4.2 to production"), (T, "U01ALICE", "C0DEPLOYS", "approve req-0001"), (T, "U02BOB", "D0BOBDM", "approve req-0001"), (T, "U02BOB", "C0DEPLOYS", "approve req-0001"), (T, "U02BOB", "C0DEPLOYS", "approve req-0001"), ("T0PARTNER", "U09EXT", "C0SHARED", "status checkout"), (T, "U07MALLORY", "C0DEPLOYS", "status checkout"), ] for i, (team, user, channel, text) in enumerate(conversation): ev = {"team_id": team, "user_id": user, "channel_id": channel, "text": text} reply = gw.handle(ev, T0 + 60 * i) print(f"{user}: {text}") print(textwrap.fill(reply, width=80, initial_indent=" -> ", subsequent_indent=" ", break_on_hyphens=False)) lines = open("audit.jsonl", encoding="utf-8").read().splitlines() print(f"\n{len(lines)} audit records; the approved deploy:") print(json.dumps(json.loads(lines[8]), indent=2)) ``` **Expected output** (the audit record printed at the end is the one shown in the [reference implementation](#reference-implementation)): ```text U03CAROL: help -> You may run: status (and 'approve ' if you are an approver) U03CAROL: status checkout -> OK: dry run - would call the status API with {'service': 'checkout'} for carol@example.com U03CAROL: deploy checkout v1.4.2 to staging -> DENIED: 'deploy' needs one of ['deployers'] U01ALICE: deploy checkout v1.4.2 to staging -> DENIED: 'deploy' may only run in C0DEPLOYS U01ALICE: deploy checkout v1.4.2 to staging; rm -rf / -> REJECTED: unknown command or invalid arguments (try: help) U01ALICE: deploy checkout v1.4.2 to staging -> OK: dry run - would call the deploy API with {'service': 'checkout', 'version': 'v1.4.2', 'env': 'staging'} for alice@example.com U01ALICE: deploy checkout v1.4.2 to production -> PENDING req-0001: deploy service=checkout version=v1.4.2 env=production; another approver must run 'approve req-0001' U01ALICE: approve req-0001 -> DENIED: the requester cannot approve their own request U02BOB: approve req-0001 -> DENIED: approve req-0001 in C0DEPLOYS, where it was requested U02BOB: approve req-0001 -> OK (req-0001, approved by bob@example.com): dry run - would call the deploy API with {'service': 'checkout', 'version': 'v1.4.2', 'env': 'production'} for alice@example.com U02BOB: approve req-0001 -> DENIED: req-0001 does not exist or has expired U09EXT: status checkout -> DENIED: commands are accepted only from our own workspace U07MALLORY: status checkout -> DENIED: your chat account is not linked to a corporate identity 12 audit records; the approved deploy: ``` **Validation.** `audit.jsonl` has 12 lines for 13 events (`help` is deliberately not audited); `grep -c denied audit.jsonl` prints 7. **Extensions.** Add the `silence` command from the catalogue; make approvals expire by advancing the clock past `approval_ttl_seconds`; make `deploy` refuse a version older than the one running. **Troubleshooting.** `ModuleNotFoundError: gateway` means you ran the script from another directory. **Teardown.** Delete the directory. ::: caution Before production The lab keeps pending approvals in process memory: a restart loses them, and two approvals handled concurrently could both pass the check before either deletes the request. A real gateway needs a durable pending store, an atomic single-use step (compare-and-delete in one transaction), and an idempotency key per request so that platform retries cannot run a command twice. ::: #### Lab 3 (advanced): put the gateway behind a Slack slash command **Objective.** Expose Lab 2 as `/ops` through Socket Mode, so the bot needs no public endpoint [^slack-socket][^bolt]. **Prerequisites.** A Slack workspace where you may install apps (Slack offers free developer sandboxes [^slack-audit]), Python 3 (see above) and `slack-bolt` 1.30 (current on PyPI in October 2026), a secret store or at least environment variables. **Cost and risk.** No cost in a sandbox; **do not** point the executor at real systems until the gateway's tests pass. 1. In the Slack app settings, create an app **from a manifest** and paste the manifest below (fields as in Slack's manifest reference [^slack-manifest]); install it to the workspace. 2. Create an app-level token with the `connections:write` scope [^slack-socket], and copy the bot token (`xoxb-`). App-level tokens do not expire by default [^slack-tokens]; store both in your secret store. 3. Copy `policy.json` and `gateway.py` from Lab 2 next to `app.py`, and replace the example IDs with your real team, user and channel IDs (`T…`, `U…`, `C…`). 4. `export SLACK_BOT_TOKEN=… SLACK_APP_TOKEN=…` from the secret store, then `pip install slack-bolt==1.30.0` and `python3 app.py`. 5. Add the bot to a sandbox channel: a new app can post with `chat.postMessage` only where it is a member, unless it has the `chat:write.public` scope [^slack-postmessage]. Run `/ops help`, `/ops status checkout`, then the production deploy and approval flow with a colleague. As the [reference architecture](#reference-architecture) advises, help text and refusals go only to the caller through `respond()` (the `response_url`), and results and pending approvals are posted with `chat.postMessage` through Bolt's `client` [^bolt], so channel posting permissions apply. The manifest sets `token_rotation_enabled: false` to keep the sandbox simple; do not copy that to production without deciding on rotation first ([the bot is a privileged system](#identity-and-trust-who-is-really-asking)). _Example status: Python parsed (AST) and YAML parsed; reply routing checked with stub Slack objects; not run against Slack (needs a workspace and tokens)._ ```yaml display_information: name: ops-gateway description: Runs allow-listed operations commands with approval and audit. features: bot_user: display_name: ops-gateway always_online: false slash_commands: - command: /ops description: Run an allow-listed operations command (try /ops help) usage_hint: deploy checkout v1.4.2 to staging should_escape: false oauth_config: scopes: bot: - commands - chat:write settings: socket_mode_enabled: true token_rotation_enabled: false org_deploy_enabled: false ``` ```python """Lab 3: expose the Lab 2 gateway as /ops in Slack, over Socket Mode.""" import json import os import time from slack_bolt import App from slack_bolt.adapter.socket_mode import SocketModeHandler from gateway import Gateway def executor(name, args, requested_by): # Replace with calls to your status/deploy APIs, using short-lived credentials. return f"dry run - would call the {name} API with {args} for {requested_by}" with open("policy.json", encoding="utf-8") as f: policy = json.load(f) gateway = Gateway(policy, os.environ.get("AUDIT_PATH", "audit.jsonl"), executor) # xoxb- bot token from the secret store, never from code app = App(token=os.environ["SLACK_BOT_TOKEN"]) @app.command("/ops") def ops(ack, respond, command, client): ack() # Slack needs this within 3,000 ms; the work happens afterwards event = {"team_id": command["team_id"], "user_id": command["user_id"], "channel_id": command["channel_id"], "text": command.get("text", "")} reply = gateway.handle(event, time.time()) if reply.startswith(("OK", "PENDING")): # Announce via the Web API; response_url would bypass posting permissions client.chat_postMessage(channel=command["channel_id"], text=reply) else: # Help text and refusals go only to the caller respond(text=reply, response_type="ephemeral") if __name__ == "__main__": SocketModeHandler(app, os.environ["SLACK_APP_TOKEN"]).start() # xapp- token ``` **Expected output.** `/ops help` returns a reply only you can see; after `/ops status checkout` the bot posts the dry-run result in the channel; `audit.jsonl` gains a line with your real `U…` ID. **Validation.** Run `/ops deploy checkout v1.4.2 to staging` from an unlisted channel: it must be refused and audited. **Troubleshooting.** If Slack shows `operation_timeout`, the listener did not call `ack()` within 3,000 ms [^slack-slash]; if nothing arrives, check that Socket Mode is enabled and the app-level token has the right scope; if refusals arrive but results do not, the bot is not a member of the channel. **Teardown.** Revoke the bot token (Slack documents `auth.revoke` [^slack-security]), uninstall and delete the app, remove both tokens from the secret store, and delete `audit.jsonl`. ### Use cases and scenarios The sourced cases describe each company's own practice at the date of the source, not today's. | Scenario | Context | Workflow | Trade-offs | Success metric | |---|---|---|---|---| | Deploy coordination (GitHub, sourced) | Hundreds of deploys a week to one monolith | Queue, guards, locks, partial deploys from chat | Visible and teachable; chat-held state stopped scaling | Deploys per week; deploy-caused incidents; time a deploy waits in the queue | | Incident commands (PagerDuty, sourced) | Paging and status during incidents | `!ic page`, `!status stalk`, scribe in Slack | Fast assembly; depends on chat and the bot being up | Time from declaration to commander joining; failed pages reported | | Team hygiene nudges (GOV.UK, sourced) | Many teams, many repositories | Scheduled bots post open PRs, Dependabot PRs, missing scans | Cheap and effective; easy to ignore if noisy | Age of open PRs; repositories missing scans | | Diagnostics-only cloud channel (illustrative) | On-call needs AWS facts during incidents | Channel with a diagnostics-only guardrail and per-user roles | Fast answers; one provider; IAM must be right | Share of diagnostics done without console access | | Access elevation request (illustrative) | Engineers need temporary production access | Request in chat, approval in chat, grant by the access system | Visible approvals; the access system, not the bot, makes and expires the grant ([AccessOps](../accessops/)) | Median time to approved access; standing privileges removed | **Case 1: GitHub, from chat-only deploys to "one command" (2015 → 2021).** In 2015 GitHub engineers deployed "several hundred times every week" from chat, with the queue, guards, locks and loud override of the [deploy playbook](#core-practices-and-playbooks), and Hubot posted elevated exceptions after a deploy [^gh-2015]. By 2021 the number of developers contributing to GitHub.com had roughly doubled in a year. A deploy now took about a dozen steps "spread across hundreds of messages" in `#dotcom-ops`, and developers were running chatops commands "multiple times in every deployment, and often getting them wrong" [^gh-2021]. GitHub's answer was not to leave chat: one chatops command starts a state-machine-backed deploy with automatic 2 % and 20 % canary stages and 5-minute timers, a UI shows its state, and developers typically start it from Slack and follow it in the UI [^gh-2021]. **Lesson:** chat is an excellent trigger and summary, and a poor place to keep state. **Case 2: PagerDuty's incident commands.** PagerDuty's public incident-response guide documents bots that react to `!` commands in Slack. `!ic page` pages the primary, backup and shadowing Incident Commanders; the commands cache on-call schedules so they work if the PagerDuty API fails, and if paging fails the bot says so and gives phone numbers; `!status stalk` polls every 30 seconds and reports only when the status changes [^pd-chatops]. Two design principles stand out: **fail loudly with a fallback** and **report changes, not states**. The channel's lifecycle around such commands looks like this (this Atlas's model; incident roles and severity rules are taught in [IncidentOps](../incidentops/)): ::: figure The life of an incident channel and the bot's job at each step ```mermaid stateDiagram-v2 [*] --> Declared: declare command Declared --> Assembling: bot creates channel, registers ID, pages Assembling --> Active: bot posts commander, roles, links Active --> Mitigated: bot posts status change Mitigated --> Resolved: bot posts resolution Resolved --> Archived: bot exports timeline, archives Active --> Handoff: attacker suspected Handoff --> Preserved: bot exports timeline, narrows access Archived --> [*] Preserved --> [*] ``` Text equivalent: a declare command opens the incident (Declared). The bot creates the channel, registers its ID with the gateway so incident commands work there, and pages on-call (Assembling). When a commander is named, the bot posts the commander, roles and links (Active). It posts each status change (Mitigated) and the resolution (Resolved), then exports the timeline and archives the channel, which also removes it from the gateway's registry (Archived). If an attacker is suspected, the incident is handed off to the security response process: the bot exports the timeline and the channel's membership is narrowed, so the record is preserved for investigation or legal hold (Preserved). ::: Suspicion of an attacker can arise in any state; the figure shows it from Active only to stay readable. Keeping the record after a security handoff matters because the channel may be evidence ([ResponseOps](../responseops/)). **Case 3: GOV.UK's team bots.** GOV.UK's developer documentation lists Slack integrations that nudge teams instead of running production commands: Seal posts each team's open pull requests, Dependapanda its open Dependabot pull requests, Release App Badger pull requests not yet deployed, and CI Bot repositories missing CodeQL or Dependency Review scans [^govuk]. "The Seal and Dependapanda can only post to public channels." [^govuk] Low-risk, high-value ChatOps: no write access to production, and each message names an owner and an action. (The page, last updated in January 2026, is flagged as possibly out of date.) ::: example Scenario 4: a diagnostics-only AWS channel for incidents A platform team creates an Amazon Q Developer channel configuration for a standing `#ops-aws` channel (a configuration names one channel ID, so short-lived incident channels are not covered) using the template in the [reference implementation](#reference-implementation): the custom diagnostics guardrail, per-user roles required. During an incident, responders query alarm states, metrics and service health there. **Success:** responders stop sharing console screenshots, and no write and no read of data contents (log lines, objects, table items) is possible from the channel even if a user's role allows it, because guardrails intersect with roles [^awsq-perms]. **Pitfall:** a later configuration created through the API without guardrails inherits AdministratorAccess [^awsq-api], so the platform team checks every configuration in its quarterly review. ::: ## Measure, secure and pay ### Metrics and KPIs No standard ChatOps metrics exist; the set below is **this Atlas's proposal**. Measure the interface (does it make work visible and safe?) and the noise (does it waste attention?), not activity. Delivery outcomes such as deployment frequency belong to [DevOps](../devops/), and incident timings such as MTTA to [IncidentOps](../incidentops/). | KPI | Definition and formula | Data source | Pitfall | Cadence | |---|---|---|---|---| | Audited change coverage | Production changes made through an audited path (gateway or pipeline) ÷ all production changes | Gateway audit log, CD system, cloud audit logs | Counting chat messages instead of real changes | Monthly | | Command success rate | Successful executions ÷ executions (refusals excluded) | Audit log | Hiding executor failures behind "OK, started" | Weekly | | Refusal rate and mix | Refusals ÷ invocations, split by reason | Audit log | Treating refusals only as noise; spikes may be probing | Weekly | | Approval latency | p50 and p95 time from `PENDING` to approval | Audit log | Approvers rubber-stamping fast to unblock | Monthly | | Responsiveness | Share of commands acknowledged within the platform deadline ([table](#proof-of-origin-differs-by-platform)); p95 time to first reply | Gateway metrics | Acknowledging fast but leaving silent gaps | Continuous, alert on breach | | Actionable notification ratio | Sampled bot messages that led to an action within an agreed time ÷ sampled messages | Monthly sample per channel, reviewed by the team | Judging "action" loosely; sample too small | Monthly | | Notification load per on-call shift | Bot messages in on-call channels ÷ shifts | Chat export or router logs | Averages hiding storm days; report the worst shift too | Monthly | | Credential hygiene | Credentials within rotation policy ÷ all bot credentials; scopes beyond policy | Secret store, app scope audit | Forgetting webhooks and app-level tokens | Quarterly | The actionable notification ratio counts **chat messages** of every kind, including deploy notices and nudges. It differs from page actionability in [ObservabilityOps](../observabilityops/#metrics-and-kpis) and the actionable page rate in [AIOps](../aiops/), which count pages: a channel can score badly here while every page is justified. **Leading and lagging.** Refusal mix, notification load and credential hygiene are leading indicators; audited change coverage and the actionable ratio are lagging. **Vanity metrics:** number of commands, number of bot messages, "bot usage" totals. ::: example Worked calculation over four weeks (illustrative numbers) - **Audited change coverage.** 412 production changes: 236 through the gateway, 151 through the pipeline with audit, 25 in the cloud console. Coverage = (236 + 151) ÷ 412 = **93.9 %**. Next step: ask why 25 changes bypassed both paths. - **Commands.** 3,180 invocations; 214 refusals (**6.7 %**): 150 wrong channel, 41 missing role, 23 malformed. Of 2,966 executions, 2,911 succeeded (**98.1 %**). The 150 wrong-channel refusals say people do not know where to deploy from: fix the help text and channel names before anything else. - **Notifications.** `#alerts-checkout` received 1,240 bot messages (310 a week, 44 per on-call shift across 28 shifts). In a random sample of 200, 46 led to action within 30 minutes: actionable ratio **23 %**, about ± 6 percentage points at 95 % confidence (± 5 with the finite-population correction, because the sample is a sixth of the messages). Roughly 240 non-actionable messages a week remain. **Target:** at least 50 % within a quarter, by moving informational alerts to a digest and grouping by service. ::: ### Security, privacy and compliance | Threat | Signal | Mitigation | Owner | |---|---|---|---| | Leaked bot or app token | Secret-scanning alert; unexpected API use or source IP; bot suddenly silent (the provider may have revoked it) | Secret store; restrict Web API use to the gateway's public egress addresses (no "Local" addresses) [^slack-security]; rotation; push protection (GitHub detects Slack API tokens [^gh-secret-scanning]); [runbook](#runbook-a-leaked-bot-token) | Bot owner | | Leaked incoming-webhook URL | Secret-scanning alert; posts nobody's router sent | Secret store; delete and recreate the webhook; IP allowlisting "does not apply to incoming webhooks" [^slack-security]; push protection covers webhook URLs too [^gh-secret-scanning] | Bot owner | | Forged or replayed request | Signature failures; old timestamps; duplicate request IDs | The platform's proof of origin ([table](#proof-of-origin-differs-by-platform)); idempotency keys where no timestamp is signed; nonces for internal RPC [^crpc-protocol] | Bot owner | | Impersonation | Commands from renamed or new accounts | Immutable IDs, not names [^slack-slash]; identity map reconciled with leavers | Bot owner, [IdentityOps](../identityops/) | | Confused deputy | Bot identity in executor logs instead of people | Gateway authorises the human; executors receive requester identity; per-user roles where available [^st2-rbac][^awsq-perms] | Bot owner, [IAMOps](../iamops/) | | Command injection | Odd characters in arguments; refusals for malformed input | Typed arguments, full-match patterns, no shell; pin immutable parameters [^st2-aliases] | Command owner | | External or guest users | Commands from other workspace IDs | Workspace and organisation allow-list; commands disabled in shared channels | Bot owner | | Compromised chat account | Bulk channel reads or downloads | Short sessions and device policy; no secrets in chat; retention limits | Security, [IdentityOps](../identityops/) | | Prompt injection via an assistant | Assistant output containing links or instructions not asked for | Assistant behind the gateway, read-only, no approvals ([Automation and AI](#automation-and-ai)) | Security | | Data leakage in notifications or command output | Personal data or secrets in alert text, log lines or query results posted in a channel | Redact at the source; link to dashboards instead of pasting logs; guardrails that exclude data-content reads ([reference implementation](#reference-implementation)) | Service team, bot owner | | Spoofed approval | Approvals by emoji, by the requester, or in another channel | Typed approvals checked by the gateway: different person, approver group, same channel, time-boxed, single use | Command owner | **Why chat content itself is sensitive.** In the Disney case, SecurityWeek reported in 2025, citing the Justice Department, that an employee downloaded a fake AI art tool to a personal computer; credentials stored there gave the attacker the employee's work Slack account and data from thousands of Slack channels [^disney]. A ChatOps programme adds to what such an account can see. Keep secrets, tokens and connection strings out of chat entirely, and treat ops channels as confidential. **Privacy and compliance.** Chat is a data store: minimise personal data in notifications, set retention deliberately, and agree with your privacy team which channels hold what ([PrivacyOps](../privacyops/)). For compliance, the gateway's audit log, not the chat transcript, is the record of who did what. Whether a regulator also expects you to retain the chat itself depends on your sector and jurisdiction; ask your compliance team ([ComplianceOps](../complianceops/)), do not assume. ### Cost, capacity and sustainability - **Direct costs** are modest and plan-dependent: hosting a gateway is cheap, but some controls sit in premium tiers, such as Slack's Audit Logs API ([above](#what-chatops-is-not-boundaries-and-naming)). Prices change; check your contract. - **The largest cost is attention**: about 13 hours a week for one noisy channel in the [illustrative calculation](#the-problem-it-solves). The actionable ratio is the cost-control metric. - **Maintenance cost** is driven by platform churn: connectors, legacy bots and whole products retired, or scheduled for retirement, between 2018 and 2027 ([origins](#origins-and-evolution)). A thin adapter keeps rewrites small. - **Capacity limits are the platform's**, not yours: per-channel posting limits with HTTP 429 and `Retry-After` [^slack-rate], acknowledgement deadlines and `response_url` limits. Design for alert storms and long-running commands. - **Sustainability.** The compute footprint of bots is negligible next to the systems they operate; we found no evidence that would justify a carbon claim either way. ### Assurance of the operating model - **Synthetic checks.** A scheduled canary command in a test channel proves the full path (platform, adapter, gateway, executor in dry-run) and alerts the bot owner when it fails; PagerDuty's bots also report their own failures in the room [^pd-chatops]. - **Reconciliation.** Monthly: gateway audit records versus executor logs (every executed command appears in both); identity map versus leavers; approver groups versus the on-call roster. - **Scope and guardrail audits.** Quarterly: installed apps and their scopes against the scope policy (Slack suggests scripting such audits [^slack-security]); cloud chat configurations against the guardrail baseline [^awsq-api]. - **Exception review.** Every break-glass use and every executed tier-3 command is reviewed in the next team meeting. - **Drills.** Twice a year, run an incident drill with the chat platform declared unavailable, and one with a revoked bot token. ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Remediation | Prevention | |---|---|---|---|---| | User sees `operation_timeout` | Work done before the acknowledgement | Gateway latency logs around the 3,000 ms deadline [^slack-slash] | Acknowledge first, then work and reply via `response_url` or the API | Responsiveness metric with an alert | | Bot silent in every channel | WebSocket disconnected, token revoked, app uninstalled | Adapter logs; check the token with Slack's `auth.test` method [^slack-authtest] | Reconnect, restore token from secret store, reinstall | Synthetic canary command | | Alerts missing during a storm | Rate limit: HTTP 429, over-limit bursts not displayed [^slack-rate] | Router logs for 429 and `Retry-After` | Group and inhibit; one updated message per incident [^am-config] | Load-test the router with a synthetic storm | | A command ran twice | Retried delivery or double submit | Two audit records with the same arguments seconds apart | Idempotency key per request; refuse duplicates | Executor-side deduplication | | Teams alerts stopped in late May 2026 | Office 365 Connectors disabled (scheduled for 18–22 May 2026) [^ms-connectors] | Senders still configured with Office 365 connector webhook URLs | Recreate as Workflows webhooks | Track vendor deprecation notices | | Executor logs show only "chatops_bot" | Confused deputy | Compare executor and gateway audit records | Pass the requester identity; per-user roles | Design review of each executor | | Everyone has muted the alert channel | Noise | Actionable ratio and per-shift load | Re-route, group, move informational alerts to a digest | Monthly noise review | ### Runbook: a leaked bot token **Illustrative scenario.** At 10:12 a secret-scanning alert reports that a `xoxb-` token for `ops-gateway` was committed to a public fork of an internal tool. The bot can run tier-2 commands and post in 40 channels. ::: figure Response to a leaked chat credential ```mermaid %%{init: {"flowchart": {"wrappingWidth": 160, "nodeSpacing": 24, "rankSpacing": 36}}}%% flowchart TB classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 D["Alert: credential exposed"]:::risk T{"Which credential, which scopes, still valid?"}:::decision K["Turn off write commands (kill switch)"]:::risk R["Revoke and replace; restart adapter; run canary"]:::core I["Review use since exposure: platform, gateway, executor logs"]:::core Q{"Misuse found?"}:::decision S["Security incident: hand off to ResponseOps"]:::risk C["Re-enable writes; review and prevention actions"]:::core D --> T --> K --> R --> I --> Q Q -->|"yes"| S Q -->|"no"| C ``` Text equivalent: an alert reports an exposed credential. Identify which credential it is, its scopes and whether the provider has already revoked it, turn off write commands with the gateway's kill switch, then revoke and replace the credential, restart the adapter and run the canary. Review all use since the exposure in platform, gateway and executor logs. If misuse is found, open a security incident and hand off to ResponseOps; otherwise re-enable writes and record review and prevention actions. ::: 1. **Triage (T+0 to T+5).** Identify the credential type (bot token, app-level token, webhook URL or signing secret) and what it can do: scopes from the app configuration, commands from the catalogue. Severity is high if it can trigger writes. Run Slack's `auth.test` with the token [^slack-authtest]: for a public repository, GitHub's partner program reports Slack tokens and webhook URLs to Slack [^gh-secret-scanning], and Slack says it "actively searches out and revokes leaked secrets" [^slack-webhooks]. If the token is already dead, the bot is silent: skip to Restore, then still investigate the exposure window. 2. **Contain (by T+10).** Flip the gateway's kill switch for tier-2 and tier-3 commands. Revoke the token (Slack's `auth.revoke` "Revokes a token." [^slack-authrevoke]); for a webhook, delete it and create a new one; for a signing secret, regenerate it. 3. **Restore (by T+20).** Put the new credential in the secret store, restart the adapter, run the canary command, re-enable reads. 4. **Investigate (by T+60).** Establish the exposure window (commit time to revocation). Check the platform's logs (Slack's Audit Logs API, where your plan has it [^slack-audit]), the gateway audit log and executor logs for activity in the window, and search channels for messages posted by the bot that nobody triggered. 5. **Decide.** Signs of misuse (unexplained executions, posts, new IP sources) make it a security incident: hand off to [ResponseOps](../responseops/). Otherwise re-enable writes. 6. **Prevent.** Enable push protection for the repository (GitHub detects Slack tokens and webhook URLs [^gh-secret-scanning]); restrict token use to the gateway's public egress addresses [^slack-security]; plan token rotation (it cannot be turned off later [^slack-tokens]); use separate credentials per environment. ### Runbook: the bot goes quiet during an incident 1. **Confirm** with the canary command or `help` in a test channel; check the chat vendor's status page. 2. **Fall back at once** to the documented paths: pipeline UI, runbook engine, phone numbers for paging. Announce the fallback in the incident channel (or on the call if chat is down). Do not debug the bot in the middle of someone else's incident. 3. **Diagnose** in order: adapter process and WebSocket connection; token validity (`auth.test` [^slack-authtest]); 429 responses; gateway errors; executor health. 4. **Restore** and run the canary; post a summary of any actions taken outside the gateway so the audit trail can be completed afterwards. 5. **Review** in the incident's follow-up: the bot is part of the incident response system and has its own action items. ### Anti-patterns and myths - **A shell with a chat UI.** StackStorm's own example alias `run {{cmd}} on {{hosts}}` wraps a remote command action [^st2-aliases]; it teaches aliases, but deployed as is it gives a remote shell to everyone allowed to use it (GitHub's own history in [Origins](#origins-and-evolution)). **Instead:** typed, allow-listed commands. - **The god-mode bot.** One token with admin scopes and production credentials. **Instead:** least privilege for the bot, separate credentials per executor and tier. - **Chat as the database.** Deploy state, approvals or incident state reconstructed by scrolling. **Instead:** keep state in a system with a UI and post summaries ([Case 1](#use-cases-and-scenarios)). - **Direct messages by default, emoji approvals, logic in the adapter.** Each breaks a [principle](#charter-and-principles): visibility, typed approvals checked by the gateway, a thin and replaceable adapter. - **The firehose channel.** Every alert, build and comment in one channel. **Instead:** one channel per audience, grouped, updated in place, with an actionable-ratio target. - **Myth: "everything is visible, so it is controlled."** Visibility helps detection; it does not stop a forged, mistaken or injected command. - **Myth: "an AI assistant makes the catalogue unnecessary."** Natural language widens what users can ask for; the catalogue and tiers are what keep the answer safe. ## Deciding and adopting ### When to use and when not to use **Use ChatOps when** operational tasks repeat often or must be coordinated fast, and have an API behind them; the team already coordinates in chat; shared visibility is valuable (deploys, incidents, on-call handovers); and the team is distributed. Newland's abstract credits it with having "enabled and enhanced our culture of remote work" [^newland]. **Do not use it, or limit it to notifications, when:** - **Inputs are rich or the review is visual.** Database migrations, capacity plans and infrastructure diffs need a form, a plan view or a pull request. Chat can link to them and announce results. - **Nobody will own the bot.** An unowned bot with production credentials is a liability; start only with a named owner and a budget for maintenance. - **You do not control identity in the workspace.** Many guests, external organisations in shared channels, or chat accounts not tied to your identity provider make command authorisation guesswork. - **Policy forbids it.** Some regulated or classified environments do not allow production control through a SaaS chat platform; self-hosted chat or no chat path may be the answer. - **The chat platform is part of what you would be fixing**, with no fallback path. - **Outputs contain secrets or personal data** that cannot be reliably redacted. **Hidden responsibilities:** running the gateway as a product, security reviews of new commands, token rotation, noise reviews and adapter rewrites when vendors change their platforms. ### Decision framework The everyday decision is not "should we do ChatOps?" but "should **this action** be a chat command, and at which tier?" ::: figure Should this action become a chat command? ```mermaid %%{init: {"flowchart": {"wrappingWidth": 150, "nodeSpacing": 22, "rankSpacing": 30}}}%% flowchart TB classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 Q1{"Output free of secrets and personal data?"}:::decision Q2{"Short, typeable inputs and an API behind it?"}:::decision Q3{"Read-only?"}:::decision Q4{"Does the team gain from seeing it?"}:::decision Q5{"Weekly, or part of a time-critical response?"}:::decision Q6{"Reversible in minutes, small blast radius?"}:::decision N1["Not a chat command; at most a notice that it ran"]:::risk N2["Form, plan view or pull request; chat links to it"]:::neutral T1["Tier 1: verified identity, audit"]:::core N3["Pipeline only; message chat only if someone must act"]:::neutral N4["Runbook or CLI; chat announces the result"]:::neutral T2["Tier 2: role, channel, audit"]:::core T3["Tier 3: plus second person, rehearsed executor"]:::core Q1 -->|"yes"| Q2 Q1 -->|"no"| N1 Q2 -->|"yes"| Q3 Q2 -->|"no"| N2 Q3 -->|"no"| Q4 Q3 -->|"yes"| T1 Q4 -->|"yes"| Q5 Q4 -->|"no"| N3 Q5 -->|"yes"| Q6 Q5 -->|"no"| N4 Q6 -->|"yes"| T2 Q6 -->|"no"| T3 N1 ~~~ N2 ~~~ T1 ~~~ N3 ~~~ N4 ``` Text equivalent: first, if the action's output can contain secrets or personal data, it is not a chat command; at most chat gets a notice that it ran. If its inputs are not short and typeable, or there is no API behind it, use a form, plan view or pull request and let chat link to it. A read-only action becomes a tier-1 command (verified identity and audit). For an action that changes something: if the team gains nothing from seeing it, keep it in the pipeline and message chat only when someone must act; if it is neither weekly nor part of a time-critical response such as an incident, keep it as a runbook or CLI and announce the result in chat. Otherwise it becomes a tier-2 command (role, channel, audit) when it is reversible within minutes with a small blast radius, and a tier-3 command (a second person, and an executor that has been rehearsed) when it is not. ::: The order matters. Output safety comes first because it applies to reads as well: a "read-only" log search can still post personal data. Time-critical incident actions such as paging or failover pass the frequency question even if they are rare, because chat is where the responders already are. **Weighted check for borderline cases (this Atlas's proposal).** It answers one question, "is this a good chat command?"; the tree's last question then sets the tier. Score each criterion from 1 (poor fit) to 5 (good fit), multiply by the weight, and treat a total under 3.0 as "not a chat command". | Criterion | Weight | Silence an alert for 1 h | Database failover | Rotate TLS certificates | |---|---|---|---|---| | Frequency or time-criticality | 0.25 | 5 | 4 | 2 | | Value of shared visibility | 0.25 | 5 | 5 | 2 | | Input simplicity | 0.20 | 5 | 4 | 2 | | Output safe to show the channel | 0.15 | 5 | 4 | 2 | | Executor maturity (API, dry run, rehearsed) | 0.15 | 4 | 3 | 3 | | **Weighted total** | | **4.85** | **4.1** | **2.15** | Silencing is tier 2 because it expires on its own. Failover is tier 3: hard to undo, large blast radius. Its weakest score, executor maturity, is a precondition for tier 3 rather than a weight: if the runbook has not been rehearsed, do not ship the command whatever the total (the catalogue entry refuses to run without a rehearsal in the last 90 days). Certificate rotation scores 2.15 and belongs in a pipeline with a chat notification. ### Maturity model A **proposal by this Atlas**, not an industry standard. Levels describe observable behaviour; skip none, and stop at the level your risk and team size justify. ::: figure ChatOps maturity levels proposed by this Atlas: exit evidence and where to stop ```mermaid %%{init: {"flowchart": {"wrappingWidth": 135, "nodeSpacing": 20, "rankSpacing": 30}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 L1["1 Notify. Exit: actionable ratio measured in every alert channel"]:::core L2["2 Read. Exit: every invocation audited with a person"]:::core L3["3 Guarded write. Exit: most routine changes audited; refusals reviewed weekly"]:::core L4["4 Governed. Exit: chat-outage and revoked-token drills passed"]:::core L5["5 Assisted: AI reads and proposes; humans issue and approve"]:::core S1["Stop at 1 if chat only needs to inform"]:::neutral S2["Stop at 2 if changes need not start in chat"]:::neutral S3["Stop at 3 if high-impact actions stay out of chat"]:::neutral S4["Stop at 4 if no assistant reads ops channels"]:::neutral L1 --> L2 --> L3 --> L4 --> L5 L1 -.- S1 L2 -.- S2 L3 -.- S3 L4 -.- S4 S1 ~~~ S2 ~~~ S3 ~~~ S4 ``` Text equivalent: five levels in sequence, each with the evidence needed to leave it, and a dotted branch saying when a team should stop there. Level 1, Notify: leave when the actionable ratio is measured in every alert channel; stop here if chat only needs to inform. Level 2, Read: leave when every command invocation is audited with a person attached; stop here if changes need not start in chat. Level 3, Guarded write: leave when most routine changes go through an audited path and refusals are reviewed weekly; stop here if high-impact actions stay out of chat (credential rotation and quarterly audits still apply). Level 4, Governed: leave when a chat-outage drill and a revoked-token drill have passed; stop here if no AI assistant reads ops channels. Level 5, Assisted: AI assistants read and propose, and humans issue and approve. ::: | Level | Observable behaviour | Exit evidence | Typical constraint | |---|---|---|---| | 1 Notify | Bots post deploys and alerts; each alert channel has an owner | Actionable ratio measured for every alert channel | Noise; unowned integrations | | 2 Read | Status and diagnostic commands; IDs mapped to the identity provider | Every command invocation in the audit log with a person attached | Identity mapping for contractors and guests | | 3 Guarded write | Tier-2 commands with typed arguments, roles and channels | Most routine changes go through an audited path; refusals reviewed weekly | Executors running as one bot identity | | 4 Governed | Two-person tier 3, catalogue reviewed like code, rotation, canary, quarterly audits | A chat-outage drill and a revoked-token drill passed | Approver availability out of hours | | 5 Assisted | AI summarises and proposes commands; humans issue and approve them | Assistant has no execute or approve rights; its tool calls are logged | Prompt-injection exposure; vendor features change fast | ### Adoption roadmap: 30/60/90 days | Window | Actions | Owner | Milestone and metric | |---|---|---|---| | Days 1–30 | Inventory existing bots, webhooks and tokens with their scopes; remove unowned ones; agree the channel taxonomy; group and route alerts for two services; baseline the actionable ratio; deploy a gateway skeleton with two read commands and the audit log | Bot owner, two service teams | Inventory complete; baseline metrics; first audited commands | | Days 31–60 | Map chat IDs from the identity provider; ship tier-2 commands for the three most frequent tasks; weekly refusal review; incident-channel automation agreed with the IncidentOps owner; secret store, IP allowlisting and push protection | Bot owner, security, service teams | Audited change coverage measured; no tokens outside the secret store | | Days 61–90 | Tier-3 approvals for production deploys; canary command; quarterly scope and guardrail audit scheduled; chat-outage drill; decide the AI assistant's scope (read and propose only) | Bot owner, engineering manager | Drill passed; actionable ratio improved against baseline; written ChatOps policy | **Quick wins:** route informational alerts out of chat; add owner and runbook links to every alert message; make every refusal message say what to do instead. **Dependencies:** an identity provider feed ([IdentityOps](../identityops/)), executors with APIs ([AutomationOps](../automationops/)), and agreed incident roles ([IncidentOps](../incidentops/)). ## Toolkit ### Checklists **Quick start (before the first command)** - [ ] A named bot owner and a backup - [ ] Inventory of existing bots, webhooks and tokens, with scopes and owners - [ ] Channel taxonomy agreed (`#deploys-…`, `#alerts-…`, `#inc-…`, `#ops-sandbox`) - [ ] Secret store holds every token; nothing in code, chat or tickets - [ ] Gateway writes an audit record for every decision, outside the chat platform **Launch readiness for a new command** - [ ] Proposal approved, tier assigned, owner named - [ ] Arguments typed and fully matched; unknown arguments refused - [ ] Roles, channel IDs and (tier 3) approver group set in the catalogue - [ ] Refusals tested: wrong user, wrong channel, malformed input, self-approval, expired approval - [ ] `help` text, example, rollback command or runbook link - [ ] Executor credential limited to this command's needs **Quarterly security and governance audit** - [ ] App scopes match the scope policy; unused apps removed - [ ] Identity map reconciled with leavers and contractors; approver groups current - [ ] Credentials rotated or within policy; IP allowlists current - [ ] Cloud chat configurations have explicit guardrails that exclude data-content reads - [ ] Break-glass uses reviewed; audit log and executor logs reconciled **Monthly notification review (per alert channel)** - [ ] Actionable ratio sampled and compared with target - [ ] Top ten noisiest alerts sent back to their owners ([ObservabilityOps](../observabilityops/)) - [ ] No `@here` or `@channel` on non-paging alerts - [ ] Every message has an owner and a runbook or dashboard link **Incident readiness** - [ ] Declare command creates the channel, pages on-call and posts links (tested this quarter) - [ ] Fallback paths documented for a chat or bot outage, including phone numbers - [ ] Canary command alerts the bot owner when the path breaks ### Templates **Command proposal (intake form).** ```text Command and syntax: deploy to Owner (team, person): team-checkout, A. Example Problem it solves: manual pipeline clicks; deploys invisible to support Frequency: ~40 per week Arguments, validation: service [a-z0-9-]{1,40} version vMAJOR.MINOR.PATCH env staging|production Risk tier and reason: 2 for staging; 3 for production (customer impact; rollback takes ~10 min) Channels and roles: #deploys-checkout; deployers; approvers: oncall-leads Executor, credential: CD pipeline API; token limited to the checkout pipelines Output shown in chat: parsed arguments, pipeline link, final status; never logs or secrets Rollback: deploy to Tests: refusal cases in CI; dry run in #ops-sandbox Review: bot owner; security (tier 3 only) ``` **Notification contract (one per service).** ```yaml service: checkout owner: team-checkout channels: page_feed: '#alerts-checkout' # one message per alert group, updated in place team: '#checkout-team' # tickets, no mentions digest: daily-email # informational alerts never go to chat required_labels: [owner, runbook_url, severity] mentions: page-only forbidden_content: [secrets, tokens, personal data, raw log dumps] target_actionable_ratio: 0.5 review: monthly ``` **Architecture decision record: adopting a ChatOps gateway.** ```text Title: ADR-012 Route all chat commands through ops-gateway Status: accepted (2026-10-16) Context: three bots with admin tokens; no audit outside Slack; deploys need approvals Decision: one gateway owns catalogue, identity map, approvals and audit; adapters stay thin; executors receive requester identity; tier 3 needs a second person in the same channel Alternatives: StackStorm with aliases and RBAC; vendor incident tool commands only; no write commands in chat Consequences: + one place to review and audit + platform change limited to the adapter - gateway is a critical service (needs canary, on-call, drills) Review: after 90 days, using audited change coverage and refusal mix ``` ### Deep comparisons **Which interface for an operational action?** | Interface | Best at | Weak at | Audit | Choose when | |---|---|---|---|---| | Chat command | Visible, quick, teachable actions with short inputs | Rich input, visual review, long-running state | Gateway log (chat is not enough) | Frequent, team-relevant, typeable actions | | Pipeline or deploy UI | Gated, reproducible change with history | Ad-hoc questions; visibility to non-users | Built in | Changes that need review artefacts | | CLI with personal credentials | Expert speed, scripting | Visibility; consistent guards | Cloud or system audit logs | Individual investigation, break-glass | | Runbook portal or engine | Complex multi-step procedures, approvals | Discoverability in the moment | Built in | Rare, risky, multi-step operations (chat can trigger it) | **Socket Mode or a public HTTPS endpoint (Slack)?** | Aspect | Socket Mode | HTTPS endpoint | |---|---|---| | Network exposure | No public Request URL; the app opens a WebSocket [^slack-socket] | Public URL that must verify every signature [^slack-verify] | | Key secret | App-level `xapp-` token (does not expire by default [^slack-tokens]) | Signing secret | | Hosting | Long-running process inside your network | Serverless or any web host | | Distribution | Not allowed in the public Slack Marketplace [^slack-socket] | Allowed | | Good fit | Internal ops bots behind a firewall | Multi-workspace or public apps | The comparison with neighbouring Ops pages is in [How ChatOps relates to other Ops](#how-chatops-relates-to-other-ops). ### FAQ **Is deploying to production from chat safe?** It can be, with this page's controls: verified requests, mapped identities, typed arguments, channel and role checks, a second person and an audit log. Without them it is a remote shell with a friendly face. **Slack or Microsoft Teams?** The platform your organisation already runs and whose identity it controls; both have SDKs for bots [^bolt][^teams-sdk]. Compare audit, retention and data residency on your plan. Share the gateway; only the adapter's trust checks differ ([proof of origin](#proof-of-origin-differs-by-platform)). **Should the bot act with users' own tokens?** Usually not. On Slack, user tokens act "as if by the user themselves" with the user's access [^slack-tokens]; storing many such tokens creates many high-value secrets. Prefer a bot token for chat, plus a gateway that passes the requester's identity to executors. **Who approves at 3 a.m.?** A named approver group that includes the on-call lead, with a documented break-glass path for when nobody is reachable. Break-glass is announced in the channel and reviewed afterwards. **Where does ChatOps end and IncidentOps begin?** ChatOps owns the tooling (channel creation, paging commands, timeline capture); [IncidentOps](../incidentops/) owns roles, severity, communication and reviews. **Will AI assistants replace slash commands?** They change how people find and phrase commands. They do not remove the need for a catalogue, tiers and approvals; OWASP recommends human approval for high-impact actions [^owasp-llm06]. ## Going further ### Advanced topics and emerging direction - **Established:** typed command catalogues, gateways separate from adapters, two-person approval, notification grouping and in-place updates [^crpc-protocol][^am-config]. - **Maturing:** passing the requester's identity end to end, so executors authorise the person rather than the bot; cloud chat integrations with per-user roles and guardrails are one implementation [^awsq-perms]. Policy engines for command authorisation belong with [IAMOps](../iamops/). - **Moving fast (re-check before relying on it):** AI assistants in chat, and protocols that let agents read and act through chat platforms. In February 2026 Slack announced an MCP server that "enables AI agents to interact with Slack content", together with separate search scopes for public channels, private channels and direct messages [^slack-mcp]; grant the narrowest. The OWASP risks above apply unchanged [^owasp-llm01][^owasp-llm06]. - **Speculative:** "conversational operations" where the assistant runs remediation unattended. No public evidence we found shows this is safe at production scale; treat vendor claims as marketing until measured in your own environment. ### Learning path - **Prerequisites.** Basic HTTP and JSON, one scripting language, how your CI/CD and alerting work ([DevOps](../devops/), [ObservabilityOps](../observabilityops/)). - **One-day primer.** Read this page's first four parts; run Labs 1 and 2; read Slack's request-verification guide [^slack-verify] and GitHub's 2015 and 2021 deploy posts [^gh-2015][^gh-2021]. - **One week.** Run Lab 3 in a sandbox; add the `silence` command; route one service's alerts with grouping and in-place updates; measure its actionable ratio. - **Thirty days.** Run the [30-day roadmap](#adoption-roadmap-306090-days) slice for one team; write the notification contract and two command proposals; run a revoked-token drill. - **Specialisation.** Identity propagation and authorisation ([IAMOps](../iamops/), [AccessOps](../accessops/)); incident tooling ([IncidentOps](../incidentops/)); safe automation behind commands ([AutomationOps](../automationops/)); AI assistant security ([LLMOps](../llmops/)). - **Portfolio ideas.** A gateway with a policy file and tests for every refusal path; a notification-noise report for a real channel (anonymised); a threat model of an existing bot. ## Wrap-up ### Executive recap and cheat sheet | Topic | Remember | |---|---| | What it is | Operations through shared chat: notifications, allow-listed commands, visible coordination | | Status | Established practice pattern since 2013; no standard, role or certification | | Architecture | Thin adapter → gateway (identity, policy, approvals, audit) → executors with scoped credentials | | Trust | Verify requests; immutable IDs, not names; authorise the person, not the bot; typed arguments only | | Tiers | Inform · read · reversible write · high impact (second person) · break-glass (loud, reviewed) | | Notifications | One channel per audience; group; update in place; mentions only for pages; measure the actionable ratio | | Records | Gateway audit log with what, when, where, source, outcome, identity; chat is context, not evidence | | AI | Assistants read and propose; humans issue and approve; assume every message may carry instructions | | Metrics | Audited change coverage, refusal mix, approval latency, actionable ratio, credential hygiene | | Watch-outs | God-mode tokens, shell passthrough, chat as database, platform deprecations, muted channels | | Next actions | Inventory bots and tokens; build the catalogue; route alerts; add approvals; drill a chat outage | ### A judgement call: should the AI assistant in the incident channel act? **Situation (illustrative).** A 200-engineer SaaS company has an AI assistant that can read incident channels and run cloud commands through a channel role. The SRE lead proposes (A) letting it restart unhealthy workloads and scale services automatically when it is confident, to cut response time. The security lead proposes (B) letting it read, summarise and **propose** commands, which a human then issues through the gateway. **Assumptions:** incident channels sometimes include a vendor's engineers; alert messages quote log lines that can contain user-supplied text; restarts and scaling are reversible, but a restart loop can deepen an outage; nobody has measured the assistant's proposal accuracy here. **Reasoning.** 1. *Worst credible failure.* Under A, text the assistant reads (a quoted log line, a guest's message, an uploaded file) can carry instructions. That is indirect prompt injection [^owasp-llm01], and it has been demonstrated against a chat assistant [^promptarmor]. The assistant would then act with the channel role's permissions. Under B, the worst case is a bad proposal that a human must still type and the gateway must still allow. 2. *Where time goes.* In most incidents the slow part is understanding, not typing a restart. B keeps most of the speed benefit: summaries, correlations and a ready-made command. 3. *Guidance.* OWASP attributes damage to excessive functionality, permissions and autonomy and recommends human approval for high-impact actions [^owasp-llm06]. Restarts look low-impact one at a time; in a loop they are not. 4. *What would change the answer.* A narrow executor that can restart only one named workload, at most once per ten minutes, with no external participants in the channel, plus a quarter of data showing that most of the assistant's proposals were accepted unchanged. Then auto-executing that **single** action could be justified. **Decision: B now, with a review after a quarter.** Measure accepted, edited and rejected proposals. **Why A is attractive and still wrong today:** it optimises the visible minute of typing and pays for it with an invisible attack surface: every author of every message in the channel. **Uncertainty:** we found no public data on assistant accuracy for remediation in comparable settings, so the measurement is the point of the trial. ### Learning objectives and mastery check After this page you should be able to **explain** what ChatOps is and is not; **design** a command with tiers, typed arguments and approvals; **implement** request verification and a gateway decision path; **operate** notification routing with measurable noise; **respond** to a leaked bot token; and **evaluate** whether an action, a tool or an AI assistant belongs in chat. ::: answer 1. "Our Slack workspace is private, so our slash-command endpoint does not need signature checks." What is wrong? The endpoint is reachable by anyone who finds the URL; the workspace's privacy does not protect it. Without the HMAC check anyone can forge a request with any `user_id`, and without the timestamp check a captured request can be replayed [^slack-verify]. Verify the raw body, reject old timestamps, or use Socket Mode so there is no public endpoint [^slack-socket]. ::: ::: answer 2. Every deploy in the executor's log was made by "chatops_bot". Why is that a problem, and what do you change? It is a confused deputy: the executor cannot tell who asked, and any authorisation it applies is the bot's, not the person's. Authorise the person in the gateway, pass the requester's identity to the executor, and use per-user roles where the platform supports them [^awsq-perms]. ::: ::: answer 3. In Errbot you restricted `deploy` to the `#deploys` room, yet people still run it. How? `allowrooms` allows the listed rooms "(and direct messages)" [^errbot]; people are messaging the bot privately. Disallow private use for that command, and check channel IDs in your own gateway as well. ::: ::: answer 4. A channel had 900 bot messages last month. You sampled 150 and 60 led to action. What is the actionable ratio, and how precise is it? 60 ÷ 150 = 40 %. The 95 % margin is about 1.96 × √(0.4 × 0.6 ÷ 150) ≈ 7.8 percentage points, so roughly 32–48 % (a little narrower with a finite-population correction). Enough to see the gap to a 50 % target, not to compare two similar channels. ::: ::: answer 5. Classify two requests: "purge one URL from the CDN cache" and "rotate the production database password". Purging one URL repeats often, is easy to type, reversible and small in blast radius: a tier-2 command. Rotating a database password is rarer and its output is a secret: it belongs in a pipeline or secret-management workflow, with a chat notification that it happened ([decision framework](#decision-framework)). ::: ::: answer 6. You port Lab 1's check to a Teams outgoing webhook and to a Mattermost slash command. What changes? Neither has Slack's signed timestamp. The Teams webhook signs a Bot Framework activity [^ms-outgoing], whose specification tells bots not to reject by `timestamp` but allows de-duplication by `id`: keep recently seen IDs [^bf-activity]. Mattermost's static token does not cover the body [^mm-slash], so whoever obtains it can forge any `user_id`: keep it out of logs and limit who can reach the endpoint. A Teams bot instead lets the Bot Framework SDK validate a JWT [^ms-botauth]. ::: ::: answer 7. An assistant in the incident channel proposes `kubectl delete namespace payments`, citing an alert message. What stops it? Under the design on this page: the assistant cannot execute or approve; a human would have to issue the command; the catalogue has no free-form `kubectl` passthrough; and a destructive action would be tier 3 with a second person. Also check the cited alert: an instruction inside log text is a sign of injection [^owasp-llm01]. ::: ### Conclusion ChatOps began as a good idea that is easy to state: put the tools in the middle of the conversation so that everyone learns by watching. Thirteen years on, the idea holds, and the open questions are about **trust** and **attention**. A chat command is a production control surface inside a product that many people, integrations and now AI assistants can write to; a notification spends the attention of everyone who sees it. Teams that do ChatOps well put every command through a gateway that knows who is asking and keeps its own record, and measure whether their messages lead to action. Teams that do it badly hand a god-mode token to a chatty bot and call the scrollback an audit trail. The difference is not the chat platform; it is whether the team treats its bot as a privileged system with an owner. ## Sources and further reading Vendor documentation pages are live pages without a publication date unless one is given; product facts are as of October 2026. Ledger with claim-by-claim notes: `research/chatops-ledger.md`. [^newland]: [ChatOps at GitHub](https://speakerdeck.com/jnewland/chatops-at-github) — Jesse Newland, slides on Speaker Deck, 2013-02-07. Checked 2026-10-16. [^hubot-2011]: [Say Hello to Hubot](https://github.blog/news-insights/the-library/say-hello-to-hubot/) — Corey Donohoe, GitHub Blog, 2011-10-25 (updated 2021-06-24). Checked 2026-10-16. [^hubot-repo]: [hubotio/hubot](https://github.com/hubotio/hubot) — GitHub; version from the [npm registry](https://registry.npmjs.org/hubot) (14.1.0, 2026-02-24). Checked 2026-10-16. [^atl-2016]: [What is ChatOps? A guide to its evolution, adoption, and significance](https://www.atlassian.com/blog/software-teams/what-is-chatops-adoption-guide) — Atlassian, 2016-01-22. Checked 2026-10-16. [^st2-chatops]: [What is ChatOps?](https://docs.stackstorm.com/chatops/chatops.html) — StackStorm 3.9.0 documentation. Checked 2026-10-16. [^hand-podcast]: [ChatOps: Supercharging DevOps with group chat](https://www.oreilly.com/content/chatops-supercharging-devops-with-group-chat/) — Brian Anderson interviewing Jason Hand, O'Reilly, 2016-11-23. Checked 2026-10-16. [^hand-report]: [ChatOps: Managing Operations in Group Chat (catalogue record)](https://www.goodreads.com/book/show/31567120.ChatOps_Managing_Operations_in_Group_Chat) — Goodreads, record of Jason Hand's O'Reilly report, 2016. Checked 2026-10-16. [^fgcs-2024]: [ChatOps for microservice systems: A low-code approach using service composition and large language models](https://doi.org/10.1016/j.future.2024.07.029) — Sheng-Kai Wang, Shang-Pin Ma, Guan-Hong Lai and Chen-Hao Chao, *Future Generation Computer Systems* 161 (2024) 518–530, Elsevier; metadata from [Crossref](https://api.crossref.org/works/10.1016/j.future.2024.07.029), abstract from the [NTOU repository record](https://scholars.ntou.edu.tw/handle/123456789/25467?mode=full). Checked 2026-10-16. [^erlenhov]: [An Empirical Study of Bots in Software Development: Characteristics and Challenges from a Practitioner's Perspective](https://arxiv.org/abs/2005.13969) — Erlenhov, Oliveira Neto and Leitner, ESEC/FSE 2020 (arXiv v2, 2020-10-29). Checked 2026-10-16. [^hipchat]: [Announcing our new partnership with Slack](https://www.atlassian.com/blog/developers/new-atlassian-slack-partnership) — Atlassian (Inside Atlassian blog), 2018-07-26. Checked 2026-10-16. [^gh-2015]: [Deploying branches to GitHub.com](https://github.blog/engineering/engineering-principles/deploying-branches-to-github-com/) — Aman Gupta, GitHub Blog, 2015-06-02 (updated 2024-06-17). Checked 2026-10-16. [^gh-2021]: [Improving how we deploy GitHub](https://github.blog/enterprise-software/devops/improving-how-we-deploy-github/) — Julian Nadeau, GitHub Blog, 2021-01-25 (updated 2021-01-26). Checked 2026-10-16. [^crpc-why]: [Why Chatops RPC?](https://github.com/github/chatops-controller/blob/master/docs/why.md) — GitHub, chatops-controller repository. Checked 2026-10-16. [^crpc-protocol]: [Chatops RPC protocol description](https://github.com/github/chatops-controller/blob/master/docs/protocol-description.md) — GitHub, chatops-controller repository (protocol version 3). Checked 2026-10-16. [^pd-chatops]: [ChatOps](https://response.pagerduty.com/resources/chatops/) — PagerDuty Incident Response documentation. Checked 2026-10-16. [^pd-during]: [During an Incident](https://response.pagerduty.com/during/during_an_incident/) — PagerDuty Incident Response documentation. Checked 2026-10-16. [^govuk]: [Slack integrations](https://docs.publishing.service.gov.uk/manual/slack-integrations.html) — GOV.UK Developer Documentation, updated 2026-01-08. Checked 2026-10-16. [^disney]: [Man Admits Hacking Disney and Leaking Data Disguised as Hacktivist](https://www.securityweek.com/man-admits-hacking-disney-and-leaking-data-disguised-as-hacktivist/) — SecurityWeek, 2025-05-05. Checked 2026-10-16. [^promptarmor]: [Data Exfiltration from Slack AI via indirect prompt injection](https://promptarmor.substack.com/p/data-exfiltration-from-slack-ai-via) — PromptArmor, 2024-08-20. Checked 2026-10-16. [^register]: [Slack AI can leak private data via prompt injection](https://www.theregister.com/2024/08/21/slack_ai_prompt_injection/) — The Register, 2024-08-21 (updated with Salesforce's statement). Checked 2026-10-16. [^slack-verify]: [Verifying requests from Slack](https://docs.slack.dev/authentication/verifying-requests-from-slack/) — Slack Developer Docs. Checked 2026-10-16. [^slack-slash]: [Implementing slash commands](https://docs.slack.dev/interactivity/implementing-slash-commands/) — Slack Developer Docs. Checked 2026-10-16. [^slack-interact]: [Handling user interaction](https://docs.slack.dev/interactivity/handling-user-interaction/) — Slack Developer Docs. Checked 2026-10-16. [^slack-socket]: [Using Socket Mode](https://docs.slack.dev/apis/events-api/using-socket-mode/) and [connections:write scope](https://docs.slack.dev/reference/scopes/connections.write/) — Slack Developer Docs. Checked 2026-10-16 (scope page 2026-10-17). [^slack-tokens]: [Tokens](https://docs.slack.dev/authentication/tokens/) — Slack Developer Docs. Checked 2026-10-16. [^slack-security]: [Security best practices](https://docs.slack.dev/concepts/security/) — Slack Developer Docs. Checked 2026-10-16. [^slack-manifest]: [App manifest reference](https://docs.slack.dev/reference/app-manifest/) — Slack Developer Docs. Checked 2026-10-16. [^bolt]: [Bolt for Python: Using Socket Mode](https://docs.slack.dev/tools/bolt-python/concepts/socket-mode/), [Listening & responding to commands](https://docs.slack.dev/tools/bolt-python/concepts/commands/) and [Using the Web API](https://docs.slack.dev/tools/bolt-python/concepts/web-api/) — Slack Developer Docs; versions from [PyPI](https://pypi.org/project/slack-bolt/) (1.30.0, 2026-07-15) and [npm](https://www.npmjs.com/package/@slack/bolt) (5.1.0, 2026-09-02). Checked 2026-10-16 (Web API page 2026-10-17). [^slack-postmessage]: [chat.postMessage method](https://docs.slack.dev/reference/methods/chat.postMessage/) — Slack Developer Docs. Checked 2026-10-17. [^slack-webhooks]: [Sending messages using incoming webhooks](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/) — Slack Developer Docs. Checked 2026-10-16. [^slack-rate]: [Rate limits](https://docs.slack.dev/apis/web-api/rate-limits/) — Slack Developer Docs. Checked 2026-10-16. [^slack-audit]: [Audit Logs API](https://docs.slack.dev/admins/audit-logs-api/) — Slack Developer Docs. Checked 2026-10-16. [^slack-legacy]: [Discontinuing support for legacy custom bots and classic apps](https://docs.slack.dev/changelog/2024-09-legacy-custom-bots-classic-apps-deprecation/) — Slack Developer Docs changelog, 2024-09. Checked 2026-10-16. [^slack-classic-pause]: [Classic apps deprecation paused](https://docs.slack.dev/changelog/2025/12/08/classic-apps-deprecation-paused/) — Slack Developer Docs changelog, 2025-12-08. Checked 2026-10-16. [^slack-usersinfo]: [users.info method](https://docs.slack.dev/reference/methods/users.info/) — Slack Developer Docs. Checked 2026-10-16. [^slack-authtest]: [auth.test method](https://docs.slack.dev/reference/methods/auth.test/) — Slack Developer Docs. Checked 2026-10-16. [^slack-authrevoke]: [auth.revoke method](https://docs.slack.dev/reference/methods/auth.revoke/) — Slack Developer Docs. Checked 2026-10-16. [^slack-mcp]: [Announcing the Slack MCP server and Real-time Search API](https://docs.slack.dev/changelog/2026/02/17/slack-mcp/) — Slack Developer Docs changelog, 2026-02-17. Checked 2026-10-16. [^ms-connectors]: [Retirement of Office 365 connectors within Microsoft Teams](https://devblogs.microsoft.com/microsoft365dev/retirement-of-office-365-connectors-within-microsoft-teams/) — Microsoft 365 Developer Blog, 2024-07-03, updated 2026-04-14. Checked 2026-10-16. [^ms-botauth]: [Authenticate requests with the Bot Connector API](https://learn.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-connector-authentication) — Microsoft Learn, updated 2024-10-09. Checked 2026-10-16. [^ms-outgoing]: [Create an Outgoing Webhook](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-outgoing-webhook) — Microsoft Learn, updated 2026-06-23. Checked 2026-10-16. [^bf-activity]: [Bot Framework -- Activity](https://github.com/microsoft/botframework-sdk/blob/main/specs/botframework-activity/botframework-activity.md) — Microsoft, botframework-sdk repository, specification version 3.1.12 (file last changed 2024-12-12); rules A2030–A2033 (`id`) and A2040–A2042 (`timestamp`). Checked 2026-10-17. [^ms-invoke]: [Explore responses to action commands](https://learn.microsoft.com/en-us/microsoftteams/platform/messaging-extensions/how-to/action-commands/respond-to-task-module-submit) — Microsoft Learn, updated 2026-10-09. Checked 2026-10-16. [^ms-conv]: [Conversations with an agent](https://learn.microsoft.com/en-us/microsoftteams/platform/bots/how-to/conversations/conversation-messages) — Microsoft Learn, updated 2026-09-28 (message payload with `from.aadObjectId`; `channelData` tenant ID); [ChannelAccount interface](https://learn.microsoft.com/en-us/javascript/api/botframework-schema/channelaccount) — Microsoft Learn (`aadObjectId`: "This account's object ID within Azure Active Directory"). Checked 2026-10-16. [^mm-slash]: [Custom slash commands](https://docs.mattermost.com/developers/integrate/slash-commands/custom) — Mattermost documentation. Checked 2026-10-16. [^teams-sdk]: [Teams SDK (Teams AI Library)](https://learn.microsoft.com/en-us/microsoftteams/platform/teams-sdk/) and [Tools and SDKs overview](https://learn.microsoft.com/en-us/microsoftteams/platform/concepts/build-and-test/tool-sdk-overview) — Microsoft Learn. Checked 2026-10-16. [^awsq-rename]: [Amazon Q Developer in chat applications rename: summary of changes](https://docs.aws.amazon.com/chatbot/latest/adminguide/service-rename.html) — AWS documentation. Checked 2026-10-16. [^awsq-perms]: [Understanding Amazon Q Developer in chat applications permissions](https://docs.aws.amazon.com/chatbot/latest/adminguide/understanding-permissions.html) — AWS documentation. Checked 2026-10-16. [^awsq-logs]: [Accessing Amazon CloudWatch Logs for Amazon Q Developer in chat applications](https://docs.aws.amazon.com/chatbot/latest/adminguide/cloudwatch-logs.html) — AWS documentation. Checked 2026-10-16. [^awsq-cli]: [Running commands using Amazon Q Developer in chat applications](https://docs.aws.amazon.com/chatbot/latest/adminguide/Things-to-know-about-cli.html) — AWS documentation. Checked 2026-10-16. [^aws-jobfn]: [AWS managed policies for job functions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html) — AWS Identity and Access Management User Guide. Checked 2026-10-16. [^aws-readonly]: [ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ReadOnlyAccess.html) — AWS Managed Policy Reference (policy version v190). Checked 2026-10-16. [^aws-viewonly]: [ViewOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ViewOnlyAccess.html) — AWS Managed Policy Reference (`arn:aws:iam::aws:policy/job-function/ViewOnlyAccess`, policy version v47, edited 2026-10-14). Checked 2026-10-16. [^awsq-api]: [CreateSlackChannelConfiguration](https://docs.aws.amazon.com/chatbot/latest/APIReference/API_CreateSlackChannelConfiguration.html), [AWS::Chatbot::SlackChannelConfiguration](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-chatbot-slackchannelconfiguration.html) and [AWS::IAM::ManagedPolicy](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-iam-managedpolicy.html) — AWS documentation. Checked 2026-10-16. [^st2-aliases]: [Action Aliases](https://docs.stackstorm.com/chatops/aliases.html) — StackStorm 3.9.0 documentation. Checked 2026-10-16. [^st2-rbac]: [Role Based Access Control](https://docs.stackstorm.com/rbac.html) — StackStorm 3.9.0 documentation. Checked 2026-10-16. [^st2-inquiries]: [Inquiries](https://docs.stackstorm.com/inquiries.html) — StackStorm 3.9.0 documentation. Checked 2026-10-16. [^errbot]: [Administration](https://errbot.readthedocs.io/en/latest/user_guide/administration.html) — Errbot documentation; version from [PyPI](https://pypi.org/project/errbot/) (6.2.1, 2026-06-06). Checked 2026-10-16. [^lita]: [lita versions](https://rubygems.org/gems/lita/versions) — RubyGems (latest 4.8.0, 2020-09-25). Checked 2026-10-16. [^am-config]: [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/) — Prometheus documentation. Checked 2026-10-17. [^am-changelog]: [Alertmanager CHANGELOG](https://github.com/prometheus/alertmanager/blob/main/CHANGELOG.md) — Prometheus project (0.34.1, 2026-09-17). Checked 2026-10-17. [^opsgenie]: [Opsgenie migration](https://www.atlassian.com/software/opsgenie/migration) — Atlassian. Checked 2026-10-16. [^botkube]: [Botkube](https://botkube.io/) — product homepage. Checked 2026-10-16. [^nist]: [SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final) — NIST, 2020 (Release 5.2.0, 2025-08-27); control text for AU-3, AC-3(2), AC-5, AC-6 and CM-5(4) read from the [OSCAL catalog](https://github.com/usnistgov/oscal-content). Checked 2026-10-16. [^owasp-llm01]: [LLM01:2025 Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) — OWASP Gen AI Security Project, 2025. Checked 2026-10-16. [^owasp-llm06]: [LLM06:2025 Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) — OWASP Gen AI Security Project, 2025. Checked 2026-10-16. [^sre-monitoring]: [Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/) — Rob Ewaschuk, ed. Betsy Beyer, in *Site Reliability Engineering*, Google, 2016. Checked 2026-10-16. [^gh-secret-scanning]: [Supported secret scanning patterns](https://docs.github.com/en/code-security/secret-scanning/introduction/supported-secret-scanning-patterns) — GitHub Docs. Checked 2026-10-16. **Further reading** - PagerDuty's open [Incident Response documentation](https://response.pagerduty.com/) shows chat commands inside a full incident process. **Changelog** - 2026-10-16: first version (research, ledger, draft, diagrams, labs 1–2 executed). - 2026-10-16 — r1 independent fact check (`reviews/chatops-r1.md`): 0 critical, 3 major, 15 minor. Revision (`reviews/chatops-r1-response.md`): decision tree and weighted check rebuilt to match the catalogue; per-platform proof-of-origin table (Slack, Socket Mode, Teams bot, Teams outgoing webhook, Mattermost); AWS template now uses a custom diagnostics guardrail instead of `ReadOnlyAccess`; NIST mapping to AC-3(2) and CM-5(4); verify-before-ack sequence; channels authorised by ID; Lab 2 shows what is pending (re-run with Python 3.8.10); literature, Disney, Opsgenie and footnote metadata corrected; repetition and overlap with AIOps, ObservabilityOps and IncidentOps trimmed. - 2026-10-17 — r2 after the independent score (`reviews/chatops-score.md`, 91/100; `reviews/chatops-r2-response.md`): Teams outgoing-webhook replay cell rewritten from the Bot Framework Activity specification; Lab 3 posts results with `chat.postMessage`; Figures 2, 3 and 8 redrawn; code blocks wrapped to 85 characters (Labs 1–2 re-run, audit file byte-identical); sample margin corrected; repetition trimmed. - 2026-10-17, post-score polish (`reviews/chatops-score-r2.md`, 94/100; `reviews/chatops-polish.md`): Figure 2 labels the executors a separate privilege zone; the Alertmanager feed posts to a channel ID and notes `app_token_file`; three code lines shortened; repetition trimmed.