--- title: "API Ops" slug: api-ops category: delivery term_status: emerging evidence_status: partially_verified summary: "API Ops, usually written APIOps, applies GitOps and DevOps practice to APIs: contracts and gateway configuration live in Git, and a pipeline lints, diffs and tests each change before promoting it to the gateways." audience: [Beginners, Backend engineers, API platform teams, API product managers, Architects] last_verified: 2026-10-19 key_takeaways: - "API Ops (usually spelled APIOps) is an emerging label with no standard and two related meanings: Kong, Microsoft and Tyk mean Git-based pipelines for API definitions and gateway configuration; the APIOps community means the operating model behind APIs. This page centres on the pipeline." - "Treat the contract (OpenAPI or AsyncAPI) as the deployable artefact: generate gateway configuration from it, and let no change reach a gateway without lint, breaking-change and test gates." - "'Breaking' is a published policy, not a law of nature, and so are your diff tool's defaults: oasdiff blocks a new response enum value but ignores a removed optional field until you change its severity levels." - "Gateway rate limits are targets, not ceilings (AWS calls them best effort, Azure never completely accurate, Kong's default local policy drifts as nodes are added): they protect capacity, but they are not cost controls, security boundaries or SLAs." - "Scope every gateway sync to what your team owns, let an approver read the planned deletions before production, and check for drift: a full decK sync deletes everything that is not in your file." - "Retire versions with evidence: Deprecation and Sunset headers, a dated sunset in the contract and per-consumer traffic data, not an email and hope." related: [gitops, integrationops, devsecops, iamops, identityops, testops, platformops, sre] --- ## What API Ops is ### API Ops in 30 seconds **API Ops** is the practice of changing APIs through a reviewed, automated pipeline. The API **contract** (an OpenAPI or AsyncAPI description) and the [gateway](#glossary-and-mental-model) configuration (routes, authentication, limits) live in Git. Each change is a pull request: gates lint it, compare it with the previous contract for breaking changes and run tests; after approval, a pipeline generates the gateway configuration and promotes the same commit through environments. Telemetry shows who calls which version. It serves API platform teams, backend engineers who publish APIs, and API product managers who promise consumers that an API will not break under them. ::: note Honest term status: emerging, two related meanings, no standard Vendors write **APIOps** as one word; this Atlas keeps the manifest's "API Ops". Kong defines it as "the practice of managing your API management configuration declaratively", following "GitOps principles such as version control, CI/CD, and pull request reviews"[^kong-apiops]. Microsoft calls it "a methodology that applies the concepts of GitOps and DevOps to API deployment"[^ms-arch]; Tyk says it "blends the best practices of GitOps and API management"[^tyk]. Microsoft also states: "There's no industry standard for setting up a DevOps pipeline for publishing APIs"[^ms-devops]. The label is older than these vendor uses: a non-vendor community uses it for the operating model behind APIs, and the open method it promotes, **APIOps Cycles**, runs from capability strategy to monitoring[^apiops-community][^apiopscycles]. No standards body defines either meaning. This page centres on the pipeline; its editorial emphasis, chosen by this Atlas, is **the contract as the deployable artefact that gates gateway changes**. ::: **Where to start.** Beginners: the [glossary](#glossary-and-mental-model) and [Lab 1](#lab-1-beginner-lint-a-contract-against-a-house-style). Engineers: the [contract gate](#the-contract-gate-what-blocks-a-merge) and [reference implementation](#reference-implementation). Leaders: the [decision framework](#decision-framework) and [judgement call](#a-judgement-call-a-new-major-version-or-a-twelve-month-migration). ### At a glance | Aspect | Summary | |---|---| | Goal | APIs that are consistent, discoverable and safe to evolve; gateways that match what Git says | | People | API owner teams, an API platform team, API product managers, security reviewers, consumers | | Prerequisites | HTTP and REST basics, YAML, Git and pull requests, one CI system; [GitOps](../gitops/) helps | | In scope | Design-first contracts, lint rules, breaking-change detection, conformance and contract tests, gateway configuration as code, drift, versioning and deprecation, limits and quotas, OAuth 2.0 and keys at the gateway, catalogues and portals, API metrics and SLAs | | Out of scope | Message flows between systems ([IntegrationOps](../integrationops/)); API security testing ([DevSecOps](../devsecops/)); running the authorisation server ([IdentityOps](../identityops/)); scope and permission design ([IAMOps](../iamops/)) | | Outputs | Contract repository, ruleset, change gate, generated gateway configuration, consumer register, deprecation calendar, scorecard | | Term status | Emerging (note above) | ### The problem it solves Hand-publishing works for one API. Trouble arrives with the tenth API, the fourth team and the first external partner: - **Four truths that disagree.** Documentation, contract, gateway and code diverge when people fix production in a portal; Microsoft's reference architecture requires such changes to be extracted back into Git through a reviewed pull request[^ms-arch]. - **Accidental breaking changes.** A renamed field breaks clients that worked yesterday, and Hyrum's Law warns that undocumented behaviour matters too: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody."[^hyrum] - **Forgotten versions.** OWASP's API9 (2023): attackers "usually get unauthorized access through old API versions or endpoints left running unpatched"[^owasp-api9]. - **Inconsistent, undiscoverable APIs.** Postman's 2025 survey (over 5,700 developers, architects and executives) reports 55% struggling with inconsistent documentation and 34% unable to find existing APIs; the report does not state the base[^postman-2025]. ::: example Illustrative scenario: one silent break A partner calls `GET /orders/{id}` 40,000 times a day. A release renames `deliveryDate` to `deliveredAt`; the partner's sync fails for a day. A gate set to block removed response fields (not every tool's default, see the [contract gate](#the-contract-gate-what-blocks-a-merge)) would have turned the rename into "add the new field, deprecate the old one" at review time. ::: ### What API Ops is not: boundaries and naming - **Not an API management product.** Gateways and portals are tools; API Ops is the change process around them. - **Not GitOps renamed.** OpenGitOps requires pull-based, continuously reconciled state ([GitOps](../gitops/)); the decK and APIOps CLI pipelines push from CI (`deck gateway sync`, `apiops publish`)[^deck-sync][^ms-arch]. Kubernetes controllers such as Tyk Operator allow a pull-based variant ([tooling](#tooling-landscape)). - **Not API security testing**, which belongs to [DevSecOps](../devsecops/); API Ops runs the cheap contract checks. - **Not integration operations.** Retries, dead-letter queues and replay belong to [IntegrationOps](../integrationops/). **Editorial distinction:** API Ops owns the contract; IntegrationOps owns the flow. - **Not the whole APIOps Cycles operating model**, which also covers strategy, platform decisions, publishing and monitoring of API capabilities[^apiopscycles]; it complements this page and feeds it the first draft of a contract. - **Misconceptions:** "we have a gateway, so we do API Ops" (a hand-configured gateway is the problem); "a version in the URL makes changes safe" (it only gives a breaking change somewhere to live). ## Where it comes from and where it sits ### Origins and evolution - **2015:** the Linux Foundation announced the Open API Initiative (5 November)[^lf-2015]; OpenAPI "was originally based on the Swagger Specification, donated by SmartBear Software"[^oai-about]. - **2016–2018:** an APIOps project for the National Library of Finland planned "APIOps Monthly Jam" sessions in 2016, and Nordic APIs describes Jarkko Moilanen as "creator of the APIOps concept"[^apiops-community]. APIOps Cycles, "Developed since 2017"[^apiopscycles], was presented at the 2018 Platform Summit as a lean API design method[^nordic-2019]. - **2021:** Kong described APIOps as applying "the solid and proven principles of DevOps and GitOps to the API and microservice lifecycle", one of the earliest vendor uses we found[^kong-2021]. - **2024–2025:** Stripe moved to monthly non-breaking API releases from `2024-09-30.acacia`[^stripe-versions]; RFC 9745 standardised the `Deprecation` header (March 2025)[^rfc9745]. - **2026:** Microsoft's reference architecture (dated 1 September) uses a single APIOps CLI and asks users of the legacy APIOps Toolkit to migrate[^ms-arch]; the OpenAPI Initiative published Arazzo 1.1.0 (May), OpenAPI 3.2.1 and Overlay 1.2.0 (September)[^arazzo][^oas][^overlay]. ### Glossary and mental model **Mental model: three artefacts, one loop.** The *contract* says what the API promises; the *gateway policy* says how each environment exposes it; the *consumer register* says who depends on which version. Gates protect the register's consumers from contract changes, the pipeline derives the policy from the contract, and telemetry keeps the register honest. ::: figure Three artefacts and the loop that keeps them consistent ```mermaid %%{init: {"flowchart": {"wrappingWidth": 150, "nodeSpacing": 18, "rankSpacing": 30, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 C["1. Contract in Git: OpenAPI or AsyncAPI"]:::core P{{"2. Gates: lint, breaking diff, tests"}}:::decision G["3. Gateway policy generated per environment"]:::core T["4. Telemetry per consumer and version"]:::neutral R["5. Consumer register: who calls what"]:::neutral C --> P P -->|"pass"| G P -->|"fail: fix, or add and deprecate"| C G --> T --> R R -->|"evidence for design and retirement"| C ``` Text equivalent: (1) A contract change in Git meets (2) the gates: lint, breaking diff, tests. On failure it returns as a fix or as an additive change that deprecates the old element. On success (3) the pipeline generates each environment's gateway policy; (4) telemetry per consumer and version keeps (5) the consumer register current, the evidence for the next design or retirement. ::: | Term | Meaning | |---|---| | **API gateway, API management** | A gateway is the single entry point that routes calls to services and can check that clients are authorised[^microservices]; API management products (Kong, Azure API Management, AWS API Gateway, Tyk) add limits, keys, portals and analytics | | **Consumer, developer portal** | A client application calling the API; the portal is where its developers find documentation and get credentials | | **Contract, OAD** | The machine-readable promise; an OpenAPI Description "formally describes the surface of an API and its semantics"[^oas]; AsyncAPI covers message-driven APIs[^asyncapi] | | **Design-first, code-first** | Contract before code (Zalando rule 100[^zalando]), or generated from code or TypeSpec[^typespec] | | **Ruleset** | The style guide written as lint rules (Spectral, `deck file lint`)[^spectral][^deck-file] | | **Breaking, additive change** | A change existing clients cannot absorb, or can; each provider publishes its own list | | **Conformance test** | Checks that the running service behaves as its contract says; Schemathesis generates such tests[^schemathesis] | | **Three "versions"** | The API's version (`v2`, `2026-03-10`), the OpenAPI version (`openapi: 3.0.4`) and `info.version`, which OpenAPI 3.2 says is distinct from both[^oas] | | **Deprecation, sunset** | Still works but do not start using it; the date it is expected to stop responding. HTTP headers exist for both[^rfc9745][^rfc8594] | | **Stability level** | Draft, alpha, beta or stable; sets how strictly oasdiff checks changes (`x-stability-level`)[^oasdiff-docs] | | **Rate limit, quota, token bucket** | Short-window capacity protection versus a plan's long-period allowance; a bucket refills at the *rate* and holds the *burst*[^aws-throttle] | | **Drift** | Any difference between a deployed gateway and the approved commit | | **Consumer-driven contract** | Consumer expectations verified against the provider (Pact)[^pact] | ### How API Ops relates to other Ops The boundaries below are **editorial distinctions** made by this Atlas (coverage matrix); no standard defines them. | Neighbour | Shared ground | Choose that page when… | |---|---|---| | [GitOps](../gitops/) | State in Git, review, drift | You design reconciliation itself | | [IntegrationOps](../integrationops/) | Schemas, AsyncAPI | The problem is message delivery, retries or replay | | [DevSecOps](../devsecops/) | OWASP API Top 10 | You need active API scans or authorisation tests ([its playbooks](../devsecops/#core-practices-and-playbooks)) | | [IdentityOps](../identityops/), [IAMOps](../iamops/) | OAuth 2.0 at the gateway | You run the authorisation server, or design scopes | | [TestOps](../testops/) | Contract tests | You run consumer-driven contract tests as a CI stage | | [PlatformOps](../platformops/) | Developer portal, catalogue | The API platform is one product of a wider internal platform | | [SRE](../sre/) | API SLOs | You set SLO targets and error-budget policy | | [ReleaseOps](../releaseops/) | Canary, rollback | You choose the release strategy behind the API | ## Principles and operating model ### Charter and principles ::: decision Proposed charter (Atlas proposal; no official manifesto exists) **Mission.** Every API change is checked against a published style and compatibility policy, reaches every environment through one pipeline, and is traceable to a commit, an approver and the consumers it affects. **Goals:** no ungated contract change; no production gateway change outside the pipeline except recorded emergencies; every deprecated version dated, with each remaining consumer named. **Non-goals:** choosing the gateway product, running the authorisation server, security testing. ::: 1. **The contract is source code.** Microsoft says the API definition "should be considered "source code""[^ms-devops]. 2. **Generate gateway configuration from the contract**, as Kong's 2021 description does[^kong-2021]; hand edits drift. 3. **Fail early.** "The earlier in the pipeline you can identify deviations from your standards, the faster they are to resolve."[^kong-2021] 4. **Publish your compatibility policy, then make the tools match it.** A gate enforces its configuration, not your intent ([contract gate](#the-contract-gate-what-blocks-a-merge)). 5. **One writer per target.** Only the pipeline writes to production gateways, with separate identities for extracting and publishing; people need no standing write access[^ms-arch]. 6. **Own only what you can name.** Scope each deployment to a team's entities (tags, workspaces)[^deck-sync]. 7. **Retire with evidence.** Deprecation is a measured migration, not an announcement. ### Building blocks | Block | What it holds | Typical owner | Examples (October 2026) | |---|---|---|---| | Contract repository | Contracts, examples, changelog | API owner team | Git; OpenAPI 3.0–3.2[^oas], AsyncAPI 3.1[^asyncapi] | | Ruleset | House style and governance rules | API platform team | Spectral 6.17.0[^spectral], `deck file lint`[^deck-file] | | Change gate | Lint, breaking diff, tests | Platform team; owners act on results | oasdiff 1.33.0[^oasdiff], Schemathesis[^schemathesis], Pact[^pact] | | Gateway policy | Routes, auth, limits, per-environment overrides | Platform team, with owners | decK 1.68.0[^deck], APIOps CLI 1.0.5[^apiops-cli] | | Runtime | Gateways, portal or catalogue | Platform team | Kong, Azure API Management, AWS API Gateway; Backstage `kind: API`[^backstage] | | Evidence | Traffic per consumer and version, errors, 429s | Owners, product manager | Gateway logs and metrics[^aws-metrics] | ### Roles, ownership and RACI | Activity | API owner team | API platform team | API product manager | Security | Consumers | |---|---|---|---|---|---| | Contract change | R/A | C | C | C (auth, data) | I (breaking or deprecating) | | Ruleset and gate | C | R/A | C | C | — | | Gateway policy (limits, auth) | R | A | C | C | I | | Production promotion | R (requests) | A (pipeline) | I | I | — | | Deprecation and sunset | R | C | A | I | C (consent for partners) | | Emergency gateway change | R | A | I | I | I | R = responsible, A = accountable, C = consulted, I = informed. A platform team runs ruleset, pipeline and gateways as a product while owner teams own contracts ([PlatformOps](../platformops/)); with few APIs, one team owns all. A board that hand-checks every change becomes a queue: automate its rules, keep it for exceptions. ### Lifecycle: from draft to retirement ::: figure API lifecycle states and the evidence each transition needs ```mermaid stateDiagram-v2 [*] --> Draft Draft --> Beta Beta --> Stable Stable --> Deprecated Deprecated --> Retired Retired --> [*] Beta: Beta Beta: contract reviewed and mocked Stable: Stable Stable: owner, SLO, docs Stable: additive changes only Deprecated: Deprecated Deprecated: successor live, dates set Retired: Retired Retired: sunset passed, traffic gone ``` Text equivalent: Draft becomes Beta once the contract is reviewed and mocked; Beta becomes Stable with an owner, an SLO and documentation; Stable absorbs additive changes; it becomes Deprecated when a successor is live and dates are set, and Retired once the sunset has passed and traffic is gone. The table lists the gate and evidence for each transition. ::: | Transition | Gate (Atlas proposal) | Evidence | |---|---|---| | Draft → Beta | Lint passes; design review for new resources | Mock or test deployment; examples | | Beta → Stable | Owner, SLO, docs, policy reviewed | `x-stability-level: stable`; catalogue entry | | Stable → Stable | Breaking diff clean at error level | Changelog entry | | Stable → Deprecated | Successor live; dates agreed | `deprecated: true` and `x-sunset`[^oasdiff-docs]; headers | | Deprecated → Retired | Sunset passed; traffic gone or waived | Register: each consumer migrated or waived | ### Operating model and governance | Cadence | Activity | |---|---| | Every pull request | Automated gates; code-owner review of contract and policy files | | Weekly, 30 minutes | API health: errors, latency, 429s per consumer, drift, blocked merges | | Monthly | Deprecation calendar: who still calls what, outreach, dates | | Quarterly | Reconcile routes, contracts and catalogue; review ruleset; expire waivers | **Exceptions** are reviewed waiver files with an owner and expiry (oasdiff reads `--err-ignore` and `--warn-ignore` files[^oasdiff-docs]). **Escalation:** a blocked breaking change goes to the API product manager, who chooses between a new version and a coordinated migration ([judgement call](#a-judgement-call-a-new-major-version-or-a-twelve-month-migration)). ## Architecture and workflow ### Reference architecture A logical reference for several teams sharing a gateway platform; the shape is the same on Azure API Management or AWS. ::: figure Reference architecture: from contract repository to gateways, with the drift loop ```mermaid %%{init: {"flowchart": {"wrappingWidth": 210, "nodeSpacing": 20, "rankSpacing": 30, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 K["Git, one folder per API: contract, ruleset, gateway policy, overrides"]:::core G{{"Pull-request gates: lint, breaking diff, conformance tests"}}:::decision B["Merge build: generate and lint gateway configuration"]:::core R["Release pipeline, sole writer: preview deletions, then sync"]:::neutral D["Development and staging gateways"]:::core P["Production gateways"]:::risk T["Telemetry, catalogue, consumer register"]:::neutral K --> G G -->|"pass and approve"| B B --> R --> D D -->|"approval"| P P --> T P -.->|"nightly drift check"| R T -.->|"evidence"| K ``` Text equivalent: Git holds, per API, the contract, ruleset, gateway policy and overrides. Pull requests pass gates (lint, breaking diff, conformance tests); after merge, a build generates and lints the gateway configuration. The release pipeline, the only writer, previews each change, deploys to development and staging, and after approval to production (the high-risk step). Production feeds telemetry, catalogue and consumer register, evidence for the contract; a nightly drift check compares production with the last deployment. ::: | Component | Key control | |---|---| | Contract repository, gates | Protected branch; code owners per API folder; required status checks | | Release pipeline | Workload identity; approver reads the dry run's deletions; one writer per target at a time[^ms-arch] | | Gateway | Admin interface on a private network; sync scoped by tags or workspaces | | Catalogue and portal | Generated from the contract in CI; visible only to authorised users where needed[^owasp-api9] | | Telemetry | Consumer identity on every request; retention policy | **Trade-offs.** A gateway adds "Increased response time due to the additional network hop through the API gateway", a cost the same author calls insignificant for most applications, and another component to run[^microservices]. Gateway features live either in vendor extensions inside the contract (`x-kong-*`)[^deck-file] or in a separate policy file (this page's choice). And reverting an APIOps commit "doesn't restore an incompatible or unavailable back end"[^ms-arch]. ### The contract gate: what blocks a merge What counts as breaking is API Ops's hardest policy call, and published policies disagree at the edges: | Change | Google AIP-180[^aip180] | GitHub REST[^github] | Stripe[^stripe-versions] | Atlas gate default | |---|---|---|---|---| | Remove or rename an operation or field | Must not, within a major version | Breaking | Major releases only | Block unless deprecated and past sunset | | Add a required parameter; make one required | Must not | Breaking | — | Block | | Change a field's type | Must not | Breaking | — | Block | | Add a validation rule to a parameter | Stricter resource-name formats make old requests fail | Breaking | — | Block | | Change a default value | "considered breaking and must not be done" | — | — | Block | | Change authentication or authorisation requirements | — | Breaking | — | Block; security review | | Add an optional parameter or a response field | May be added | Additive | Backward-compatible | Allow | | Add a value to a response enum | "appropriate caution"; document that it may grow | Additive | — | Warn; allow if the enum is documented as open | | Change ID length or format | Must not change format; treat larger size limits as incompatible | — | Backward-compatible (opaque strings) | Warn; allow only if documented as opaque | A dash means the source does not address the change. oasdiff's documentation states the pattern behind most rows: narrowing what requests accept, or widening what responses return, breaks clients[^oasdiff-docs]. Stripe puts the counterpart on consumers: "Design your integration to ignore unfamiliar fields and event types."[^stripe-versions] Write down who carries which burden, then encode it: - **Error-level findings block** the merge (`oasdiff breaking --fail-on ERR` exits 1)[^oasdiff-docs]. - **Warnings need a named reviewer's acknowledgement.** - **Deprecation needs a date:** the house Spectral rule (an Atlas rule) rejects `deprecated: true` without `x-sunset`. oasdiff blocks removing an operation or parameter before its sunset, and with `--deprecation-days-stable` also rejects a deprecation whose sunset is missing or too close[^oasdiff-docs]. - **Stability sets strictness:** oasdiff checks beta and stable by default, not draft or alpha[^oasdiff-docs]. - **Waivers are files** with owner and expiry (`--err-ignore`), never a disabled check. **Tool defaults are a policy too.** oasdiff 1.33.0 does not read this table. It rates removing an optional response property as INFO, which `oasdiff breaking` never shows, and adding a response enum value as ERR, which blocks[^oasdiff-docs]. List every check's level with `oasdiff checks changelog`, then record your differences in a file passed with `--severity-levels`: ```text response-optional-property-removed err response-property-enum-value-added warn ``` Removing a field after its sunset then needs a waiver line, which is the review you want. Zalando agrees that extending a closed `enum` is incompatible; where values may grow, its rule 112 lists them under `examples` instead[^zalando], so no enum finding arises. ### End-to-end workflow ::: figure From pull request to production gateway, with the breaking-change exception path ```mermaid --- config: sequence: actorMargin: 30 width: 80 diagramMarginX: 6 wrap: true boxMargin: 6 messageMargin: 30 --- sequenceDiagram autonumber participant Team as Team participant Git as Git participant CI as Pipeline participant GW as Gateways Team->>Git: Pull request Git->>CI: Run gates opt Break found CI-->>Git: Fail Git-->>Team: Red check Team->>Git: Deprecate Git->>CI: Re-run end CI-->>Git: Pass Team->>Git: Merge Git->>CI: Build config CI->>GW: Staging CI->>GW: Preview Note over Team,CI: Approver reads the planned deletions CI->>GW: Sync prod CI->>GW: Verify ``` Text equivalent: (1–2) A pull request triggers the gates. (3–6) Exception path: the breaking diff finds a removed field, so the team adds the new field, deprecates the old one and the gates re-run. (7–8) Gates pass; owners approve and merge. (9–10) The pipeline builds the configuration from the main branch and syncs staging. (11–12) It previews production; an approver reads the planned deletions and approves before the sync. (13) Verification (smoke tests and a drift diff) closes the release. ::: Every decision leaves a record: gate results, waivers, approvals and the commit deployed to each environment. In Microsoft's version, "The release approver reviews the resources that the dry run creates, updates, deletes, or skips", a dry run is "a deployment gate, not a substitute for automated API tests", and only one pipeline writes to a target at a time[^ms-arch]. ## Practices, frameworks and tools ### Core practices and playbooks Each playbook ends with an acceptance test and a rollback. #### Playbook 1: the style guide as a ruleset 1. Start from published guidelines (Azure, Google AIPs, Zalando)[^ms-devops][^aip180][^zalando]. Microsoft's own Spectral ruleset is archived because "Azure API specifications have moved to TypeSpec"[^azure-style], so rulesets need owners. 2. Encode 10–20 rules consumers feel (naming, RFC 9457 errors[^rfc9457], pagination, security schemes, deprecation dates). 3. Extend `spectral:oas`; lower or switch off inherited rules with `rule-name: warn` or `off`[^spectral-docs]. 4. Lint the contract locally and in the pull request, and the generated gateway file with `deck file lint`[^deck-file]. **Acceptance:** every contract lints clean; every rule says why. **Rollback:** introduce new rules as `warn` for one cycle. #### Playbook 2: the breaking-change gate 1. Publish the compatibility policy and the matching levels file ([contract gate](#the-contract-gate-what-blocks-a-merge)). 2. In each pull request run `oasdiff breaking base.yaml head.yaml --fail-on ERR --severity-levels `[^oasdiff-docs]. 3. Set `x-stability-level` per operation; require `deprecated: true` plus `x-sunset` before removal[^oasdiff-docs]. 4. Test the running service against the contract (Schemathesis)[^schemathesis]; Pact suits internal APIs with known consumers, not public APIs or pass-through gateways[^pact] (see [TestOps](../testops/)). **Acceptance:** a pull request removing a response field fails; one adding an optional field passes. **Rollback:** warn-only with a dated issue. #### Playbook 3: versioning and deprecation Pick one scheme per API family and record it in an ADR; each has a price: | Scheme | Who uses it | Strength | Price | |---|---|---|---| | Major version in the path (`/v1/`) | Google AIP-185[^aip185] | Visible; easy to route | Whole-API version for one breaking change | | Dated version in a header | GitHub, Stripe[^github][^stripe-versions] | Small, opt-in changes | Provider keeps every supported shape | | Media type; avoid versioning | Zalando rules 113–115[^zalando] | Forces compatible evolution | Hard for simple clients | | Versionless; deprecate fields | GraphQL guidance and `@deprecated`[^graphql] | No version sprawl | Needs per-field usage data | Then run deprecation as a protocol: 1. Mark the element `deprecated: true` with an `x-sunset` date in the contract[^oas][^oasdiff-docs]. 2. Send `Deprecation` (a structured date such as `@1796083199`) and `Sunset` (an HTTP date) headers and a `rel="deprecation"` link to the migration guide; the sunset must not precede the deprecation[^rfc9745][^rfc8594]. 3. Measure traffic **per consumer** ([Lab 4](#lab-4-advanced-executed-api-kpis-from-a-gateway-log)) and contact each; Zalando requires usage monitoring and client approval before shut-down[^zalando]. 4. Choose the window: GitHub keeps a previous REST version "at least 24 more months"[^github]; Meta guarantees each Graph API version at least two years of operation[^meta]; Google recommends 180 days between deprecating beta functionality and removing it[^aip185]. 5. Decide post-sunset behaviour: an error such as `410 Gone`, or Meta's forwarding to "the next oldest, usable version"[^meta], which avoids outages but can change behaviour silently. **Acceptance:** each deprecated element has a date, a guide and a per-consumer chart. **Rollback:** extend the sunset (RFC 8594 calls it a hint[^rfc8594]); never shorten it without consent. #### Playbook 4: rate limits and quotas 1. Rate limits protect capacity; quotas express a plan; provider spending limits stop bill shocks (OWASP API4)[^owasp-api4]. 2. Size the token bucket: from full, it admits up to *burst + rate × T* requests in *T* seconds, a target, not a guarantee[^aws-throttle]. 3. Key limits by consumer, not IP, when clients share proxies; Kong limits by consumer by default[^kong-rl]. 4. Return `429` with `Retry-After`[^rfc6585]; `RateLimit-Policy` and `RateLimit` headers are still an Internet-Draft (October 2026)[^ratelimit-draft]. 5. Expect approximation: AWS applies limits "on a best-effort basis"[^aws-throttle]; Azure's rate limiting "is never completely accurate" and counts per gateway[^apim-rl]. Kong's default `local` policy "diverges when scaling the number of nodes"; its shared `cluster` and `redis` policies are accurate only with synchronous counter updates[^kong-rl]. 6. Choose failure behaviour: Kong's `fault_tolerant` defaults to `true`, proxying without limits while the counter store is down[^kong-rl]. **Acceptance:** a load test yields 429s with `Retry-After` at the expected rate per consumer. **Rollback:** change limits in Git only. #### Playbook 5: authentication as gateway configuration 1. **Keys identify; tokens authorise.** AWS: "Don't use API keys for authentication or authorization to control access to your APIs"; a key for one API in a usage plan works for every API in it[^aws-plans]. 2. **Validate JWT access tokens fully:** `typ` `at+jwt`, exact issuer, audience naming this API, signature from the authorisation server's keys, no `alg: none`, expiry[^rfc9068]. Tokens "SHOULD be audience-restricted" and minimally privileged[^rfc9700]. 3. **Close side doors.** Kong's OpenID Connect plugin (Enterprise) reads bearer tokens from body, header and query string by default (`bearer_token_param_type`); Key Auth accepts query-string keys by default (`key_in_query`)[^kong-auth]. Allow headers only: RFC 9700 says "Clients MUST NOT pass access tokens in a URI query parameter", and recommends sender-constrained tokens (mutual TLS or DPoP), an option for partner APIs that [IdentityOps](../identityops/) would issue[^rfc9700]. 4. **Leave object-level checks to the service**, which alone knows ownership ([IAMOps](../iamops/) designs scopes; [DevSecOps](../devsecops/) tests authorisation). **Acceptance:** tokens with wrong issuer, wrong audience, expired, or sent in a query string are refused in test. **Rollback:** revert the commit. #### Playbook 6: drift and emergency changes 1. The pipeline is the only writer; people get read access and time-bound elevation[^ms-arch]. 2. Scope every sync with `--select-tag`: an unscoped sync deletes whatever the file lacks; `deck gateway apply` never deletes and suits bootstrap only[^deck-sync]. 3. Schedule `deck gateway diff --non-zero-exit-code`; exit code 2 means drift[^deck-diff]. 4. Extract each emergency change and merge it through a pull request so Git stays authoritative[^ms-arch]. **Acceptance:** a deliberate manual change turns the nightly drift job red. **Rollback:** re-sync the last approved commit per tag. ### Methods, standards and frameworks | Item | Kind | Use in API Ops | |---|---|---| | OpenAPI 3.2.1, Overlay 1.2.0, Arazzo 1.1.0[^oas][^overlay][^arazzo] | Specifications (OpenAPI Initiative) | Contract; separate variants; multi-call workflows | | AsyncAPI 3.1.0[^asyncapi] | Specification | Event and message contracts (flows: [IntegrationOps](../integrationops/)) | | RFC 9457, RFC 6585, RFC 9745, RFC 8594[^rfc9457][^rfc6585][^rfc9745][^rfc8594] | IETF RFCs (8594 is Informational) | Error format, 429, deprecation and sunset signals | | RFC 9068, RFC 9700[^rfc9068][^rfc9700] | Standards Track; Best Current Practice | JWT access-token validation; OAuth security | | OWASP API Security Top 10 (2023)[^owasp-top10] | Industry risk list | API9 inventory and API4 resource limits drive gates and limits | | Google AIPs, Zalando and Azure guidelines[^aip180][^zalando][^ms-devops] | Published company guidelines | Starting points for your ruleset; not standards | | APIOps Cycles[^apiopscycles] | Community method (CC BY-SA 4.0) | Operating model around the pipeline: strategy, design, publishing, monitoring | Only the RFCs and specifications are normative, for what they cover; company guidelines are optional. None defines "APIOps". ### Tooling landscape | Job | Options (October 2026) | Watch out for | |---|---|---| | Write contracts | Editors; TypeSpec emits OpenAPI 3.0 by default, 3.1 or 3.2 on request[^typespec] | Generated contracts still need review | | Lint | Spectral 6.17.0 (Node 22 or 24)[^spectral]; `deck file lint`[^deck-file] | Rulesets need owners; archived rulesets rot | | Breaking diff | oasdiff 1.33.0[^oasdiff]; Microsoft openapi-diff[^ms-devops]; AsyncAPI CLI `diff` (6.2.1)[^asyncapi-cli] | Pin versions; default severities are policy choices | | Conformance, mocks | Schemathesis (OpenAPI 2.0–3.2)[^schemathesis]; Prism (up to 3.1)[^prism] | Active tests mutate data: test systems only | | Consumer contracts | Pact[^pact] | Not for public APIs or pass-through gateways[^pact] | | Gateway as code (push) | decK 1.68.0 for Kong[^deck]; APIOps CLI 1.0.5 for Azure[^apiops-cli]; Tyk Sync[^tyk]; Terraform or Bicep for infrastructure[^ms-arch] | decK needs a database-backed Kong, not DB-less[^kong-dbless] | | Gateway as code (pull) | Tyk Operator with Argo CD or Flux[^tyk-operator]; Kong Ingress Controller fed by `deck file kong2kic`[^kong-apiops] | Needs Kubernetes; the controller, not the pipeline, corrects drift | | Catalogue | Backstage `kind: API` with lifecycle `deprecated`[^backstage] | Catalogue entries drift unless generated | **Selection criteria:** your gateways, per-team scoping, preview of deletions, offline CI use, and OpenAPI version support across the whole chain. Not every tool lists OpenAPI 3.2 (Prism stops at 3.1[^prism]), so the labs use 3.0.4. **AsyncAPI:** the gate chain above is OpenAPI-only; the AsyncAPI CLI's `diff` classifies event-contract changes as breaking, non-breaking or unclassified[^asyncapi-cli]. This Atlas did not test its rules: run it as a warning first and review event changes by hand. ### Automation and AI Automate the mechanical checks (style, compatibility, conformance, drift, inventory); keep humans for domain fit, whether a break is worth a new version, and migration timing: "Tools won't catch all problems."[^ms-devops] AI assistants can draft descriptions or summarise a diff; their output passes the same gates. Agents are also becoming consumers: 51% of developers in Postman's 2025 survey worry about unauthorised or excessive calls from AI agents[^postman-2025]. Per-consumer identity, quotas and telemetry make such traffic visible. ## API Ops in practice ### Reference implementation A **representative design**, not a recipe: an `orders` API on Kong Gateway 3.16, decK 1.68, OpenAPI 3.0.4 and GitHub Actions; hosts, names and dates are placeholders. ```text apis/orders/openapi.yaml contract (owners: orders team) apis/orders/catalog-info.yaml catalogue entry that embeds the contract apis/orders/gateway/plugins.yaml gateway policy layered onto the generated config .spectral.yaml house ruleset (owner: API platform team) .oasdiff-levels.txt severity levels matching the compatibility policy .github/workflows/api-gate.yaml pull-request gates, preview and release ops/kong-rules.yaml rules for generated gateway config (platform team) CODEOWNERS /apis/orders/ @example/orders-team ``` #### 1. The contract _Example status: YAML syntax-checked by the Atlas validator; not linted with Spectral or diffed with oasdiff here._ ```yaml openapi: 3.0.4 info: title: Orders API version: 2.3.0 contact: {name: Orders team, email: orders-team@example.com} servers: - url: http://orders-mock:4010 paths: /v2/orders/{orderId}: get: operationId: getOrder tags: [orders] x-stability-level: stable security: [{apiKey: []}] parameters: - {name: orderId, in: path, required: true, schema: {type: string}} responses: "200": description: The order. content: application/json: schema: {$ref: "#/components/schemas/Order"} "404": {$ref: "#/components/responses/Problem"} /v1/orders/{orderId}: get: operationId: getOrderV1 tags: [orders] deprecated: true x-sunset: "2027-06-30" x-stability-level: stable security: [{apiKey: []}] parameters: - {name: orderId, in: path, required: true, schema: {type: string}} responses: "200": description: The order (legacy shape). content: application/json: schema: {$ref: "#/components/schemas/Order"} "404": {$ref: "#/components/responses/Problem"} components: securitySchemes: apiKey: {type: apiKey, in: header, name: apikey} responses: Problem: description: RFC 9457 problem details. content: application/problem+json: schema: {$ref: "#/components/schemas/Problem"} schemas: Order: type: object required: [id, status] properties: id: {type: string, description: Opaque identifier.} status: type: string description: Open set; clients must accept unknown values. enum: [created, paid, shipped] deliveryDate: {type: string, format: date, deprecated: true, x-sunset: "2027-06-30"} deliveredAt: {type: string, format: date-time} Problem: type: object properties: type: {type: string} title: {type: string} status: {type: integer} detail: {type: string} ``` The API key identifies the consumer for limits and analytics; production authorisation adds OAuth 2.0 tokens ([Playbook 5](#playbook-5-authentication-as-gateway-configuration)). #### 2. The house ruleset _Example status: YAML syntax-checked; rule syntax per the Spectral docs[^spectral-docs]; the JSONPath filters were not executed. The `~` key selector and the numeric `@property` filter are documented jsonpath-plus syntax[^jsonpath], untested under Spectral._ ```yaml extends: ["spectral:oas"] rules: info-contact: error operation-id-camel-case: description: operationId is camelCase so generated clients read well. severity: error given: "$.paths[*][get,put,post,delete,patch].operationId" then: function: casing functionOptions: {type: camel} paths-kebab-case: description: Path segments are kebab-case. severity: warn given: "$.paths[*]~" then: function: pattern functionOptions: {match: "^(/[a-z0-9-]+|/[{][a-zA-Z]+[}])+$"} errors-use-problem-json: description: 4xx and 5xx responses use application/problem+json (RFC 9457). severity: error given: "$.paths[*][*].responses[?(@property >= 400 && @property < 600)].content" then: field: application/problem+json function: truthy deprecation-needs-sunset: description: Deprecated operations, parameters and properties state x-sunset. severity: error given: - "$.paths[*][?(@.deprecated == true)]" - "$.paths[*][*].parameters[?(@.deprecated == true)]" - "$.components.schemas[*].properties[?(@.deprecated == true)]" then: field: x-sunset function: truthy ``` #### 3. Gateway policy layered onto the generated configuration _Example status: YAML syntax-checked; file format from Kong's add-plugins page[^deck-file]; plugin fields from the Rate Limiting and Key Auth references[^kong-rl][^kong-auth]; not applied to a gateway here._ ```yaml add-plugins: - selectors: - $..services[*] overwrite: false plugins: - name: key-auth config: key_names: [apikey] key_in_query: false - name: rate-limiting config: minute: 600 policy: redis redis: host: redis.internal.example port: 6379 limit_by: consumer fault_tolerant: false ``` `fault_tolerant: false` returns errors instead of proxying unlimited traffic when Redis is down[^kong-rl]; choose it only if that outage is cheaper than an unprotected backend. #### 4. Pull-request gates, preview and release _Example status: YAML syntax-checked; not run on GitHub. Action majors are the latest releases on 2026-10-18, all on Node 24, which runners require since Node 20's removal on 2026-09-23[^gh-actions]; pin them by commit SHA in production. Assumes a self-hosted runner that reaches the Admin API privately and has `jq`; Admin API credentials are deployment-specific and omitted; the Admin API must never be reachable from the internet._ ```yaml name: api-gate on: pull_request: paths: - "apis/**" - "ops/**" - ".spectral.yaml" - ".oasdiff-levels.txt" - ".github/workflows/api-gate.yaml" push: branches: [main] paths: ["apis/**", "ops/**", ".spectral.yaml", ".github/workflows/api-gate.yaml"] permissions: contents: read jobs: gates: runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v7 with: {fetch-depth: 0} - uses: actions/setup-node@v7 with: {node-version: 24} - uses: actions/setup-go@v7 with: {go-version: stable} - name: Install pinned tools run: | npm install --global @stoplight/spectral-cli@6.17.0 go install github.com/oasdiff/oasdiff@v1.33.0 go install github.com/kong/deck@v1.68.0 - name: Lint contract run: | spectral lint apis/orders/openapi.yaml --ruleset .spectral.yaml \ --fail-severity error --format github-actions - name: Breaking changes against the base branch if: github.event_name == 'pull_request' run: | if ! git show "origin/${GITHUB_BASE_REF}:apis/orders/openapi.yaml" > base.yaml then echo "::notice::New API, no base contract: breaking diff skipped" exit 0 fi oasdiff breaking base.yaml apis/orders/openapi.yaml --fail-on ERR \ --severity-levels .oasdiff-levels.txt --deprecation-days-stable 180 \ --format githubactions - name: Generate and lint gateway config run: | deck file openapi2kong --spec apis/orders/openapi.yaml \ --select-tag orders-api --output-file kong.generated.yaml deck file add-plugins -s kong.generated.yaml \ apis/orders/gateway/plugins.yaml > kong.plugins.yaml deck file add-tags -s kong.plugins.yaml orders-api > kong.yaml deck file lint -s kong.yaml ops/kong-rules.yaml - uses: actions/upload-artifact@v7 with: {name: kong-orders, path: kong.yaml} preview: if: github.event_name == 'push' needs: gates runs-on: [self-hosted, gateway-network] env: DECK_KONG_ADDR: ${{ vars.KONG_ADMIN_URL }} steps: - uses: actions/download-artifact@v8 with: {name: kong-orders} - uses: actions/setup-go@v7 with: {go-version: stable} - run: go install github.com/kong/deck@v1.68.0 - name: Plan production changes; stop above the deletion budget run: | deck gateway diff kong.yaml --select-tag orders-api --json-output > plan.json jq -c '.summary' plan.json >> "$GITHUB_STEP_SUMMARY" jq -r '.changes.deleting[] | "- delete \(.kind) \(.name)"' plan.json \ >> "$GITHUB_STEP_SUMMARY" if [ "$(jq '.summary.deleting' plan.json)" -gt 3 ]; then echo "::error::More than 3 deletions: split the change or approve a waiver" exit 1 fi - uses: actions/upload-artifact@v7 with: name: release-orders path: | kong.yaml plan.json sync: needs: preview runs-on: [self-hosted, gateway-network] environment: production concurrency: {group: kong-production, cancel-in-progress: false} env: DECK_KONG_ADDR: ${{ vars.KONG_ADMIN_URL }} steps: - uses: actions/download-artifact@v8 with: {name: release-orders} - uses: actions/setup-go@v7 with: {go-version: stable} - run: go install github.com/kong/deck@v1.68.0 - name: Sync the reviewed plan if the gateway has not changed since run: | deck gateway diff kong.yaml --select-tag orders-api --json-output > now.json a=$(jq -S '.changes' now.json | sha256sum | cut -d' ' -f1) b=$(jq -S '.changes' plan.json | sha256sum | cut -d' ' -f1) if [ "$a" != "$b" ]; then echo "::error::Gateway changed after the reviewed preview; re-run" exit 1 fi deck gateway sync kong.yaml --select-tag orders-api ``` With required reviewers and self-review prevention set on the `production` environment, `sync` waits until someone other than the person who triggered the run approves[^gh-actions]; that reviewer reads the plan, deletions first, on the run's summary page. `sync` stops unless the reviewed change set is unchanged — the hash covers the whole list, so a same-count swap of deletions stops the run too — and `concurrency` keeps one writer per target. Real estates add a staging stage and signed artefacts; `ops/kong-rules.yaml` can require `key-auth` and `rate-limiting` on every service. Pin tool tags you have tested[^oasdiff][^deck]. **Environments and secrets.** Keep one generated file and put only what must differ in reviewed per-environment values, as Microsoft's override files do for "back-end URLs, resource IDs, and secret references"[^ms-arch]. decK reads a state-file value written as `${{ env "DECK_REDIS_PASSWORD" }}` from the environment at sync time[^deck], so the pipeline injects it from its secret store and Git holds none. The resolved value still reaches the gateway, so restrict who can read the Admin API too. #### 5. Catalogue entry from the contract _Example status: YAML syntax-checked; fields from the Backstage descriptor reference[^backstage]; not loaded into Backstage._ ```yaml apiVersion: backstage.io/v1alpha1 kind: API metadata: name: orders-api spec: type: openapi lifecycle: production owner: orders-team definition: $text: ./openapi.yaml ``` `$text` embeds the contract file itself, so the catalogue shows the merged contract rather than a copy that can drift; move `lifecycle` to `deprecated` when the deprecation protocol starts. ### Hands-on labs Labs 1–3 were **not executed** here (no third-party tools were installed on the Atlas build host); Lab 4 was. Use a laptop or a disposable VM, never a shared gateway. #### Lab 1 (beginner): lint a contract against a house style **Objective.** See style rules fail and pass. **Prerequisites.** Node.js 22 or 24; the contract and ruleset from the [reference implementation](#reference-implementation) saved as `openapi.yaml` and `.spectral.yaml`. **Cost and risk.** None. ```bash npx --yes @stoplight/spectral-cli@6.17.0 lint openapi.yaml --ruleset .spectral.yaml echo "exit code: $?" ``` 1. Run it: expect no errors, possibly warnings from `spectral:oas` defaults such as missing descriptions. 2. Delete `x-sunset` from `/v1/orders/{orderId}` and run again. 3. Rename `getOrder` to `get_order` and run again. **Expected output.** Step 2 reports `deprecation-needs-sunset`, step 3 `operation-id-camel-case`, each with exit code 1 (default fail severity `error`)[^spectral-docs]. **Validation.** With `--fail-severity warn`, warnings fail the run too. **Troubleshooting.** Silent filter rule: test the JSONPath on a smaller file; engine error: Node too old. **Teardown.** Delete the folder. #### Lab 2 (intermediate): predict, then detect breaking changes **Objective.** Calibrate your judgement against your policy and a tool. **Prerequisites.** oasdiff, pinned: `go install github.com/oasdiff/oasdiff@v1.33.0` — `brew install oasdiff` installs the latest release, and v1.34.0 (2026-10-18) changes default levels for checks this lab does not use[^oasdiff]; the two-line levels file from the [contract gate](#the-contract-gate-what-blocks-a-merge) saved as `levels.txt`. Copy `openapi.yaml` to `base.yaml`, then make these edits in `head.yaml` one at a time. **First predict** each result: | Edit | Your prediction | |---|---| | A. Remove `deliveryDate` from `Order` | | | B. Add optional query parameter `expand` | | | C. Add a required query parameter `fields` | | | D. Add `cancelled` to the `status` enum | | | E. Remove `/v1/orders/{orderId}` today (sunset 2027-06-30) | | | F. Change `id` from `string` to `integer` | | ::: answer Show the reasoning for Lab 2 Under this page's policy A, C, E and F block: GitHub lists removing a response field, adding a required parameter and changing a type as breaking[^github], and E removes an operation before its sunset[^oasdiff-docs]. B is additive. D is the policy edge (GitHub: additive; Google: caution[^aip180]), so it warns. ::: ```bash oasdiff breaking base.yaml head.yaml --fail-on ERR echo "defaults, exit code: $?" oasdiff breaking base.yaml head.yaml --fail-on ERR --severity-levels levels.txt echo "policy levels, exit code: $?" ``` **Expected output** (from oasdiff 1.33.0's rule levels, not executed here[^oasdiff-docs]; the six rules involved are unchanged in v1.34.0's source, checked 2026-10-19). With defaults: errors for C, D, E and F (exit code 1) and nothing for A or B, because removing an optional response property is INFO and a new response enum value is ERR. With `levels.txt`: errors for A, C, E and F, a warning for D (exit code 0) and nothing for B. The gap between your predictions, the defaults and the policy is the lesson. **Validation.** Reverted edits give no findings. **Troubleshooting.** Nothing for E: use an RFC 3339 sunset date[^oasdiff-docs]; nothing at all: check both files parse (`oasdiff validate`). **Teardown.** None. #### Lab 3 (intermediate): gateway configuration as code with decK **Objective.** Generate Kong configuration from the contract, sync it, then detect drift. **Prerequisites.** Docker with Compose; decK 1.68.0; the contract from the reference implementation in `./spec/openapi.yaml`. decK needs a database-backed Kong, not DB-less mode[^kong-dbless]. **Cost and risk.** Local only; ports bound to 127.0.0.1 (Kong's own example binds all interfaces[^kong-docker]). _Example status: YAML and bash syntax-checked; not executed. Image tags from Docker Hub on 2026-10-18; PostgreSQL 16 is in Kong 3.16's tested range[^kong-docker]._ ```yaml services: db: image: postgres:16 environment: {POSTGRES_USER: kong, POSTGRES_DB: kong, POSTGRES_PASSWORD: kong} healthcheck: test: ["CMD", "pg_isready", "-U", "kong"] interval: 5s migrate: image: kong/kong-gateway:3.16.0.1 command: kong migrations bootstrap environment: {KONG_DATABASE: postgres, KONG_PG_HOST: db, KONG_PG_PASSWORD: kong} depends_on: {db: {condition: service_healthy}} kong: image: kong/kong-gateway:3.16.0.1 environment: KONG_DATABASE: postgres KONG_PG_HOST: db KONG_PG_PASSWORD: kong KONG_ADMIN_LISTEN: "0.0.0.0:8001" depends_on: {migrate: {condition: service_completed_successfully}} ports: ["127.0.0.1:8000:8000", "127.0.0.1:8001:8001"] orders-mock: image: stoplight/prism:5.15.10 command: mock -h 0.0.0.0 /spec/openapi.yaml volumes: ["./spec:/spec:ro"] ``` ```bash docker compose up -d && sleep 20 deck gateway ping deck file openapi2kong --spec spec/openapi.yaml --select-tag orders-api \ --output-file kong.yaml deck gateway diff kong.yaml --select-tag orders-api deck gateway sync kong.yaml --select-tag orders-api curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8000/v2/orders/42 curl -s -o /dev/null -w "%{http_code}\n" -H "apikey: lab-key" \ http://127.0.0.1:8000/v2/orders/42 # Drift: change a route by hand, then ask decK what differs (exit code 2 = drift). deck gateway diff kong.yaml --select-tag orders-api --non-zero-exit-code echo "diff exit code: $?" ``` **Expected output.** The first `diff` lists entities to create. After `sync`, the first `curl` prints `401`: the Prism mock, which builds responses from the contract, fails requests whose declared security information is missing[^prism]. With any `apikey` header it prints `200` (this lab adds no Key Auth plugin, so Kong does not check the key). **Validation.** Change a route's `paths` through the Admin API: the final `diff` exits 2[^deck-diff] and `sync` restores it. **Troubleshooting.** `405` from the Admin API: DB-less mode; connection refused: `KONG_ADMIN_LISTEN` or port bindings. **Teardown.** `docker compose down -v` removes containers and the database volume. #### Lab 4 (advanced, executed): API KPIs from a gateway log **Objective.** Compute this page's SLIs and deprecation evidence from gateway logs. **Prerequisites.** Python 3.8 or later; a JSON Lines extract with consumer, version, status, latencies and who produced the status. **Cost and risk.** Read-only; strip client IPs and tokens first. _Example status: executed on 2026-10-18 with Python 3.8.10 against a 20-line synthetic log written for this lab (revision r1); not run on real gateway logs. Percentiles checked by hand._ ```python #!/usr/bin/env python3 """API KPIs from gateway access logs in JSON Lines (standard library only). Usage: python3 api_kpis.py access.jsonl [threshold_ms] [deprecated_versions] One request per line, for example: {"consumer": "acme-mobile", "version": "v2", "route": "GET /orders/{id}", "status": 200, "latency_ms": 120, "upstream_ms": 105, "source": "upstream"} "source" names who produced the status: "gateway" (auth, limits) or "upstream". """ import json import math import sys from collections import Counter def p95(values): """Nearest-rank 95th percentile (no interpolation).""" ordered = sorted(values) return ordered[math.ceil(0.95 * len(ordered)) - 1] def load(path): with open(path, encoding="utf-8") as handle: return [json.loads(line) for line in handle if line.strip()] def main(): rows = load(sys.argv[1]) threshold = float(sys.argv[2]) if len(sys.argv) > 2 else 300.0 deprecated = set((sys.argv[3] if len(sys.argv) > 3 else "v1").split(",")) total = len(rows) if not total: sys.exit("no requests in the log") bad = sum(1 for r in rows if r["status"] >= 500) throttled = Counter(r["consumer"] for r in rows if r["status"] == 429) by_consumer = Counter(r["consumer"] for r in rows) rejected = sum(1 for r in rows if r["source"] == "gateway") # Latency only for proxied calls: fast gateway rejections would flatter it. proxied = [r for r in rows if r["source"] == "upstream"] old = Counter(r["consumer"] for r in rows if r["version"] in deprecated) print(f"requests {total}") print(f"availability SLI {(total - bad) / total:.2%} (5xx count as bad)") if proxied: fast = sum(1 for r in proxied if r["latency_ms"] <= threshold) overhead = [r["latency_ms"] - r["upstream_ms"] for r in proxied] print(f"latency SLI (<= {threshold:.0f} ms) {fast / len(proxied):.2%}" f" ({len(proxied)} proxied)") print(f"p95 latency {p95([r['latency_ms'] for r in proxied])} ms") print(f"p95 gateway overhead {p95(overhead)} ms") print(f"rejected at the gateway {rejected / total:.2%} (auth, limits)") print(f"throttled (429) {sum(throttled.values()) / total:.2%}") for consumer, count in throttled.most_common(): share = count / by_consumer[consumer] print(f" {consumer}: {count} of {by_consumer[consumer]} ({share:.0%})") print(f"deprecated-version share {sum(old.values()) / total:.2%}") for consumer, count in old.most_common(): print(f" still calling {'/'.join(sorted(deprecated))}: {consumer} ({count})") if __name__ == "__main__": main() ``` Run `python3 -I api_kpis.py access.jsonl 300 v1`. **Actual output** on the synthetic log (20 requests; 2 server errors; 2 throttled and 1 unauthenticated request rejected by the gateway; 5 requests on `v1`): ```text requests 20 availability SLI 90.00% (5xx count as bad) latency SLI (<= 300 ms) 82.35% (17 proxied) p95 latency 520 ms p95 gateway overhead 20 ms rejected at the gateway 15.00% (auth, limits) throttled (429) 10.00% partner-shipco: 2 of 5 (40%) deprecated-version share 25.00% still calling v1: legacy-erp (4) still calling v1: acme-mobile (1) ``` **Validation.** By hand: 18 of 20 requests are not 5xx (90%); 14 of 17 proxied requests take 300 ms or less (82.35%), while counting the three fast rejections would have reported 85%. **Interpretation.** The availability SLI counts 429s as good, like Google's worked example[^sre-workbook], yet one partner lost 40% of its calls: report rejections per consumer beside it. **Troubleshooting.** `KeyError`: map your gateway's field names. **Teardown.** Delete the log extract. ### Use cases and scenarios The first four are documented by the providers; the measures are this Atlas's suggestions. | Case | Context and method | Trade-off | Measure to watch | |---|---|---|---| | **Stripe: dated versions** | Version from the `Stripe-Version` header, SDK pin or account default; since `2024-09-30.acacia`, monthly releases are non-breaking and twice-yearly majors may break[^stripe-versions]; webhooks migrate via a second endpoint[^stripe-upgrades] | Provider keeps many shapes; consumers must ignore unknown fields | Traffic on current versions; failed webhook deliveries | | **GitHub: calendar versions** | `X-GitHub-Api-Version`; unversioned calls get `2022-11-28`; published breaking and additive lists[^github] | Two versions to run | Calls without a version header | | **Meta: forward, do not fail** | Each Graph API version works at least two years; calls to an unusable version go to the next oldest usable one[^meta] | No hard outage, but silent behaviour change | Calls arriving on expired versions | | **Regulated estate on Azure** (reference architecture, not a case study) | Regulated sectors named as use cases: review, dry run, one promoted commit, emergency changes extracted back[^ms-arch] | Overhead per change | Production changes traceable to a reviewed commit | | **Internal platform, 40 APIs** (illustrative) | One ruleset, a breaking gate, generated configuration, team tags | Waiver churn while legacy contracts are fixed | Blocked breaking merges; drift trending to zero | ## Measure, govern, pay ### Metrics and KPIs | KPI | Definition and formula | Data source | Pitfall | |---|---|---|---| | Availability SLI | Non-5xx requests ÷ all requests, per API, 28 days[^sre-workbook] | Gateway logs | Counts 401 and 429 as good: pair it with rejections | | Latency SLI | Proxied requests at or under a threshold ÷ proxied requests[^sre-workbook] | Gateway latency, which includes backend time[^aws-metrics] | Fast gateway rejections flatter it; averages hide the tail | | Gateway rejections per consumer | Gateway-produced 401, 403, 429 ÷ that consumer's requests | Gateway logs | Invisible in API-wide totals | | Breaking-change escape rate (Atlas KPI) | Consumer incidents caused by contract changes ÷ contract changes merged, per quarter | Incident tags, Git | Needs honest tagging | | Gate catch rate (Atlas KPI) | Breaking changes blocked ÷ (blocked + escaped) | Gate results, incidents | Read with waiver counts | | Contract drift | Entities differing between live gateways and the approved commit | Nightly diff[^deck-diff] | Covers tagged entities only | | Inventory coverage | Routes with a contract and owner ÷ all routes (API9)[^owasp-api9] | Gateway export, catalogue | Misses APIs outside the gateway | | Deprecated traffic | Requests and consumers per deprecated version | Gateway logs ([Lab 4](#lab-4-advanced-executed-api-kpis-from-a-gateway-log)) | Anonymous consumers cannot be contacted | | Time to first call | Discovery to first successful call (public APIs)[^postman-ttfc] | Portal and gateway events | "Be careful of artificially hacking a TTFC"[^postman-ttfc] | **Worked calculation (illustrative month).** An API serves 48,600,000 requests; 31,590 are 5xx. Availability = 1 − 31,590 ÷ 48,600,000 = **99.935%**. Against a 99.9% SLO the month allows 48,600 bad requests, so **65%** of the error budget is spent (see [SRE](../sre/) for budget policy). 291,600 requests (0.6%) were throttled, but one partner had **18%** of its calls throttled: a wrong limit or a client retrying without backoff. Version `v1` carried 1,652,400 requests (**3.4%**) from seven consumers, 71% from one, so one conversation decides the sunset. Over the quarter, 140 contract changes merged and 2 caused consumer incidents: escape rate **1.4%**; the gate blocked 23, so catch rate = 23 ÷ 25 = **92%**. Review the first three weekly, the rest monthly; set targets after a baseline quarter. **SLO or SLA?** An SLO is your internal target; an SLA is "an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain"[^sre-book]. Promise partners an SLA looser than the SLO you meet, measured the same way, and never present rate limits, which AWS calls targets, as guaranteed throughput. ### Quality, security, privacy and compliance - **Inventory first:** every exposed host and version with owner and data sensitivity; "Include the documentation build in your CI/CD pipeline"; protect old versions or retire them[^owasp-api9]. - **Resource limits** on payload size, page size, execution time and paid downstream calls (API4)[^owasp-api4]. - **Credentials:** keys identify, tokens authorise ([Playbook 5](#playbook-5-authentication-as-gateway-configuration)); keep secrets out of contracts and generated files, and inspect extracted artefacts for them[^ms-arch]. - **The control plane is production:** admin interfaces on a private network (decK passes credentials with `--headers`[^deck]); separate least-privilege identities for extracting and publishing; workload identity where available[^ms-arch]. - **Privacy:** synthetic data in examples and mocks; set retention and access for gateway logs, which hold consumer identifiers and IP addresses. - **Compliance evidence:** commit, approval, gate results and deployed commit form a change record auditors can sample; whether it satisfies a framework is for your compliance function ([ComplianceOps](../complianceops/)). ### Cost, capacity and sustainability - **Gateway and pipeline costs** depend on tier, units, regions and usage; Microsoft gives no fixed estimate[^ms-arch]. AWS charges extra for per-method metrics[^aws-metrics]. - **Limits are not budgets.** AWS: "Don't rely on usage plan quotas or throttling to control costs or block access to an API."[^aws-plans] Use billing alerts and provider spending limits[^owasp-api4]. - **Capacity arithmetic.** Rate 50 per second, burst 100: the token-bucket bound for one minute from a full bucket is 100 + 50 × 60 = **3,100** requests, which AWS treats as a target, not a ceiling[^aws-throttle]. With Kong's per-node `local` counters[^kong-rl], a 600-a-minute consumer behind four round-robin nodes can get up to 4 × 600 = **2,400** (illustrative). - **Version sprawl** is the hidden cost: "Running multiple versions of an API requires additional management resources"[^owasp-api9]; retiring versions is also the sustainability lever. ### Assurance of the operating model Healthy signals: zero drift findings; no human writes to production gateways except recorded emergencies; inventory coverage near 100%; no expired waivers; each deprecated version dated with falling per-consumer traffic. Monthly, trace one merged change from pull request to deployed configuration. ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Fix and prevention | |---|---|---|---| | A consumer breaks after a "compatible" release | New enum value, tighter validation, changed default (Hyrum's Law) | Diff the contracts; sample responses | Roll back or shim; document open enums | | Another team's routes vanish after a deploy | Sync without select tags | `deck gateway diff --json-output` shows deletions[^deck-diff] | [Runbook below](#runbook-a-release-deleted-another-teams-routes) | | One consumer suddenly gets many 429s | Limit keyed by IP behind a shared proxy; lowered limit; client retry storm | Plugin diff in Git; `limit_by`; per-consumer rate | Key by consumer; `Retry-After`; client backoff | | Rate limits silently stop | Counter store down with `fault_tolerant: true`[^kong-rl] | Plugin errors in gateway logs | Restore the store; alert on it; decide fail-open or fail-closed | | Live configuration differs from Git | Hand edits in a portal or Admin API | Drift job exit code 2 | Extract and merge, or re-sync; remove standing write access | | Gate blocks every pull request after a tool upgrade | New checks or levels in a new linter or diff version | Compare versions in CI logs | Pin versions; upgrade in its own pull request | ### Runbook: a release deleted another team's routes ::: figure Diagnosing and recovering from an over-broad gateway sync ```mermaid %%{init: {"flowchart": {"wrappingWidth": 160, "nodeSpacing": 16, "rankSpacing": 26, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 A["404 spike on paths owned by another team, minutes after a release"]:::risk B["Freeze: pause all gateway pipelines"]:::core C{{"Did the release log or diff report deletions?"}}:::decision D["Re-sync each affected team's last approved commit with its own tag"]:::core E["Look elsewhere: backend or DNS change, expired certificate"]:::core F["Verify: drift diff clean for every tag; 404 rate back to baseline"]:::core G["Prevent: mandatory select tags; owner-tag lint; deletion budget in the preview"]:::core A --> B --> C C -->|"yes"| D --> F --> G C -->|"no"| E ``` Text equivalent: 404s spike on another team's paths after a release. Freeze all gateway pipelines and check whether the release reported deletions. If so, re-sync each affected team's last approved commit with its own tag, verify clean drift diffs and baseline 404 rates, then add mandatory tags, an owner-tag lint rule and a deletion budget. If not, look at backend, DNS or certificates. ::: **Worked runbook (illustrative).** 14:02, the `payments` release ran `deck gateway sync payments.yaml` without `--select-tag`; decK deleted the `orders` and `catalog` routes, which were not in the file[^deck-sync]. 14:06, the 404 alert fires. 1. **Contain (5 minutes).** Pause every gateway pipeline; name an incident owner. 2. **Confirm.** The release log lists deletions; `deck gateway diff orders.yaml --select-tag orders-api` from the orders team's last approved commit lists entities to create[^deck-diff]. 3. **Recover.** Sync each affected team's last approved artefact with its own tag, not by hand in the Admin API (new drift). 4. **Verify.** Clean drift diff for every tag; 404 and 5xx rates at baseline; smoke tests pass. 5. **Prevent.** Mandatory `--select-tag` in the shared release template, an owner-tag lint rule, and the preview job's deletion budget ([release workflow](#4-pull-request-gates-preview-and-release)). 6. **Review.** Blameless review with owned actions ([IncidentOps](../incidentops/)). ### Anti-patterns and myths - **Portal-first, export later:** the contract documents the past. Git first; extract only to bootstrap. - **Contract as documentation only:** if no gate reads it, it drifts. - **The 200-rule ruleset:** hundreds of warnings teach people to ignore them. Few rules, each with a reason. - **A major version per change:** `/v7/` means six sets of tests and docs. Prefer additive change and deprecation. - **Deprecation by email:** without headers, dates and per-consumer traffic, sunsets slip forever. - **Business logic in the gateway:** transformations hide behaviour outside the contract and the service's tests. - **Quotas as budgets, keys as authentication** (AWS warns against both[^aws-plans]). - **Myth: design-first is waterfall.** A contract can be written in an hour, mocked, and changed many times before code exists. - **Myth: GraphQL needs no governance.** It avoids versions[^graphql] but still needs deprecation, usage tracking and limits. ## Deciding and adopting ### When to use and when not to use **Use it** when several teams share gateways, when consumers you do not control (partners, customers, mobile apps) depend on your contracts, when changes are frequent, or when you must show who approved each production change. Microsoft frames the decision around ownership, review requirements and deployment boundaries, which "don't have fixed thresholds"[^ms-arch]. **Do not build the full pipeline** for one internal API with two known consumers that changes twice a year: a contract in Git, a linter and a reviewed pull request suffice; Microsoft advises small, slow-changing estates to "begin with a manual pull request workflow"[^ms-arch]. Skip gateway-as-code without a gateway, and breaking gates for drafts nobody calls. If infrastructure and API configuration always change together, one IaC pipeline may serve both[^ms-arch]. **Hidden responsibility:** someone owns the ruleset, levels file, waivers and consumer register indefinitely. ### Decision framework ::: figure How much API Ops does this API estate need? ```mermaid %%{init: {"flowchart": {"wrappingWidth": 150, "nodeSpacing": 16, "rankSpacing": 26, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 Q1{{"Do consumers outside the owning team depend on the API?"}}:::decision N1["Contract in Git plus lint in CI; revisit yearly"]:::neutral QG{{"Is there an API gateway in front of it?"}}:::decision Q2{{"External consumers, several teams on one gateway, or weekly changes?"}}:::decision L2["Breaking-change gate, deprecation dates, per-consumer telemetry"]:::core L3["Full pipeline: generated config, scoped sync, drift checks, sole writer"]:::core Q1 -->|"no"| N1 Q1 -->|"yes"| QG QG -->|"no"| L2 QG -->|"yes"| Q2 Q2 -->|"no"| L2 Q2 -->|"yes"| L3 ``` Text equivalent: If no one outside the owning team depends on the API, keep the contract in Git with linting in CI and revisit yearly. If others depend on it but no gateway fronts it, add a breaking-change gate, deprecation dates and per-consumer telemetry. With a gateway, adopt the full pipeline (generated configuration, scoped sync, drift checks, the pipeline as sole writer) when consumers sit outside your organisation, several teams share the gateway or it changes weekly; otherwise take the lighter option. ::: ### Maturity model This is **this Atlas's model**, not an industry standard. | Level | Observable behaviour | Exit criterion | |---|---|---| | 1. Ad hoc | APIs configured in portals; specs exported, if at all | Every gateway API has a contract in Git | | 2. Described | Contracts in Git; documentation generated in CI | Ruleset runs on every pull request | | 3. Gated | Lint and breaking diff block merges; waivers are files | Zero ungated contract changes for a quarter | | 4. Reconciled | Pipeline-only, team-scoped gateway sync; nightly drift check | No unrecorded human writes to production | | 5. Evidence-led | Consumer register; deprecation by traffic data; API SLOs and scorecards | Every deprecated version dated, consumer count falling | ### Adoption roadmap: 30/60/90 days | Window | Actions | Milestone | |---|---|---| | Days 1–30 | Inventory routes and owners; two pilot contracts in Git; 10-rule ruleset in warn mode; versioning ADR | Coverage measured; pilots linted | | Days 31–60 | Rules to error; breaking gate; generated configuration and team tags for pilots; nightly drift job | A break caught in review; drift visible | | Days 61–90 | Pipeline is sole production writer for pilots; per-consumer dashboards; deprecation calendar; next five APIs | 30 days without manual production edits | ## Toolkit ### Checklists **Preflight for a new API** - [ ] Contract in Git, design reviewed, linted clean - [ ] Versioning scheme and compatibility policy referenced - [ ] Owner, consumers, stability level and SLO recorded - [ ] Error format RFC 9457; size and page limits set - [ ] Authentication scheme, scopes and rate limits reviewed **Pull-request review** - [ ] Gates green; any waiver has an owner and expiry - [ ] Warnings acknowledged by a named reviewer - [ ] Deprecations carry a sunset date and migration note - [ ] Gateway policy diff read, especially deletions **Deprecation and sunset** - [ ] Successor live and documented - [ ] Deprecation and Sunset headers configured; `rel="deprecation"` link - [ ] Every remaining consumer identified and contacted - [ ] Post-sunset behaviour (error or forwarding) tested **Quarterly audit** - [ ] Inventory reconciled: gateway, contracts, catalogue - [ ] Drift findings and manual-write audit reviewed - [ ] Ruleset and tool versions reviewed; expired waivers removed ### Templates **Pull-request template for an API change** ```text API: Stability: draft | beta | stable Change type: additive | deprecating | breaking (needs version + approval) Consumers affected: Evidence: Gate results: lint breaking diff tests Waivers: or none Gateway policy changes: Rollback: ``` **Deprecation notice** ```text Subject: deprecated on , sunset on What changes: is replaced by . Migration guide: Why: Your traffic: made calls to the deprecated element in the last 30 days Signals: responses carry Deprecation and Sunset headers from After sunset: requests return | are forwarded to Contact: ; request an extension before ``` **Architecture decision record: versioning scheme** ```text Context: consumers (internal/partner/public), change rate, gateway capabilities Options: URL major version | dated header | media type | versionless + deprecation Decision: