---
title: "AppOps"
slug: appops
category: delivery
term_status: ambiguous
evidence_status: verified
summary: "AppOps (application operations) is how the team that owns a service runs it in production: health checks, timeouts, retries, circuit breakers, graceful degradation, emergency levers, runbooks and on-call. The label is ambiguous; this page teaches operability built into the code."
audience: [Application developers on call, Tech leads and service owners, Architects, Engineering managers, Beginners]
last_verified: 2026-10-19
key_takeaways:
  - "AppOps is an ambiguous label: enterprise application management (closest to ITIL's function, though ITIL does not use the word), vendor products and pitches (Citrix, Shipa, Prodly) and Android's App Ops permission framework. This page teaches one reading: operability built into a service by the team that owns it."
  - "Give every remote call a deadline. Common clients do not time out by default: gRPC sets no deadline, and Python Requests can hang for minutes or more unless told otherwise."
  - "Retry at one layer only, with capped exponential backoff, jitter and a budget. Four attempts at each of three layers turn one failing database call into 64 (Lab 2); one retry layer plus a 10% budget holds it near 1.1. Count the retries your SDK and mesh already make."
  - "Liveness is about the process, readiness is about traffic: a database check in a liveness probe can restart every replica when the database is slow. On shutdown, keep serving until routers stop sending traffic, then stop listening and drain."
  - "Degraded modes are code that rarely runs. Google warns that such paths often do not work; Amazon avoids fallback because it is hard to prove and test: keep them simple and exercise them on purpose."
  - "Attach the owner's runbook to every alert that can page; Google reports roughly a 3x improvement in mean time to repair with a playbook."
  - "\"You build it, you run it\" works when on-call funding and accountability sit with the owning team and out-of-hours cover matches what downtime costs."
related: [sre, observabilityops, incidentops, releaseops, serviceops, kubernetesops, devops, webops]
---

## What AppOps is

### In 30 seconds

**AppOps** (application operations) is the work of keeping one application or service healthy in production, done by the people
who own its code. In this Atlas it means **operability built into the code**: the service reports honestly whether it can take
traffic, bounds every wait on another system, retries only when retrying helps, degrades instead of collapsing, exposes levers for
the on-call engineer, and ships with a runbook its owners keep current. Werner Vogels described the ownership model in 2006: "You
build it, you run it. This brings developers into contact with the day-to-day operation of their software."[^vogels2006]

::: note Honest term status: ambiguous
"AppOps" has no standard definition, governing body or certification, and it names unrelated things:
1. **Application management** in enterprise IT. ITIL's 2011 glossary defines it as "The function responsible for managing
   applications throughout their lifecycle".[^itilglossary] ITIL does not say "AppOps"; equating the two is this Atlas's reading.
   ITIL 4 (2019) replaced processes with practices,[^itpmitil4] and PeopleCert now also offers ITIL (Version 5)
   (as of October 2026).[^peoplecertitil]
2. **Vendor categories.** DevOps.com (2016, quoting a Citrix executive) presented AppOps as agility for existing IT services and
   asked whether it was just a buzzword;[^devopscom2016] a 2021 Shipa-sponsored post proposed "AppOps as a New
   Category";[^tns2021] Prodly sold an "AppOps" product for Salesforce administrators and now markets Salesforce
   DevOps.[^prodly2021][^prodlyhome]
3. **Android App Ops**: "App-ops are used for two purposes: Access control and tracking" (`AppOpsManager`, API level
   19),[^androidappops] first a hidden permissions screen in 2013.[^commonsware2013]

We found no literature that treats "AppOps" as a discipline in its own right (October 2026).
:::

::: note Editorial distinction used on this page
This page teaches meaning 1 as product teams practise it. Its unique scope is **resilience patterns in application code and the
owner's runbook**; its boundaries with neighbouring pages ([table](#how-appops-relates-to-other-ops)) are the Atlas's, not an
industry standard.
:::

**Where to start.** Beginners: [glossary](#glossary-and-mental-model), [Lab 1](#lab-1-beginner-health-endpoints-that-tell-the-truth);
engineers: [core practices](#core-practices-and-playbooks); leaders: [decision framework](#decision-framework).
**Last verified** 2026-10-19.

### At a glance

| Aspect | Summary |
|---|---|
| Term status | Ambiguous (see the note above); this page's reading is an editorial choice |
| Goal | Services that keep their promises when dependencies misbehave, and that owners can diagnose and steer under pressure |
| People | Developers who carry a pager, tech leads, service owners; application support teams |
| In scope | Health signals, timeouts, retries, idempotency, breakers, bulkheads, shedding, degradation, graceful shutdown, config and secrets, levers, runbooks, app-tier capacity, on-call |
| Out of scope | SLOs → [SRE](../sre/); telemetry pipelines → [ObservabilityOps](../observabilityops/); incident roles → [IncidentOps](../incidentops/); CDN, TLS, DNS → [WebOps](../webops/); release flags → [ReleaseOps](../releaseops/) |
| Key outputs | A dependency register with policies; truthful health endpoints; tested levers; runbooks linked from alerts; sustainable on-call |

### The problem it solves

The SRE book defines a **cascading failure** as "a failure that grows over time as a result of positive feedback".[^srecascade]
Application code supplies much of that feedback: calls without timeouts empty thread pools, retries pile load on a drowning
dependency, and a liveness check on a slow database restarts every replica. Marc Brooker of Amazon: "Retries are "selfish.""
They spend the server's time on one client's chances, harmful during overload.[^bltimeouts]

| When a dependency… | Without operability in the code | With it |
|---|---|---|
| hangs | Threads wait on default timeouts; healthy routes fail too | Deadlines cut waits short; a bulkhead caps the damage |
| is overloaded | Retries at every layer multiply load | One retry point, a budget, a breaker that stops calling |
| is fine but a deploy runs | Requests reach a stopping instance | It serves until routers stop sending, then stops listening and drains |

### What AppOps is not

- **Not Android App Ops** (mobile delivery is in [MobileOps](../mobileops/)), and **not a product category**: libraries and meshes
  implement patterns that someone must choose, tune and test.
- **Not SRE under another name.** SRE sets targets, budgets and on-call policy; AppOps is what the code and its owners do to meet
  them. **Not monitoring:** telemetry shows that a dependency is slow; it does not stop your threads waiting on it.
- **Not an unconditional pager for developers.** Who answers at night is an operating-model decision with costs
  ([operating model](#operating-model-and-governance)).

## Where AppOps comes from and where it sits

### Origins and evolution of the label and the practices

2006: Vogels states the ownership model behind this page, "You build it, you run it".[^vogels2006] 2014: Fowler credits Nygard's
*Release It!* with popularising the circuit breaker.[^fowlercb] 2016: Google's SRE book on cascading failures and
overload.[^srecascade][^sreoverload] 2019: Amazon Builders' Library on timeouts, health checks, shedding and
fallback.[^bltimeouts][^blhealth]

### Glossary and mental model

| Term | Meaning on this page |
|---|---|
| **Hard / soft dependency** | Without a hard dependency the request fails; without a soft one it succeeds in reduced form |
| **Timeout, deadline** | A bound on one attempt; the time by which the whole request must finish, passed down the chain |
| **Backoff, jitter, retry budget** | Waiting longer after each failure; randomising the wait; capping retries as a share of requests |
| **Idempotent** | Repeating has the same effect as doing it once (an idempotency key makes a write so) |
| **Circuit breaker, bulkhead** | Stops calling a failing dependency for a while; a bounded pool per dependency |
| **Liveness, readiness** | *Is the process working?* (restart if not); *should it get traffic now?* (remove from rotation if not) |

**Mental model: a remote call has three outcomes, not two.** It succeeds, fails, or *does not answer*. The patterns handle the
third: how long to wait (deadline), whether to try again (retry policy), whether to keep trying (breaker), how much of the service
may wait (bulkhead), and what the user gets meanwhile (degradation).

::: figure Where each resilience control acts on one request
```mermaid
flowchart TB
  IN["Request arrives with a deadline"] --> ADM{"Over capacity?"}
  ADM -- "yes" --> SHED["Shed: fast 503 with Retry-After"]
  ADM -- "no" --> CB{"Breaker closed?"}
  CB -- "no" --> DEG["Degrade or fail fast"]
  CB -- "yes" --> BH{"Bulkhead slot free?"}
  BH -- "no" --> DEG
  BH -- "yes" --> CALL["Call with the remaining deadline"]
  CALL -- "success" --> OK["Respond"]
  CALL -- "error or timeout" --> RT{"Safe and allowed to retry?"}
  RT -- "yes" --> BO["Back off with jitter"]
  RT -- "no" --> GIVE["Give up: degrade or fail"]
  BO -- "retry" --> CB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class IN,CALL,OK,BO core
  class SHED,DEG,GIVE risk
  class ADM,BH,CB,RT decision
```
Text equivalent: a request arrives with a deadline. Over capacity, it is shed with a fast 503 and Retry-After. Otherwise it needs a
closed breaker, then a free bulkhead slot (Resilience4j's default order[^r4jspring]), or it degrades or fails fast. After an error
or timeout it retries only if the operation is idempotent and budget and deadline allow, after a jittered backoff that passes
breaker and bulkhead again; otherwise it degrades or fails.
:::

### How AppOps relates to other Ops

Each line is an **editorial distinction**, chosen so that each topic is taught once in the Atlas.

| Neighbour | What AppOps owns | Go there for |
|---|---|---|
| [SRE](../sre/) | Patterns in the code that meet the targets | SLIs, SLOs, error budgets, on-call policy |
| [ObservabilityOps](../observabilityops/) | Signals about the service's own decisions | Pipelines, storage, cardinality, telemetry budgets |
| [IncidentOps](../incidentops/) | The service runbook and its levers | Incident roles, communications, MTTA and MTTR |
| [ReleaseOps](../releaseops/) | Operational kill switches | Release flags, progressive delivery, canaries |
| [KubernetesOps](../kubernetesops/) | What health endpoints check; app-tier concurrency | Probe configuration, autoscalers |
| [ServiceOps](../serviceops/) | How the owning team operates | Service record, catalogue, tiers, SLAs |

::: figure AppOps between the pages that feed it and the pages it hands work to
```mermaid
flowchart LR
  subgraph IN["Feeds AppOps"]
    SRE["SRE: targets, budgets"]
    SVC["ServiceOps: owner, tier"]
    PLAT["PlatformOps: defaults"]
  end
  APP["AppOps: resilience in code, levers, runbook"]
  subgraph OUT["AppOps hands work to"]
    OBS["ObservabilityOps: telemetry"]
    INC["IncidentOps: coordination"]
    REL["ReleaseOps: exposure"]
    K8S["KubernetesOps: probes, scaling"]
  end
  SRE --> APP
  SVC --> APP
  PLAT --> APP
  APP --> OBS
  APP --> INC
  APP --> REL
  APP --> K8S
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class APP core
  class SRE,SVC,PLAT,OBS,INC,REL,K8S neutral
```
Text equivalent: SRE, ServiceOps and PlatformOps feed AppOps, which builds resilience into the code and keeps levers and the
runbook; it hands work to ObservabilityOps, IncidentOps, ReleaseOps and KubernetesOps.
:::

## Principles and operating model

### A proposed charter and principles

**This Atlas's proposal**; no AppOps manifesto exists. **Mission:** every service keeps its promises when its dependencies do not,
and whoever is on call can steer it without reading the code.

1. **Bound every wait:** a deadline per request; per-attempt timeouts from measured latency, not library defaults.
2. **Retry in one place, only what is safe,** with backoff, jitter and a budget.
3. **Fail fast and keep failure local** with breakers and bulkheads.
4. **Health signals tell the truth to their audience:** liveness to the restarter, readiness to the load balancer.
5. **Prefer the path that always runs:** degraded modes stay simple and are exercised on purpose.
6. **No page without a runbook and a tested lever.** Google reports that a written playbook gives "roughly a 3x improvement in
   MTTR".[^sreintro]
7. **Owners operate, within a load they can sustain,** funded and sized by the organisation.

**Non-goals:** SLO targets, the telemetry platform and major-incident command belong to [SRE](../sre/),
[ObservabilityOps](../observabilityops/) and [IncidentOps](../incidentops/).

### Building blocks of an operable service

Five building blocks carry the discipline: a dependency register with policies (Artefact 1), the same policies in library or mesh
configuration (Artefact 3), a truthful health model and lifecycle (Lab 1, Artefact 4), diagnosability signals for the code's
decisions (playbook and metrics below), and alerts that link the runbook (Artefact 5).

### Roles, ownership and RACI

R = responsible, A = accountable, C = consulted, I = informed — a proposal to adapt.

| Activity | Owning team | Service owner | Platform team | SRE partner | Dependency owner |
|---|---|---|---|---|---|
| Dependency register and policies | R | A | C (defaults) | C | C (limits, idempotency) |
| Health endpoints and drain | R | A | C (probes) | I | – |
| Levers, tested | R | A | C | C | I |
| Runbook and alert rules | R | A | C (tooling) | C (SLO alerts) | C |
| Answer pages | R | A | I | C (escalation) | C |
| Fund and size on-call | C | R, A | – | C | – |

The **service owner** is accountable: funding and the feature-versus-operability trade-off are theirs. When an operations manager
stays accountable while product teams carry the pager, Equal Experts calls it "responsible but unaccountable", with inflated
availability targets and operability losing to features.[^eepitfalls]

### Lifecycle: Day 0, Day 1, Day 2

| Phase | AppOps work | Exit evidence |
|---|---|---|
| **Day 0: design** | Dependencies, hard or soft; deadline and timeouts from latency; retry points; degraded responses and levers | Register reviewed; policy check passes |
| **Day 1: launch** | Health endpoints and drain; alerts with runbook links; tested levers; a drill per hard dependency; readiness review | Review signed off; rota published |
| **Day 2: operate** | Watch retries, breaker events, degraded responses, incidents per shift; fix runbooks after incidents; prune levers | Monthly operability review |

At Google, "A PRR is considered a prerequisite for an SRE team to accept responsibility for managing the production aspects of a
service";[^sreprr] AWS's Operational Readiness Review distils "the learnings from AWS operational incidents into curated questions
with best practice guidance".[^awsorr] Both work as question banks even when no SRE team takes over.

### Operating model and governance

Three models differ in who answers when the service breaks, not in whether the patterns are needed.

| Model | Who runs it | Fits | Watch for |
|---|---|---|---|
| **Central application operations** (ITIL) | An application support team | Packaged or rarely changed software | Hand-offs slow fixes |
| **You build it, you run it** | The product team, including on-call | Custom services that change often | On-call cost and fatigue |
| **SRE-supported** | The team, plus SRE for the most critical services | Risk that justifies specialists | Entry and hand-back rules |

Equal Experts' playbook keeps operations teams for "self-hosted foundational systems, such as COTS applications" (commercial
off-the-shelf),[^eeybiyri] warns of **linear run costs** (standby cost *R(n)*, where *n* is the number of teams on call) and advises
"Fund on-call costs from product team budgets", choosing out-of-hours cover by financial exposure (95.0% no night rota, 99.0% a
domain rota, 99.9% the team's own).[^eepitfalls][^eescale]

Sustainable load is arithmetic: Google caps on-call at 25% of an SRE's time, needs eight engineers for a single-site 24/7 rota,
and sets two incidents per 12-hour shift as the upper bound.[^sreoncall] A team of five cannot run 24/7 on those terms.
**Governance:** the register check runs in CI ([Artefact 2](#artefact-2-a-policy-check-in-ci)), readiness reviews precede launches,
a monthly operability review runs, and every lever has an owner and a test date.

## Architecture and workflow

### Reference architecture

::: figure Reference architecture of an operable service
```mermaid
flowchart TB
  LB["Load balancer or orchestrator"]
  subgraph SVC["checkout-api process"]
    ADM["Admission and shedding"]
    HE["Health: /livez, /readyz"]
    H["Handlers set a deadline"]
    LEV["Levers: flags, config"]
    OUT["Outbound clients: timeout, retry, breaker, bulkhead"]
  end
  PAY["payments (hard)"]
  INV["inventory (hard)"]
  REC["recommendations (soft)"]
  LB --> ADM
  LB -. "probes" .-> HE
  ADM --> H
  H --> OUT
  LEV -.-> OUT
  OUT --> PAY
  OUT --> INV
  OUT --> REC
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class ADM,H,OUT,HE core
  class LEV decision
  class LB,PAY,INV,REC neutral
```
Text equivalent: a load balancer or orchestrator sends traffic to checkout-api and probes its health endpoints. Inside, admission
control sheds excess load, handlers set a deadline, and one outbound client layer applies timeouts, retries, breakers and bulkheads
to two hard dependencies (payments, inventory) and one soft one (recommendations); levers change that layer's behaviour.
:::

| Component | Responsibility | Trade-off |
|---|---|---|
| Admission | Refuse work that cannot finish in time | Static limits go stale; adaptive ones are harder to reason about |
| Handlers | Set the deadline; stop work once the caller has timed out[^blshedding] | Needs deadline-aware code |
| Outbound client layer | One place for every resilience policy | Library knows semantics; a mesh needs no code ([tooling](#tooling-landscape)) |
| Health endpoints | Truthful signals | More checks catch more faults and risk fleet-wide false alarms |
| Levers | Change behaviour without a deploy | Each lever is code to test and access-control |


### End-to-end workflow: a soft dependency browns out

::: figure Brownout of a soft dependency, from first timeout to recovery
```mermaid
sequenceDiagram
  participant U as User
  participant A as checkout-api
  participant R as recommendations
  U->>A: GET /checkout
  A->>R: call, 200 ms timeout
  Note over R: stops answering
  A-->>U: page without the panel
  Note over A: breaker opens
  U->>A: GET /checkout
  A-->>U: reduced page, no call
  A->>R: 3 trial calls after 5 s
  Note over A: a trial times out: open again
  A->>R: rounds every 5 s keep failing
  Note over A: not closed 5 min after open: ticket
  A->>R: 3 trial calls, later
  R-->>A: trials succeed
  Note over A: breaker closes
```
Text equivalent: checkout-api calls recommendations with a 200 ms timeout; when it stops answering, the page is served without the
panel. Once half of the last 20 calls fail the breaker opens and later requests get the reduced page without a call. Trial calls
every five seconds time out while the dependency hangs, so the breaker opens again; five minutes after it first opened, still not
closed, a ticket goes to the on-call engineer. A later round of trials succeeds and the breaker closes.
:::

**Exception path: a hard dependency fails.** Without payments, checkout fails fast with a 503 and Retry-After
("how long the user agent ought to wait before making a follow-up request"),[^rfc9110] stops retrying once the breaker opens, and
pages. If the cause is outside the team, [IncidentOps](../incidentops/) roles coordinate.

## Practices, frameworks and tools

### Core practices and playbooks

#### Playbook: deadlines and timeouts

1. **Find the defaults you live with.** "By default, gRPC does not set a deadline",[^grpcdeadlines] and Python Requests does "not
   time out unless a timeout value is set explicitly".[^requests] AWS names "Not being aware of system timeouts or default
   timeouts" an anti-pattern.[^warel05bp05]
2. **Derive timeouts from latency.** Amazon picks an acceptable false-timeout rate (say 0.1%) and uses the matching downstream
   percentile (p99.9), padding when p99.9 is close to p50.[^bltimeouts]
3. **Pass one deadline down.** In the SRE book's example a 30-second deadline leaves 23 seconds for the next hop after 7 seconds of
   work.[^srecascade] gRPC propagates deadlines by default in Java and Go, not in C++.[^grpcdeadlines]
4. **Check the sum** of sequential timeouts times attempts against the deadline ([Artefact 2](#artefact-2-a-policy-check-in-ci)).

#### Playbook: retries that help instead of hurt

1. **One layer.** Three layers of four attempts produce "64 attempts (4^3) on the database" per user action;[^srecascade] Amazon's
   five-deep stack reaches 243 times the load, and it retries "at a single point in the stack".[^bltimeouts]
2. **Count retries you did not write** (as of October 2026). AWS SDKs in standard mode make three attempts (DynamoDB four), but
   Java, Python, Ruby, PHP, C++ and the CLI still default to legacy mode until the switch announced for November
   2026;[^awssdkretry][^awsretry2026] Boto3's legacy default is five.[^boto3retries] .NET's standard handler retries three times
   by default;[^dotnetresilience] Istio's mesh-wide default retries twice.[^istiomesh] Pin what you rely on; a loop around any of
   them is a second layer.
3. **Only what is safe to repeat.** Side-effecting APIs "aren't safe to retry unless they provide idempotency";[^bltimeouts]
   idempotency keys let a client "safely repeat the request without risk of creating a second object or performing the update
   twice".[^stripeidem]
4. **Backoff with jitter:** "Always use randomized exponential backoff when scheduling retries";[^srecascade] "Full Jitter" sleeps
   `random_between(0, min(cap, base * 2 ** attempt))`.[^brookerjitter]
5. **A budget.** Google caps each request at three attempts and retries per client only while retries stay below 10% of requests.
   In the book's worst-case example the per-request cap alone allows almost 3x growth; the budget brings it to "just 1.1x in the
   general case".[^sreoverload]
6. **Not what will not change** (permission, configuration errors);[^warel05bp03] honour 429[^rfc6585] and 503 with
   Retry-After.[^rfc9110]

#### Playbook: circuit breakers and bulkheads

Above its failure threshold a breaker trips and all further calls fail at once, without the protected call being made.[^fowlercb]
Its states are Closed (calls flow, failures counted), Open (calls fail at once) and Half-Open (a few trial calls decide).[^azurecb]

- **Tune for your traffic.** Resilience4j's defaults keep the last 100 calls, open at a 50% failure rate and wait 60 seconds
  (October 2026).[^r4jcb] Time to open is failures needed ÷ failing calls per second: a full window opens after 50 failures (25 s
  at two calls a second); a fresh one needs 100 calls. Count slow calls if slowness is the risk.
- **Decide what open means:** degrade for a soft dependency, fail fast with a clear error for a hard one.
- **Keep breaker state out of liveness.** Resilience4j's Spring Boot health indicators are off by default "because the application
  status is DOWN, when a CircuitBreaker is OPEN".[^r4jspring]

**Bulkheads**, named after a ship's hull partitions, give each dependency a bounded pool.[^azurebulkhead] Size them with Little's
law (in-flight = arrival rate × time in system):[^little] 50 calls a second at 100 ms need about 5 slots, so a limit of 20 leaves
headroom and caps a hang at 20 stuck calls.

#### Playbook: load shedding and graceful degradation

Load shedding "drops some proportion of load by dropping traffic as the server approaches overload conditions"; graceful degradation
"takes the concept of load shedding one step further by reducing the amount of work that needs to be performed".[^srecascade]

1. Shed at admission with a cheap 503 and Retry-After, least important work first (Google's criticality levels run from
   `CRITICAL_PLUS` to `SHEDDABLE`).[^sreoverload]
2. Write and test the degraded response for each soft dependency; AWS calls skipping that test an anti-pattern.[^warel05bp01]
3. Keep it rare and simple: "the code path you never use is the code path that (often) doesn't work".[^srecascade]

#### Playbook: health endpoints, startup and shutdown

Google separates process health ("is this binary responding at all?") from service health (able to respond to this class of
requests right now?).[^srecascade] Kubernetes liveness probes "determine when to restart a container", readiness probes
"determine when a container is ready to accept traffic", and wrong liveness probes "can lead to cascading failures".[^k8sprobes]

- **Liveness is process-local:** it "should not depend on health checks for external systems" (Spring Boot), or a database outage can
  restart every instance;[^springprobes] Microsoft agrees.[^azurehealth]
- **Readiness for a shared dependency is a judgement call** (Spring Boot's words): include it and all instances leave rotation
  together; leave it out and handle failure with a breaker.[^springprobes] Some load balancers **fail open** when every server
  fails;[^blhealth] a Kubernetes Service does not — with no ready pod it refuses connections, with no 503 at all.[^springprobes]
- **Beware the fast-failing server:** under "least requests" balancing it attracts traffic and becomes a "black hole".[^blhealth]
- **Start unready; stop in three steps:** keep serving while routers stop sending traffic, stop listening, then finish in-flight
  work and exit,[^12fdisposability] within the grace period (30 s by default in Kubernetes).[^k8spodlifecycle] Kubernetes removes
  a terminating pod from Service endpoints as shutdown begins, whatever readiness says, but traffic can still arrive for a while:
  add a `preStop` sleep before SIGTERM, long enough for routing to stop **and** for the longest in-flight request, and a grace
  period covering sleep plus drain (Spring Boot's guide uses the built-in `sleep` action from Kubernetes 1.32).[^springcloud]
  Probe settings: [KubernetesOps](../kubernetesops/).

#### Playbook: emergency levers

Levers "are rapid processes that can mitigate availability impact on your workload" by "disabling, throttling, or changing the
behavior of components or dependencies using known and tested mechanisms" (AWS).[^warel05bp07] Hodgson calls the long-lived ones
"Kill Switches".[^hodgsontoggles] Typical levers switch off a soft dependency, force a breaker open, lower a rate limit or cut
retries to zero; each needs an owner, an access rule, a runbook entry and a test date. Flag systems: [ReleaseOps](../releaseops/).

#### Playbook: make the service diagnosable

When prevention fails, owners must explain what the code decided. Zero-code instrumentation sees "the edges of your application";
decisions need code-based instrumentation.[^otelinstr] Pipelines and storage belong to [ObservabilityOps](../observabilityops/).

1. **Propagate trace context through every call, retries included.** Each resent request is its own client span carrying
   `http.request.resend_count`,[^otelhttpspans] so one trace shows a retry storm's fan-out.
2. **Record decisions, not just outcomes.** A breaker transition, a rejected or shed call and a degraded response each become a
   span event plus a log line with fixed fields (dependency, decision, reason, attempt); Resilience4j publishes state-transition
   events.[^r4jcb]
3. **Put the trace ID in every log line;** an active OpenTelemetry SDK correlates existing logs with the current trace and
   span.[^otellogs]
4. **Link metrics to traces.** An exemplar attaches trace context to a metric observation,[^otelexemplars] so a latency spike
   opens an example trace.
5. **Plan a debug lever.** Raising one logger's level at runtime (Spring Boot's `/actuator/loggers`)[^springloggers] beats a
   redeploy; time-box it, restrict access and keep personal data out ([PrivacyOps](../privacyops/)).

_Example status: JSON syntax-checked; the field names are this Atlas's convention._

```json
{"time": "2026-10-18T03:12:07.412Z", "level": "WARN", "service": "checkout-api",
 "event": "dependency.degraded", "dependency": "recommendations",
 "decision": "skip_call", "reason": "breaker_open", "attempt": 0,
 "route": "GET /checkout", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"}
```

### Methods, standards and frameworks

Only the RFCs and the OpenTelemetry semantic conventions are specifications; the rest are recommendations to adapt.

| Source | Kind | What it gives AppOps |
|---|---|---|
| RFC 9110[^rfc9110]; RFC 6585[^rfc6585] | IETF standards | 503 and Retry-After; 429 Too Many Requests |
| OpenTelemetry semantic conventions[^otelhttp] | Specification | Names such as `http.server.request.duration` |
| *Site Reliability Engineering* (Google, 2016)[^srecascade][^sreoverload] | Practitioner book | Cascading failures, overload, budgets, criticality |
| AWS Well-Architected REL05, OPS07, ORR[^warel05bp01][^waops07][^awsorr] | Vendor framework | Degradation, retries, timeouts, levers, runbooks |
| Azure Architecture Center patterns[^azurecb][^azurebulkhead][^azurehealth] | Vendor patterns | Breaker, bulkhead, health endpoints |
| The Twelve-Factor App[^12fconfig][^12fdisposability] | Methodology | Config outside code, disposability |

### Tooling landscape

Choose the category first; brands change faster than patterns. As of October 2026:

| Category | Verified examples | Strengths | Limits |
|---|---|---|---|
| In-process library | Resilience4j (Java);[^r4jcb] .NET `Microsoft.Extensions.Http.Resilience`[^dotnetresilience] | Knows idempotency, deadlines, degraded responses | One per language; defaults need tuning |
| SDK and framework built-ins | AWS SDK retry modes;[^awssdkretry] gRPC deadlines[^grpcdeadlines] | Already there | Hidden retry layers; defaults differ |
| Proxy or mesh | Envoy circuit breaking and retry budgets;[^envoycb] Istio, which retries by default[^istiomesh] | Language-neutral, no code change | Blind to business meaning |
| Instrumentation | OpenTelemetry, code-based or zero-code[^otelinstr] | Vendor-neutral | Breaker and degradation events need code |
| APM back ends | Many commercial back ends accept OpenTelemetry natively[^otelvendors] | Traces, errors, dashboards | Cost grows with volume ([ObservabilityOps](../observabilityops/)) |

Envoy "enforces circuit breaking limits at the network level as opposed to having to configure and code each application
independently",[^envoycb] but cannot know which call is safe to repeat; with both layers, make sure only one retries.

### Automation and AI

Automation makes the policies dependable: the check runs in CI, and budgets, breakers and adaptive throttling act without a
human.[^sreoverload] Repeated runbook steps can be automated with [AutomationOps](../automationops/)'s safeguards. **This page's
view on AI:** models can draft runbook entries and summarise traces, but production levers stay with a human or a tested rule.
Event correlation: [AIOps](../aiops/).

## AppOps in practice

### Reference implementation

A fictional service, `checkout-api`, with placeholder names and URLs. The register and its check need only Python; the others
assume Kubernetes, Prometheus and Spring Boot 4.1 with Resilience4j 2.4 (October 2026: open-source support for Spring Boot 3.5
ended on 2026-06-30, and Resilience4j added a Spring Boot 4 module in 2.4.0).[^springsupport][^r4jrelease] Non-JVM stacks map the
same policies across: .NET's resilience pipelines appear in the [tooling table](#tooling-landscape), and the register and its
checker (Artefacts 1–2) are language-neutral.

#### Artefact 1: the dependency register

One record per outbound dependency, reviewed with the code. `sequential` marks calls made one after another on the request path;
`sdk_attempts` and `mesh_attempts` record attempts a client library or sidecar already makes (the AWS SDK pinned to standard
mode, mesh retries off).

_Example status: JSON syntax-checked; input of Artefact 2, which was executed._

```json
{
  "service": "checkout-api",
  "owner": "team-checkout",
  "inbound_deadline_ms": 2000,
  "dependencies": [
    {
      "name": "payments", "criticality": "hard", "sequential": true,
      "timeout_ms": 800, "max_attempts": 2, "sdk_attempts": 1,
      "mesh_attempts": 1, "idempotent": false, "idempotency_key": true,
      "breaker": true, "runbook": "runbooks/checkout.md#payments"
    },
    {
      "name": "inventory", "criticality": "hard", "sequential": true,
      "timeout_ms": 150, "max_attempts": 3, "sdk_attempts": 1,
      "mesh_attempts": 1, "idempotent": true, "breaker": true,
      "runbook": "runbooks/checkout.md#inventory"
    },
    {
      "name": "recommendations", "criticality": "soft", "sequential": false,
      "timeout_ms": 200, "max_attempts": 1, "sdk_attempts": 1,
      "mesh_attempts": 1, "idempotent": true, "breaker": true,
      "degraded_behaviour": "render the page without the panel",
      "runbook": "runbooks/checkout.md#recommendations"
    },
    {
      "name": "receipt-store", "criticality": "soft", "sequential": false,
      "timeout_ms": 1000, "max_attempts": 2, "sdk_attempts": 3,
      "mesh_attempts": 1, "idempotent": true, "breaker": false,
      "runbook": "runbooks/checkout.md#receipts"
    }
  ]
}
```

#### Artefact 2: a policy check in CI

The [charter](#a-proposed-charter-and-principles) as build-enforceable rules: one retry layer (app, SDK or mesh), no retried side
effects without a key, a degraded behaviour per soft dependency, sequential worst case within the inbound deadline.

_Example status: executed on 2026-10-18 with Python 3.8.10 against Artefact 1 (exit code 1, output below)._

```python
#!/usr/bin/env python3
"""Check a dependency register against the team's resilience policy."""
import json
import sys

REQUIRED = ("name", "criticality", "timeout_ms", "max_attempts",
            "sdk_attempts", "mesh_attempts", "idempotent", "breaker",
            "runbook")
LAYERS = {"max_attempts": "app", "sdk_attempts": "SDK",
          "mesh_attempts": "mesh"}


def check(reg):
    problems, budget_ms = [], 0
    for d in reg["dependencies"]:
        name = d.get("name", "?")
        missing = [k for k in REQUIRED if k not in d]
        if missing:
            problems.append(f"{name}: missing {', '.join(missing)}")
            continue
        attempts = d["max_attempts"] * d["sdk_attempts"] * d["mesh_attempts"]
        retrying = [label for k, label in LAYERS.items() if d[k] > 1]
        if len(retrying) > 1:
            problems.append(f"{name}: retries at {len(retrying)} layers "
                            f"({' x '.join(retrying)} = {attempts} attempts)")
        if (attempts > 1 and not d["idempotent"]
                and not d.get("idempotency_key")):
            problems.append(f"{name}: retried but not idempotent")
        if d["criticality"] == "soft" and not d.get("degraded_behaviour"):
            problems.append(f"{name}: soft dependency without a "
                            "degraded behaviour")
        if d.get("sequential"):
            budget_ms += d["timeout_ms"] * attempts
    if budget_ms > reg["inbound_deadline_ms"]:
        problems.append(f"worst case {budget_ms} ms of sequential calls "
                        f"exceeds the {reg['inbound_deadline_ms']} ms "
                        "inbound deadline")
    return problems


if __name__ == "__main__":
    with open(sys.argv[1], encoding="utf-8") as fh:
        found = check(json.load(fh))
    for p in found:
        print("POLICY", p)
    print(f"{len(found)} problem(s)")
    sys.exit(1 if found else 0)
```

`python3 check_policy.py dependencies.json` prints:

```text
POLICY receipt-store: retries at 2 layers (app x SDK = 6 attempts)
POLICY receipt-store: soft dependency without a degraded behaviour
POLICY worst case 2050 ms of sequential calls exceeds the 2000 ms inbound deadline
3 problem(s)
```

All three are real design bugs: `receipt-store` makes six calls per failure and has no degraded behaviour, and `payments`
(800 ms × 2) plus `inventory` (150 ms × 3) can take 2,050 ms of the 2,000 ms deadline before any backoff sleep. Fixes: drop the
application retry, queue receipts, cut inventory to two attempts.

#### Artefact 3: library configuration

The register as Resilience4j configuration (keys from its documentation;[^r4jspring] `maxAttempts` counts the first
call[^r4jretry]). TimeLimiter bounds only calls that return a `CompletableFuture`,[^r4jtimeout] so the blocking-call timeouts
belong in each HTTP client. Retry has no ratio budget: the breaker stops retries in an outage, and Envoy offers retry
budgets.[^envoycb] In the default aspect order the bulkhead sits inside the breaker,[^r4jspring] so the breaker ignores bulkhead
rejections instead of counting them as dependency failures.

_Example status: YAML syntax-checked; not executed (no Spring application here). Property prefixes checked
against the resilience4j-spring-boot4 2.4.0 sources._

```yaml
# application.yml: Spring Boot 4.1 with resilience4j-spring-boot4 2.4.0
management:
  endpoint:
    health:
      probes:
        add-additional-paths: true    # /livez and /readyz on the main port
spring:
  lifecycle:
    timeout-per-shutdown-phase: 15s   # drain; the pod's grace period covers it
resilience4j:
  circuitbreaker:
    configs:
      checkout:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 20
        minimumNumberOfCalls: 10
        failureRateThreshold: 50
        waitDurationInOpenState: 5s
        permittedNumberOfCallsInHalfOpenState: 3
        registerHealthIndicator: false  # an open breaker must not fail health
        ignoreExceptions:               # a full bulkhead is not a dependency fault
          - io.github.resilience4j.bulkhead.BulkheadFullException
    instances:
      payments: {baseConfig: checkout}
      inventory: {baseConfig: checkout}
      recommendations:
        baseConfig: checkout
        slowCallDurationThreshold: 150ms
        slowCallRateThreshold: 50
  retry:
    instances:
      payments:                       # idempotency key created outside the retry
        maxAttempts: 2                # includes the first call
        waitDuration: 100ms
        enableRandomizedWait: true
        randomizedWaitFactor: 0.5
        retryExceptions: [java.io.IOException]
      inventory:
        maxAttempts: 3
        waitDuration: 50ms
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        enableRandomizedWait: true    # jitter: each wait varies by up to 50%
        randomizedWaitFactor: 0.5
        retryExceptions: [java.io.IOException]
  bulkhead:
    instances:
      recommendations:
        maxConcurrentCalls: 20
        maxWaitDuration: 0
  timelimiter:
    instances:
      recommendations:                # only for calls returning CompletableFuture
        timeoutDuration: 200ms
        cancelRunningFuture: true
```

#### Artefact 4: probes and graceful shutdown

The service on Kubernetes (placeholder image). `/livez` and `/readyz` are Spring Boot's probe groups on the main port, enabled in
Artefact 3 and used by Lab 1 too.[^springprobes] The startup probe covers slow starts; readiness gates warm-up. The pod meets the
Kubernetes Restricted security profile.[^k8spss] Depth: [KubernetesOps](../kubernetesops/), [KubernetesSecOps](../kubernetessecops/).

_Example status: YAML syntax-checked; not applied to a cluster. Resource sizes are placeholders for a load test._

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 3
  selector:
    matchLabels: {app: checkout-api}
  template:
    metadata:
      labels: {app: checkout-api}
    spec:
      terminationGracePeriodSeconds: 30   # >= preStop 10 s + drain 15 s
      automountServiceAccountToken: false # the app never calls the API server
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: app
          image: registry.example.com/checkout-api:1.4.2
          ports:
            - containerPort: 8080
          resources:                        # pools and bulkheads assume this
            requests: {cpu: 500m, memory: 768Mi}
            limits: {memory: 768Mi}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: [ALL]}
          volumeMounts:
            - {name: tmp, mountPath: /tmp}  # writable scratch for the server
          lifecycle:
            preStop:                    # routing stops + the longest in-flight request
              sleep: {seconds: 10}
          startupProbe:                     # slow start: up to 60 s to listen
            httpGet: {path: /livez, port: 8080}
            periodSeconds: 5
            failureThreshold: 12
          readinessProbe:                   # traffic on/off; may fail often
            httpGet: {path: /readyz, port: 8080}
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:                    # restart only if wedged
            httpGet: {path: /livez, port: 8080}
            periodSeconds: 10
            failureThreshold: 3
      volumes:
        - {name: tmp, emptyDir: {}}
```

#### Artefact 5: alerts that link the runbook

Prometheus annotations "can be used to store longer additional information such as alert descriptions or runbook
links".[^promalerting] An open breaker on a soft dependency is a ticket, not a page; SLO paging is in [SRE](../sre/). An alert
fires only if its condition holds at every evaluation of the `for` window,[^promalerting] and a long outage alternates open and
half-open, so the first rule tests *not closed*.

_Example status: YAML syntax-checked; both PromQL expressions parsed by the public Prometheus demo server (3.14.0) on 2026-10-19;
rules not run with `promtool`. Metric names follow the metric table of the Resilience4j Micrometer page;[^r4jmicrometer] its older
sample output shows another counter name, so check your own `/actuator/prometheus` output._

```yaml
groups:
  - name: checkout-api-dependencies
    rules:
      - alert: DependencyBreakerNotClosed
        # open and half_open alternate in a long outage: test "not closed"
        expr: min by (name) (resilience4j_circuitbreaker_state{state="closed"}) == 0
        for: 5m
        labels:
          severity: ticket            # degraded, not down: no page at night
        annotations:
          summary: "Breaker {{ $labels.name }} not closed for 5 minutes"
          runbook_url: "https://runbooks.example.com/checkout-api#breaker-open"
      - alert: DependencyCallsRejected
        expr: |
          sum by (name) (
            rate(resilience4j_circuitbreaker_not_permitted_calls_total[5m])
          ) > 5
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.name }}: over 5 calls/s short-circuited for 15 min"
          runbook_url: "https://runbooks.example.com/checkout-api#breaker-open"
```

### Hands-on labs

Each lab runs on one machine with Python 3.8+, `bash` and `curl`, uses only the loopback interface and costs nothing. Outputs were
produced on 2026-10-18.

#### Lab 1 (beginner): health endpoints that tell the truth

**Objective.** Watch liveness and readiness disagree during warm-up, then shut down in order: readiness off, keep serving for
`DEREGISTER_S`, stop listening, drain.
**Prerequisites.** Linux or macOS, Python 3.8+, `curl`. **Cost and risk.** None; the server listens on 127.0.0.1:18080 only.

1. Save the server as `health_server.py`.

_Example status: executed on 2026-10-18 (Python 3.8.10)._

```python
#!/usr/bin/env python3
"""Lab 1: liveness and readiness endpoints that tell the truth.

/livez  - process-local: 200 while the server can answer at all.
/readyz - 200 only when warmed up and not shutting down.
/work   - a 1.5-second request, to show draining on SIGTERM.
"""
import signal
import sys
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

WARMUP_S = 2.0          # e.g. loading caches or opening connection pools
DEREGISTER_S = 1.0      # keep serving while routers stop sending traffic
GRACE_S = 10.0          # stay below the platform's termination grace period
STARTED = time.monotonic()
state = {"draining": False, "in_flight": 0}
lock = threading.Lock()


def ready():
    warmed = time.monotonic() - STARTED >= WARMUP_S
    return warmed and not state["draining"]


class Handler(BaseHTTPRequestHandler):
    def reply(self, code, body):
        data = (body + "\n").encode()
        self.send_response(code)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        if self.path == "/livez":
            self.reply(200, "alive")
        elif self.path == "/readyz":
            if ready():
                self.reply(200, "ready")
            else:
                why = "draining" if state["draining"] else "warming up"
                self.reply(503, "not ready: " + why)
        elif self.path == "/work":
            with lock:
                state["in_flight"] += 1
            try:
                time.sleep(1.5)
                self.reply(200, "done")
            finally:
                with lock:
                    state["in_flight"] -= 1
        else:
            self.reply(404, "not found")

    def log_message(self, fmt, *args):
        sys.stderr.write("%s %s\n" % (self.command, self.path))


def main(port):
    server = ThreadingHTTPServer(("127.0.0.1", port), Handler)

    def stop_accepting_later():
        time.sleep(DEREGISTER_S)          # 2. still serving new requests
        server.shutdown()                 # 3. then stop the accept loop

    def on_sigterm(signum, frame):
        state["draining"] = True          # 1. readiness fails at once
        threading.Thread(target=stop_accepting_later, daemon=True).start()

    signal.signal(signal.SIGTERM, on_sigterm)
    print("listening on 127.0.0.1:%d" % port, flush=True)
    server.serve_forever(poll_interval=0.05)
    server.server_close()                 # new connections are now refused
    print("stopped listening", flush=True)
    deadline = time.monotonic() + GRACE_S
    while time.monotonic() < deadline:    # 4. finish in-flight work
        with lock:
            if state["in_flight"] == 0:
                break
        time.sleep(0.05)
    print("drained, exiting", flush=True)


if __name__ == "__main__":
    main(int(sys.argv[1]) if len(sys.argv) > 1 else 8080)
```

2. Save the driver as `lab1.sh` next to it and run `bash lab1.sh`.

_Example status: executed on 2026-10-18; output below._

```bash
#!/usr/bin/env bash
set -euo pipefail
PORT=18080
probe() {  # usage: probe <path>
  curl -s -o /dev/null -w "$1 %{http_code}\n" "http://127.0.0.1:${PORT}$1" || true
}
python3 health_server.py "${PORT}" 2>server.log &
PID=$!
sleep 0.5
probe /livez; probe /readyz            # alive, still warming up
sleep 2
probe /readyz                          # ready
work() { curl -s "http://127.0.0.1:${PORT}/work" || echo "failed"; }
work > work1.out &
W1=$!
sleep 0.2
kill -TERM "${PID}"                    # the platform asks it to stop
sleep 0.2
probe /readyz                          # failing: routers stop sending
work > work2.out &
W2=$!                                  # a late request is still served
sleep 1.2
probe /readyz                          # 000: no longer listening
wait "${W1}" "${W2}"; wait "${PID}"; CODE=$?
echo "request sent before SIGTERM: $(cat work1.out)"
echo "request sent after SIGTERM: $(cat work2.out)"
echo "server exit code: ${CODE}"
```

**Expected output.**

```text
listening on 127.0.0.1:18080
/livez 200
/readyz 503
/readyz 200
/readyz 503
stopped listening
/readyz 000
drained, exiting
request sent before SIGTERM: done
request sent after SIGTERM: done
server exit code: 0
```

**Validation.** Liveness stays 200; readiness goes 503 (warming up), 200, 503 after SIGTERM, yet a request sent after SIGTERM is
served; a second later the server stops listening (`000`: connection refused), finishes both requests and exits 0.
**Exercise.** Set `DEREGISTER_S = 0` and watch the late request fail; make `/livez` fail on a pretend database flag and predict
what a platform does to every replica.[^springprobes] **Troubleshooting.** `Address already in use`: change `PORT`; an early
`000`: read `server.log`. **Teardown.** The server exits by itself (else `kill %1`); delete the files.

#### Lab 2 (intermediate): measure retry amplification and contain it

**Objective.** Reproduce the SRE book's 4^3 amplification[^srecascade] and see what one retry layer and a 10% budget[^sreoverload]
change and cost. **Prerequisites.** Python 3.8+. **Cost and risk.** None (simulation).

1. Save the script as `retry_lab.py`.

_Example status: executed on 2026-10-18 (Python 3.8.10); seeded, so your output should match exactly._

```python
#!/usr/bin/env python3
"""Lab 2: measure retry amplification, then contain it.

A user action passes through three layers (browser code, front end,
back end) before it reaches a database. The database fails each
attempt with probability p. We count database attempts per action.
"""
import random

ACTIONS = 10_000


class Budget:
    """Per-client retry budget: retries stay below `ratio` of sends."""

    def __init__(self, ratio):
        self.ratio, self.sent, self.retries = ratio, 0, 0

    def allow_retry(self):
        return self.retries + 1 <= self.ratio * (self.sent + 1)


def call(layer, attempts, rng, p, stats, budget=None):
    """Call the layer below with up to attempts[layer] tries."""
    mine = budget if layer == 0 else None       # the top client owns it
    for i in range(attempts[layer]):
        if mine and i > 0:
            if not mine.allow_retry():
                return False
            mine.retries += 1
        if mine:
            mine.sent += 1
        if layer == len(attempts) - 1:          # next hop is the database
            stats["db"] += 1
            ok = rng.random() >= p
        else:
            ok = call(layer + 1, attempts, rng, p, stats, budget)
        if ok:
            return True
    return False


def run(name, attempts, p, budget_ratio=None):
    rng = random.Random(42)
    stats = {"db": 0}
    budget = Budget(budget_ratio) if budget_ratio else None
    ok = sum(call(0, attempts, rng, p, stats, budget)
             for _ in range(ACTIONS))
    print(f"{name:<28} p={p:<4} db attempts/action="
          f"{stats['db'] / ACTIONS:6.2f}  success={ok / ACTIONS:6.1%}")


JITTER_RNG = random.Random(7)


def full_jitter(attempt, base=0.1, cap=2.0):
    """AWS 'Full Jitter': random_between(0, min(cap, base * 2**attempt))."""
    return JITTER_RNG.uniform(0, min(cap, base * 2 ** attempt))


if __name__ == "__main__":
    for p in (0.05, 0.3, 1.0):
        run("A: 4 attempts at 3 layers", [4, 4, 4], p)
        run("B: 4 attempts at top only", [4, 1, 1], p)
        run("C: B plus 10% budget", [4, 1, 1], p, budget_ratio=0.1)
    print("full-jitter sleeps (s):",
          [round(full_jitter(a), 3) for a in range(6)])
```

2. Run `python3 retry_lab.py`. **Expected output.**

```text
A: 4 attempts at 3 layers    p=0.05 db attempts/action=  1.05  success=100.0%
B: 4 attempts at top only    p=0.05 db attempts/action=  1.05  success=100.0%
C: B plus 10% budget         p=0.05 db attempts/action=  1.05  success=100.0%
A: 4 attempts at 3 layers    p=0.3  db attempts/action=  1.42  success=100.0%
B: 4 attempts at top only    p=0.3  db attempts/action=  1.41  success= 99.1%
C: B plus 10% budget         p=0.3  db attempts/action=  1.11  success= 78.2%
A: 4 attempts at 3 layers    p=1.0  db attempts/action= 64.00  success=  0.0%
B: 4 attempts at top only    p=1.0  db attempts/action=  4.00  success=  0.0%
C: B plus 10% budget         p=1.0  db attempts/action=  1.11  success=  0.0%
full-jitter sleeps (s): [0.032, 0.03, 0.26, 0.058, 0.857, 0.731]
```

**Validation.** At p=1.0: 64 attempts per action with layered retries, 4 with one layer, about 1.1 with the budget. At p=0.3 the
budget costs success (78.2% against 99.1%): availability traded for overload protection. The simulation treats failures as
independent of load; in real overload extra attempts raise p itself. Use your library's retries: AWS calls custom retry code an
anti-pattern when built-in support suffices.[^warel05bp03] **Troubleshooting.** Different numbers mean a changed seed or
`ACTIONS`. **Teardown.** Delete the file.

#### Lab 3 (advanced): a circuit breaker during a brownout

**Objective.** Compare *timeout only* with *timeout plus breaker* while a soft dependency hangs for 30 seconds, and find the
breaker's cost. **Prerequisites.** Python 3.8+; Lab 2. **Cost and risk.** None (simulated clock).

1. Save the script as `breaker_lab.py`.

_Example status: executed on 2026-10-18 (Python 3.8.10); deterministic, so your output should match exactly._

```python
#!/usr/bin/env python3
"""Lab 3: a circuit breaker and a timeout during a dependency brownout.

Simulated time, no sleeping. A checkout page calls a *soft* dependency
(recommendations) 100 times a second for 60 s. Between t=10 s and
t=40 s the dependency hangs. Each call has a 200 ms timeout; on any
failure the page degrades (renders without recommendations).
"""
from collections import deque

RATE, DURATION = 100, 60.0           # requests per second, seconds
TIMEOUT, HEALTHY_MS = 0.200, 0.040   # seconds
BROWNOUT = (10.0, 40.0)


class Breaker:
    """Count-based breaker: closed -> open -> half-open -> closed."""

    def __init__(self, window=20, min_calls=10, threshold=0.5,
                 open_s=5.0, trial_calls=3):
        self.window, self.min_calls = window, min_calls
        self.threshold, self.open_s = threshold, open_s
        self.trial_calls = trial_calls
        self.state, self.results, self.opened_at = "closed", deque(), 0.0
        self.trials = self.trial_ok = 0
        self.log = []

    def _move(self, now, state):
        self.log.append(f"t={now:5.2f}s {self.state} -> {state}")
        self.state = state

    def allow(self, now):
        if self.state == "open" and now - self.opened_at >= self.open_s:
            self._move(now, "half_open")
            self.trials = self.trial_ok = 0
        if self.state == "half_open":
            if self.trials < self.trial_calls:
                self.trials += 1
                return True
            return False
        return self.state == "closed"

    def record(self, now, ok):
        if self.state == "open":
            return                        # late result of an earlier call
        if self.state == "half_open":
            self.trial_ok += ok
            if not ok:
                self._move(now, "open")
                self.opened_at = now
            elif self.trial_ok == self.trial_calls:
                self._move(now, "closed")
                self.results.clear()
            return
        self.results.append(ok)
        if len(self.results) > self.window:
            self.results.popleft()
        failures = self.results.count(False)
        if (len(self.results) >= self.min_calls
                and failures / len(self.results) >= self.threshold):
            self._move(now, "open")
            self.opened_at = now
            self.results.clear()


def simulate(use_breaker):
    breaker = Breaker() if use_breaker else None
    pending = []                          # (finish time, ok) not yet seen
    calls = sick_calls = degraded = 0
    waits = []
    for i in range(int(RATE * DURATION)):
        now = i / RATE
        for done in [x for x in pending if x[0] <= now]:
            breaker.record(*done)         # outcomes arrive after the wait
            pending.remove(done)
        sick = BROWNOUT[0] <= now < BROWNOUT[1]
        if breaker and not breaker.allow(now):
            degraded += 1                 # fail fast: no call, no wait
            waits.append(0.0)
            continue
        calls += 1
        sick_calls += sick
        waits.append(TIMEOUT if sick else HEALTHY_MS)
        degraded += sick
        if breaker:
            pending.append((now + waits[-1], not sick))
    waits.sort()
    p99 = waits[int(len(waits) * 0.99) - 1]
    stuck = sick_calls * TIMEOUT / (BROWNOUT[1] - BROWNOUT[0])  # Little
    name = "with breaker" if use_breaker else "timeout only"
    print(f"{name:<13} calls={calls:5d} to sick={sick_calls:5d} "
          f"degraded={degraded:5d} p99 wait={p99 * 1000:3.0f} ms "
          f"stuck={stuck:4.1f}")
    if breaker:
        print("\n".join(breaker.log))


if __name__ == "__main__":
    simulate(False)
    simulate(True)
```

2. Run `python3 breaker_lab.py`. **Expected output.**

```text
timeout only  calls= 6000 to sick= 3000 degraded= 3000 p99 wait=200 ms stuck=20.0
with breaker  calls= 2911 to sick=   44 degraded= 3133 p99 wait= 40 ms stuck= 0.3
t=10.29s closed -> open
t=15.29s open -> half_open
t=15.49s half_open -> open
t=20.49s open -> half_open
t=20.69s half_open -> open
t=25.69s open -> half_open
t=25.89s half_open -> open
t=30.89s open -> half_open
t=31.09s half_open -> open
t=36.09s open -> half_open
t=36.29s half_open -> open
t=41.30s open -> half_open
t=41.36s half_open -> closed
```

**Validation.** With a timeout alone all 3,000 brownout calls wait 200 ms, about 20 stuck at any moment (Little's law: 100/s ×
0.2 s). With the breaker only 44 reach the sick dependency and p99 wait falls from 200 ms to 40 ms. The cost: 133 extra degraded
responses after recovery, as trials come only every 5 s. **Exercises.** Try `open_s=1.0`, then `window=100, min_calls=100` (the
library defaults: it opens after 50 failures, at t=10.69 s). **Troubleshooting.** Other numbers mean edited constants.
**Teardown.** Delete the file.

### Use cases and scenarios

#### Case 1 (sourced): a latent back-off bug at AWS, December 2021

On 2021-12-07 an automated scaling activity on AWS's network triggered "a large surge of connection activity", and the delays
caused "even more connection attempts and retries". The clients had well-tested back-off, "but, a latent issue prevented these
clients from adequately backing off during this event".[^aws2021] **Lesson:** back-off is
code with its own failure modes; test it under triggering conditions and keep dashboards off the failing path.

#### Case 2 (sourced): from fixed settings to adaptive limits at Netflix

Netflix's Hystrix isolated calls to remote systems to stop cascading failure. Its README says it "is no longer in active
development, and is currently in maintenance mode": Netflix's focus shifted to "more adaptive implementations that react to an
application's real time performance rather than pre-configured settings", and it recommends resilience4j for new projects.[^hystrix]
**Lesson:** static thresholds go stale; whatever the mechanism, someone owns its tuning.

#### Case 3 (illustrative): two situations

::: example Illustrative scenarios
- **A queue worker that retries forever.** A malformed invoice job fails and is redelivered endlessly. The team retries only
  transient errors, caps deliveries, sends poison messages to an alerting dead-letter queue and keys the job for idempotency.
  Success: no duplicate invoices. Pitfall: an unread dead-letter queue is silent data loss.
- **A packaged HR system** (the ITIL meaning):[^itilglossary] nobody in-house builds it, so AppOps lives in the integration code
  (timeouts, retries, breakers), runbooks for vendor patches and a vendor escalation path.
:::

## Measure, govern, pay

### Metrics and KPIs

SLIs and targets: [SRE](../sre/); MTTA and MTTR: [IncidentOps](../incidentops/). These measures show whether the resilience code
works.

| KPI | Definition and formula | Pitfall | Review |
|---|---|---|---|
| Goodput ratio | (responses without error **and** within the latency target) ÷ requests,[^blshedding] from a request histogram such as `http.server.request.duration`[^otelhttp] | Degraded 200s count as good: track them separately | Weekly |
| Degraded share | degraded responses ÷ responses, per degraded mode | Invisible in error rates | Weekly |
| Retry ratio | retries ÷ first attempts, per dependency | High ratio with high success hides a sick dependency | Daily alert |
| Breaker time not closed | minutes open or half-open per dependency per day; short-circuited calls[^r4jmicrometer] | Zero may mean *never opens* | Monthly |
| Runbook coverage | paging alerts with a working runbook link ÷ paging alerts | Outdated pages still "count" | Monthly |
| Incidents per shift | incidents ÷ shifts (one incident may page many times); Google's upper bound is two per 12-hour shift[^sreoncall] | Silenced alerts vanish from the count | Monthly |
| Lever freshness | levers tested in 90 days ÷ levers | An untested lever is a hope | Quarterly |

Retry ratio, breaker events, coverage and lever freshness lead; goodput and incidents lag. Vanity trap: fewer pages because
alerts were silenced.

### Quality, security, privacy and compliance

- **Config versus secrets.** Twelve-Factor requires "strict separation of config from code" and recommends environment
  variables;[^12fconfig] OWASP warns they "are generally accessible to all processes and may be included in logs or system
  dumps".[^owaspsecrets] Workable split: settings in environment or files; secrets from a store, rotated, never in images.
- **Kubernetes Secrets** are "stored unencrypted in the API server's underlying data store (etcd)" by default, and anyone who can
  create a Pod in a namespace can read its Secrets;[^k8ssecrets] hardening is in [KubernetesSecOps](../kubernetessecops/).
- **Health endpoints and levers are attack surface:** no hostnames, versions or stack traces on public endpoints; lever changes
  restricted and logged. **Error paths log too:** keep personal data out ([PrivacyOps](../privacyops/)).
- **Compliance.** Equal Experts argues team-run operations are compatible with ITIL v3, PCI-DSS and SOX;[^eeybiyri] confirm
  with your auditors. Change approval is in [ITOps](../itops/).

### Cost, capacity and sustainability

Application-tier capacity is concurrency as much as CPU: Little's law sizes thread pools, connection pools and bulkheads,[^little]
and more replicas mean more database connections ([DBOps](../dbops/)). Autoscaling waits for new pods; shedding acts on the next
request, so use both ([KubernetesOps](../kubernetesops/)). Retries cost twice in incidents; standby pay grows with every
rota.[^eescale] Skipping work for callers that have timed out[^blshedding] saves capacity and energy ([GreenOps](../greenops/)).

### Assurance of the operating model

The policy check runs on every register change; each hard dependency gets a quarterly drill in a test environment (production
experiments follow [SRE](../sre/)'s chaos-engineering guidance); a monthly review covers the metrics above; incidents end with a
runbook check; major changes trigger a new readiness review.[^sreprr][^awsorr]

## When things go wrong

### Common failures and troubleshooting

| Symptom | Likely cause | Diagnosis | Remediation and prevention |
|---|---|---|---|
| Every replica restarts during a database outage | Liveness checks the database | Restarts line up with database errors | Process-local liveness[^springprobes] |
| One slow dependency fails every route | No timeout or bulkhead | Pool at its limit; threads wait on one client | Timeouts from latency; bulkhead |
| Dependency recovers but stays overloaded | Retry storm | Attempts per request above 1.5 | Retries to zero, then one layer, jitter, budget[^srecascade] |
| Errors on every deploy | Stops before routers stop sending | Errors or refused connections at pod termination | `preStop` sleep or deregistration delay, then stop listening and drain[^springcloud] |
| Degraded mode fails in the incident | Path never exercised | No recent drill or traffic | Exercise or simplify it[^blfallback] |

### Worked runbook: a slow dependency starts a retry storm

**Trigger:** checkout error ratio above target, or an SLO burn alert. **Levers:** `inventory.retries`, `recommendations.enabled`,
admission limit, forced breaker open.

::: figure Runbook for a retry storm against one dependency
```mermaid
flowchart TB
  P["Page: checkout errors up"] --> S{"Traces and logs: one dependency slow or failing?"}
  S -- "no" --> G["Check own deploys and capacity"]
  S -- "yes" --> L1["Retry ratio above 10%: its retries to zero"]
  L1 --> K{"Is the dependency soft?"}
  K -- "yes" --> L2["Switch it off or force its breaker open"]
  K -- "no" --> L3["Shed sheddable traffic; call its owner; incident after 15 min"]
  L2 --> V["Verify goodput; restore levers one at a time"]
  L3 --> V
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class P risk
  class S,K decision
  class L1,L2,L3,V,G core
```
Text equivalent: on a page for rising checkout errors, use traces and logs to check whether one dependency is slow or failing; if
not, check own deploys and capacity. If its retry ratio is above 10%, set its retries to zero, switch a soft dependency off or force
its breaker open — for a hard one, shed sheddable traffic, call its owner, declare an incident after 15 minutes. Verify goodput,
then restore levers one at a time.
:::

1. **Scope and diagnose (5 minutes):** failing routes, start time, per-dependency latency, error and retry panels; an exemplar
   trace from the spike; logs filtered on dependency and decision.
2. **Stop amplification:** above a 10% retry ratio (this Atlas's threshold, borrowed from Google's client budget[^sreoverload]),
   retries to zero; attempts per request should near 1.0 within two minutes.
3. **Protect users:** soft dependency off or breaker forced open; hard dependency: fast 503s with Retry-After for sheddable traffic.
4. **Escalate** to the dependency owner; past 15 minutes or across teams, to [IncidentOps](../incidentops/) roles.
5. **Recover** one lever at a time, retries last, logging each change; then a postmortem and register, runbook and drill updates.

### Anti-patterns and myths

| Anti-pattern or myth | Why it fails | Do instead |
|---|---|---|
| *More retries, more reliability* | Layers multiply load ([retries](#playbook-retries-that-help-instead-of-hurt)) | One layer, jitter, budget |
| *The health check should check everything* | Shared dependencies restart or drain whole fleets[^springprobes] | Liveness local; readiness deliberate |
| *The mesh handles resilience* | A proxy cannot know idempotency or degraded responses | Mesh for limits, code for semantics |
| *We have fallbacks, so we are safe* | Rare paths fail, and fallback is hard to prove and test, so Amazon avoids it[^blfallback] | Exercise them or harden the primary |
| *The runbook is the launch wiki page* | Runbooks drift, an AWS-listed anti-pattern[^waops07] | Version with code; check after incidents |

## Deciding and adopting

### When to use and when not to use

**Invest fully** when a service calls other systems while a user waits, changes often, and costs money or trust when down.
**Keep a light baseline** (truthful health, drain, config outside code, runbooks) for batch jobs, internal tools and prototypes; for
**packaged software**, the work is in your integration code and vendor runbooks. **Do not adopt team on-call** when the team cannot
staff a sustainable rota, downtime cost does not justify night cover, or a central team fits better ([NoOps](../noops/)). Budget for
hidden work: re-tuning thresholds, testing levers and degraded paths, current runbooks, standby pay.

### Decision framework

::: figure Who runs the service, and how much AppOps it needs
```mermaid
flowchart TB
  Q1{"Do you change this service's code?"}
  Q1 -- "no" --> C1["Central application support; AppOps in integration code"]
  Q1 -- "yes" --> C2["Baseline: health, ordered shutdown, config, runbook; a register if it calls other systems"]
  C2 --> Q3{"Does downtime cost justify night cover?"}
  Q3 -- "no" --> R1["Team owns it in working hours only"]
  Q3 -- "yes" --> Q4{"Enough people for a sustainable rota?"}
  Q4 -- "yes" --> R2["Team on-call: you build it, you run it"]
  Q4 -- "no" --> R3["Shared domain rota or SRE support"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q1,Q3,Q4 decision
  class C2,R1,R2,R3 core
  class C1 neutral
```
Text equivalent: if you do not change the code, central application support runs the service and AppOps lives in integration code.
Otherwise every service needs the baseline, plus a register and policies if it calls other systems; without a cost case for night
cover the team owns it in working hours, with one, on call if it can staff a sustainable rota, else a domain rota or SRE support.
:::

Then choose patterns per dependency; a starting point, not a rule:

| Dependency | Timeout | Retries | Breaker, bulkhead | When it fails |
|---|---|---|---|---|
| Hard, idempotent read | From p99.9 | Up to 2, one layer, budget | Yes | Fail fast, clear error |
| Hard, side effects (payment) | From p99.9 plus margin | Only with an idempotency key | Yes | Fail fast; never guess the outcome |
| Soft (recommendations) | Short | None | Yes | Degraded response |
| Asynchronous write | Generous | Capped queue redelivery | Queue as bulkhead | Deferred; alert on dead letters |
| Rate-limited third party | From its limits | Honour 429 and Retry-After[^rfc6585] | Client throttling[^sreoverload] | Shed sheddable work |

### Maturity model (this Atlas's model)

Levels and thresholds are this Atlas's proposal, not a standard.

| Level | Observable behaviour | Evidence | Next step |
|---|---|---|---|
| **0 Unbounded** | Library defaults; liveness checks dependencies; no runbooks | Nothing written | Inventory; timeouts |
| **1 Hygiene** | Explicit timeouts; split probes; ordered shutdown; runbook per paging alert | Dependency list | Retry policy |
| **2 Designed** | Register with policies; one budgeted retry layer; breakers and bulkheads; degraded modes written; decisions in traces and logs | Policy check in CI | Drills, levers |
| **3 Verified** | Quarterly drills; levers tested within 90 days; runbook coverage ≥ 90% | Drill notes; trends | Adaptive controls |
| **4 Adaptive** | Adaptive limits or throttling; criticality-aware shedding; degraded paths always exercised | Tuning linked to data | Keep it explainable |

Level 3 suits anything that pages; few services need level 4.

### Adoption roadmap: 30/60/90 days

| Window | Actions | Exit metric |
|---|---|---|
| **Days 1–30** | Three most-paging services: registers; find default timeouts and hidden SDK and mesh retries; process-local liveness; explicit timeouts; runbook per paging alert | Explicit timeouts on every critical-path call |
| **Days 31–60** | One budgeted retry layer; idempotency keys; breakers, bulkheads and degraded responses; first levers; policy check in CI | Retry ratio visible; runbook coverage ≥ 80% |
| **Days 61–90** | A drill per hard dependency; readiness review; on-call model chosen on cost and coverage; monthly review | Every hard dependency drilled |

Dependencies: telemetry ([ObservabilityOps](../observabilityops/)), SLOs ([SRE](../sre/)), flags ([ReleaseOps](../releaseops/))
and a recorded owner ([ServiceOps](../serviceops/)).

## Toolkit

### Checklists

#### Production readiness for a service

- [ ] Register complete (hard or soft per dependency); policy check passes
- [ ] Explicit timeouts; the deadline propagates
- [ ] One budgeted, jittered retry layer (SDK and mesh retries counted); retried writes idempotent or keyed
- [ ] Breakers and bulkheads on critical paths; open-breaker behaviour defined
- [ ] Liveness process-local; readiness covers warm-up; shutdown serves until routers stop, then drains in the grace period
- [ ] Retries, breaker transitions and degraded responses visible in traces and logs by dependency; trace ID in every log line
- [ ] Degraded responses written and exercised; levers access-controlled and tested
- [ ] Every paging alert links a runbook entry
- [ ] Secrets from a store; on-call rota published and sustainable

#### Dependency onboarding

- [ ] Owner, escalation contact, measured latency and chosen timeout recorded; retryable errors, rate limits and Retry-After known
- [ ] Degraded behaviour agreed with the product owner

#### Monthly operability review and after incidents

- [ ] Retry ratio, breaker time and degraded share per dependency; incidents per shift; readiness flaps outside deploys
- [ ] Runbook coverage and lever freshness
- [ ] After each incident: runbook fixed within a week; missing lever added; register and drill updated

### Templates

#### Service operability one-pager

```text
SERVICE OPERABILITY ONE-PAGER                 version / date
Service:            owner team:          service owner:
Tier / SLO link:    on-call model: team | domain | central | none at night
Inbound deadline:   ___ ms         admission limit: ___ concurrent
Dependencies:       see register (hard: __, soft: __), policy check: pass/fail
Health:             /livez = process only; /readyz = warm-up + draining + ___
Levers:             name - effect - who may pull - last tested
Degraded modes:     dependency - what the user gets - how it is exercised
Runbook:            link; last reviewed ____; alerts without runbook: __
Open risks:         ...
```

#### Runbook entry

```text
RUNBOOK ENTRY: <alert name>                         reviewed: YYYY-MM-DD
What it means:     one sentence in user terms
Impact:            who is affected, degraded or failing
First checks:      dashboards / queries (links), in order
Decision points:   if X then lever A; if Y then lever B; else escalate
Levers:            name - how to pull - expected effect - how to undo
Escalate to:       dependency owner / incident roles after __ minutes
Verify recovery:   metric and threshold
Afterwards:        ticket for follow-up, register and drill updates
```

#### Lever record

```text
LEVER: recommendations.enabled                 owner: team-checkout
Effect when off:   checkout renders without the panel; no calls made
Who may change:    on-call engineers of team-checkout (audited)
How:               flag system UI or CLI; takes effect within 30 s
Undo:              set back to on; watch error rate for 10 minutes
Last tested:       YYYY-MM-DD in production during a drill
```

#### Architecture decision record: resilience in the library or the mesh

```text
ADR-NNN  Where do timeouts, retries and breakers live?     status: proposed
Context:   languages in use; mesh present?; who tunes policies?
Options:   (1) library per language  (2) mesh/proxy  (3) both, split by concern
Trade-offs: a library knows idempotency, deadlines and degraded responses but
           means one implementation per language; a mesh is uniform but blind
           to business meaning; platform defaults are a start for both
Decision:  e.g. (3): mesh enforces connection limits and outlier ejection;
           libraries own deadlines, idempotent retries and degraded responses;
           mesh retries off for routes that retry in code (Istio's default: 2)
Consequences: one retry layer per route documented in the register;
           policy check fails if both layers retry
Review:    after the next two incidents involving a dependency
```

### Deep comparisons

**Ways to survive a failing dependency.** Amazon names four and avoids the last because
"it's difficult to prove and its effectiveness is hard to test".[^blfallback]

| Option | Works when | Main risk |
|---|---|---|
| Retry | Failures are rare and transient | Amplifies overload |
| Proactive retry (hedging) | Tail latency matters; spare capacity | Constant extra load |
| Failover | A healthy copy exists elsewhere | The copy shares the failure |
| Graceful degradation | A reduced answer is still useful | Rarely exercised path |
| Fallback | The alternative runs and is tested regularly | Entered "only in the most chaotic moments"[^blfallback] |

### FAQ

**Should readiness check the database?** Only if taking every instance out of rotation at once helps. Spring Boot calls it a
judgement call, and behind a Kubernetes Service clients then get refused connections, not a 503.[^springprobes]

**What timeout should I use?** The dependency's latency at the percentile matching your acceptable false-timeout rate, plus network
margin, short enough that the sum fits your deadline.[^bltimeouts]

**Do I need a breaker if I have a timeout?** For important dependencies, yes: a timeout limits each wait; a breaker stops the
waiting (Lab 3: calls to the sick dependency fell from 3,000 to 44).

**Does every service need 24/7 on-call?** No; match cover to downtime cost (Equal Experts' 95% example has no night
rota).[^eescale]

**What if we cannot change the code?** Put timeouts, retries and breakers in integration code or a proxy you control, keep
runbooks for the product, and agree an escalation path with the vendor.

## Going further

### Advanced topics and emerging direction

- **Established and documented:** client-side adaptive throttling (reject locally with probability
  max(0, (requests − K × accepts) ÷ (requests + 1)), K = 2 preferred) and criticality-aware shedding;[^sreoverload] hedging for
  tail latency (.NET ships a hedging handler[^dotnetresilience]); and Amazon's constant-work redundancy, which adds no retry load
  as the need for redundant requests grows.[^blfallback]
- **Active practice, still settling:** adaptive concurrency limits borrowed from TCP congestion control, Netflix's direction after
  Hystrix.[^hystrix][^concurrencylimits] Less tuning, but behaviour that is harder to predict and explain at 03:00.
- **Speculative or marketing-led:** AI agents that fix incidents alone. This page's view: treat their actions as lever changes
  (reviewed, reversible, logged) until safe autonomy is shown for your kind of system ([AIOps](../aiops/)).

### Learning path

| Stage | What to do |
|---|---|
| 1 day | HTTP basics, one language, the glossary, Lab 1, the SRE book's "Addressing Cascading Failures"[^srecascade] |
| 1 week | Labs 2 and 3; Brooker on timeouts and retries;[^bltimeouts] a register for one real service |
| 30 days | Artefacts 1–5 for that service; one drill; a readiness review with PRR or ORR questions[^sreprr][^awsorr] |
| Advanced | "Handling Overload";[^sreoverload] *Release It!*;[^releaseit] adaptive limits;[^concurrencylimits] SLO design in [SRE](../sre/) |
| Portfolio | A small service with a register, a CI policy check and injected faults |

## Wrap-up

### Executive recap and cheat sheet

| Area | Remember |
|---|---|
| Waiting | Deadline per request; timeouts from measured latency, never library defaults |
| Retrying | One layer; idempotent or keyed; jitter; a budget; count SDK and mesh retries |
| Containing and degrading | Breakers, bulkheads sized by Little's law; shed early; simple, exercised degraded modes |
| Health | Liveness process-local; readiness for traffic; on shutdown serve until routers stop, then drain |
| Diagnosing | Trace context through retries; decisions as span events and log fields; trace ID in logs |
| Operating | Runbook per paging alert; tested levers; on-call matched to downtime cost and team size |

### A judgement call: build a fallback or harden the primary path?

**Situation (illustrative).** Checkout depends on one payment provider, which failed twice last year for about 45 minutes; the
team has one quarter. **Option A:** integrate a second provider with automatic switch-over (most of the quarter). **Option B:**
harden the primary path: idempotency keys, timeouts from measured latency, a breaker with a clear *try again shortly* page, a lever
to pause non-urgent payments, and a runbook.

**Why A can be wrong.** A switch used only in failures is "the code path you never use", which "(often) doesn't work";[^srecascade]
it runs at the worst moment, with unclear payment states, when double charges are likeliest without idempotency. **A better A**
sends real traffic to the second provider all the time, so both paths are "exercised in production continuously rather than
rarely":[^blfallback] active-active, with ongoing fees and a larger compliance scope.

**Decision.** Weigh about 90 minutes a year of failed payments against a quarter of engineering plus a second provider's running
cost. A without idempotency is dangerous, so B comes first; add active-active only if the expected loss pays for it, never an
untested switch. Two outages are a small sample; revisit if the provider's record worsens.

### Learning objectives and mastery check

You should now be able to **explain** why "AppOps" is ambiguous, **design** per-dependency resilience and truthful health
endpoints, **operate** with levers and runbooks, and **evaluate** a sustainable on-call model.

1. A liveness endpoint runs `SELECT 1` on the shared database, which is slow for ten minutes. What happens, and what do you change?

::: answer Show answer for question 1
Every replica fails liveness and is restarted while the database is still slow, the cascading failure Kubernetes and Spring Boot
warn about.[^k8sprobes][^springprobes] Make liveness process-local; handle the outage with timeouts and a breaker.
:::

2. A teammate wraps an AWS SDK call, at default settings, in a loop of three tries. Worst case?

::: answer Show answer for question 2
It depends on the SDK (October 2026): standard mode makes three attempts, so up to 9 calls; Boto3's legacy default of five gives
15.[^awssdkretry][^boto3retries] Remove the loop, or turn SDK retries off: `AWS_MAX_ATTEMPTS=1`, or
`Config(retries={'total_max_attempts': 1})` in Boto3 (a `Config`'s `max_attempts` counts retries only, so 1 still allows two
calls). Record the retry layer in the register.
:::

3. A breaker keeps Resilience4j's default window and minimum of 100 calls; the dependency is called once a second and starts timing
   out. How long before the breaker can open?

::: answer Show answer for question 3
A running breaker holds the last 100 outcomes, so it opens once 50 are failures: about 50 seconds, plus the last call's timeout.
Just after a start or reset it needs 100 calls: about 100 seconds.[^r4jcb] Time to open = failures needed ÷ failing calls per
second. Use a smaller or time-based window, or a slow-call threshold, for low-volume calls.
:::

4. A job advert asks for an "AppOps engineer". What settles what the job is?

::: answer Show answer for question 4
Which meaning: application support for packaged systems,[^itilglossary] operability in a product team, or a vendor's "AppOps"
product.[^prodly2021] Ask what the person changes, who carries the pager and which systems they run.
:::

### Conclusion

"AppOps" is a weak label for strong practices. It means application support in one company, a product in another and a permission
screen on a phone, so say which you mean. The practices here are well documented: bound every wait, retry in one place
and safely, stop calling what is failing, keep failures local, tell the truth in health signals, make decisions visible,
and give the person on call a runbook and levers that work.

Most of this is invisible until a dependency misbehaves. That is the test: a well-operated service looks unremarkable on the worst
day of its dependencies, and its owners sleep through the incidents that do not need them.

## Sources and further reading

Every source below was opened on 2026-10-18 (rows marked 2026-10-19 re-checked in the post-score polish); dates are the source's
own. Claim-level support, fetch methods and caveats are in the Atlas evidence ledger (`research/appops-ledger.md`).

[^vogels2006]: [A Conversation with Werner Vogels](https://queue.acm.org/detail.cfm?id=1142065) — interview by Jim Gray, ACM Queue 4(4), 2006-06-30 (read end-to-end from the Internet Archive snapshot of 2025-12-01; the live page blocks automated access). Checked 2026-10-19.
[^itilglossary]: [ITIL glossary and abbreviations, English](https://htmlscript.auburn.edu/oit/ITILV3_Glossary.pdf) — © Crown Copyright 2011, copy hosted by Auburn University. Checked 2026-10-18.
[^itpmitil4]: [ITIL 4](https://wiki.en.it-processmaps.com/index.php/ITIL_4) — IT Process Maps wiki, last edited 2024-02-18 (secondary summary). Checked 2026-10-18.
[^peoplecertitil]: [ITIL certifications](https://www.peoplecert.org/browse-certifications/it-governance-and-service-management/ITIL-1) — PeopleCert, as of 2026-10-18. Checked 2026-10-18.
[^devopscom2016]: [AppOps: The Next Big Thing or Just the Next Buzzword?](https://devops.com/appops-next-big-thing-just-next-buzzword/) — Tony Bradley, DevOps.com, 2016-07-07 (paraphrased; automated extract only). Checked 2026-10-18.
[^tns2021]: [Why an AppOps Model Empowers Developers](https://thenewstack.io/why-an-appops-model-empowers-developers/) — Vivek Pandey (Shipa), The New Stack, sponsored post, 2021-09-17. Checked 2026-10-18.
[^prodly2021]: [AppOps: NextGen DevOps Built for Admins](https://automationchampion.com/2021/08/03/appops-nextgen-devops-built-for-admins/) — Rakesh Gupta, Automation Champion, 2021-08-03 (paraphrased; automated extract only). Checked 2026-10-18.
[^prodlyhome]: [Prodly home page](https://prodly.co/) — Prodly, as of 2026-10-18. Checked 2026-10-18.
[^androidappops]: [AppOpsManager](https://developer.android.com/reference/android/app/AppOpsManager) — Android Developers API reference, last updated 2026-08-28. Checked 2026-10-18.
[^commonsware2013]: [App Ops Developer FAQ](https://commonsware.com/blog/2013/07/26/app-ops-developer-faq.html) — Mark Murphy, CommonsWare, 2013-07-26 (updated 2013-11-26). Checked 2026-10-18.
[^srecascade]: [Addressing Cascading Failures](https://sre.google/sre-book/addressing-cascading-failures/) — Mike Ulrich, *Site Reliability Engineering* (Google, O'Reilly, 2016). Checked 2026-10-18.
[^sreoverload]: [Handling Overload](https://sre.google/sre-book/handling-overload/) — Alejandro Forero Cuervo, *Site Reliability Engineering* (2016). Checked 2026-10-18.
[^sreintro]: [Introduction](https://sre.google/sre-book/introduction/) — Benjamin Treynor Sloss, *Site Reliability Engineering* (2016). Checked 2026-10-18.
[^sreoncall]: [Being On-Call](https://sre.google/sre-book/being-on-call/) — Andrea Spadaccini, *Site Reliability Engineering* (2016). Checked 2026-10-18.
[^sreprr]: [The Evolving SRE Engagement Model](https://sre.google/sre-book/evolving-sre-engagement-model/) — Acacio Cruz, Ashish Bhambhani, *Site Reliability Engineering* (2016). Checked 2026-10-18.
[^bltimeouts]: [Timeouts, retries, and backoff with jitter](https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter) — Marc Brooker, Amazon Builders' Library, 2019. Checked 2026-10-18.
[^blhealth]: [Implementing health checks](https://builder.aws.com/content/3Ev53O39izHCtWLzp4XU6t8PC1O/implementing-health-checks) — David Yanacek, Amazon Builders' Library, 2019. Checked 2026-10-18.
[^blshedding]: [Using load shedding to avoid overload](https://builder.aws.com/content/3Eun1EEyX6p2e3VYNyRLSJzLuMV/using-load-shedding-to-avoid-overload) — David Yanacek, Amazon Builders' Library, 2019. Checked 2026-10-18.
[^blfallback]: [Avoiding fallback in distributed systems](https://builder.aws.com/content/3EuS9Sakq7L3VLQIF3qzfMfke1Y/avoiding-fallback-in-distributed-systems) — Jacob Gabrielson, Amazon Builders' Library, 2019. Checked 2026-10-18.
[^brookerjitter]: [Exponential Backoff And Jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) — Marc Brooker, AWS Architecture Blog, 2015-03-04 (updated May 2023; the formulas are images, transcribed from the figures). Checked 2026-10-19.
[^fowlercb]: [Circuit Breaker](https://martinfowler.com/bliki/CircuitBreaker.html) — Martin Fowler, 2014-03-06. Checked 2026-10-18.
[^releaseit]: [Release It! Second Edition](https://pragprog.com/titles/mnee2/release-it-second-edition/) — Michael Nygard, Pragmatic Bookshelf, 2018-01. Checked 2026-10-18.
[^hodgsontoggles]: [Feature Toggles (aka Feature Flags)](https://martinfowler.com/articles/feature-toggles.html) — Pete Hodgson, martinfowler.com, 2017-10-09. Checked 2026-10-18.
[^azurecb]: [Circuit Breaker pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker) — Microsoft Azure Architecture Center, 2025-02-05. Checked 2026-10-18.
[^azurebulkhead]: [Bulkhead pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead) — Microsoft Azure Architecture Center, 2026-03-19. Checked 2026-10-18.
[^azurehealth]: [Health Endpoint Monitoring pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/health-endpoint-monitoring) — Microsoft Azure Architecture Center, 2026-09-24. Checked 2026-10-18.
[^warel05bp01]: [REL05-BP01 Implement graceful degradation to transform applicable hard dependencies into soft dependencies](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_mitigate_interaction_failure_graceful_degradation.html) — AWS Well-Architected Framework, Reliability Pillar. Checked 2026-10-18.
[^warel05bp03]: [REL05-BP03 Control and limit retry calls](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_mitigate_interaction_failure_limit_retries.html) — AWS Well-Architected Framework, Reliability Pillar. Checked 2026-10-18.
[^warel05bp05]: [REL05-BP05 Set client timeouts](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_mitigate_interaction_failure_client_timeouts.html) — AWS Well-Architected Framework, Reliability Pillar. Checked 2026-10-18.
[^warel05bp07]: [REL05-BP07 Implement emergency levers](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_mitigate_interaction_failure_emergency_levers.html) — AWS Well-Architected Framework, Reliability Pillar. Checked 2026-10-18.
[^waops07]: [OPS07-BP03 Use runbooks to perform procedures](https://docs.aws.amazon.com/wellarchitected/latest/operational-excellence-pillar/ops_ready_to_support_use_runbooks.html) — AWS Well-Architected Framework, Operational Excellence Pillar. Checked 2026-10-18.
[^awsorr]: [Operational Readiness Reviews (ORR)](https://docs.aws.amazon.com/wellarchitected/latest/operational-readiness-reviews/wa-operational-readiness-reviews.html) — AWS Well-Architected, 2022-06-30. Checked 2026-10-18.
[^aws2021]: [Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/12721/) — Amazon Web Services, 2021-12-10. Checked 2026-10-18.
[^eeybiyri]: [What is You Build It You Run It](https://playbooks.equalexperts.com/you-build-it-you-run-it/what-is-you-build-it-you-run-it) — Bethan Timmins, Steve Smith, Equal Experts playbook, undated. Checked 2026-10-18.
[^eepitfalls]: [Pitfalls](https://playbooks.equalexperts.com/you-build-it-you-run-it/pitfalls) — Equal Experts, You Build It You Run It playbook, undated. Checked 2026-10-18.
[^eescale]: [Scale](https://playbooks.equalexperts.com/you-build-it-you-run-it/practices/scale) — Equal Experts, You Build It You Run It playbook, undated. Checked 2026-10-18.
[^rfc9110]: [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html) — R. Fielding, M. Nottingham, J. Reschke (eds.), IETF, 2022-06. Checked 2026-10-18.
[^rfc6585]: [RFC 6585: Additional HTTP Status Codes](https://www.rfc-editor.org/rfc/rfc6585.html) — M. Nottingham, R. Fielding, IETF, 2012-04. Checked 2026-10-18.
[^stripeidem]: [Idempotent requests](https://docs.stripe.com/api/idempotent_requests) — Stripe API reference. Checked 2026-10-18.
[^grpcdeadlines]: [Deadlines](https://grpc.io/docs/guides/deadlines/) — gRPC documentation. Checked 2026-10-18.
[^requests]: [Advanced Usage: Timeouts](https://requests.readthedocs.io/en/latest/user/advanced/) — Requests documentation (latest). Checked 2026-10-18.
[^awssdkretry]: [Retry behavior](https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html) — AWS SDKs and Tools Reference Guide. Checked 2026-10-19.
[^awsretry2026]: [Announcing updated retry behavior for AWS SDKs and tools](https://aws.amazon.com/blogs/developer/announcing-updated-retry-behavior-for-aws-sdks-and-tools/) — Matthew Miller, AWS Developer Tools Blog, 2026-05-20. Checked 2026-10-19.
[^boto3retries]: [Retries](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html) — Boto3 1.43.114 documentation. Checked 2026-10-18.
[^dotnetresilience]: [Build resilient HTTP apps: Key development patterns](https://learn.microsoft.com/en-us/dotnet/core/resilience/http-resilience) — Microsoft Learn, .NET, 2026-02-24. Checked 2026-10-18.
[^r4jcb]: [CircuitBreaker](https://resilience4j.readme.io/docs/circuitbreaker) — Resilience4j documentation. Checked 2026-10-18.
[^r4jretry]: [Retry](https://resilience4j.readme.io/docs/retry) — Resilience4j documentation. Checked 2026-10-18.
[^r4jtimeout]: [TimeLimiter](https://resilience4j.readme.io/docs/timeout) — Resilience4j documentation. Checked 2026-10-18.
[^r4jrelease]: [Release v2.4.0](https://github.com/resilience4j/resilience4j/releases/tag/v2.4.0) — Resilience4j, GitHub, 2026-03-14 (adds Spring Boot 4 support). Checked 2026-10-18.
[^r4jspring]: [Getting Started: Spring Boot](https://resilience4j.readme.io/docs/getting-started-3) — Resilience4j documentation. Checked 2026-10-18.
[^r4jmicrometer]: [Micrometer](https://resilience4j.readme.io/docs/micrometer) — Resilience4j documentation, updated about four years before checking; metric table current, sample output not. Checked 2026-10-18.
[^envoycb]: [Circuit breaking](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/circuit_breaking) — Envoy proxy documentation (latest). Checked 2026-10-18.
[^istiomesh]: [Global Mesh Options](https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/) — Istio documentation (latest, when 1.31.2 was current). Checked 2026-10-18.
[^hystrix]: [Netflix/Hystrix README](https://github.com/Netflix/Hystrix) — Netflix, GitHub. Checked 2026-10-18.
[^concurrencylimits]: [Netflix/concurrency-limits README](https://github.com/Netflix/concurrency-limits) — Netflix, GitHub. Checked 2026-10-18.
[^k8sprobes]: [Liveness, Readiness, and Startup Probes](https://kubernetes.io/docs/concepts/workloads/pods/probes/) — Kubernetes documentation, last modified 2026-06-30. Checked 2026-10-18.
[^k8spodlifecycle]: [Pod Lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) — Kubernetes documentation, last modified 2026-07-27. Checked 2026-10-18.
[^k8ssecrets]: [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) — Kubernetes documentation, last modified 2026-07-23. Checked 2026-10-18.
[^k8spss]: [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) — Kubernetes documentation, last modified 2026-10-16. Checked 2026-10-18.
[^springprobes]: [Endpoints: Kubernetes Probes](https://docs.spring.io/spring-boot/reference/actuator/endpoints.html) — Spring Boot reference documentation (current). Checked 2026-10-18.
[^springcloud]: [Deploying to the Cloud: Kubernetes](https://docs.spring.io/spring-boot/how-to/deployment/cloud.html) — Spring Boot documentation (current). Checked 2026-10-18.
[^springloggers]: [Loggers (loggers)](https://docs.spring.io/spring-boot/api/rest/actuator/loggers.html) — Spring Boot Actuator API documentation (current). Checked 2026-10-18.
[^springsupport]: [Spring Boot support generations](https://api.spring.io/projects/spring-boot/generations) — Spring, the data behind the project's support page. Checked 2026-10-18.
[^12fconfig]: [III. Config](https://12factor.net/config) — The Twelve-Factor App. Checked 2026-10-18.
[^12fdisposability]: [IX. Disposability](https://12factor.net/disposability) — The Twelve-Factor App. Checked 2026-10-18.
[^owaspsecrets]: [Secrets Management Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html) — OWASP Cheat Sheet Series. Checked 2026-10-18.
[^otelinstr]: [Instrumentation](https://opentelemetry.io/docs/concepts/instrumentation/) — OpenTelemetry documentation. Checked 2026-10-18.
[^otelhttp]: [Semantic conventions for HTTP metrics](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/) — OpenTelemetry. Checked 2026-10-18.
[^otelhttpspans]: [Semantic conventions for HTTP spans](https://opentelemetry.io/docs/specs/semconv/http/http-spans/) — OpenTelemetry. Checked 2026-10-18.
[^otellogs]: [Logs](https://opentelemetry.io/docs/concepts/signals/logs/) — OpenTelemetry documentation, last modified 2026-08-07. Checked 2026-10-18.
[^otelexemplars]: [Metrics data model: Exemplars](https://opentelemetry.io/docs/specs/otel/metrics/data-model/) — OpenTelemetry specification. Checked 2026-10-18.
[^otelvendors]: [Vendors](https://opentelemetry.io/ecosystem/vendors/) — OpenTelemetry ecosystem (list frozen at the time of checking). Checked 2026-10-18.
[^promalerting]: [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) — Prometheus documentation. Checked 2026-10-18.
[^little]: [Little's law](https://en.wikipedia.org/wiki/Little%27s_law) — Wikipedia. Checked 2026-10-18.

**Changelog.** 2026-10-18, r0: first draft; term status confirmed as ambiguous; three labs and the policy check executed.
2026-10-18, r1: first-pass fact check (`reviews/appops-r1-superseded-20261018T0506.md`: 0 critical, 5 major, 12 minor; response in
`reviews/appops-r1-response.md`): ordered shutdown with `preStop`, dated AWS SDK, .NET and Istio retry defaults, current
Resilience4j metrics and a *not closed* alert, a diagnosability playbook, corrected breaker timing, Spring Boot 4.1,
15 new sources.
2026-10-19, r2: second-pass review (`reviews/appops-r1.md`: 0 critical, 0 major, 6 minor; response in
`reviews/appops-r1-response.md`): fallback attribution (Amazon's reason is provability), `preStop` sizing, ITIL 5 dated "as of",
*n* defined as teams on call, specifications sentence, ticket note dated to 5 minutes after open.
2026-10-19, r3 (post-score polish): re-verified the fast-aging rows (Vogels interview end-to-end via the Wayback copy; AWS jitter
formula read from the blog's equation image and the SDK reference guide; SDK retry defaults; PromQL re-parsed on the demo, now
Prometheus 3.14.0); added a non-JVM pointer; trimmed into the 4,000–9,000 band without changing claims.