AppOps

AppOps (application operations) is the work of keeping one application or service healthy in production, done by the people who own its code. In this Atlas it means operability built into the code: the service reports honestly whether it can take traffic, bounds every wait on another system, retries only when retrying helps, degrades instead of collapsing, exposes levers for the on-call engineer, and ships with a runbook its owners keep current. Werner Vogels described the ownership model in 2006: “You build it, you run it. This brings developers into contact with the day-to-day operation of their software.”^[1]

Where to start. Beginners: glossary, Lab 1; engineers: core practices; leaders: decision framework. Last verified 2026-10-19.

The SRE book defines a cascading failure as “a failure that grows over time as a result of positive feedback”.^[11] Application code supplies much of that feedback: calls without timeouts empty thread pools, retries pile load on a drowning dependency, and a liveness check on a slow database restarts every replica. Marc Brooker of Amazon: “Retries are “selfish.”” They spend the server’s time on one client’s chances, harmful during overload.^[12]

• Not Android App Ops (mobile delivery is in MobileOps), and not a product category: libraries and meshes implement patterns that someone must choose, tune and test.
• Not SRE under another name. SRE sets targets, budgets and on-call policy; AppOps is what the code and its owners do to meet them. Not monitoring: telemetry shows that a dependency is slow; it does not stop your threads waiting on it.
• Not an unconditional pager for developers. Who answers at night is an operating-model decision with costs (operating model).

2006: Vogels states the ownership model behind this page, “You build it, you run it”.^[1] 2014: Fowler credits Nygard’s Release It! with popularising the circuit breaker.^[13] 2016: Google’s SRE book on cascading failures and overload.^[11][14] 2019: Amazon Builders’ Library on timeouts, health checks, shedding and fallback.^[12][15]

Mental model: a remote call has three outcomes, not two. It succeeds, fails, or does not answer. The patterns handle the third: how long to wait (deadline), whether to try again (retry policy), whether to keep trying (breaker), how much of the service may wait (bulkhead), and what the user gets meanwhile (degradation).

flowchart TB
  IN["Request arrives with a deadline"] --> ADM{"Over capacity?"}
  ADM -- "yes" --> SHED["Shed: fast 503 with Retry-After"]
  ADM -- "no" --> CB{"Breaker closed?"}
  CB -- "no" --> DEG["Degrade or fail fast"]
  CB -- "yes" --> BH{"Bulkhead slot free?"}
  BH -- "no" --> DEG
  BH -- "yes" --> CALL["Call with the remaining deadline"]
  CALL -- "success" --> OK["Respond"]
  CALL -- "error or timeout" --> RT{"Safe and allowed to retry?"}
  RT -- "yes" --> BO["Back off with jitter"]
  RT -- "no" --> GIVE["Give up: degrade or fail"]
  BO -- "retry" --> CB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class IN,CALL,OK,BO core
  class SHED,DEG,GIVE risk
  class ADM,BH,CB,RT decision

Each line is an editorial distinction, chosen so that each topic is taught once in the Atlas.

flowchart LR
  subgraph IN["Feeds AppOps"]
    SRE["SRE: targets, budgets"]
    SVC["ServiceOps: owner, tier"]
    PLAT["PlatformOps: defaults"]
  end
  APP["AppOps: resilience in code, levers, runbook"]
  subgraph OUT["AppOps hands work to"]
    OBS["ObservabilityOps: telemetry"]
    INC["IncidentOps: coordination"]
    REL["ReleaseOps: exposure"]
    K8S["KubernetesOps: probes, scaling"]
  end
  SRE --> APP
  SVC --> APP
  PLAT --> APP
  APP --> OBS
  APP --> INC
  APP --> REL
  APP --> K8S
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class APP core
  class SRE,SVC,PLAT,OBS,INC,REL,K8S neutral

This Atlas’s proposal; no AppOps manifesto exists. Mission: every service keeps its promises when its dependencies do not, and whoever is on call can steer it without reading the code.

1. Bound every wait: a deadline per request; per-attempt timeouts from measured latency, not library defaults.
2. Retry in one place, only what is safe, with backoff, jitter and a budget.
3. Fail fast and keep failure local with breakers and bulkheads.
4. Health signals tell the truth to their audience: liveness to the restarter, readiness to the load balancer.
5. Prefer the path that always runs: degraded modes stay simple and are exercised on purpose.
6. No page without a runbook and a tested lever. Google reports that a written playbook gives “roughly a 3x improvement in MTTR”.^[17]
7. Owners operate, within a load they can sustain, funded and sized by the organisation.

Non-goals: SLO targets, the telemetry platform and major-incident command belong to SRE, ObservabilityOps and IncidentOps.

Five building blocks carry the discipline: a dependency register with policies (Artefact 1), the same policies in library or mesh configuration (Artefact 3), a truthful health model and lifecycle (Lab 1, Artefact 4), diagnosability signals for the code’s decisions (playbook and metrics below), and alerts that link the runbook (Artefact 5).

R = responsible, A = accountable, C = consulted, I = informed — a proposal to adapt.

The service owner is accountable: funding and the feature-versus-operability trade-off are theirs. When an operations manager stays accountable while product teams carry the pager, Equal Experts calls it “responsible but unaccountable”, with inflated availability targets and operability losing to features.^[18]

At Google, “A PRR is considered a prerequisite for an SRE team to accept responsibility for managing the production aspects of a service”;^[19] AWS’s Operational Readiness Review distils “the learnings from AWS operational incidents into curated questions with best practice guidance”.^[20] Both work as question banks even when no SRE team takes over.

Three models differ in who answers when the service breaks, not in whether the patterns are needed.

Equal Experts’ playbook keeps operations teams for “self-hosted foundational systems, such as COTS applications” (commercial off-the-shelf),^[21] warns of linear run costs (standby cost R(n), where n is the number of teams on call) and advises “Fund on-call costs from product team budgets”, choosing out-of-hours cover by financial exposure (95.0% no night rota, 99.0% a domain rota, 99.9% the team’s own).^[18][22]

Sustainable load is arithmetic: Google caps on-call at 25% of an SRE’s time, needs eight engineers for a single-site 24/7 rota, and sets two incidents per 12-hour shift as the upper bound.^[23] A team of five cannot run 24/7 on those terms. Governance: the register check runs in CI (Artefact 2), readiness reviews precede launches, a monthly operability review runs, and every lever has an owner and a test date.

flowchart TB
  LB["Load balancer or orchestrator"]
  subgraph SVC["checkout-api process"]
    ADM["Admission and shedding"]
    HE["Health: /livez, /readyz"]
    H["Handlers set a deadline"]
    LEV["Levers: flags, config"]
    OUT["Outbound clients: timeout, retry, breaker, bulkhead"]
  end
  PAY["payments (hard)"]
  INV["inventory (hard)"]
  REC["recommendations (soft)"]
  LB --> ADM
  LB -. "probes" .-> HE
  ADM --> H
  H --> OUT
  LEV -.-> OUT
  OUT --> PAY
  OUT --> INV
  OUT --> REC
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class ADM,H,OUT,HE core
  class LEV decision
  class LB,PAY,INV,REC neutral

sequenceDiagram
  participant U as User
  participant A as checkout-api
  participant R as recommendations
  U->>A: GET /checkout
  A->>R: call, 200 ms timeout
  Note over R: stops answering
  A-->>U: page without the panel
  Note over A: breaker opens
  U->>A: GET /checkout
  A-->>U: reduced page, no call
  A->>R: 3 trial calls after 5 s
  Note over A: a trial times out: open again
  A->>R: rounds every 5 s keep failing
  Note over A: not closed 5 min after open: ticket
  A->>R: 3 trial calls, later
  R-->>A: trials succeed
  Note over A: breaker closes

Exception path: a hard dependency fails. Without payments, checkout fails fast with a 503 and Retry-After (“how long the user agent ought to wait before making a follow-up request”),^[25] stops retrying once the breaker opens, and pages. If the cause is outside the team, IncidentOps roles coordinate.

1. Find the defaults you live with. “By default, gRPC does not set a deadline”,^[26] and Python Requests does “not time out unless a timeout value is set explicitly”.^[27] AWS names “Not being aware of system timeouts or default timeouts” an anti-pattern.^[28]
2. Derive timeouts from latency. Amazon picks an acceptable false-timeout rate (say 0.1%) and uses the matching downstream percentile (p99.9), padding when p99.9 is close to p50.^[12]
3. Pass one deadline down. In the SRE book’s example a 30-second deadline leaves 23 seconds for the next hop after 7 seconds of work.^[11] gRPC propagates deadlines by default in Java and Go, not in C++.^[26]
4. Check the sum of sequential timeouts times attempts against the deadline (Artefact 2).

1. One layer. Three layers of four attempts produce “64 attempts (4^3) on the database” per user action;^[11] Amazon’s five-deep stack reaches 243 times the load, and it retries “at a single point in the stack”.^[12]
2. Count retries you did not write (as of October 2026). AWS SDKs in standard mode make three attempts (DynamoDB four), but Java, Python, Ruby, PHP, C++ and the CLI still default to legacy mode until the switch announced for November 2026;^[29][30] Boto3’s legacy default is five.^[31] .NET’s standard handler retries three times by default;^[32] Istio’s mesh-wide default retries twice.^[33] Pin what you rely on; a loop around any of them is a second layer.
3. Only what is safe to repeat. Side-effecting APIs “aren’t safe to retry unless they provide idempotency”;^[12] idempotency keys let a client “safely repeat the request without risk of creating a second object or performing the update twice”.^[34]
4. Backoff with jitter: “Always use randomized exponential backoff when scheduling retries”;^[11] “Full Jitter” sleeps random_between(0, min(cap, base * 2 ** attempt)).^[35]
5. A budget. Google caps each request at three attempts and retries per client only while retries stay below 10% of requests. In the book’s worst-case example the per-request cap alone allows almost 3x growth; the budget brings it to “just 1.1x in the general case”.^[14]
6. Not what will not change (permission, configuration errors);^[36] honour 429^[37] and 503 with Retry-After.^[25]

Above its failure threshold a breaker trips and all further calls fail at once, without the protected call being made.^[13] Its states are Closed (calls flow, failures counted), Open (calls fail at once) and Half-Open (a few trial calls decide).^[38]

• Tune for your traffic. Resilience4j’s defaults keep the last 100 calls, open at a 50% failure rate and wait 60 seconds (October 2026).^[39] Time to open is failures needed ÷ failing calls per second: a full window opens after 50 failures (25 s at two calls a second); a fresh one needs 100 calls. Count slow calls if slowness is the risk.
• Decide what open means: degrade for a soft dependency, fail fast with a clear error for a hard one.
• Keep breaker state out of liveness. Resilience4j’s Spring Boot health indicators are off by default “because the application status is DOWN, when a CircuitBreaker is OPEN”.^[16]

Bulkheads, named after a ship’s hull partitions, give each dependency a bounded pool.^[40] Size them with Little’s law (in-flight = arrival rate × time in system):^[41] 50 calls a second at 100 ms need about 5 slots, so a limit of 20 leaves headroom and caps a hang at 20 stuck calls.

Load shedding “drops some proportion of load by dropping traffic as the server approaches overload conditions”; graceful degradation “takes the concept of load shedding one step further by reducing the amount of work that needs to be performed”.^[11]

1. Shed at admission with a cheap 503 and Retry-After, least important work first (Google’s criticality levels run from CRITICAL_PLUS to SHEDDABLE).^[14]
2. Write and test the degraded response for each soft dependency; AWS calls skipping that test an anti-pattern.^[42]
3. Keep it rare and simple: “the code path you never use is the code path that (often) doesn’t work”.^[11]

Google separates process health (“is this binary responding at all?”) from service health (able to respond to this class of requests right now?).^[11] Kubernetes liveness probes “determine when to restart a container”, readiness probes “determine when a container is ready to accept traffic”, and wrong liveness probes “can lead to cascading failures”.^[43]

• Liveness is process-local: it “should not depend on health checks for external systems” (Spring Boot), or a database outage can restart every instance;^[44] Microsoft agrees.^[45]
• Readiness for a shared dependency is a judgement call (Spring Boot’s words): include it and all instances leave rotation together; leave it out and handle failure with a breaker.^[44] Some load balancers fail open when every server fails;^[15] a Kubernetes Service does not — with no ready pod it refuses connections, with no 503 at all.^[44]
• Beware the fast-failing server: under “least requests” balancing it attracts traffic and becomes a “black hole”.^[15]
• Start unready; stop in three steps: keep serving while routers stop sending traffic, stop listening, then finish in-flight work and exit,^[46] within the grace period (30 s by default in Kubernetes).^[47] Kubernetes removes a terminating pod from Service endpoints as shutdown begins, whatever readiness says, but traffic can still arrive for a while: add a preStop sleep before SIGTERM, long enough for routing to stop and for the longest in-flight request, and a grace period covering sleep plus drain (Spring Boot’s guide uses the built-in sleep action from Kubernetes 1.32).^[48] Probe settings: KubernetesOps.

Levers “are rapid processes that can mitigate availability impact on your workload” by “disabling, throttling, or changing the behavior of components or dependencies using known and tested mechanisms” (AWS).^[49] Hodgson calls the long-lived ones “Kill Switches”.^[50] Typical levers switch off a soft dependency, force a breaker open, lower a rate limit or cut retries to zero; each needs an owner, an access rule, a runbook entry and a test date. Flag systems: ReleaseOps.

When prevention fails, owners must explain what the code decided. Zero-code instrumentation sees “the edges of your application”; decisions need code-based instrumentation.^[51] Pipelines and storage belong to ObservabilityOps.

1. Propagate trace context through every call, retries included. Each resent request is its own client span carrying http.request.resend_count,^[52] so one trace shows a retry storm’s fan-out.
2. Record decisions, not just outcomes. A breaker transition, a rejected or shed call and a degraded response each become a span event plus a log line with fixed fields (dependency, decision, reason, attempt); Resilience4j publishes state-transition events.^[39]
3. Put the trace ID in every log line; an active OpenTelemetry SDK correlates existing logs with the current trace and span.^[53]
4. Link metrics to traces. An exemplar attaches trace context to a metric observation,^[54] so a latency spike opens an example trace.
5. Plan a debug lever. Raising one logger’s level at runtime (Spring Boot’s /actuator/loggers)^[55] beats a redeploy; time-box it, restrict access and keep personal data out (PrivacyOps).

Example status: JSON syntax-checked; the field names are this Atlas’s convention.

{"time": "2026-10-18T03:12:07.412Z", "level": "WARN", "service": "checkout-api",
 "event": "dependency.degraded", "dependency": "recommendations",
 "decision": "skip_call", "reason": "breaker_open", "attempt": 0,
 "route": "GET /checkout", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"}

Only the RFCs and the OpenTelemetry semantic conventions are specifications; the rest are recommendations to adapt.

Choose the category first; brands change faster than patterns. As of October 2026:

Envoy “enforces circuit breaking limits at the network level as opposed to having to configure and code each application independently”,^[59] but cannot know which call is safe to repeat; with both layers, make sure only one retries.

Automation makes the policies dependable: the check runs in CI, and budgets, breakers and adaptive throttling act without a human.^[14] Repeated runbook steps can be automated with AutomationOps’s safeguards. This page’s view on AI: models can draft runbook entries and summarise traces, but production levers stay with a human or a tested rule. Event correlation: AIOps.

A fictional service, checkout-api, with placeholder names and URLs. The register and its check need only Python; the others assume Kubernetes, Prometheus and Spring Boot 4.1 with Resilience4j 2.4 (October 2026: open-source support for Spring Boot 3.5 ended on 2026-06-30, and Resilience4j added a Spring Boot 4 module in 2.4.0).^[61][62] Non-JVM stacks map the same policies across: .NET’s resilience pipelines appear in the tooling table, and the register and its checker (Artefacts 1–2) are language-neutral.

One record per outbound dependency, reviewed with the code. sequential marks calls made one after another on the request path; sdk_attempts and mesh_attempts record attempts a client library or sidecar already makes (the AWS SDK pinned to standard mode, mesh retries off).

Example status: JSON syntax-checked; input of Artefact 2, which was executed.

{
  "service": "checkout-api",
  "owner": "team-checkout",
  "inbound_deadline_ms": 2000,
  "dependencies": [
    {
      "name": "payments", "criticality": "hard", "sequential": true,
      "timeout_ms": 800, "max_attempts": 2, "sdk_attempts": 1,
      "mesh_attempts": 1, "idempotent": false, "idempotency_key": true,
      "breaker": true, "runbook": "runbooks/checkout.md#payments"
    },
    {
      "name": "inventory", "criticality": "hard", "sequential": true,
      "timeout_ms": 150, "max_attempts": 3, "sdk_attempts": 1,
      "mesh_attempts": 1, "idempotent": true, "breaker": true,
      "runbook": "runbooks/checkout.md#inventory"
    },
    {
      "name": "recommendations", "criticality": "soft", "sequential": false,
      "timeout_ms": 200, "max_attempts": 1, "sdk_attempts": 1,
      "mesh_attempts": 1, "idempotent": true, "breaker": true,
      "degraded_behaviour": "render the page without the panel",
      "runbook": "runbooks/checkout.md#recommendations"
    },
    {
      "name": "receipt-store", "criticality": "soft", "sequential": false,
      "timeout_ms": 1000, "max_attempts": 2, "sdk_attempts": 3,
      "mesh_attempts": 1, "idempotent": true, "breaker": false,
      "runbook": "runbooks/checkout.md#receipts"
    }
  ]
}

The charter as build-enforceable rules: one retry layer (app, SDK or mesh), no retried side effects without a key, a degraded behaviour per soft dependency, sequential worst case within the inbound deadline.

Example status: executed on 2026-10-18 with Python 3.8.10 against Artefact 1 (exit code 1, output below).

#!/usr/bin/env python3
"""Check a dependency register against the team's resilience policy."""
import json
import sys

REQUIRED = ("name", "criticality", "timeout_ms", "max_attempts",
            "sdk_attempts", "mesh_attempts", "idempotent", "breaker",
            "runbook")
LAYERS = {"max_attempts": "app", "sdk_attempts": "SDK",
          "mesh_attempts": "mesh"}


def check(reg):
    problems, budget_ms = [], 0
    for d in reg["dependencies"]:
        name = d.get("name", "?")
        missing = [k for k in REQUIRED if k not in d]
        if missing:
            problems.append(f"{name}: missing {', '.join(missing)}")
            continue
        attempts = d["max_attempts"] * d["sdk_attempts"] * d["mesh_attempts"]
        retrying = [label for k, label in LAYERS.items() if d[k] > 1]
        if len(retrying) > 1:
            problems.append(f"{name}: retries at {len(retrying)} layers "
                            f"({' x '.join(retrying)} = {attempts} attempts)")
        if (attempts > 1 and not d["idempotent"]
                and not d.get("idempotency_key")):
            problems.append(f"{name}: retried but not idempotent")
        if d["criticality"] == "soft" and not d.get("degraded_behaviour"):
            problems.append(f"{name}: soft dependency without a "
                            "degraded behaviour")
        if d.get("sequential"):
            budget_ms += d["timeout_ms"] * attempts
    if budget_ms > reg["inbound_deadline_ms"]:
        problems.append(f"worst case {budget_ms} ms of sequential calls "
                        f"exceeds the {reg['inbound_deadline_ms']} ms "
                        "inbound deadline")
    return problems


if __name__ == "__main__":
    with open(sys.argv[1], encoding="utf-8") as fh:
        found = check(json.load(fh))
    for p in found:
        print("POLICY", p)
    print(f"{len(found)} problem(s)")
    sys.exit(1 if found else 0)

python3 check_policy.py dependencies.json prints:

POLICY receipt-store: retries at 2 layers (app x SDK = 6 attempts)
POLICY receipt-store: soft dependency without a degraded behaviour
POLICY worst case 2050 ms of sequential calls exceeds the 2000 ms inbound deadline
3 problem(s)

All three are real design bugs: receipt-store makes six calls per failure and has no degraded behaviour, and payments (800 ms × 2) plus inventory (150 ms × 3) can take 2,050 ms of the 2,000 ms deadline before any backoff sleep. Fixes: drop the application retry, queue receipts, cut inventory to two attempts.

The register as Resilience4j configuration (keys from its documentation;^[16] maxAttempts counts the first call^[63]). TimeLimiter bounds only calls that return a CompletableFuture,^[64] so the blocking-call timeouts belong in each HTTP client. Retry has no ratio budget: the breaker stops retries in an outage, and Envoy offers retry budgets.^[59] In the default aspect order the bulkhead sits inside the breaker,^[16] so the breaker ignores bulkhead rejections instead of counting them as dependency failures.

Example status: YAML syntax-checked; not executed (no Spring application here). Property prefixes checked against the resilience4j-spring-boot4 2.4.0 sources.

# application.yml: Spring Boot 4.1 with resilience4j-spring-boot4 2.4.0
management:
  endpoint:
    health:
      probes:
        add-additional-paths: true    # /livez and /readyz on the main port
spring:
  lifecycle:
    timeout-per-shutdown-phase: 15s   # drain; the pod's grace period covers it
resilience4j:
  circuitbreaker:
    configs:
      checkout:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 20
        minimumNumberOfCalls: 10
        failureRateThreshold: 50
        waitDurationInOpenState: 5s
        permittedNumberOfCallsInHalfOpenState: 3
        registerHealthIndicator: false  # an open breaker must not fail health
        ignoreExceptions:               # a full bulkhead is not a dependency fault
          - io.github.resilience4j.bulkhead.BulkheadFullException
    instances:
      payments: {baseConfig: checkout}
      inventory: {baseConfig: checkout}
      recommendations:
        baseConfig: checkout
        slowCallDurationThreshold: 150ms
        slowCallRateThreshold: 50
  retry:
    instances:
      payments:                       # idempotency key created outside the retry
        maxAttempts: 2                # includes the first call
        waitDuration: 100ms
        enableRandomizedWait: true
        randomizedWaitFactor: 0.5
        retryExceptions: [java.io.IOException]
      inventory:
        maxAttempts: 3
        waitDuration: 50ms
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        enableRandomizedWait: true    # jitter: each wait varies by up to 50%
        randomizedWaitFactor: 0.5
        retryExceptions: [java.io.IOException]
  bulkhead:
    instances:
      recommendations:
        maxConcurrentCalls: 20
        maxWaitDuration: 0
  timelimiter:
    instances:
      recommendations:                # only for calls returning CompletableFuture
        timeoutDuration: 200ms
        cancelRunningFuture: true

The service on Kubernetes (placeholder image). /livez and /readyz are Spring Boot’s probe groups on the main port, enabled in Artefact 3 and used by Lab 1 too.^[44] The startup probe covers slow starts; readiness gates warm-up. The pod meets the Kubernetes Restricted security profile.^[65] Depth: KubernetesOps, KubernetesSecOps.

Example status: YAML syntax-checked; not applied to a cluster. Resource sizes are placeholders for a load test.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 3
  selector:
    matchLabels: {app: checkout-api}
  template:
    metadata:
      labels: {app: checkout-api}
    spec:
      terminationGracePeriodSeconds: 30   # >= preStop 10 s + drain 15 s
      automountServiceAccountToken: false # the app never calls the API server
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: app
          image: registry.example.com/checkout-api:1.4.2
          ports:
            - containerPort: 8080
          resources:                        # pools and bulkheads assume this
            requests: {cpu: 500m, memory: 768Mi}
            limits: {memory: 768Mi}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: [ALL]}
          volumeMounts:
            - {name: tmp, mountPath: /tmp}  # writable scratch for the server
          lifecycle:
            preStop:                    # routing stops + the longest in-flight request
              sleep: {seconds: 10}
          startupProbe:                     # slow start: up to 60 s to listen
            httpGet: {path: /livez, port: 8080}
            periodSeconds: 5
            failureThreshold: 12
          readinessProbe:                   # traffic on/off; may fail often
            httpGet: {path: /readyz, port: 8080}
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:                    # restart only if wedged
            httpGet: {path: /livez, port: 8080}
            periodSeconds: 10
            failureThreshold: 3
      volumes:
        - {name: tmp, emptyDir: {}}

Prometheus annotations “can be used to store longer additional information such as alert descriptions or runbook links”.^[66] An open breaker on a soft dependency is a ticket, not a page; SLO paging is in SRE. An alert fires only if its condition holds at every evaluation of the for window,^[66] and a long outage alternates open and half-open, so the first rule tests not closed.

Example status: YAML syntax-checked; both PromQL expressions parsed by the public Prometheus demo server (3.14.0) on 2026-10-19; rules not run with promtool. Metric names follow the metric table of the Resilience4j Micrometer page;^[67] its older sample output shows another counter name, so check your own /actuator/prometheus output.

groups:
  - name: checkout-api-dependencies
    rules:
      - alert: DependencyBreakerNotClosed
        # open and half_open alternate in a long outage: test "not closed"
        expr: min by (name) (resilience4j_circuitbreaker_state{state="closed"}) == 0
        for: 5m
        labels:
          severity: ticket            # degraded, not down: no page at night
        annotations:
          summary: "Breaker {{ $labels.name }} not closed for 5 minutes"
          runbook_url: "https://runbooks.example.com/checkout-api#breaker-open"
      - alert: DependencyCallsRejected
        expr: |
          sum by (name) (
            rate(resilience4j_circuitbreaker_not_permitted_calls_total[5m])
          ) > 5
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.name }}: over 5 calls/s short-circuited for 15 min"
          runbook_url: "https://runbooks.example.com/checkout-api#breaker-open"

Each lab runs on one machine with Python 3.8+, bash and curl, uses only the loopback interface and costs nothing. Outputs were produced on 2026-10-18.

Objective. Watch liveness and readiness disagree during warm-up, then shut down in order: readiness off, keep serving for DEREGISTER_S, stop listening, drain. Prerequisites. Linux or macOS, Python 3.8+, curl. Cost and risk. None; the server listens on 127.0.0.1:18080 only.

1. Save the server as health_server.py.

Example status: executed on 2026-10-18 (Python 3.8.10).

#!/usr/bin/env python3
"""Lab 1: liveness and readiness endpoints that tell the truth.

/livez  - process-local: 200 while the server can answer at all.
/readyz - 200 only when warmed up and not shutting down.
/work   - a 1.5-second request, to show draining on SIGTERM.
"""
import signal
import sys
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

WARMUP_S = 2.0          # e.g. loading caches or opening connection pools
DEREGISTER_S = 1.0      # keep serving while routers stop sending traffic
GRACE_S = 10.0          # stay below the platform's termination grace period
STARTED = time.monotonic()
state = {"draining": False, "in_flight": 0}
lock = threading.Lock()


def ready():
    warmed = time.monotonic() - STARTED >= WARMUP_S
    return warmed and not state["draining"]


class Handler(BaseHTTPRequestHandler):
    def reply(self, code, body):
        data = (body + "\n").encode()
        self.send_response(code)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        if self.path == "/livez":
            self.reply(200, "alive")
        elif self.path == "/readyz":
            if ready():
                self.reply(200, "ready")
            else:
                why = "draining" if state["draining"] else "warming up"
                self.reply(503, "not ready: " + why)
        elif self.path == "/work":
            with lock:
                state["in_flight"] += 1
            try:
                time.sleep(1.5)
                self.reply(200, "done")
            finally:
                with lock:
                    state["in_flight"] -= 1
        else:
            self.reply(404, "not found")

    def log_message(self, fmt, *args):
        sys.stderr.write("%s %s\n" % (self.command, self.path))


def main(port):
    server = ThreadingHTTPServer(("127.0.0.1", port), Handler)

    def stop_accepting_later():
        time.sleep(DEREGISTER_S)          # 2. still serving new requests
        server.shutdown()                 # 3. then stop the accept loop

    def on_sigterm(signum, frame):
        state["draining"] = True          # 1. readiness fails at once
        threading.Thread(target=stop_accepting_later, daemon=True).start()

    signal.signal(signal.SIGTERM, on_sigterm)
    print("listening on 127.0.0.1:%d" % port, flush=True)
    server.serve_forever(poll_interval=0.05)
    server.server_close()                 # new connections are now refused
    print("stopped listening", flush=True)
    deadline = time.monotonic() + GRACE_S
    while time.monotonic() < deadline:    # 4. finish in-flight work
        with lock:
            if state["in_flight"] == 0:
                break
        time.sleep(0.05)
    print("drained, exiting", flush=True)


if __name__ == "__main__":
    main(int(sys.argv[1]) if len(sys.argv) > 1 else 8080)

2. Save the driver as lab1.sh next to it and run bash lab1.sh.

Example status: executed on 2026-10-18; output below.

#!/usr/bin/env bash
set -euo pipefail
PORT=18080
probe() {  # usage: probe <path>
  curl -s -o /dev/null -w "$1 %{http_code}\n" "http://127.0.0.1:${PORT}$1" || true
}
python3 health_server.py "${PORT}" 2>server.log &
PID=$!
sleep 0.5
probe /livez; probe /readyz            # alive, still warming up
sleep 2
probe /readyz                          # ready
work() { curl -s "http://127.0.0.1:${PORT}/work" || echo "failed"; }
work > work1.out &
W1=$!
sleep 0.2
kill -TERM "${PID}"                    # the platform asks it to stop
sleep 0.2
probe /readyz                          # failing: routers stop sending
work > work2.out &
W2=$!                                  # a late request is still served
sleep 1.2
probe /readyz                          # 000: no longer listening
wait "${W1}" "${W2}"; wait "${PID}"; CODE=$?
echo "request sent before SIGTERM: $(cat work1.out)"
echo "request sent after SIGTERM: $(cat work2.out)"
echo "server exit code: ${CODE}"

Expected output.

listening on 127.0.0.1:18080
/livez 200
/readyz 503
/readyz 200
/readyz 503
stopped listening
/readyz 000
drained, exiting
request sent before SIGTERM: done
request sent after SIGTERM: done
server exit code: 0

Validation. Liveness stays 200; readiness goes 503 (warming up), 200, 503 after SIGTERM, yet a request sent after SIGTERM is served; a second later the server stops listening (000: connection refused), finishes both requests and exits 0. Exercise. Set DEREGISTER_S = 0 and watch the late request fail; make /livez fail on a pretend database flag and predict what a platform does to every replica.^[44] Troubleshooting. Address already in use: change PORT; an early 000: read server.log. Teardown. The server exits by itself (else kill %1); delete the files.

Objective. Reproduce the SRE book’s 4^3 amplification^[11] and see what one retry layer and a 10% budget^[14] change and cost. Prerequisites. Python 3.8+. Cost and risk. None (simulation).

1. Save the script as retry_lab.py.

Example status: executed on 2026-10-18 (Python 3.8.10); seeded, so your output should match exactly.

#!/usr/bin/env python3
"""Lab 2: measure retry amplification, then contain it.

A user action passes through three layers (browser code, front end,
back end) before it reaches a database. The database fails each
attempt with probability p. We count database attempts per action.
"""
import random

ACTIONS = 10_000


class Budget:
    """Per-client retry budget: retries stay below `ratio` of sends."""

    def __init__(self, ratio):
        self.ratio, self.sent, self.retries = ratio, 0, 0

    def allow_retry(self):
        return self.retries + 1 <= self.ratio * (self.sent + 1)


def call(layer, attempts, rng, p, stats, budget=None):
    """Call the layer below with up to attempts[layer] tries."""
    mine = budget if layer == 0 else None       # the top client owns it
    for i in range(attempts[layer]):
        if mine and i > 0:
            if not mine.allow_retry():
                return False
            mine.retries += 1
        if mine:
            mine.sent += 1
        if layer == len(attempts) - 1:          # next hop is the database
            stats["db"] += 1
            ok = rng.random() >= p
        else:
            ok = call(layer + 1, attempts, rng, p, stats, budget)
        if ok:
            return True
    return False


def run(name, attempts, p, budget_ratio=None):
    rng = random.Random(42)
    stats = {"db": 0}
    budget = Budget(budget_ratio) if budget_ratio else None
    ok = sum(call(0, attempts, rng, p, stats, budget)
             for _ in range(ACTIONS))
    print(f"{name:<28} p={p:<4} db attempts/action="
          f"{stats['db'] / ACTIONS:6.2f}  success={ok / ACTIONS:6.1%}")


JITTER_RNG = random.Random(7)


def full_jitter(attempt, base=0.1, cap=2.0):
    """AWS 'Full Jitter': random_between(0, min(cap, base * 2**attempt))."""
    return JITTER_RNG.uniform(0, min(cap, base * 2 ** attempt))


if __name__ == "__main__":
    for p in (0.05, 0.3, 1.0):
        run("A: 4 attempts at 3 layers", [4, 4, 4], p)
        run("B: 4 attempts at top only", [4, 1, 1], p)
        run("C: B plus 10% budget", [4, 1, 1], p, budget_ratio=0.1)
    print("full-jitter sleeps (s):",
          [round(full_jitter(a), 3) for a in range(6)])

2. Run python3 retry_lab.py. Expected output.

A: 4 attempts at 3 layers    p=0.05 db attempts/action=  1.05  success=100.0%
B: 4 attempts at top only    p=0.05 db attempts/action=  1.05  success=100.0%
C: B plus 10% budget         p=0.05 db attempts/action=  1.05  success=100.0%
A: 4 attempts at 3 layers    p=0.3  db attempts/action=  1.42  success=100.0%
B: 4 attempts at top only    p=0.3  db attempts/action=  1.41  success= 99.1%
C: B plus 10% budget         p=0.3  db attempts/action=  1.11  success= 78.2%
A: 4 attempts at 3 layers    p=1.0  db attempts/action= 64.00  success=  0.0%
B: 4 attempts at top only    p=1.0  db attempts/action=  4.00  success=  0.0%
C: B plus 10% budget         p=1.0  db attempts/action=  1.11  success=  0.0%
full-jitter sleeps (s): [0.032, 0.03, 0.26, 0.058, 0.857, 0.731]

Validation. At p=1.0: 64 attempts per action with layered retries, 4 with one layer, about 1.1 with the budget. At p=0.3 the budget costs success (78.2% against 99.1%): availability traded for overload protection. The simulation treats failures as independent of load; in real overload extra attempts raise p itself. Use your library’s retries: AWS calls custom retry code an anti-pattern when built-in support suffices.^[36] Troubleshooting. Different numbers mean a changed seed or ACTIONS. Teardown. Delete the file.

Objective. Compare timeout only with timeout plus breaker while a soft dependency hangs for 30 seconds, and find the breaker’s cost. Prerequisites. Python 3.8+; Lab 2. Cost and risk. None (simulated clock).

1. Save the script as breaker_lab.py.

Example status: executed on 2026-10-18 (Python 3.8.10); deterministic, so your output should match exactly.

#!/usr/bin/env python3
"""Lab 3: a circuit breaker and a timeout during a dependency brownout.

Simulated time, no sleeping. A checkout page calls a *soft* dependency
(recommendations) 100 times a second for 60 s. Between t=10 s and
t=40 s the dependency hangs. Each call has a 200 ms timeout; on any
failure the page degrades (renders without recommendations).
"""
from collections import deque

RATE, DURATION = 100, 60.0           # requests per second, seconds
TIMEOUT, HEALTHY_MS = 0.200, 0.040   # seconds
BROWNOUT = (10.0, 40.0)


class Breaker:
    """Count-based breaker: closed -> open -> half-open -> closed."""

    def __init__(self, window=20, min_calls=10, threshold=0.5,
                 open_s=5.0, trial_calls=3):
        self.window, self.min_calls = window, min_calls
        self.threshold, self.open_s = threshold, open_s
        self.trial_calls = trial_calls
        self.state, self.results, self.opened_at = "closed", deque(), 0.0
        self.trials = self.trial_ok = 0
        self.log = []

    def _move(self, now, state):
        self.log.append(f"t={now:5.2f}s {self.state} -> {state}")
        self.state = state

    def allow(self, now):
        if self.state == "open" and now - self.opened_at >= self.open_s:
            self._move(now, "half_open")
            self.trials = self.trial_ok = 0
        if self.state == "half_open":
            if self.trials < self.trial_calls:
                self.trials += 1
                return True
            return False
        return self.state == "closed"

    def record(self, now, ok):
        if self.state == "open":
            return                        # late result of an earlier call
        if self.state == "half_open":
            self.trial_ok += ok
            if not ok:
                self._move(now, "open")
                self.opened_at = now
            elif self.trial_ok == self.trial_calls:
                self._move(now, "closed")
                self.results.clear()
            return
        self.results.append(ok)
        if len(self.results) > self.window:
            self.results.popleft()
        failures = self.results.count(False)
        if (len(self.results) >= self.min_calls
                and failures / len(self.results) >= self.threshold):
            self._move(now, "open")
            self.opened_at = now
            self.results.clear()


def simulate(use_breaker):
    breaker = Breaker() if use_breaker else None
    pending = []                          # (finish time, ok) not yet seen
    calls = sick_calls = degraded = 0
    waits = []
    for i in range(int(RATE * DURATION)):
        now = i / RATE
        for done in [x for x in pending if x[0] <= now]:
            breaker.record(*done)         # outcomes arrive after the wait
            pending.remove(done)
        sick = BROWNOUT[0] <= now < BROWNOUT[1]
        if breaker and not breaker.allow(now):
            degraded += 1                 # fail fast: no call, no wait
            waits.append(0.0)
            continue
        calls += 1
        sick_calls += sick
        waits.append(TIMEOUT if sick else HEALTHY_MS)
        degraded += sick
        if breaker:
            pending.append((now + waits[-1], not sick))
    waits.sort()
    p99 = waits[int(len(waits) * 0.99) - 1]
    stuck = sick_calls * TIMEOUT / (BROWNOUT[1] - BROWNOUT[0])  # Little
    name = "with breaker" if use_breaker else "timeout only"
    print(f"{name:<13} calls={calls:5d} to sick={sick_calls:5d} "
          f"degraded={degraded:5d} p99 wait={p99 * 1000:3.0f} ms "
          f"stuck={stuck:4.1f}")
    if breaker:
        print("\n".join(breaker.log))


if __name__ == "__main__":
    simulate(False)
    simulate(True)

2. Run python3 breaker_lab.py. Expected output.

timeout only  calls= 6000 to sick= 3000 degraded= 3000 p99 wait=200 ms stuck=20.0
with breaker  calls= 2911 to sick=   44 degraded= 3133 p99 wait= 40 ms stuck= 0.3
t=10.29s closed -> open
t=15.29s open -> half_open
t=15.49s half_open -> open
t=20.49s open -> half_open
t=20.69s half_open -> open
t=25.69s open -> half_open
t=25.89s half_open -> open
t=30.89s open -> half_open
t=31.09s half_open -> open
t=36.09s open -> half_open
t=36.29s half_open -> open
t=41.30s open -> half_open
t=41.36s half_open -> closed

Validation. With a timeout alone all 3,000 brownout calls wait 200 ms, about 20 stuck at any moment (Little’s law: 100/s × 0.2 s). With the breaker only 44 reach the sick dependency and p99 wait falls from 200 ms to 40 ms. The cost: 133 extra degraded responses after recovery, as trials come only every 5 s. Exercises. Try open_s=1.0, then window=100, min_calls=100 (the library defaults: it opens after 50 failures, at t=10.69 s). Troubleshooting. Other numbers mean edited constants. Teardown. Delete the file.

On 2021-12-07 an automated scaling activity on AWS’s network triggered “a large surge of connection activity”, and the delays caused “even more connection attempts and retries”. The clients had well-tested back-off, “but, a latent issue prevented these clients from adequately backing off during this event”.^[68] Lesson: back-off is code with its own failure modes; test it under triggering conditions and keep dashboards off the failing path.

Netflix’s Hystrix isolated calls to remote systems to stop cascading failure. Its README says it “is no longer in active development, and is currently in maintenance mode”: Netflix’s focus shifted to “more adaptive implementations that react to an application’s real time performance rather than pre-configured settings”, and it recommends resilience4j for new projects.^[69] Lesson: static thresholds go stale; whatever the mechanism, someone owns its tuning.

SLIs and targets: SRE; MTTA and MTTR: IncidentOps. These measures show whether the resilience code works.

Retry ratio, breaker events, coverage and lever freshness lead; goodput and incidents lag. Vanity trap: fewer pages because alerts were silenced.

• Config versus secrets. Twelve-Factor requires “strict separation of config from code” and recommends environment variables;^[58] OWASP warns they “are generally accessible to all processes and may be included in logs or system dumps”.^[70] Workable split: settings in environment or files; secrets from a store, rotated, never in images.
• Kubernetes Secrets are “stored unencrypted in the API server’s underlying data store (etcd)” by default, and anyone who can create a Pod in a namespace can read its Secrets;^[71] hardening is in KubernetesSecOps.
• Health endpoints and levers are attack surface: no hostnames, versions or stack traces on public endpoints; lever changes restricted and logged. Error paths log too: keep personal data out (PrivacyOps).
• Compliance. Equal Experts argues team-run operations are compatible with ITIL v3, PCI-DSS and SOX;^[21] confirm with your auditors. Change approval is in ITOps.

Application-tier capacity is concurrency as much as CPU: Little’s law sizes thread pools, connection pools and bulkheads,^[41] and more replicas mean more database connections (DBOps). Autoscaling waits for new pods; shedding acts on the next request, so use both (KubernetesOps). Retries cost twice in incidents; standby pay grows with every rota.^[22] Skipping work for callers that have timed out^[24] saves capacity and energy (GreenOps).

The policy check runs on every register change; each hard dependency gets a quarterly drill in a test environment (production experiments follow SRE’s chaos-engineering guidance); a monthly review covers the metrics above; incidents end with a runbook check; major changes trigger a new readiness review.^[19][20]

Trigger: checkout error ratio above target, or an SLO burn alert. Levers: inventory.retries, recommendations.enabled, admission limit, forced breaker open.

flowchart TB
  P["Page: checkout errors up"] --> S{"Traces and logs: one dependency slow or failing?"}
  S -- "no" --> G["Check own deploys and capacity"]
  S -- "yes" --> L1["Retry ratio above 10%: its retries to zero"]
  L1 --> K{"Is the dependency soft?"}
  K -- "yes" --> L2["Switch it off or force its breaker open"]
  K -- "no" --> L3["Shed sheddable traffic; call its owner; incident after 15 min"]
  L2 --> V["Verify goodput; restore levers one at a time"]
  L3 --> V
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class P risk
  class S,K decision
  class L1,L2,L3,V,G core

1. Scope and diagnose (5 minutes): failing routes, start time, per-dependency latency, error and retry panels; an exemplar trace from the spike; logs filtered on dependency and decision.
2. Stop amplification: above a 10% retry ratio (this Atlas’s threshold, borrowed from Google’s client budget^[14]), retries to zero; attempts per request should near 1.0 within two minutes.
3. Protect users: soft dependency off or breaker forced open; hard dependency: fast 503s with Retry-After for sheddable traffic.
4. Escalate to the dependency owner; past 15 minutes or across teams, to IncidentOps roles.
5. Recover one lever at a time, retries last, logging each change; then a postmortem and register, runbook and drill updates.

Invest fully when a service calls other systems while a user waits, changes often, and costs money or trust when down. Keep a light baseline (truthful health, drain, config outside code, runbooks) for batch jobs, internal tools and prototypes; for packaged software, the work is in your integration code and vendor runbooks. Do not adopt team on-call when the team cannot staff a sustainable rota, downtime cost does not justify night cover, or a central team fits better (NoOps). Budget for hidden work: re-tuning thresholds, testing levers and degraded paths, current runbooks, standby pay.

flowchart TB
  Q1{"Do you change this service's code?"}
  Q1 -- "no" --> C1["Central application support; AppOps in integration code"]
  Q1 -- "yes" --> C2["Baseline: health, ordered shutdown, config, runbook; a register if it calls other systems"]
  C2 --> Q3{"Does downtime cost justify night cover?"}
  Q3 -- "no" --> R1["Team owns it in working hours only"]
  Q3 -- "yes" --> Q4{"Enough people for a sustainable rota?"}
  Q4 -- "yes" --> R2["Team on-call: you build it, you run it"]
  Q4 -- "no" --> R3["Shared domain rota or SRE support"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q1,Q3,Q4 decision
  class C2,R1,R2,R3 core
  class C1 neutral

Then choose patterns per dependency; a starting point, not a rule:

Levels and thresholds are this Atlas’s proposal, not a standard.

Level 3 suits anything that pages; few services need level 4.

Dependencies: telemetry (ObservabilityOps), SLOs (SRE), flags (ReleaseOps) and a recorded owner (ServiceOps).

• Register complete (hard or soft per dependency); policy check passes
• Explicit timeouts; the deadline propagates
• One budgeted, jittered retry layer (SDK and mesh retries counted); retried writes idempotent or keyed
• Breakers and bulkheads on critical paths; open-breaker behaviour defined
• Liveness process-local; readiness covers warm-up; shutdown serves until routers stop, then drains in the grace period
• Retries, breaker transitions and degraded responses visible in traces and logs by dependency; trace ID in every log line
• Degraded responses written and exercised; levers access-controlled and tested
• Every paging alert links a runbook entry
• Secrets from a store; on-call rota published and sustainable

• Owner, escalation contact, measured latency and chosen timeout recorded; retryable errors, rate limits and Retry-After known
• Degraded behaviour agreed with the product owner

• Retry ratio, breaker time and degraded share per dependency; incidents per shift; readiness flaps outside deploys
• Runbook coverage and lever freshness
• After each incident: runbook fixed within a week; missing lever added; register and drill updated

SERVICE OPERABILITY ONE-PAGER                 version / date
Service:            owner team:          service owner:
Tier / SLO link:    on-call model: team | domain | central | none at night
Inbound deadline:   ___ ms         admission limit: ___ concurrent
Dependencies:       see register (hard: __, soft: __), policy check: pass/fail
Health:             /livez = process only; /readyz = warm-up + draining + ___
Levers:             name - effect - who may pull - last tested
Degraded modes:     dependency - what the user gets - how it is exercised
Runbook:            link; last reviewed ____; alerts without runbook: __
Open risks:         ...

RUNBOOK ENTRY: <alert name>                         reviewed: YYYY-MM-DD
What it means:     one sentence in user terms
Impact:            who is affected, degraded or failing
First checks:      dashboards / queries (links), in order
Decision points:   if X then lever A; if Y then lever B; else escalate
Levers:            name - how to pull - expected effect - how to undo
Escalate to:       dependency owner / incident roles after __ minutes
Verify recovery:   metric and threshold
Afterwards:        ticket for follow-up, register and drill updates

LEVER: recommendations.enabled                 owner: team-checkout
Effect when off:   checkout renders without the panel; no calls made
Who may change:    on-call engineers of team-checkout (audited)
How:               flag system UI or CLI; takes effect within 30 s
Undo:              set back to on; watch error rate for 10 minutes
Last tested:       YYYY-MM-DD in production during a drill

ADR-NNN  Where do timeouts, retries and breakers live?     status: proposed
Context:   languages in use; mesh present?; who tunes policies?
Options:   (1) library per language  (2) mesh/proxy  (3) both, split by concern
Trade-offs: a library knows idempotency, deadlines and degraded responses but
           means one implementation per language; a mesh is uniform but blind
           to business meaning; platform defaults are a start for both
Decision:  e.g. (3): mesh enforces connection limits and outlier ejection;
           libraries own deadlines, idempotent retries and degraded responses;
           mesh retries off for routes that retry in code (Istio's default: 2)
Consequences: one retry layer per route documented in the register;
           policy check fails if both layers retry
Review:    after the next two incidents involving a dependency

Ways to survive a failing dependency. Amazon names four and avoids the last because “it’s difficult to prove and its effectiveness is hard to test”.^[72]

Should readiness check the database? Only if taking every instance out of rotation at once helps. Spring Boot calls it a judgement call, and behind a Kubernetes Service clients then get refused connections, not a 503.^[44]

What timeout should I use? The dependency’s latency at the percentile matching your acceptable false-timeout rate, plus network margin, short enough that the sum fits your deadline.^[12]

Do I need a breaker if I have a timeout? For important dependencies, yes: a timeout limits each wait; a breaker stops the waiting (Lab 3: calls to the sick dependency fell from 3,000 to 44).

Does every service need 24/7 on-call? No; match cover to downtime cost (Equal Experts’ 95% example has no night rota).^[22]

What if we cannot change the code? Put timeouts, retries and breakers in integration code or a proxy you control, keep runbooks for the product, and agree an escalation path with the vendor.

• Established and documented: client-side adaptive throttling (reject locally with probability max(0, (requests − K × accepts) ÷ (requests + 1)), K = 2 preferred) and criticality-aware shedding;^[14] hedging for tail latency (.NET ships a hedging handler^[32]); and Amazon’s constant-work redundancy, which adds no retry load as the need for redundant requests grows.^[72]
• Active practice, still settling: adaptive concurrency limits borrowed from TCP congestion control, Netflix’s direction after Hystrix.^[69][73] Less tuning, but behaviour that is harder to predict and explain at 03:00.
• Speculative or marketing-led: AI agents that fix incidents alone. This page’s view: treat their actions as lever changes (reviewed, reversible, logged) until safe autonomy is shown for your kind of system (AIOps).

Situation (illustrative). Checkout depends on one payment provider, which failed twice last year for about 45 minutes; the team has one quarter. Option A: integrate a second provider with automatic switch-over (most of the quarter). Option B: harden the primary path: idempotency keys, timeouts from measured latency, a breaker with a clear try again shortly page, a lever to pause non-urgent payments, and a runbook.

Why A can be wrong. A switch used only in failures is “the code path you never use”, which “(often) doesn’t work”;^[11] it runs at the worst moment, with unclear payment states, when double charges are likeliest without idempotency. A better A sends real traffic to the second provider all the time, so both paths are “exercised in production continuously rather than rarely”:^[72] active-active, with ongoing fees and a larger compliance scope.

Decision. Weigh about 90 minutes a year of failed payments against a quarter of engineering plus a second provider’s running cost. A without idempotency is dangerous, so B comes first; add active-active only if the expected loss pays for it, never an untested switch. Two outages are a small sample; revisit if the provider’s record worsens.

You should now be able to explain why “AppOps” is ambiguous, design per-dependency resilience and truthful health endpoints, operate with levers and runbooks, and evaluate a sustainable on-call model.

1. A liveness endpoint runs SELECT 1 on the shared database, which is slow for ten minutes. What happens, and what do you change?

2. A teammate wraps an AWS SDK call, at default settings, in a loop of three tries. Worst case?

3. A breaker keeps Resilience4j’s default window and minimum of 100 calls; the dependency is called once a second and starts timing out. How long before the breaker can open?

4. A job advert asks for an “AppOps engineer”. What settles what the job is?

“AppOps” is a weak label for strong practices. It means application support in one company, a product in another and a permission screen on a phone, so say which you mean. The practices here are well documented: bound every wait, retry in one place and safely, stop calling what is failing, keep failures local, tell the truth in health signals, make decisions visible, and give the person on call a runbook and levers that work.

Most of this is invisible until a dependency misbehaves. That is the test: a well-operated service looks unremarkable on the worst day of its dependencies, and its owners sleep through the incidents that do not need them.

Every source below was opened on 2026-10-18 (rows marked 2026-10-19 re-checked in the post-score polish); dates are the source’s own. Claim-level support, fetch methods and caveats are in the Atlas evidence ledger (research/appops-ledger.md).

Changelog. 2026-10-18, r0: first draft; term status confirmed as ambiguous; three labs and the policy check executed. 2026-10-18, r1: first-pass fact check (reviews/appops-r1-superseded-20261018T0506.md: 0 critical, 5 major, 12 minor; response in reviews/appops-r1-response.md): ordered shutdown with preStop, dated AWS SDK, .NET and Istio retry defaults, current Resilience4j metrics and a not closed alert, a diagnosability playbook, corrected breaker timing, Spring Boot 4.1, 15 new sources. 2026-10-19, r2: second-pass review (reviews/appops-r1.md: 0 critical, 0 major, 6 minor; response in reviews/appops-r1-response.md): fallback attribution (Amazon’s reason is provability), preStop sizing, ITIL 5 dated “as of”, n defined as teams on call, specifications sentence, ticket note dated to 5 minutes after open. 2026-10-19, r3 (post-score polish): re-verified the fast-aging rows (Vogels interview end-to-end via the Wayback copy; AWS jitter formula read from the blog’s equation image and the SDK reference guide; SDK retry defaults; PromQL re-parsed on the demo, now Prometheus 3.14.0); added a non-JVM pointer; trimmed into the 4,000–9,000 band without changing claims.