SRE (Site Reliability Engineering)

Site Reliability Engineering (SRE) is a way of running production services that treats operations as a software-engineering problem. It sets explicit reliability targets, called service level objectives (SLOs), based on what users experience. It measures how much unreliability is still acceptable (the error budget) and uses that budget to decide how fast to ship and when to stop and fix. It also caps repetitive manual work so engineers have time to automate it away.

Ben Treynor Sloss, who founded Google’s SRE organisation, put it in one line: “SRE is what happens when you ask a software engineer to design an operations team.”^[1] Developers want to ship and operators want stability; SRE gives both one agreed number, such as “99.9% of checkout requests succeed over four weeks”. Above target, teams ship. Below it, the agreed policy shifts effort to reliability, and nobody wins the argument by authority.

For beginners (vocabulary), engineers (SLOs and alerts), architects (dependencies) and managers (policies). Research verified 2026-10-16; selected sources re-checked 2026-10-17.

Without explicit targets, reliability is negotiated by mood: expectations ratchet towards “never fail” after each outage, release pace slows, and the people on call absorb whatever cannot be automated. The SRE book locates the root conflict “between pace of innovation and product stability” and argues that an error budget “resolves the structural conflict of incentives between development and SRE”.^[1] Three failure modes are common before SRE practices exist:

1. Unbounded targets. Nobody can say how reliable is enough, so every outage justifies more process. The book’s counterpoint: “100% is the wrong reliability target for basically everything”,^[1] and an incremental improvement “may cost 100x more than the previous increment”.^[2]
2. Alerting on causes. Pages fire on CPU, disk and queue depth that users never notice, so engineers learn to ignore pages.
3. Toil that grows with the service. Manual work scales with traffic; nothing improves and people burn out.

Before and after (illustrative)

• Not a renamed operations team. If engineers get the title and nothing else changes (no SLOs, no toil cap, no power to hand work back), the result is what DevOps Topologies calls the “Fake SRE” anti-type: a dev–ops silo “but 10 years later”.^[3]
• Not “maximum reliability”. SRE deliberately spends reliability. Google ran planned outages of its global Chubby lock service so that it met, “but does not significantly exceed”, its SLO, flushing out dependants that assumed it never failed.^[4]
• Not monitoring or incident management. Monitoring supplies data and incident management coordinates people; SRE decides what “good enough” means and what happens when it is not met (ObservabilityOps, IncidentOps).
• Not hardware reliability engineering, the older discipline of component failure rates and MTBF predictions.

Naming confusion. “SRE” names a discipline (this page), a role and a team; you can adopt the discipline without the other two. Despite “site”, it applies to any production service, including APIs, data pipelines and internal platforms.

Two cautions. Much of the canon describes Google’s scale; when not to use says what to drop at small scale. And this page cites the free first edition and the workbook; the second edition was not read for it and may update specific numbers (its announced direction is in advanced topics).

Evidence for the term status. Google’s books define the practices in detail,^[1][13] Google Cloud documents several team implementations,^[10] USENIX has run SREcon since 2014,^[5] and Catchpoint’s SRE Report, a vendor survey in its eighth year, drew 418 responses from reliability, DevOps and IT professionals for its 2026 edition (survey run July–August 2025).^[14]

Mental model: reliability is a budget, not a feeling. Pick what users care about (SLI), agree how often it may fail (SLO), and spend the allowed failures on change. When the budget is gone, invest in reliability. Alerting, toil limits, postmortems and readiness reviews protect that budget or give engineers time to defend it.

The workbook puts the DevOps relationship in programming terms: “class SRE implements interface DevOps”. DevOps is a broader philosophy, “relatively silent on how to run operations at a detailed level”; SRE is a concrete, opinionated implementation.^[23] Each line below is an editorial distinction (a split chosen so that each topic is taught once), not an industry standard.

flowchart LR
  DEV["DevOps"]
  OBS["ObservabilityOps"]
  SRE["SRE: SLOs, budgets, toil"]
  PLAT["PlatformOps"]
  APP["AppOps"]
  SVC["ServiceOps"]
  INC["IncidentOps"]
  DR["DR and backup"]
  DEV -- "implemented by" --> SRE
  OBS -- "telemetry" --> SRE
  SRE -- "SLO method" --> PLAT
  SRE -- "targets" --> APP
  SRE -- "SLOs under SLAs" --> SVC
  SRE -- "roles, comms" --> INC
  SRE -- "RTO, RPO" --> DR
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class SRE core
  class DEV,PLAT,APP,INC,OBS,SVC,DR neutral

Responsibilities. “An SRE team is responsible for the availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning of their service(s)”; its tenets start with a durable focus on engineering (the 50% cap on operational work) and maximum change velocity without violating the SLO.^[1]

Principles (workbook). Operations is a software problem; manage by SLOs; work to minimise toil; automate this year’s job away; move fast by reducing the cost of failure; share ownership with developers; use the same tooling regardless of job title.^[23] Both lists are Google’s guidance, not a formal standard.

A synthesis to copy into a team charter and argue about.

1. Mission. Keep each service as reliable as its users need, and no more, at the lowest cost in toil and lost velocity.
2. Users define reliability. SLIs measure user-visible outcomes, not component health.
3. Targets are explicit and agreed. Every tier-1 service has an SLO document approved by product, developers and on-call.
4. The budget has consequences. A policy signed in advance says what happens when the budget runs out and who decides disputes.
5. Engineering time is protected. Toil is measured. Above the cap, work is redirected rather than absorbed.
6. Failure is information. Postmortems are blameless and their actions are prioritised like features.
7. Shared ownership is conditional. SRE support depends on developers keeping their commitments; the last lever is handing the pager back.

Non-goals. 100% availability; SRE as the gatekeeper of every release; owning incident communications or the telemetry platform.

Roles. A service owner (usually a product engineering lead) owns the SLO and the code; a product manager agrees it is good enough for users; SREs design SLIs, alerting, capacity and automation, and share or carry the pager; an engineering director or CTO arbitrates disputes (the workbook’s example escalates to the CTO).^[21] Google hires SREs as 50–60% software engineers and 40–50% near-qualified candidates with rarer skills, most often UNIX internals and networking.^[1]

Team models. Google Cloud’s Customer Reliability Engineers describe six implementations and stress that the list is not exhaustive; Google’s own SRE organisation has every type except the kitchen sink.^[10]

RACI for core SRE activities (R responsible, A accountable, C consulted, I informed); a starting point, not a standard.

A production readiness review (PRR) is the typical first step before SRE takes on a service: engagement, analysis against a checklist, improvements, training, onboarding and continuous improvement. The checklist covers architecture and dependencies, monitoring, emergency response, capacity, change management and performance. Two later models are early engagement in design, and frameworks that make new services reliable by default.^[18]

The relationship is conditional. Introducing Google’s CRE team, Dave Rensin described the deal: developers maintain critical support systems such as monitoring and take part in reviews and postmortems; if they do not, the SREs “offboard” the system and hand back the pagers.^[8]

stateDiagram-v2
  direction TB
  state "Developers on call" as DevOwned
  state "Readiness review (PRR)" as PRR
  state "Joint improvements" as Improve
  state "SRE onboarded" as Onboarded
  state "Steady state" as Steady
  [*] --> DevOwned: new service launches
  DevOwned --> PRR: reliability needs grow
  PRR --> Improve: gaps found
  Improve --> PRR: re-check
  PRR --> Onboarded: checklist met
  Onboarded --> Steady: SLOs met, toil under cap
  Steady --> DevOwned: commitments broken, pager handed back
  Steady --> Steady: continuous improvement

The operating model is a set of feedback loops; the cadences are this Atlas’s suggestion. The workbook suggests reviewing SLO documents monthly at first and quarterly later.^[13]

Escalation. Put a named role, not a committee, at the top of the policy, as the workbook’s example does with the CTO.^[21] Record every invocation, the cause assigned and any escalated decision: an auditor or a new manager will ask for them first.

SRE’s architecture is a control loop, not a product stack: it turns user experience into a number, compares it with a target, and feeds the result back into engineering decisions. Tools named are examples; the tooling landscape lists others.

flowchart TB
  subgraph Points["Measurement points (closest to the user first)"]
    RUM["Client instrumentation"]
    PROBE["Synthetic prober"]
    LB["Load balancer logs"]
    APPM["Service metrics"]
  end
  SPEC["SLO spec in Git (OpenSLO or Sloth)"]
  TSDB["Metrics store, e.g. Prometheus"]
  REC["Recording rules: error ratio per window"]
  ALERT["Burn-rate alerts: page or ticket"]
  DASH["SLO dashboard and budget report"]
  ONCALL["On-call engineer"]
  REVIEW["Weekly and monthly budget review"]
  POLICY["Error-budget policy decision"]
  RUM --> TSDB
  PROBE --> TSDB
  LB --> TSDB
  APPM --> TSDB
  SPEC -- "generates" --> REC
  TSDB --> REC
  REC --> ALERT
  REC --> DASH
  ALERT --> ONCALL
  DASH --> REVIEW
  REVIEW --> POLICY
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class REC,ALERT,DASH core
  class POLICY decision
  class RUM,PROBE,LB,APPM,SPEC,TSDB,ONCALL,REVIEW neutral

Trust boundaries. The SLO pipeline only reads production, but the policy decision governs releases. Keep the out-of-scope exclusion in the code-reviewed spec, not an ad hoc query, or whoever edits the query quietly decides whether a halt applies.

Trigger: a burn-rate alert or the scheduled budget review. Actors: on-call engineer, service owner, SRE, product manager, escalation owner. Outputs: a release halt with a recorded cause and work mix, reliability work, a postmortem where required. Feedback: the next budget review checks whether the actions restored compliance.

flowchart TB
  T["Budget review or burn-rate alert"] --> Q1{"Budget exhausted over trailing 4 weeks?"}
  Q1 -- "No" --> N["Normal release pace"]
  Q1 -- "Yes" --> H["Halt changes and releases except P0 and security fixes"]
  H --> Q2{"What caused the miss?"}
  Q2 -- "Ours to fix" --> RW["Must work on reliability: own bug or process, a hard dependency to soften, a miss the SLI hid"]
  Q2 -- "Outside the team" --> FW["May keep building features, unreleased: network-wide fault, frozen dependency, out-of-scope users"]
  Q2 -- "Disputed" --> E["Named exec decides the calculation or the actions"]
  E --> RW
  E --> FW
  RW --> Q3{"Back within SLO?"}
  FW --> Q3
  Q3 -- "No" --> H
  Q3 -- "Yes" --> N
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class Q1,Q2,Q3 decision
  class H risk
  class N,RW,FW,E core

This flow follows the workbook’s example policy, which has two tiers.^[21] First, if the service has exceeded its budget for the preceding four weeks, the team will “halt all changes and releases other than P0 issues or security fixes until the service is back within its SLO”; no cause exempts it. Second, the cause decides the work. The team must work on reliability after its own code bug or procedural error, when a postmortem reveals a hard dependency that could be softened, or when miscategorised errors hid a miss. It may continue feature work after a company-wide network problem, an outage in another team’s service whose owners have frozen their own releases, budget use by out-of-scope users (load tests, penetration testers), or miscategorised errors that affected no users. The last two are measurement errors: fix the SLI in the reviewed spec, recompute, and send disputes about the calculation to the named executive (the CTO in the example). Two triggers do not wait for exhaustion: a single incident consuming more than 20% of the four-week budget requires a postmortem with at least one P0 action, and a class of outage consuming more than 20% in a quarter requires a P0 item in quarterly planning. The policy “is not intended to serve as a punishment for missing SLOs”.^[21]

Start from user journeys (search, add to basket, pay), not from the metrics you already have, and ask what “it worked” means to the user; Google calls these critical user journeys (CUJs).^[25] Then write two things, as the workbook recommends:^[13]

• an SLI specification: the outcome that matters to users, independent of measurement (“ratio of home-page requests that loaded in under 100 ms”);
• an SLI implementation: the specification plus a way to measure it (server or load-balancer logs, a prober, client code).

The workbook recommends expressing an SLI as good events divided by total events, so that it runs from 0% (nothing works) to 100% (nothing is broken).^[13] This page writes good events ÷ valid events to make exclusions such as load tests explicit. The shape makes budgets, burn rates and alerts uniform across services.

Decisions inside “valid” and “good”, where SLOs quietly go wrong:

• Which responses are bad? Server errors (5xx) almost always count; client errors (4xx) usually do not. A 429 depends on why it was sent. Shedding overload, or refusing a client that is within its quota, is your service failing; refusing a client that exceeded its own quota is correct enforcement, like a declined card. Sloth’s getting-started example counts 5xx and 429 as errors without giving a reason.^[26] Use one rule in every spec, alert, dashboard and runbook query.
• Which requests are valid? Exclude load tests, penetration tests and health checks in the reviewed spec. The workbook’s example policy lets feature work continue when out-of-scope users consumed the budget;^[21] excluding them up front avoids the argument.
• Latency as a ratio, not an average. Use “the proportion of requests faster than X”; the SRE book prefers percentiles to the mean, which hides the slow tail.^[4] Prometheus classic histograms need a bucket boundary at exactly X. Native histograms answer with histogram_fraction, an interpolated estimate the documentation calls “quite accurate” at decent resolution, and Prometheus recommends them where you can use them (as of October 2026).^[27]
• Successful versus failed latency. The SRE book warns to distinguish the latency of successful and failed requests.^[28] A fast error is not a good event: count it as bad in the availability SLI and filter it out of the latency SLI.

The four golden signals (latency, traffic, errors, saturation) are a good starting set for telemetry,^[28] but traffic and saturation explain why an SLI moved; they rarely describe user happiness on their own.

The SRE book’s rules of thumb: don’t pick a target based on current performance, keep it simple, avoid absolutes, have as few SLOs as possible, perfection can wait, keep a safety margin, and don’t overachieve.^[4] Ask what users expect, whether the service ties to revenue, whether it is paid or free, what competitors offer, and whether users are consumers or enterprises. Users impose a ceiling: “a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability”.^[2]

Reconciling two pieces of advice. “Don’t pick a target based on current performance” seems to clash with the workbook’s example SLO document, which derived targets from a month of data, rounding availability down to the nearest 1% and latency up to the nearest 50 ms.^[15] Both hold: user needs say what the target should be, data whether it is achievable. When they disagree, publish the achievable SLO and track the better one as an aspirational SLO, “explicitly called out in your error budget policy as not requiring action”.^[13]

Dependencies cap what you can promise. “A service cannot be more available than the intersection of all its critical dependencies”, so critical dependencies need one more nine than the service (the “rule of the extra 9”). Frequency and duration matter: three 20-minute outages a year already exceed the roughly 53 minutes that 99.99% allows.^[9] For a quick feasibility check, multiply the measured availabilities of hard dependencies: three at 99.95% each give 0.9995³ ≈ 99.85% if their failures are independent. Correlated failures (shared infrastructure) overlap and lift the figure towards the weakest dependency, 99.95%; failures that never overlap lower it to 1 − Σ unavailability, also about 99.85% here. Your service’s own failures come on top, so the SLO must sit below whichever figure applies.

Windows. The workbook found “a four-week rolling window to be a good general-purpose interval”; rolling windows follow user experience, calendar windows suit business planning.^[13] A 28-day window also holds the same number of each weekday.

Allowed bad minutes = (1 − SLO) × 40,320. Prefer request-based SLIs where traffic allows: a time-based SLI treats a minute at 3 a.m. like a minute at peak. Managed tools support both; Cloud Monitoring offers request-based and windows-based SLIs over rolling (1 to 30 days) or calendar periods.^[29]

An SLO without a policy is a dashboard. The workbook’s prerequisites are SLOs that all stakeholders approve and consider achievable, a commitment to use the budget for decisions, and a process for refining the SLO. Three groups must agree: product managers (good enough for users), developers (they will reduce risk when the budget is spent) and the production team (defensible “without Herculean effort, excessive toil, and burnout”).^[13]

Minimum contents (workbook): authors, reviewers and approvers; approval and review dates; a short service description; the actions when the budget is exhausted; an escalation path.^[13] Graduated actions (this Atlas’s proposal) reduce the gaming that a single cliff edge invites:

A halt is not the goal: the workbook’s example says “Halting change is undesirable” and frames the policy as permission to focus on reliability when data shows it matters more than features.^[21] Write down in advance which causes require reliability work and which let feature work continue, and record which applied each time.

The workbook judges SLO alerts on precision (share of alerts that were significant), recall (share of significant events that alerted), detection time and reset time, and favours multiwindow, multi-burn-rate alerting.^[16] Burn rate = error ratio ÷ (1 − SLO). Burn rate 1 spends exactly the budget over the window; 14.4 spends a 30-day budget in about two days. The share of budget a burn consumes is burn rate × alert window ÷ SLO period.

These are the workbook’s starting parameters for a 99.9% SLO; its example rules add a ticket on a 24-hour and 2-hour pair at burn rate 3.^[16] Both windows must exceed the threshold: the long one gives precision, and the short one (about 1/12 of the long) stops the alert soon after the problem stops. Over 28 days the same burn rates consume 2.1%, 5.4% and 10.7%, close enough to keep, as Lab 3 does. Sloth’s 28-day catalogue keeps the budget shares instead and lowers the burn rates to 13.44 and 5.6 for pages and 2.8 and about 0.93 for tickets (Sloth v0.16.0).^[30] Either is defensible; write down which you use.

flowchart LR
  R["Error ratio per window"]
  P1["1 h and 5 min both<br>above 14.4 × 0.1%"]
  P2["6 h and 30 min both<br>above 6 × 0.1%"]
  T1["24 h and 2 h both<br>above 3 × 0.1%"]
  T2["3 d and 6 h both<br>above 1 × 0.1%"]
  PAGE["Page on-call now"]
  TICKET["Ticket for working hours"]
  R --> P1
  R --> P2
  R --> T1
  R --> T2
  P1 --> PAGE
  P2 --> PAGE
  T1 --> TICKET
  T2 --> TICKET
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class P1,P2,T1,T2 decision
  class PAGE risk
  class R,TICKET neutral

Where the parameters break. At a 90% SLO the 2%-in-an-hour page can never fire, because a total outage spends only 1.4% of a month’s budget in an hour; at 99.999% a total outage exhausts the budget in 26 seconds, so design for a small blast radius (roll out to 1% of users first). Low-traffic services get noisy ratios: use synthetic traffic (which can hide real user failures), combine services, change the product (client retries), or lower the SLO or lengthen the window. At scale, group request types into a few buckets with one parameter set each.^[16]

The SRE book defines toil as work “tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows”.^[17] Google’s target is to keep operational work (toil) below 50% of each SRE’s time. Its quarterly surveys found an average of about 33%, but “the average doesn’t capture outliers: some SREs claim 0% toil (pure development projects with no on-call work) and others claim 80% toil”.^[17] Catchpoint’s 2026 SRE Report (a vendor survey, 68% of respondents in North America) reports a median of 34% of engineers’ time.^[14] Both figures sit near a third. Neither is a target, and averages hide teams far above the cap.

Some toil is unavoidable, but it “becomes toxic when experienced in large quantities”, leading to career stagnation, low morale and attrition.^[17] The workbook’s method:^[31]

1. Identify toil with the people who do it: business processes, production interrupts, release shepherding, migrations, cost engineering and capacity planning, and troubleshooting opaque architectures.
2. Measure it in one consistent unit (hours, tickets, manual changes) before, during and after reduction.
3. Reduce it: engineer it out, reject toil not worth doing, use SLOs to decide which interrupts matter, move from human-backed interfaces to self-service, increase uniformity, and assess the risk inside automation.

The cap needs a mechanism. Google redirects excess operational work to product teams by reassigning bugs and tickets and putting developers back into on-call rotations.^[1] Without such a mechanism, a toil limit is an aspiration. Automation’s own safety controls are taught in AutomationOps.

A blameless postmortem “must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior”. The SRE book’s triggers are user-visible downtime or degradation beyond a threshold, any data loss, on-call intervention such as a rollback, resolution time above a threshold, and a monitoring failure; any stakeholder may request one. “Writing a postmortem is not punishment—it is a learning opportunity for the entire company.”^[19]

Culture shows in behaviour, not templates. In this Atlas’s view, healthy signs are volunteers writing postmortems, readers outside the team, actions completed at their stated priority, and senior engineers writing up their own mistakes. Google adds a postmortem of the month, reading clubs and “Wheel of Misfortune” re-enactments,^[19] and now applies CAST (Causal Analysis based on System Theory), a retrospective method from the STAMP safety model, to “supercharge our postmortems”.^[32]

The mechanics (timelines, severity, owners, action tracking) and the review template live in IncidentOps, whose skeleton already records the SLO budget consumed. This page’s concern is the budget link: the policy decides which incidents demand a postmortem with priority actions.

Google’s published numbers make on-call load explicit:^[33]

• At least 50% of SRE time goes to engineering; of the rest, no more than 25% is on-call and up to another 25% other operational work.
• At most two incidents per 12-hour shift, because handling one, including follow-up, takes about six hours on average.
• A single-site rotation with a primary and a secondary needs at least eight engineers; a dual-site team needs at least six per site.
• Typical response targets are 5 minutes for user-facing or time-critical services and 30 minutes for less time-sensitive ones.
• Underload is a risk too: everyone should be on call once or twice a quarter to stay familiar with production.

Treat these as reference points. A three-person team cannot staff 24/7 by them: either the service does not need night paging, or it is under-staffed. Google also reports that playbooks written in advance give “roughly a 3x improvement in MTTR” over improvising,^[1] an internal observation, not an independent study. MTTR here is mean time to repair, the book’s own expansion; IncidentOps explains its caveats and owns paging tools and incident roles.

Capacity. The SRE book makes three steps mandatory: an organic demand forecast beyond the lead time for acquiring capacity, inorganic demand (launches, marketing events) in the forecast, and regular load tests that map raw resources to service capacity; provisioning is riskier than shifting load.^[1] Google’s planning moved to intent: “Specify the requirements, not the implementation”, for example “meet service Foo’s demand in each geographic region, and have N + 2 redundancy” rather than “50 cores in clusters X, Y, and Z”.^[34] Saturation is the early warning that capacity is running out.^[28] Cost and commitments are covered in FinOps.

Release engineering. The SRE book names four principles: a self-service model, high velocity (frequent, small releases), hermetic builds, and enforcement of policies and procedures through access controls.^[35] Google found that “roughly 70% of outages are due to changes in a live system”, so its change management relies on progressive rollouts, fast and accurate detection, and safe rollback.^[1] Canarying links releases to the budget: the impact “is directly proportional to the amount of traffic exposed to defects”, so a 5% canary serving 20% errors produces a 1% overall error rate. Run one canary at a time and compare it with a concurrent control group, not with the period before the deployment.^[20] Release strategy and progressive delivery are taught in ReleaseOps.

No standards body defines SRE. Keep the tiers of reference separate when you justify a decision.

RED and USE belong to ObservabilityOps and DORA’s delivery metrics to DevOps; DORA itself calls reliability “more a measure of operational performance than a measure of software delivery performance”.^[11]

Choose tool categories first; each can be built from open-source parts or bought.

Feature lists change quickly; keep SLO definitions portable so that changing back ends does not mean renegotiating targets.

Automation is SRE’s core tool, and it can itself cause outages at scale, so apply the AutomationOps controls (dry runs, rate limits, approvals for destructive actions). Good candidates are high-volume, low-judgement toil: provisioning, routine capacity changes, certificate renewals, first diagnostic steps.

Vendors market AI assistants for incident summaries, log search and suggested remediations, and Google’s announcement of the second edition says the book covers generative AI for incident triage and postmortem generation.^[12] Treat these tools as automation with an unusual failure mode: plausible but wrong output. Keep a human approval before any AI-suggested production change; measure the tool by the SLO outcomes it affects, not by how many summaries it writes; and do not let it set SLO targets, which are a product decision for the stakeholders in Practice 3. ML event correlation is evaluated in AIOps.

Assumptions. One HTTP service, checkout, exposes a counter http_requests_total{job, code, traffic} and a classic histogram http_request_duration_seconds{job, code, traffic} with a bucket at le="0.4". The service sets traffic="loadtest" on requests from the load generator (for example, from a header it accepts only from that client) and traffic="user" on all others. Two SLOs over a rolling 28 days: 99.9% of valid requests succeed, and 99% of valid, successful requests complete within 400 ms. One error definition everywhere: a request is bad if it returns a 5xx. The service sheds overload with 503, which counts, and returns 429 only to clients above their own quota, which is correct enforcement and counts as good (Practice 1). The SLO lives in Git, rules are generated from it, and the policy (Templates) sits next to it.

checkout/
  slo/checkout.openslo.yaml      # portable definition (documentation and vendor import)
  slo/checkout.sloth.yaml        # input to the rule generator
  rules/checkout-slo.rules.yaml  # generated by Sloth; reviewed, never hand-edited
  slo/SLO.md                     # SLO document and error-budget policy, signed

1. The portable definition (OpenSLO v1) records intent in a vendor-neutral format, following the v1 schema.^[41][44] The latency SLO has the same shape as the availability SLO shown.

Example status: YAML syntax-checked by the validator; not loaded into an OpenSLO implementation.

apiVersion: openslo/v1
kind: Service
metadata:
  name: checkout
  displayName: Checkout API
spec:
  description: Takes payment and creates orders for the web shop.
---
apiVersion: openslo/v1
kind: SLO
metadata:
  name: checkout-availability
  displayName: Checkout availability
spec:
  description: 99.9% of valid checkout requests succeed (non-5xx, excluding load tests).
  service: checkout
  indicator:
    metadata:
      name: checkout-non-5xx
    spec:
      ratioMetric:
        counter: true
        good:
          metricSource:
            type: Prometheus
            spec:
              query: >-
                sum(http_requests_total{job="checkout",code!~"5..",
                traffic!="loadtest"})
        total:
          metricSource:
            type: Prometheus
            spec:
              query: sum(http_requests_total{job="checkout",traffic!="loadtest"})
  budgetingMethod: Occurrences
  timeWindow:
    - duration: 28d
      isRolling: true
  objectives:
    - displayName: Availability
      target: 0.999

2. The generator input (Sloth prometheus/v1). Sloth turns this into SLI recording rules, metadata rules and page and ticket alerts.^[26] Both latency queries carry code!~"5..", so a fast error never counts as a fast success.

Example status: YAML syntax-checked by the validator; field names match the prometheus/v1 spec in Sloth v0.16.0 (April 2026); not run through Sloth for this page.

version: "prometheus/v1"
service: "checkout"
labels:
  owner: "payments-team"
  tier: "1"
slos:
  - name: "requests-availability"
    objective: 99.9
    description: "Valid checkout requests that do not fail with a 5xx."
    sli:
      events:
        error_query: >-
          sum(rate(http_requests_total{job="checkout",code=~"5..",
          traffic!="loadtest"}[{{.window}}]))
        total_query: >-
          sum(rate(http_requests_total{job="checkout",
          traffic!="loadtest"}[{{.window}}]))
    alerting:
      name: CheckoutAvailabilityBudgetBurn
      annotations:
        summary: "Checkout is burning its availability error budget"
        runbook: "https://runbooks.example.com/checkout/budget-burn"
      page_alert:
        labels:
          severity: page
      ticket_alert:
        labels:
          severity: ticket
  - name: "requests-latency"
    objective: 99.0
    description: "Valid, successful checkout requests that complete within 400 ms."
    sli:
      events:
        error_query: >-
          sum(rate(http_request_duration_seconds_count{job="checkout",
          code!~"5..",traffic!="loadtest"}[{{.window}}]))
          - sum(rate(http_request_duration_seconds_bucket{job="checkout",
          code!~"5..",traffic!="loadtest",le="0.4"}[{{.window}}]))
        total_query: >-
          sum(rate(http_request_duration_seconds_count{job="checkout",
          code!~"5..",traffic!="loadtest"}[{{.window}}]))
    alerting:
      name: CheckoutLatencyBudgetBurn
      page_alert:
        labels:
          severity: page
      ticket_alert:
        labels:
          severity: ticket

3. Generate, check and review the rules. Sloth’s prometheus/v1 spec has no field for the SLO period and the default is 30 days, so the command selects the 28-day catalogue explicitly (flag present in Sloth v0.16.0).^[30]

Example status: bash syntax-checked; not executed (needs the sloth and promtool binaries).

#!/usr/bin/env bash
set -euo pipefail
# Generate Prometheus rules for a 28-day SLO period (Sloth's default is 30d).
sloth generate --default-slo-period=28d \
  -i slo/checkout.sloth.yaml -o rules/checkout-slo.rules.yaml
# Validate the generated file before it reaches Prometheus.
promtool check rules rules/checkout-slo.rules.yaml
# Show reviewers exactly what changed in the alerting behaviour.
git diff --stat -- rules/

Design notes. The latency SLI needs a bucket boundary at exactly 0.4 s.^[27] With the 28-day catalogue, Sloth’s page thresholds are 13.44 and 5.6, not Lab 3’s 14.4 and 6 (Practice 4 explains both). The traffic!="loadtest" filter puts the out-of-scope rule in the reviewed spec, but only if the label exists: a != matcher also selects series that lack the label,^[45] so an uninstrumented service silently excludes nothing. Check that the selector {job="checkout",traffic=""} returns no series. The generated file is committed so that reviewers see alerting changes in the pull request.

Labs 1 and 2 need only a pen or Python and cost nothing; Lab 3 needs Prometheus’s promtool.

• Objective: Turn “the shop must be reliable” into an SLI specification, an SLI implementation, an SLO and a budget.
• Prerequisites: A pen or spreadsheet; 45 minutes; the glossary.
• Cost and risk: None.

Steps

1. Pick one user journey: “a signed-in user pays for the basket”.
2. Write the SLI specification in user terms: what counts as success, and how fast is fast enough?
3. Choose an SLI implementation: where you count events, and what that point misses.
4. List what is valid and what is excluded (load tests, blocked bots, health checks).
5. Pick a target and window; compute the budget for 28 days with 30 million valid requests.
6. Write one sentence for each stakeholder group saying what it agrees to.

Expected output (worked answer).

• Validation: A correct decline (“card refused”) counts as good, because the service behaved correctly; if your SLI counts it as bad, you are measuring the bank, not your service. The same test settles a 429: good if the client exceeded its own quota, bad if the service shed load or refused a client within quota (with overload shed as 503, “non-5xx” applies this rule). Check that numerator and denominator use the same filters.
• Troubleshooting: If nobody would sign step 6, you have found the real, organisational blocker. If the budget feels huge, 30,000 failed payments may be 30,000 lost orders: the number starts the business conversation.
• Teardown: None; keep the sheet as the first draft of your SLO document.

• Objective: See how a budget is spent and when each of the workbook’s alert pairs would fire, minute by minute.^[16]
• Prerequisites: Python 3.8 or later, standard library only; 30 minutes.
• Cost and risk: None; the script generates its own synthetic traffic.

Steps

1. Save the script as slo_budget.py.

Example status: executed on 2026-10-16 with Python 3.8.10; syntax-checked by the validator.

"""Error budget and multiwindow burn-rate alerts for ONE service, minute by minute.

Usage: python3 slo_budget.py [--slo 0.999] [--days 28]
Generates a deterministic synthetic traffic series (no input file), then reports
SLO compliance, budget consumption and when each burn-rate alert would fire.
Alert parameters follow the Site Reliability Workbook, "Alerting on SLOs".
Standard library only.
"""
import argparse
import math

MIN_PER_DAY = 1440


def synthetic_minutes(days):
    """Yield (good, total) per minute: diurnal traffic, 0.02% errors, two incidents."""
    # Fractional errors are carried forward so the 0.02% baseline is not rounded away.
    carry = 0.0
    for m in range(days * MIN_PER_DAY):
        day, minute_of_day = divmod(m, MIN_PER_DAY)
        # Diurnal traffic between 400 and 1,600 requests per minute, peak at 14:00.
        phase = 2 * math.pi * (minute_of_day - 840) / MIN_PER_DAY
        total = int(1000 + 600 * math.cos(phase))
        error_ratio = 0.0002
        # Incident A: day 10, 14:00-14:40, 5% errors.
        if day == 9 and 840 <= minute_of_day < 880:
            error_ratio = 0.05
        # Incident B: day 20, 02:00-18:00, 0.5% errors.
        if day == 19 and 120 <= minute_of_day < 1080:
            error_ratio = 0.005
        carry += total * error_ratio
        bad = int(carry)
        carry -= bad
        yield total - bad, total


def prefix_sums(series):
    bad, tot = [0], [0]
    for good, total in series:
        bad.append(bad[-1] + (total - good))
        tot.append(tot[-1] + total)
    return bad, tot


def ratio(bad, tot, end, window):
    """Error ratio over the `window` minutes ending at minute index `end` (exclusive)."""
    start = max(0, end - window)
    t = tot[end] - tot[start]
    return (bad[end] - bad[start]) / t if t else 0.0


def stamp(minute):
    day, rest = divmod(minute, MIN_PER_DAY)
    return f"day {day + 1:>2} {rest // 60:02d}:{rest % 60:02d}"


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--slo", type=float, default=0.999)
    ap.add_argument("--days", type=int, default=28)
    args = ap.parse_args()
    budget_ratio = 1 - args.slo
    series = list(synthetic_minutes(args.days))
    bad, tot = prefix_sums(series)
    n = len(series)

    total_req, total_bad = tot[n], bad[n]
    allowed = budget_ratio * total_req
    print(f"window: {args.days} days, SLO {args.slo:.3%}, "
          f"requests {total_req:,}, failed {total_bad:,}")
    verdict = "met" if total_bad <= allowed else "MISSED"
    print(f"SLI (good/total): {1 - total_bad / total_req:.4%}  -> SLO {verdict}")
    print(f"error budget: {allowed:,.0f} failed requests; "
          f"consumed {total_bad / allowed:.1%}; remaining {1 - total_bad / allowed:.1%}")

    # Multiwindow, multi-burn-rate rules: (name, long, short, burn rate), in minutes.
    rules = [("page fast", 60, 5, 14.4), ("page slow", 360, 30, 6.0),
             ("ticket", 1440, 120, 3.0), ("ticket slow", 4320, 360, 1.0)]
    for name, long_w, short_w, burn in rules:
        threshold = burn * budget_ratio
        episodes, firing, began = [], False, None
        for end in range(1, n + 1):
            on = (ratio(bad, tot, end, long_w) > threshold
                  and ratio(bad, tot, end, short_w) > threshold)
            if on and not firing:
                firing, began = True, end - 1
            elif not on and firing:
                firing = False
                episodes.append((began, end - 1))
        if firing:
            episodes.append((began, n - 1))
        text = "; ".join(f"{stamp(a)} -> {stamp(b)}" for a, b in episodes)
        text = text or "never fired"
        print(f"{name:<11} ({long_w // 60}h/{short_w}m at {burn}x): {text}")


if __name__ == "__main__":
    main()

2. Run python3 slo_budget.py.

Expected output (copied from the real run):

window: 28 days, SLO 99.900%, requests 40,299,952, failed 16,415
SLI (good/total): 99.9593%  -> SLO met
error budget: 40,300 failed requests; consumed 40.7%; remaining 59.3%
page fast   (1h/5m at 14.4x): day 10 14:17 -> day 10 14:43
page slow   (6h/30m at 6.0x): day 10 14:37 -> day 10 15:06
ticket      (24h/120m at 3.0x): day 20 15:15 -> day 20 18:52
ticket slow (72h/360m at 1.0x): day 20 14:00 -> day 20 23:17

Reading the result. Incident A (5% errors, burn rate 50, 40 minutes at peak) paged after 17 minutes, matching 60 min × 14.4 ÷ 50 ≈ 17, and the page stopped 3 minutes after the incident, thanks to the 5-minute window. Incident B (0.5% errors, burn rate 5, 16 hours) never paged and opened a ticket after 12 hours, by design: at burn rate 5 the budget lasts about 5.6 days. The quiet cost: the 0.02% baseline spent about half of the consumed budget.

• Validation: Check one number by hand. The budget is 0.1% × 40,299,952 ≈ 40,300 requests, and 16,415 ÷ 40,300 ≈ 40.7%.
• Exercises: (a) Run python3 slo_budget.py --slo 0.9995. Does incident B page now, and how much budget is left? (b) Change incident B’s ratio to 0.008 (burn rate 8). Which alerts fire, and when? © Why did incident A trigger only pages, and incident B only tickets?
• Troubleshooting: with real data, empty minutes make ratio return 0 rather than fail.
• Teardown: Delete the file.

• Objective: Write the workbook’s multiwindow page alert as Prometheus rules and prove with promtool test rules that a 1% error rate pages (via the 6-hour pair) and a 0.05% error rate does not.^[16][46]
• Prerequisites: promtool from Prometheus 3.x (the files follow the 3.15 documentation, current in October 2026); no running server is needed.
• Cost and risk: None; unit tests run offline on synthetic series.
• Status: not executed (promtool is not installed in the authoring environment); YAML syntax-checked, written to the documented unit-test format.

1. Save the rules as checkout-slo.rules.yaml. The recording rules sum numerators and denominators before dividing, as the Prometheus naming guide recommends.^[24] For brevity they leave out the reference implementation’s traffic filter.

groups:
  - name: checkout-slo
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total{job="checkout"}[5m]))
      - record: job:slo_errors_per_request:ratio_rate30m
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout",code=~"5.."}[30m]))
          /
          sum by (job) (rate(http_requests_total{job="checkout"}[30m]))
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total{job="checkout"}[1h]))
      - record: job:slo_errors_per_request:ratio_rate6h
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout",code=~"5.."}[6h]))
          /
          sum by (job) (rate(http_requests_total{job="checkout"}[6h]))
      - alert: CheckoutErrorBudgetBurn
        expr: |
          (
              job:slo_errors_per_request:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
            and
              job:slo_errors_per_request:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
          )
          or
          (
              job:slo_errors_per_request:ratio_rate6h{job="checkout"} > (6 * 0.001)
            and
              job:slo_errors_per_request:ratio_rate30m{job="checkout"} > (6 * 0.001)
          )
        labels:
          severity: page
        annotations:
          summary: Checkout is burning its 99.9% error budget too fast

2. Save the test as checkout-slo.test.yaml. Each counter series grows by a fixed amount per minute, so the error ratio is constant.

rule_files:
  - checkout-slo.rules.yaml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout",code="200"}'
        values: '0+990x480'
      - series: 'http_requests_total{job="checkout",code="500"}'
        values: '0+10x480'
    alert_rule_test:
      - eval_time: 7h
        alertname: CheckoutErrorBudgetBurn
        exp_alerts:
          - exp_labels:
              severity: page
              job: checkout
            exp_annotations:
              summary: Checkout is burning its 99.9% error budget too fast
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout",code="200"}'
        values: '0+1999x480'
      - series: 'http_requests_total{job="checkout",code="500"}'
        values: '0+1x480'
    alert_rule_test:
      - eval_time: 7h
        alertname: CheckoutErrorBudgetBurn
        exp_alerts: []

3. Run promtool check rules checkout-slo.rules.yaml, then promtool test rules checkout-slo.test.yaml.

• Expected output: check rules reports success and 5 rules found; test rules reports success. In the first group the error ratio is 1% (burn rate 10): below 14.4, so the 1-hour pair is false, but above 6, so the 6-hour pair fires. In the second group the ratio is 0.05% (burn rate 0.5), and no alert fires.
• Validation: Prove the threshold is real. In the first group, change the increments to 0+995x480 and 0+5x480 (0.5% errors, burn rate 5, below the 6× threshold) and rerun: the test should now fail, with promtool reporting that the expected alert did not fire. Restore the values afterwards.
• Troubleshooting: a label mismatch means the alert carries labels you did not list; expected labels include the sample’s labels, without __name__ and alertname.^[46] Empty recording rules usually mean the regex or job label is wrong.
• Teardown: Delete the two files; deploy real rules through your normal change process.

Cases A and B are sourced from the workbook, written by engineers at those companies; C is an illustrative composite.

A. Evernote: SLOs as the shared language between ops and dev (sourced).^[47] Problem: operations protected production while developers built features. Method: after moving to Google Cloud, a simple first SLO of 99.95% uptime over a monthly window for certain services, measured by an external prober calling a built-in status page; a second, geographically separate prober confirms failures, and scheduled maintenance counts as downtime. Monthly reviews; SLO revisions every six months. Outcome: a common measure of success for ops and dev; in 2017 an SLO calculation helped cut a set of release windows from five to two. Trade-off: the probe misses some user interactions, accepted on the principle “Perfect is the enemy of good”.

B. The Home Depot: SLOs at scale through a simple framework (sourced).^[47] Problem: many development teams unfamiliar with SLOs. Method: the VALET acronym (volume, availability, latency, errors, tickets), executive backing, training and internal marketing, annual SLO goals for development managers, and tooling that turned logs into VALET metrics. Outcome: from about 50 services reporting SLOs to 800 within a year. Trade-off: SLOs were a trending tool, not wired to alerting; an error-budget culture was still a next step.

C. A data pipeline with a freshness SLO (illustrative). SLI: share of reads of a pricing feed served from data updated within 10 minutes; SLO 99% over 28 days. Late batches, not crashes, spend the budget; when it is gone, schema changes halt. Pitfall: measuring job success instead of data age hides retries that succeed late. Pipeline practice belongs to DataOps; platform SLOs, such as pipeline start times on an internal CI platform, to PlatformOps.

SRE metrics cover the service’s reliability, the health of the reliability process, and the people doing it.

Postmortem action closure is defined once, in IncidentOps.

Worked calculation (illustrative data). Checkout handled 50,000,000 valid requests in the last 28 days, with an SLO of 99.9%.

1. Budget = (1 − 0.999) × 50,000,000 = 50,000 failed requests.
2. Observed failures = 31,000, so SLI = 1 − 31,000 ÷ 50,000,000 = 99.938%: met, with a margin of 0.038 percentage points.
3. Budget consumed = 31,000 ÷ 50,000 = 62%; remaining 38%, or 19,000 requests.
4. One incident caused 12,500 failures = 25% of the budget, above the example policy’s 20% postmortem trigger.^[21]
5. Right now the 1-hour error ratio is 0.6%, a burn rate of 0.006 ÷ 0.001 = 6. At an average 50,000,000 ÷ 672 h ≈ 74,400 requests per hour, that is about 446 failures an hour, so the remaining 19,000 last about 43 hours. The 6-hour page fires only if the 6-hour and 30-minute ratios also exceed 0.6%.
6. Toil: a team of six logged 190 toil hours in a two-week sprint of 6 × 80 = 480 hours, a toil share of 39.6%: under Google’s 50% cap, but worth a trend line.

Leading versus lagging. Compliance lags; burn rate, coverage and toil share lead. Vanity traps: SLO counts, “100% compliance every month” (a loose target or a blind SLI), fewer pages (possibly lost recall). Data limitations: server-side SLIs miss requests that never arrive, low traffic makes ratios jumpy, and changing the SLI implementation breaks the trend, so log it.

• SLOs versus SLAs. An SLA is a contract with consequences; an SLO is an internal target.^[4] Keep SLOs stricter than any SLA so you act before you pay (ServiceOps).
• Least privilege. Rollbacks, failovers and capacity changes need production access: grant it through audited, time-bound roles and one-button safe actions, with break-glass credentials for emergencies (AccessOps); the runbook shows the exact permissions for one rollback.
• Security incidents. When an outage is an attack, handling moves to ResponseOps; preserving evidence can conflict with restoring fast.
• Auditability. Keep the signed SLO document and policy, every invocation and escalated decision, and the postmortems; regulatory needs depend on jurisdiction and sector (ComplianceOps).
• Privacy. Client-side instrumentation and logs can carry personal data; count events, not people, where you can.

Reliability is paid for in engineering time, infrastructure and lost features, and each increment can cost far more than the last.^[2] Three drivers dominate:

Over-provisioning wastes money and energy; under-provisioning spends the budget. Rightsizing belongs to FinOps, carbon to GreenOps.

• Do SLOs predict user pain? Complaints while the SLO is green mean a blind SLI; burns nobody notices may mean the target is too strict.
• Is the policy real? A policy never invoked, or overridden without a recorded decision, is not operating.
• Are alerts healthy? Track precision (pages that needed action) and recall (incidents caught by alerts rather than by users).^[16]
• Is toil falling? A rising toil share on a stable service means the work is not being engineered away.
• Are postmortems learning? Repeated contributing causes mean actions miss the underlying conditions.

Illustrative scenario. Commands are syntax-checked, not executed; they assume Kubernetes and the Prometheus HTTP API.

Trigger: CheckoutAvailabilityBudgetBurn pages at 14:17. The incident process starts as in IncidentOps; this runbook covers the reliability side.

Access (least privilege). In namespace shop only, the on-call identity needs get, list and watch on Deployments and ReplicaSets and patch on Deployments: kubectl’s rollback reads the Deployment, lists its ReplicaSets and patches the Deployment.^[49] Grant this Role just in time through an audited process; check it with kubectl auth can-i patch deployments -n shop. Prometheus access is read-only, over TLS.

flowchart TB
  P["Burn-rate page fires"] --> C["Confirm impact: SLI by route and version"]
  C --> Q1{"Change in the last 2 hours?"}
  Q1 -- "Yes" --> RB["Roll back or turn off the flag"]
  Q1 -- "No" --> Q2{"A dependency's SLI degraded?"}
  Q2 -- "Yes" --> DG["Degrade gracefully, shed load, call the owner"]
  Q2 -- "No" --> ESC["Escalate: incident commander, more responders"]
  RB --> V{"Burn rate below 1 on 5-min window?"}
  DG --> V
  ESC --> V
  V -- "No" --> C
  V -- "Yes" --> B["Budget accounting: share consumed"]
  B --> Q3{"More than 20% of budget?"}
  Q3 -- "Yes" --> PM["Postmortem with a P0 action"]
  Q3 -- "No" --> R["Review at weekly budget meeting"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class Q1,Q2,Q3,V decision
  class P risk
  class C,RB,DG,ESC,B,PM,R core

Step 1: confirm impact (2 minutes). Is the error concentrated in one version, route or zone?

#!/usr/bin/env bash
set -euo pipefail
# Read-only query endpoint over TLS (placeholder URL; add your proxy's authentication).
PROM_URL="${PROM_URL:-https://prometheus.example.internal}"
# 5-minute error ratio by version (the version label must exist in your metrics).
NUM='sum by (version) (rate(http_requests_total{job="checkout",code=~"5.."}[5m]))'
DEN='sum by (version) (rate(http_requests_total{job="checkout"}[5m]))'
curl -sG "$PROM_URL/api/v1/query" --data-urlencode "query=$NUM / $DEN" \
  | jq -r '.data.result[] | "\(.metric.version) \(.value[1])"'

Step 2: look for a change. Most outages follow a change (Practice 8), so check deployments, configuration pushes and feature flags first.

#!/usr/bin/env bash
set -euo pipefail
kubectl -n shop rollout history deployment/checkout
kubectl -n shop get deployment checkout \
  -o jsonpath='{.metadata.annotations.kubernetes\.io/change-cause}{"\n"}'

Step 3: mitigate before you diagnose. If the new version is the outlier, roll back now; root-cause analysis can wait for the postmortem.^[50] The undo is itself a change: only the Pod template is rolled back (scaling is not),^[51] pods are replaced through the rollout strategy, and the patch appears in the API audit log if auditing is on. Record it in the incident timeline; never roll back across a schema or data migration without the service owner.

#!/usr/bin/env bash
set -euo pipefail
kubectl -n shop rollout undo deployment/checkout
kubectl -n shop rollout status deployment/checkout --timeout=5m

Exception path. No recent change, and the errors are timeouts to the payment gateway? Shed non-critical load, serve a degraded path if one exists, and call the gateway’s owner. Watch for retry storms: limit retries per request, use randomised exponential backoff and a server-wide retry budget.^[48] If the budget is exhausted, the halt applies whatever the cause. The team may keep building features if the gateway’s owners have frozen their own releases, but a hard dependency that could be softened (a fallback, a queue) is reliability work for the team.^[21]

Step 4: verify. Rerun Step 1. The page resets once the 5-minute ratio drops (3 minutes after the incident in Lab 2); watch the 6-hour pair for another half hour.

Step 5: account for the budget. Record the share consumed for the weekly review; above 20%, the policy’s postmortem trigger applies.

Step 6: learn. Ask blamelessly why a normal deployment could do this much harm: canary too short, no version split, slow rollback.^[19]

Use SRE practices when users or the business notice unreliability; several teams change the same production system; operations work grows with traffic or customers; release arguments are settled by seniority rather than data; or you depend on, or are depended on by, other services with reliability expectations.

Do not build an SRE team (yet) when:

• The product has no users who would notice. A prototype needs basic monitoring and a rollback button, not a budget policy.
• Nobody with authority will sign the policy. Start with an SLO pilot and an executive sponsor instead of hiring.
• You cannot staff a sustainable rotation. By Google’s numbers a single-site 24/7 rotation needs about eight people.^[33] With three engineers, reduce what must page out of hours, buy managed services, or accept business-hours support.
• The work is mostly a vendor’s. For SaaS-heavy stacks, spend the effort on SLAs, dependency SLIs and fallbacks; the NoOps page estimates the operations work that remains.
• You want a gatekeeper. A team that says no to releases becomes a bottleneck and the blame target; SRE makes the budget the gate.

Hidden responsibilities. Adopting SRE means maintaining SLO specs, keeping latency thresholds measurable (a bucket boundary per threshold for classic histograms), reviewing SLOs quarterly, enforcing the toil cap, and defending the policy when a big launch collides with an empty budget.

Use the tree to choose a starting point, then the weighted matrix to choose a team model. Both are this Atlas’s tools, not industry standards.

flowchart TB
  Q1["Would users or the business notice an outage today?"]
  Q1 -- "No" --> A1["Basic monitoring and rollback; revisit at launch"]
  Q1 -- "Yes" --> Q2["Will leaders sign an error-budget policy?"]
  Q2 -- "Yes" --> Q3["After cutting toil, is ops load still over half of engineers' time?"]
  Q2 -- "No" --> A2["SLO pilot on one journey plus an exec sponsor"]
  Q3 -- "No" --> A3["SRE practices in product teams"]
  Q3 -- "Yes" --> Q4["Can you staff a sustainable rotation?"]
  Q4 -- "Yes" --> Q5["Shared infrastructure behind many teams?"]
  Q4 -- "No" --> A6["Narrow paging hours, managed services or business-hours support"]
  Q5 -- "Yes" --> A4["Infrastructure SRE team"]
  Q5 -- "No" --> A5["Embedded or kitchen-sink team with a charter"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q1,Q2,Q3,Q4,Q5 decision
  class A3,A4,A5 core
  class A1,A2,A6 neutral

Weighted matrix for choosing a team model (illustrative weights). Score each model 1–5 per criterion, multiply by the weight and add up. The example is a mid-sized company with no SRE team yet, 12 product teams on one shared Kubernetes platform and one revenue-critical checkout.

An infrastructure SRE team wins, with checkout’s SLOs owned by its product team on shared tooling. Google’s article says starting with an infrastructure or tools team “makes the most sense when the application teams are able and willing to define and maintain SLOs”, and recommends product/application teams as a second or later team,^[10] so the last column is for comparison only. Different weights can change the answer; that is why you write them down.

This Atlas’s proposed model, not an industry standard. Levels are cumulative; move up only when the exit criteria hold for your tier-1 services.

flowchart TB
  L1["1. Reactive: users find outages"] --> L2["2. Measured: SLIs on key journeys"]
  L2 --> L3["3. Targeted: agreed SLOs, burn-rate alerts"]
  L3 --> L4["4. Governed: policy changes release pace"]
  L4 --> L5["5. Engineered: reliability designed in"]
  S1["Stall: dashboards nobody owns"] -.-> L2
  S2["Stall: policy never signed"] -.-> L3
  S3["Stall: toil cap without teeth"] -.-> L4
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class L1,L2,L3,L4,L5 core
  class S1,S2,S3 risk

The plan assumes one product area with three to five services, an engaged service owner and an executive sponsor.

Quick wins: deleting pages nobody acts on; a version label on request metrics; the first budget chart. Hard parts: the signature, and the first time the budget runs out before a launch (see the judgement call).

SLO readiness (before you publish an SLO)

• The SLI describes a user-visible outcome of a named user journey.
• Specification and implementation are written down, with known blind spots.
• “Valid” excludes load tests and other out-of-scope traffic, in the reviewed spec.
• Latency thresholds are measurable: a bucket boundary at each threshold (classic histograms) or histogram_fraction accepted as an estimate (native histograms).
• At least four weeks of data show the target is achievable, or an aspirational SLO is declared.
• Product manager, developers and on-call have each agreed in writing.
• Critical dependencies’ measured availability supports the target.

Production readiness review (adapted from the SRE book’s PRR areas)^[18]

• Architecture: critical dependencies and their SLOs documented.
• Monitoring: SLIs and burn-rate alerts live; every page links to a runbook.
• Emergency response: rotation staffed, escalation path, playbooks for top failure modes.
• Capacity: forecast beyond procurement lead time; load test this quarter.
• Change management: progressive rollout, canary with a control group, tested rollback.
• Performance: availability, latency and efficiency measured against targets.

On-call health (monthly)

• No shift exceeded two incidents, or the cause has an action.^[33]
• Every page led to action; others deleted or demoted.
• Everyone was on call this quarter; nobody above the agreed share.
• Toil share recorded and below the cap.

Quarterly SLO review

• Budget burns compared with support tickets and user feedback.
• SLO coverage of critical journeys checked.
• Every policy invocation and escalated decision reviewed.
• Decision recorded (keep, tighten, loosen, change the SLI), with reason and next review date.

1. SLO document (structure adapted from the workbook’s example SLO document).^[15]

SLO DOCUMENT: <service name>
Status: draft | approved          Version: 1.0
Authors: <names>                  Reviewers (technical): <names>
Approvers (business): <names>
Approved: YYYY-MM-DD              Next review: YYYY-MM-DD

1. Service overview   <two or three sentences: users, journeys, dependencies>
2. SLIs and SLOs      one block per journey:
   Journey:             Pay for basket
   SLI specification:   confirmed order or correct decline within 2 s
   SLI implementation:  load-balancer logs, POST /checkout/pay
   SLO and window:      99.9%, rolling 28 days
3. Rationale          <user need; data used (dates); rounding; dependency check>
4. Error budget       <(1 - SLO) x valid events; how it is calculated; link to policy>
5. Clarifications     <what counts as valid; known blind spots; maintenance treatment>
6. Changelog          <date, change to SLI or target, reason>

2. Error-budget policy (structure adapted from the workbook’s example policy; the thresholds are examples to negotiate).^[21]

ERROR-BUDGET POLICY: <service name>
Authors / reviewers / approvers: <names>
Approved: YYYY-MM-DD   Review: YYYY-MM-DD

Goals:      protect users from repeated SLO misses; balance reliability with features.
Non-goals:  punishing teams for missing SLOs.

When the budget is exhausted over the trailing 4 weeks:
  - Halt all changes and releases except P0 and security fixes until the service
    is back within SLO. The cause of the miss does not lift the halt.
  - The cause decides the work. Record which case applied.
    The team MUST work on reliability if:
      - a code bug or procedural error in the service caused the miss;
      - a postmortem shows a hard dependency that could be softened;
      - miscategorised errors hid budget use that would have shown a miss.
    The team MAY continue feature work (unreleased) if:
      - a company-wide network problem caused the outage;
      - another team's service caused it, and that team has frozen its releases;
      - out-of-scope users (load tests, penetration tests) used the budget;
      - miscategorised errors used budget although no users were affected.
    Fix measurement errors in the reviewed SLO spec, then recompute.
When a single incident consumes more than 20% of the 4-week budget:
  - Postmortem required, with at least one P0 action.
When one class of outage consumes more than 20% of the budget in a quarter:
  - P0 item in the next quarter's plan.
Escalation: disagreements about the calculation or the actions go to
  <named executive role>, who decides; record the decision.

3. Toil log (one row per occurrence; units per the workbook’s advice to pick one consistent measure).^[31]

date,engineer,category,task,minutes,automatable,ticket
2026-10-12,alex,production interrupts,restart stuck queue consumer,25,yes,OPS-1412
2026-10-12,sam,release shepherding,manual canary checks for v4.18,60,yes,OPS-1415
2026-10-13,alex,business processes,grant read access to analytics bucket,15,yes,OPS-1419

4. SRE team charter (one page)

SRE TEAM CHARTER: <team name>
Mission:      keep <services> as reliable as their users need, at the lowest
              cost in toil and velocity.
Scope:        <services or platforms>. Out of scope: <list>.
Model:        <kitchen sink | infrastructure | tools | product | embedded | consulting>
Commitments from SRE:
              SLO tooling; burn-rate alerting; PRRs; capacity planning;
              shared on-call for onboarded services.
Commitments from development teams:
              own SLOs with product; keep monitoring and runbooks current;
              join postmortems; take back operational work above the cap.
Toil cap:     <e.g. 50%> of SRE time; measured <monthly>;
              excess redirected by <mechanism>.
Hand-back:    if development commitments lapse for <period>, SRE hands back the
              pager after notice to <role>.
Review:       charter reviewed <yearly> by <sponsor>.

The postmortem template and its review checklist live in IncidentOps; for budget-triggered reviews, fill in the budget consumed and include the P0 action the policy requires.

The approaches combine; they are not alternatives you must pick between.

Is SRE just operations with a new name? It should not be. The differences are mechanisms: explicit SLOs, a budget that changes release behaviour, an enforced cap on operational work, and engineers expected to automate toil away. Without them, you have the “Fake SRE” anti-type.^[3]

Do we need an SRE team to practise SRE? No. Many organisations start with SLOs and error budgets owned by product teams; Google’s team-model article recommends allocating some engineering time and at least one part-time advocate before choosing a team model.^[10]

What is the difference between an SLO and an SLA? Ask what happens when it is missed. If there is no explicit consequence, it is almost certainly an SLO; an SLA carries consequences, usually financial.^[4]

Should SLO alerts replace all other alerts? For pages, mostly: pages should reflect user impact and need human action.^[28] Keep a few pages for imminent risks that SLIs cannot see yet, such as an expiring certificate or a filling disk.

Does SRE apply to data pipelines and ML? Yes, with different SLIs: freshness, correctness and coverage for pipelines.^[13] Model quality is harder to express as good/valid events; see MLOps and LLMOps.

Established capability.

• Dependency budgeting. The Calculus paper corrects a misreading of the extra-9 rule: deeper dependencies do not each need another nine. Count each unique critical dependency once; with N of them, each gets 1/N of the dependency share. Its example splits a 0.01% budget half and half between the service’s own failures and its dependencies, and typical services have about 5 to 10 critical dependencies.^[9]
• Overload engineering. Cascading failures grow through positive feedback, and overload is their most common cause. Deadline propagation, load shedding, graceful degradation and retry budgets are reliability features.^[48]

Active practice, still settling. The first two items are Google’s own practice, published on sre.google, not yet an industry consensus.

• Systems-theoretic incident analysis. Google says it is “reexamining our beliefs about why incidents occur” and uses two methods based on the STAMP causality model: STPA, which is forward-looking and looks for risks nobody is seeking, and CAST, which is retrospective and feeds postmortems.^[32] Both extend Practice 6.
• Product-focused reliability. SREs accept accountability for a product’s critical functionality rather than for a set of services, define reliability in the product’s terms (critical user journeys), and combine service, client-side and end-to-end SLOs.^[25] It extends Practice 1.
• Composite SLOs. OpenSLO v1 lets an SLO weight several objectives (compositeWeight),^[41] and a v2alpha schema is published.^[38] Check your back end before relying on composites for alerting.
• Chaos engineering tied to budgets. The principles of chaos call for steady-state hypotheses, real-world variables, production experiments, automation and a minimal blast radius;^[40] experiment only when the budget can absorb it. In Catchpoint’s 2026 report only 17% of respondents ran such experiments regularly in production, and nearly two-thirds said performance degradations are as serious as outages, an argument for latency SLIs on every user-facing journey.^[14]

Claimed direction, not independently evidenced. Google’s announcement of the second edition says SLO practice has matured into “user-centric, multi-tiered SLOs with dynamic burn-rate alerting and automated policy enforcement”, and that the book covers generative AI for incident triage and postmortems.^[12] Vendors market AI agents that investigate or remediate incidents, but independent evidence on outcomes is thin. Keep human approval for production changes and for automated enforcement of a halt.

Leaders’ route: the 30-second definition, Practice 3, the decision framework, the judgement call and the cheat sheet.

Assumptions. The leak is the team’s own code, so the policy requires reliability work, and no cause lifts the halt. The feature can sit behind a flag for that customer alone. The business values the missed-date penalty above a few more days of overspend; nobody can quantify either side.

Option A: ship as planned. Attractive because the contract is concrete and the budget abstract. Wrong as a default: once a policy is ignored for a deadline, every later halt becomes a negotiation by seniority, and the change lands on a service already failing users.

Option B: halt absolutely. Faithful to the text, but it may protect the policy at the expense of its purpose: a flag limited to one tenant, after the leak fix, may put few users at risk.

Decision process. Use the policy’s escalation path, which covers disputes about its actions: the named executive decides on a written request that states the risk and the conditions. A defensible outcome is a recorded exception with conditions: ship the leak fix first and confirm the burn rate has fallen below 1; launch behind a flag for the one customer, with a canary compared against a control group;^[20] rehearse the rollback; keep every other release halted until the service is back within SLO; and review the exception at the next budget meeting.

Why this is not a loophole. The exception is visible, narrow, conditional and decided by the role the policy names; if such exceptions become frequent, the SLO, policy or release process must change. Remaining uncertainty: if the leak fix does not reduce the burn, the conditions fail and the launch slips; say so up front.

After this page you should be able to explain SLI, SLO, SLA, error budget, burn rate and toil as one control loop; design an SLI for a user journey, including what counts as valid and good; calculate budgets, burn rates and time to exhaustion, and check a target against dependencies; configure and test multiwindow, multi-burn-rate alerts; write an SLO document and an error-budget policy that stakeholders can sign; and evaluate whether to adopt SRE practices, build an SRE team, or neither.

Mastery check

1. A service has an SLO of 99.95% over 28 days and serves 20 million valid requests in that window. What is its error budget in requests, and roughly how many minutes of total outage would spend it if traffic were even?

2. With a 99.9% SLO, the error ratio has been 0.4% for the last 30 hours (and near zero before). What is the burn rate, and which of the workbook’s alert pairs fire?

3. Users complain about failed checkouts all week, yet the SLO is green. Give two likely causes and a check for each.

4. Your service has three unique critical dependencies whose measured availability is 99.9% each. The product manager wants a 99.95% SLO. Is it realistic, and what are your options?

5. Over the trailing four weeks a service has used 120% of its error budget. Failures from a partner’s load test against production account for 50 points of that. Does the release halt apply?

6. Which of these are toil? (a) Writing a tool that resizes volumes automatically; (b) resizing a volume by hand for each new customer; © a postmortem; (d) approving the same access request type every day.

7. A vice-president asks for a 100% availability SLO on payments “because money is involved”. How do you respond?

SRE’s lasting contribution is not a job title or a tool. It makes reliability negotiable in numbers: decide what users need, admit that 100% is neither possible nor desirable, and treat the gap as a budget that funds change. The rest (burn-rate alerts, toil caps, readiness reviews, blameless postmortems, sustainable on-call) protects that budget and gives engineers time to engineer.

The hard part is organisational. Measuring an SLI takes a week; getting product, engineering and an executive to sign a policy that can stop a launch takes months, and keeping it after the first painful halt takes more. Teams that skip that step end up with dashboards, not SRE. Teams that do it gain something rare: a shared, explicit rule for one of the most contested trade-offs in software, and the evidence to change the rule when it is wrong.

All sources were opened on 2026-10-16; those marked 2026-10-17 were re-checked that day. “SRE book” is Site Reliability Engineering: How Google Runs Production Systems (O’Reilly, 2016) and “Workbook” The Site Reliability Workbook (O’Reilly, 2018), cited from their free editions at sre.google; quotations were checked against the page text. Re-check version-specific details before use.

Further reading

• Ben Treynor Sloss on SRE and SRE team responsibilities — Google, interview page, n.d.
• Prometheus HTTP API and promtool command reference — Prometheus documentation.
• Sloth CLI usage and the OpenSLO v1 Service schema.

Changelog

• 2026-10-16 — First version (r0): researched, written and diagram-checked by the managing author; Lab 2 executed; Labs 1 and 3 and the runbook commands not executed. Awaiting independent fact-check review.
• 2026-10-17 — Revision r1 after the independent review: error-budget policy restated as the workbook’s two tiers; one error definition across the reference implementation and labs; Sloth set to 28 days; second edition, SREcon, STAMP/CAST and product-focused reliability sourced; postmortem template and action-closure KPI handed to IncidentOps; runbook least privilege; repetition cut. Lab 2 re-executed after reformatting.
• 2026-10-17, post-score polish: traffic label and its missing-label caveat added to the reference implementation; survey fieldwork dated; Sloth v0.16.0 and Prometheus 3.15 named as the versions checked; figure 5 labels re-wrapped; four glossary quotations restored to the sources’ lower case.