---
title: "ObservabilityOps"
slug: observabilityops
category: reliability
term_status: ambiguous
evidence_status: verified
summary: "ObservabilityOps is this Atlas's name for running observability as an internal product: the telemetry pipeline, standards, budgets and alert hygiene that let engineers answer new questions about production at a predictable cost. The label itself is not standardised."
audience: [Observability and platform engineers, SREs and service owners, Engineering leaders, Architects, Beginners]
last_verified: 2026-10-17
key_takeaways:
  - "ObservabilityOps is not an established term: no definition, standards body or certification uses it. This page reads it as operating the observability system itself, which organisations do under names such as observability team, observability platform or observability centre of excellence."
  - "OpenTelemetry (a CNCF graduated project since May 2026) is the vendor-neutral way to produce and move telemetry. It is not a backend, so storage, query and alerting remain separate choices."
  - "Cardinality is the main cost and stability risk: every unique label combination is a new time series, so unbounded values such as user IDs belong in traces or logs, not in metric labels."
  - "Sampling, retention tiers and deleting unused signals control cost. Google's SRE book treats signals that no dashboard or alert uses as candidates for removal."
  - "Page on user-visible symptoms, make every page actionable, and test the alert path end to end instead of alerting on each monitoring component."
  - "Run the platform with its own SLOs, per-team budgets and a change process. A central team that gatekeeps every query or owns everyone's alerts is an anti-pattern."
  - "Route spans by trace ID for tail sampling and by service for span metrics: every metric stream needs a single writer, and a trace survives only if every hop propagates its context."
related: [sre, incidentops, aiops, secdataops, appops, kubernetesops, platformops, finops]
---

## What ObservabilityOps is

### In 30 seconds: the short answer

**ObservabilityOps** has no standard definition: searches on 2026-10-16 found no definition, standards body, certification or
vendor programme for the label (see [term status](#term-status-and-the-two-readings)). **In this Atlas it means running
observability as an internal product:** operating the telemetry pipeline and back ends, setting instrumentation standards, keeping
cardinality and cost within budgets and keeping alerts useful, so that any engineer can answer a new question about production
quickly, from trustworthy data, at a predictable cost.

Observability itself is well defined. OpenTelemetry describes it as "the ability to understand the internal state of a system by
examining its outputs", typically traces, metrics and logs.[^otelwhat] DORA separates *monitoring*, which watches predefined metrics
or logs, from *observability*, which explores properties and patterns "not defined in advance".[^dora] ObservabilityOps is the
operational work that keeps that ability available and affordable.

**For** engineers who run the stack, SREs and service owners who depend on it, and leaders who pay for it.

### At a glance

| Item | Summary |
|---|---|
| Term status | **Ambiguous.** Almost unused, with two plausible readings; this page takes "operating the observability system". The practice exists under other names: GitLab's Observability team, the UK Ministry of Justice's Observability Platform team, New Relic's "Observability Center of Excellence".[^glteam][^moj][^nrcoe] |
| Goals | Telemetry that answers new questions; alerts worth acting on; predictable, allocated cost; trustworthy data (no silent loss, no secrets). |
| In scope | Instrumentation standards; telemetry pipelines, including log collection and trace propagation; metric, log, trace and profile back ends; cardinality, sampling and retention; dashboard and alert design; monitoring the monitoring; cost governance; tools. Terms are defined in the [glossary](#glossary-and-mental-model). |
| Out of scope | SLO targets and error-budget policy → [SRE](../sre/); incident roles and communications → [IncidentOps](../incidentops/); ML event correlation → [AIOps](../aiops/); security telemetry and forensic retention → [SecDataOps](../secdataops/); resilience in code → [AppOps](../appops/); the cloud bill → [FinOps](../finops/). |
| Prerequisites | Running services in production ([DevOps](../devops/)); SLI and SLO vocabulary ([SRE](../sre/)). |
| Key outputs | Instrumentation standard, pipeline with SLOs, per-team telemetry budgets, reviewed alerts, unit-cost report. |

### The problem it solves

When nobody owns the observability system, three failures recur.

1. **Cost grows faster than value.** Telemetry volume follows traffic and code, not the questions anyone asks. In Grafana Labs'
   2025 survey (1,255 self-selected respondents, so indicative at best) mean reported observability spend was 17% of compute
   infrastructure spend (median 10%), and 37% named "Costs too much" as a concern.[^gsurvey2025]
2. **Noise drowns signal.** In the same vendor's 2026 survey, the top three concerns were complexity/overhead (38%),
   signal-to-noise challenges (34%) and cost (31%).[^gsurvey2026] The human side of alert noise is covered in [AIOps](../aiops/)
   and [IncidentOps](../incidentops/).
3. **The tools fail when they are needed most.** One label filled with user-supplied values can create millions of series. Cloudflare
   describes how such a "cardinality explosion" can make Prometheus run out of memory, so that "you lose all observability as a
   result".[^cfprom]

| Operating reality | Nobody owns it | Run as a product |
|---|---|---|
| Instrumentation | Each team picks libraries and names | One standard (OpenTelemetry and semantic conventions), defaults in templates |
| Data volume | Whatever the code emits | Budgets per team; sampling and retention set by value |
| Alerts | Accumulate; nobody deletes them | Reviewed against "Every page should be actionable"[^sremon] |
| Platform health | Discovered during an incident | Platform SLOs and end-to-end checks |
| Cost | One shared bill | Shown per team and per unit of work |

### Term status and the two readings

The label could mean either of two things:

1. **Operating *with* observability:** running production by looking at telemetry, which duplicates [SRE](../sre/),
   [IncidentOps](../incidentops/) and [AIOps](../aiops/).
2. **Operating the observability system:** treating telemetry as an internal product with users, SLOs, a budget and governance.
   **This page takes this reading.**

::: note Editorial distinction
Choosing reading 2 is this Atlas's decision, not industry usage, grounded in practice under other names. Google's SRE book calls
monitoring "a significant engineering endeavor in and of itself"; a Google SRE team of 10–12 typically has one or two members
assigned mainly to monitoring.[^sremon] GitLab's Observability team states its mission as
"to deliver and maintain a world-class observability offering and frictionless operational experience for team members at
GitLab".[^glteam] Grafana Labs' 2026 survey asks about "observability operations" and describes a mature, centralised practice as one
where "a centralized observability team runs the observability platform and provides best practices and support to product teams,
but does not directly manage observability for individual services".[^gsurvey2026] That is the operating model taught here. The
nearest established names for the work are "observability engineering" and "observability platform
team"; the second edition of *Observability Engineering* adds a section on observability governance.[^oebook]
:::

### What it is not

- **Not the product "ObserveOps".** Motadata sells an observability platform with that name.[^motadata]
- **Not SRE.** SRE sets reliability targets and policy; ObservabilityOps runs the measurement system they depend on.
- **Not monitoring under a new name.** Exploring what was "not defined in advance"[^dora] needs high-dimensional data, which is
  where most cost and governance problems sit.
- **Not a team that does observability *for* everyone.** DORA writes: "Monitoring systems should not be confined to a single
  individual or team within an organization."[^dora] A central team can run those systems, provided every team can query and alert
  on them itself. Service teams still instrument their code and own their alerts.

## Where it comes from and where it sits

### Origins and evolution

| Year | Milestone |
|---|---|
| 1960 | Kalman defines observability in control theory: internal state inferred from outputs (as cited by the CNCF whitepaper)[^tagobs] |
| 2012, 2015 | Brendan Gregg publishes the USE method; Tom Wilkie creates the RED method[^use][^red] |
| 2016 | Google's SRE book: four golden signals, paging on symptoms[^sremon] |
| 2018 | Prometheus graduates in the CNCF[^cncfprom] |
| 2019 | OpenTracing and OpenCensus merge into OpenTelemetry, accepted by the CNCF in May[^otelwhat][^cncfotel] |
| 2020 | W3C Trace Context becomes a Recommendation (edited Recommendation 2021)[^w3ctc] |
| 2023 | Prometheus 2.47 adds experimental OTLP ingestion;[^promcl] the CNCF TAG Observability whitepaper v1.0 prefers "primary signals" to "three pillars"[^tagobs] |
| 2024 | Prometheus 3.0 adds OpenTelemetry interoperability features; Jaeger v2 is rebuilt on the OpenTelemetry Collector[^prom3][^cncfjaeger] |
| 2026 | OpenTelemetry Profiles enters public alpha (March); OpenTelemetry graduates (CNCF moved it on 11 May; announced 21 May)[^otelprof][^cncfotel][^otelgrad] |

No source dates the first observability teams; one practitioner wrote in 2022 that the concept was "relatively new".[^mustin]

### Glossary and mental model

| Term | Meaning on this page |
|---|---|
| **Telemetry, signal** | Data a system emits about itself: metrics, logs, traces, profiles ("primary signals" in the CNCF whitepaper).[^tagobs] |
| **Time series, sample** | A metric name plus one unique set of labels is one time series; each timestamped value is a sample.[^cfprom] |
| **Label, attribute, cardinality** | A key-value dimension (Prometheus label, OpenTelemetry attribute); cardinality is its number of distinct values, or of a metric's label combinations. |
| **Active series** | Series a time-series database holds in memory; the main driver of its memory use. |
| **Span, trace** | One timed operation; the tree of spans for one request, linked by propagated context such as the W3C `traceparent` header.[^w3ctc] |
| **Exemplar** | A trace ID attached to a metric sample, linking a spike to an example request.[^tagobs] |
| **Semantic conventions** | OpenTelemetry's standard names for attributes, spans and metrics (v1.44.0, October 2026).[^semconv] |
| **OTLP, Collector** | OpenTelemetry's protocol, and its proxy that receives, processes and exports telemetry in pipelines.[^otelcolcfg] |
| **Head or tail sampling** | Keeping a trace or not at its start (cheap, blind to the outcome) or after seeing it (can keep every error; stateful).[^otelsampling] |
| **RED, USE, golden signals** | Rate, errors, duration per service;[^red] utilisation, saturation, errors per resource;[^use] latency, traffic, errors, saturation.[^sremon] |
| **Context propagation, baggage** | Passing trace context, and optional key-value baggage, to the next service in HTTP or message headers.[^otelctx] |
| **Telemetry budget** | This Atlas's term: an agreed limit per team on series, log volume or spend, with an exception path. |

**One slow request, four signals (illustrative).** A checkout request takes 4 seconds instead of 300 ms.

| Signal | What you see | What it answers | Main cost driver |
|---|---|---|---|
| Metric | p99 of `http.server.request.duration` rises for one route | Is it happening, how often, since when? | Active series (label combinations) |
| Trace | The request's spans: 3.6 s waiting in the payment client | Where in the call path did the time go? | Spans kept (sampling rate) |
| Log | A structured line with `trace_id` and "retrying payment provider" | What did the code say at that moment? | Bytes ingested and indexed |
| Profile | CPU and memory per function across many requests | Which code is expensive in aggregate? | Sampling frequency and retention |

**Mental model: a telemetry supply chain.** Producers emit signals; a pipeline moves, enriches, redacts and samples them; back ends
store them; consumers (dashboards, alert rules, queries, [AIOps](../aiops/)) turn them into decisions. ObservabilityOps owns the
middle. Its unit of value is a question answered quickly; its unit of cost is the bytes and series kept.

### How ObservabilityOps relates to other Ops

::: figure ObservabilityOps between telemetry producers and consumers
```mermaid
flowchart TB
  subgraph prod["Producers"]
    A["`AppOps:
    service code`"]
    K["`KubernetesOps, CloudOps:
    infrastructure`"]
  end
  F["`FinOps:
  cost allocation`"]
  O["`ObservabilityOps: pipeline,
  storage, standards, budgets`"]
  SD["`SecDataOps:
  security telemetry`"]
  subgraph cons["Consumers"]
    S["`SRE:
    SLOs`"]
    I["`IncidentOps:
    response`"]
    AI["`AIOps:
    correlation`"]
  end
  A --> O
  K --> O
  F -. "budgets" .-> O
  O -. "shared tools" .- SD
  O --> S
  O --> I
  O --> AI
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class O core
  class A,K,S,I,AI,F,SD neutral
```
Text equivalent: service owners (AppOps) and infrastructure teams (KubernetesOps, CloudOps) produce telemetry; ObservabilityOps runs
the pipeline, storage, standards and budgets; SRE, IncidentOps and AIOps consume the data. Dotted lines connect FinOps, which sets
budgets, and SecDataOps, which runs security telemetry on similar tools.
:::

| Neighbour | Shared scope | This page's focus | Go there for |
|---|---|---|---|
| [SRE](../sre/) | SLIs, alerting | The measurement system; alert hygiene | SLO targets, error-budget policy, burn-rate thresholds |
| [IncidentOps](../incidentops/) | Dashboards during incidents | Available, trustworthy data | Roles, communications, MTTA and MTTR |
| [AIOps](../aiops/) | Event data | Clean, labelled, affordable input | Correlation models and how to evaluate them |
| [SecDataOps](../secdataops/) | Pipelines and tools | Operational telemetry | Security schemas, coverage, forensic retention |
| [AppOps](../appops/) | Instrumentation | Standards, libraries, defaults | Resilience patterns, the owner's runbook |
| [PlatformOps](../platformops/) | Platform as a product | One capability (the CNCF lists observability)[^cncfplat] | Portals, golden paths |
| [FinOps](../finops/) | Cost | Telemetry unit cost and budgets | Allocating the cloud bill, commitments |

## Principles and operating model

### Charter and principles (this Atlas's proposal)

No official ObservabilityOps manifesto exists. **Mission:** every engineer can answer a new
question about production within minutes, from trustworthy data, at an agreed cost.

1. **Collect for a consumer.** Each signal serves a dashboard, an alert or a known debugging question. Signals used by no prebaked
   dashboard and no alert "are candidates for removal" (Google SRE book).[^sremon]
2. **Open standards at the edges.** Instrument with OpenTelemetry so the back end can change: "You own the data that you
   generate."[^otelwhat]
3. **Bounded dimensions in metrics, rich context in events.** Unbounded identifiers go into traces and logs, never metric
   labels.[^promnaming][^tagobs]
4. **Symptoms page; causes explain.** Page on what users feel and use cause-level data to diagnose.[^sremon][^promalert]
5. **The platform has SLOs and is monitored end to end.**[^promalert]
6. **Cost is visible and allocated** to the teams and units of work that generate it.
7. **Enable, do not gatekeep.** Self-service within guardrails; the platform team is not the only team that can query or alert.[^dora]
8. **Minimise and redact at the edge.** Personal data and secrets do not enter the pipeline unless needed.[^otelsensitive]

**Invariants:** dropped data is counted and alerted on; no secrets in telemetry; every page has an owner and a runbook. **Non-goals:**
owning other teams' SLOs, being on call for their services, reading every log.

### Building blocks

| Layer | Typical components |
|---|---|
| Instrumentation | OpenTelemetry SDKs and zero-code agents, infrastructure exporters, eBPF instrumentation |
| Collection and pipeline | Collector agents and gateways; optionally a message queue |
| Storage | Time-series database, log, trace and profile stores, object storage |
| Query, dashboards, alerting | PromQL, LogQL and similar; dashboards as code; rule evaluation and paging |
| Governance | Standards, budgets, signal catalogue, access control, cost reports |

### Roles, ownership and RACI

- **Observability platform team:** a product owner and engineers who run the pipeline and back ends. In a small organisation this may be
  one person, as at Mastodon (Case 1).[^otelmastodon]
- **Community structure:** New Relic's guidance proposes a core team (typically one to three people), a council that ratifies
  standards and a volunteer guild.[^nrcoe] It is one vendor's model, useful when a full team is not justified.

| Activity | Platform team | Service team | SRE | Security / DPO | FinOps |
|---|---|---|---|---|---|
| Instrumentation standard and SDK defaults | A/R | C | C | C | I |
| Instrumenting a service; its dashboards and alerts | C | A/R | C | I | I |
| Pipeline and backend operations; platform on-call | A/R | I | C | I | I |
| SLO targets and error-budget policy | C | R | A | I | I |
| Redaction rules and retention policy | R | C | C | A | I |
| Telemetry budget and cost allocation | R | C | I | I | A |
| Tool selection and contracts | A/R | C | C | C | C |

R = responsible, A = accountable, C = consulted, I = informed. Adjust to your organisation, but leave no row blank.

### Lifecycle of a signal (Day 0, Day 1, Day 2)

::: figure Lifecycle of a telemetry signal, with budget and retirement loops
```mermaid
flowchart TB
  P["Propose: owner, question it answers, expected series or volume"]
  R{"Meets standard and budget?"}
  X["Redesign: move IDs to span attributes, or request a time-limited exception"]
  I["Instrument and ship through the pipeline"]
  U["Use: dashboards, alerts, ad hoc queries"]
  Q{"Used in 90 days?"}
  T["Downsample, move to a cheaper tier, or retire"]
  P --> R
  R -- "yes" --> I
  R -- "no" --> X
  X --> P
  I --> U
  U --> Q
  Q -- "yes" --> U
  Q -- "no" --> T
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class P,I,U core
  class R,Q decision
  class X,T risk
```
Text equivalent: a new signal is proposed with its owner, question and expected size. If it meets the standard and budget it is
instrumented and used; otherwise it is redesigned (for example, identifiers moved to span attributes) or given a time-limited
exception, and proposed again. A signal unused for 90 days is downsampled, moved to a cheaper tier or retired.
:::

- **Day 0 (design):** choose standards, architecture, tools, budget rules and retention per data class.
- **Day 1 (onboarding):** templates give each service OpenTelemetry defaults, a RED dashboard and a symptom alert.
- **Day 2 (operations):** capacity, upgrades, reviews and pruning. Collector releases are frequent and rename components
  (see the tooling landscape).[^colchangelog][^contribchangelog] Mastodon upgrades "within a day or two of each
  release" and sees breaking changes as a sign of healthy development, provided you stay current.[^otelmastodon]

### Operating model and governance

| Ceremony | Cadence | Inputs | Output |
|---|---|---|---|
| Platform health review | Weekly | Platform SLIs, dropped data, queue use, pending upgrades | Actions, upgrade plan |
| Budget review (showback) | Monthly | Series, log volume and cost per team | Budget changes, exceptions with expiry |
| Alert review | Monthly per team | Pages, each labelled actionable or not | Deleted, merged or fixed alerts |
| Signal pruning | Quarterly | Signals unused for 90 days | Retirements and tier moves |
| Standards council | Quarterly | Proposals (new conventions, tools) | Architecture decision records |

**Change control:** pipeline and rule configuration lives in Git, is validated in CI (`otelcol validate`, `promtool`, a linter such as
pint)[^otelcolcfg][^promunittest][^pint] and is deployed by reconciliation, as Mastodon does with the OpenTelemetry Operator and Argo
CD.[^otelmastodon] **Escalation:** a platform outage is an incident like any other ([IncidentOps](../incidentops/)); budget exceptions
end with whoever owns the cost.

## Architecture and workflow

### Reference architecture

::: figure Reference architecture of a telemetry platform
```mermaid
flowchart TB
  subgraph src["1. Producers"]
    APP["`Services with
    OpenTelemetry SDKs`"]
    INF["`Hosts, clusters,
    container logs`"]
  end
  subgraph agt["2. Agents"]
    AG["`Agent per node: metadata,
    memory limit, logs, routing`"]
  end
  subgraph gwt["3. Gateway replicas, mTLS"]
    TS["`Tail sampling:
    spans by trace ID`"]
    SM["`Span metrics:
    spans by service`"]
    PL["`Metrics, logs, profiles:
    redact, filter`"]
  end
  T[("`Trace
  store`")]
  M[("`Metric
  store`")]
  L[("`Log
  store`")]
  PR[("`Profile
  store`")]
  D["`Dashboards
  and queries`"]
  R["`Alert rules
  and routing`"]
  W["`Meta-monitoring:
  independent watcher`"]
  APP --> AG
  INF --> AG
  AG -- "by trace ID" --> TS
  AG -- "by service" --> SM
  AG -- "any replica" --> PL
  TS --> T
  SM --> M
  PL --> M
  PL --> L
  PL --> PR
  M --> D
  L --> D
  T --> D
  PR --> D
  M --> R
  W -.-> PL
  W -.-> R
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class AG,TS,SM,PL core
  class APP,INF,M,L,T,PR,D,R neutral
  class W risk
```
Text equivalent: services and infrastructure send telemetry to a Collector agent on each node, which adds metadata, limits memory,
collects container logs and sends data to gateway replicas over mutual TLS. Spans travel twice: by trace ID to tail sampling, which
stores the kept traces, and by service to the span-metrics connector, which writes RED metrics. Metrics, logs and profiles go to any
replica for redaction and filtering. Dashboards read all stores, alert rules mostly metrics; an independent watcher checks the
gateways and the alert path.
:::

| Component | Responsibility | Main risk | Control |
|---|---|---|---|
| SDKs | Emit signals with semantic conventions; `service.name` is required[^semres] | Missing names; unbounded attributes; lost context | Templates, CI checks |
| Agent | Host signals, container logs, Kubernetes metadata; route spans | Memory pressure on nodes | `memory_limiter` first;[^memlim] resource limits |
| Gateway | Central credentials and policy | One more thing that can fail; latency; cost[^otelgw] | Two or more replicas; sending queues; persistent queue on critical paths[^otelresil] |
| Tail sampling | Keep errors and slow traces, sample the rest | Stateful; all spans of a trace must reach one instance[^tailsamp] | `load_balancing` exporter with `routing_key: traceID`[^lbexp] |
| Span metrics | RED metrics from every span, before sampling | Several replicas writing the same series[^otelgw][^spanmetrics] | `load_balancing` exporter with `routing_key: service`[^lbexp] |
| Back ends | Store and query | Cardinality; retention cost; local Prometheus is "not clustered or replicated"[^promstorage] | Limits, long-term store, tiers |
| Alerting | Evaluate rules, route, page | Noise; silent failure | Symptom-based rules; meta-monitoring |

**Alternatives.** Sending from SDKs straight to a back end is simplest but leaves no place for central policy; a single Collector
tier, as at Mastodon (Case 1), avoids routing altogether. A gateway tier buys central credentials and policy at the price of "one more
thing to maintain and that can fail", latency and resources. It must also respect the single-writer rule: every metric stream needs
"a single writer and a globally unique identity", or the series shows gaps, jumps or out-of-order errors.[^otelgw] That rule is why
span metrics are routed by service.

**Trust boundaries.** Anything that can reach an OTLP endpoint can create series and cost. The Collector binds to localhost by
default; documentation examples use `0.0.0.0` only for convenience.[^otelcolcfg] Artefact 1 states who can reach each endpoint.
Logs and traces can contain personal data, so query access needs roles and tenancy.

### End-to-end workflow: onboarding a service

::: figure Onboarding a service, with the over-budget exception path
```mermaid
sequenceDiagram
  participant BO as Budget owner
  participant ST as Team
  participant OB as ObsOps
  participant CI as CI
  ST->>OB: Request: signals, size
  OB->>OB: Check standard, PII, budget
  alt Within budget
    OB->>CI: Merge routes, limits
    CI->>CI: Validate, test rules
    CI-->>OB: Deployed
    OB-->>ST: Dashboard, alerts live
  else Over budget or risky
    OB-->>ST: Redesign or seek exception
    ST->>BO: Cost and expiry date
    BO-->>ST: Approve or reject
    ST->>OB: Approved exception
    OB->>CI: Merge with expiry date
  end
  OB->>ST: Review after 30 days
```
Text equivalent: the service team (Team) asks the platform team (ObsOps) to onboard signals of a given size. ObsOps checks the
standard, personal-data rules and budget. On the happy path ObsOps merges routes and limits, CI validates them and tests the alert
rules, and the team gets a template dashboard and alerts. On the exception path the team redesigns or asks the budget owner for a
time-limited exception, which is merged with its expiry date. ObsOps reviews usage and noise after 30 days.
:::

**Acceptance criteria for "onboarded":** telemetry arrives under the right `service.name`; the standard dashboard renders; at least
one symptom alert has a runbook and an owner; series and log volume are inside budget; a test request yields one unbroken trace;
nothing in the debug output matches the redaction patterns.

## Practices, frameworks and tools

### Core practices and playbooks

| Practice | Goal | Owner | Verification |
|---|---|---|---|
| Instrumentation standard | Same names everywhere | Platform team | Share of telemetry under `unknown_service` near zero |
| Cardinality control | Stable, affordable metrics | Platform and service teams | Series per team within budget |
| Sampling | Keep the informative traces | Platform team | No dropped-too-early traces; RED metrics unaffected |
| Context propagation | Whole traces across every hop | Platform team sets it; service teams apply it | A test request through every hop yields one trace |
| Log pipeline | Parsed, correlated, affordable logs | Platform team | Share of log records with severity and `trace_id` set |
| Retention tiers | Pay for age only when needed | Platform team with data protection | Storage cost per tier; policy met |
| Dashboard and alert design | Fast diagnosis, few pages | Service teams | Page actionability trend |
| Meta-monitoring | Know when the platform is blind | Platform team | Alert-path heartbeat never silently stops |

**Instrumentation standard.** Use OpenTelemetry SDKs and semantic conventions. `service.name` is required; when it is missing, SDKs
fall back to `unknown_service:` plus the executable name,[^semres] so its share is a cheap hygiene metric. Record HTTP routes as templates: the
conventions say "the URI path can NOT substitute" `http.route`.[^semhttp] Emit events as logs correlated with the current span;
OpenTelemetry is deprecating the Span Event API.[^otelspanev]

**Cardinality control.** Prometheus advises that "every unique combination of key-value label pairs represents a new time series" and
that labels must not hold "user IDs, email addresses, or other unbounded sets of values".[^promnaming] Keep a metric's cardinality
below about 10 and investigate anything above 100.[^prominstr] Loki asks the same of log stream labels: dynamic labels "ideally limited
to tens of values", and no trace or order IDs.[^lokibp] Scrape limits (`sample_limit`, `label_limit`) fail the *whole* scrape when
exceeded,[^promcfg] so watch `scrape_series_added` and `scrape_samples_post_metric_relabeling`.[^promjobs] High-cardinality context is
not bad in itself, only in metric labels: event stores treat user or order IDs as the key to isolating a problem.[^honeyhc]

**Sampling.** OpenTelemetry suggests sampling from about 1,000 traces per second, and avoiding it at tens of small traces per second
or where regulation forbids dropping data. Head sampling is cheap but cannot guarantee that every error trace is kept; tail sampling
can, but it is "difficult to operate".[^otelsampling] The Collector's
tail sampler needs every span of a trace on one instance, and its `sampling_trace_dropped_too_early` metric shows traces lost before
a decision.[^tailsamp] Derive request rates from all spans before the sampler drops any, since rates counted from sampled spans fall by
the sampling factor (Artefact 1). Mastodon's published configuration uses the same order: trace statistics from every span
first, then tail sampling.[^otelmastodon]

**Context propagation.** A trace survives only if every hop passes its context on. OpenTelemetry SDKs default to the W3C
`tracecontext` and `baggage` propagators (`OTEL_PROPAGATORS`); B3 is an option.[^otelenv] Fix the list in the standard, and while
some services still use B3 configure both (`tracecontext,baggage,b3multi`). Traces break in three places:

- **Queues and batches.** A span has one parent, so OpenTelemetry's messaging conventions (still in development) link consumer spans
  to each message's creation context with span links.[^semmsg] The queue client must carry context in message headers.
- **Intermediaries.** W3C Trace Context requires a vendor that receives `traceparent` and `tracestate` to send them on;[^w3ctc] a
  proxy, API gateway or mesh that drops them starts a new trace behind it.
- **Mixed instrumentation.** A service on another propagator produces fragments that look like separate traces.

*Trust and baggage.* OpenTelemetry advises caution with incoming context from external sources (forged headers can manipulate
tracing data) and with sending context to external endpoints (IDs and baggage reveal internal details).[^otelctx] Instrumentation
adds baggage to most outgoing requests, so it can reach third-party APIs, and nothing checks its integrity.[^otelbaggage] Keep
credentials, API keys and personal data out of it,[^otelctx] although OpenTelemetry's baggage page lists user IDs among typical
uses:[^otelbaggage] allow-list baggage keys. A workable policy starts a new trace at the public edge, which W3C Trace Context
permits,[^w3ctc] and strips `baggage` at egress.

**Log pipelines.** Four decisions keep logs useful and affordable.

1. **Structure.** OpenTelemetry defines a structured log by its stable schema, not by valid JSON alone; its log records carry
   `TraceId`, `SpanId` and severity as top-level fields.[^otellogs] Require them in every log line: SDKs can inject the IDs;[^otelctx]
   services that print to stdout must write them as fields.
2. **Collection.** Kubernetes describes a node-level logging agent, run as a DaemonSet, that reads container log files (by default
   under `/var/log/pods`).[^k8slog] For the Collector, OpenTelemetry calls the `file_log` receiver "the de facto solution"; its
   `container` parser reads docker, CRI-O and containerd formats and adds pod metadata.[^otelk8s][^container] Fluent Bit is an alternative
   agent.[^fluentbit]
   Without a storage extension, offsets live only in memory and a restarted agent cannot resume.[^filelog]
3. **Parsing and multiline.** Parse JSON lines into severity and trace fields at the agent (Artefact 1). Join stack traces with the
   `recombine` operator after the container parser,[^recombine] or better, log each exception as one record.
4. **Routing by level.** The filter processor drops records matching a condition such as
   `log.severity_number < SEVERITY_NUMBER_WARN`.[^filter] Records whose severity was never parsed have severity number 0
   (`SEVERITY_NUMBER_UNSPECIFIED`) and match it too,[^ottllog] so exclude 0 or parse severity first.

**Retention tiers.** Prometheus keeps 15 days by default on single-node local storage.[^promstorage] A typical design keeps raw data
hot briefly, downsampled metrics warm for months and compressed logs in object storage as policy requires; security retention belongs
to [SecDataOps](../secdataops/).

**Dashboards and alerts.** Give every request-driven service a RED view and every resource a USE view, as Wilkie
recommends.[^red][^use] Design the drill-down: alert, dashboard, exemplar trace, that trace's logs.[^tagobs] Page on symptoms, use
tickets for slow problems, and test each page against the SRE book's questions (urgent, actionable, user-visible).[^sremon]
Burn-rate thresholds belong to the SLO owner ([SRE](../sre/)).[^srewbslo]

**Meta-monitoring.** Prometheus recommends alerts that the monitoring works, and notes that one black-box test of the alert path "is
better than individual alerts on each" component.[^promalert] The usual implementation is a dead man's switch: an
always-firing alert whose *absence* pages through another channel.

### Methods, standards and frameworks

| Name | Kind | Status (October 2026) | Use it for |
|---|---|---|---|
| OpenTelemetry specification, OTLP, semantic conventions | Open standard (CNCF graduated) | Varies by language and signal; logs still "Development" in the Python and JavaScript SDKs[^otelstatus] | Instrumentation and transport |
| W3C Trace Context | W3C Recommendation (2020; edited 2021)[^w3ctc] | Stable; Level 2 in progress[^otelsampm] | Trace propagation across services |
| Prometheus exposition format and PromQL | De facto open standard | Prometheus 3.x; native histograms stable since v3.8.0[^promnh] | Metrics |
| Golden signals, RED, USE | Methods | Stable | Deciding what to measure |
| Google SRE monitoring and SLO-alerting guidance | Industry recommendation | 2016 book, 2018 workbook | Alert design |
| CNCF TAG Observability whitepaper | Community guidance | v1.0 (2023) | Concepts and signal trade-offs |

None is mandatory; what binds comes from contracts and law, such as privacy rules on personal data in logs.

### Tooling landscape (as of October 2026)

| Job | Representative open-source options (latest release checked 2026-10-16) | Notes |
|---|---|---|
| Collect and process | OpenTelemetry Collector v0.163.x (v0.163.1, 2026-10-16);[^relotel] Grafana Alloy, a Collector distribution;[^alloy] Fluent Bit (CNCF graduated, under Fluentd);[^fluentbit] Vector[^vector] | Pick one primary pipeline; the Collector is the vendor-neutral default |
| Metrics | Prometheus v3.15.0 (CNCF graduated 2018);[^relprom][^cncfprom] Thanos (CNCF incubating) for long-term storage[^cncfthanos] | Local Prometheus storage is single-node[^promstorage] |
| Logs | Grafana Loki v3.7.8;[^relloki] Elasticsearch v9.5.5[^reles] | Loki needs low-cardinality labels; put high-cardinality fields in structured metadata[^lokibp] |
| Traces | Jaeger v2.22.0, rebuilt on the Collector;[^reljaeger][^cncfjaeger] Grafana Tempo v3.1.0[^reltempo] | Jaeger recommends the standard Collector as agent[^jaegerarch] |
| Profiles | OpenTelemetry Profiles (public alpha since March 2026)[^otelprof] | Early; expect format changes |
| Visualise and alert | Grafana v13.2.4;[^relgrafana] Prometheus Alertmanager; pint for rule linting[^pint] | Keep rules and dashboards in Git |
| Commercial back ends | Many accept OTLP natively (non-exhaustive list)[^otelvendors] | Compare pricing on your own data |

**Selection criteria.** OTLP at the edge, so you can switch back ends; the cost model (per series, per GB, per host); query
languages your engineers can learn; licence (Grafana, Loki and Tempo moved from Apache 2.0 to AGPLv3 in 2021;[^grafanaagpl] Elastic added AGPL next to ELv2 and SSPL in 2024[^elasticagpl]); and the burden of running stateful databases.
**Version churn is real:** the Collector renamed `otlp` to `otlp_grpc` (v0.144.0) and many contrib components to
snake_case,[^colchangelog][^contribchangelog] and native histograms went from experimental in Prometheus 3.0 to stable in
3.8.[^prom3][^promnh] Pin versions. Defaults move too: current Collector documentation batches in the exporter's sending queue (`sending_queue` →
`batch`), not in the beta `batch` processor, which holds data in memory outside any persistent queue.[^otelcolcfg][^exhelper]

### Automation and AI

Automate governance first: configuration validation, rule unit tests, cardinality checks in CI, budget reports and lists of unused
signals. Anomaly detection and event correlation, and when simple rules beat them, are the subject of [AIOps](../aiops/).
Natural-language query assistants need their generated queries checked, and anything that reads telemetry follows the same
redaction rules. Observability *of* LLM applications is covered in [LLMOps](../llmops/).

## ObservabilityOps in practice

### Reference implementation

Assumptions: services on Kubernetes, instrumented with OpenTelemetry; Collector contrib 0.162 or later; Prometheus 3.x; an
OTLP-capable back end at the placeholder `backend.example.internal`; certificates from a platform CA. Three artefacts follow; Lab 1 runs the third.

#### Artefact 1: agent and gateway Collector configuration

Two files implement the [reference architecture](#reference-architecture). Agents send each span twice: by trace ID to the tail
sampler and by service to a second receiver for span metrics;[^otelgw][^lbexp] metrics and logs go to any replica. The agent also
collects container logs, parses JSON lines and drops DEBUG and TRACE records. Names follow the 2026 snake_case
renames.[^contribchangelog] `span_metrics` and `filter` are alpha, and the span-metrics duration unit is changing from milliseconds to seconds;[^spanmetrics][^filter]
`redaction` is beta for traces, alpha for logs and metrics.[^redaction] `MY_POD_IP` comes from the downward API. The DaemonSet
mounts `/var/log/pods` read-only[^otelk8s] and a writable hostPath for `file_storage`, so log offsets survive a replaced
pod.[^filestorage]

_Example status: YAML syntax-checked; not executed (requires Kubernetes, certificates from a platform CA and a back end)._

```yaml
# agent.yaml: DaemonSet, one Collector per node
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317   # hostPort; no TLS or auth (see below)
  file_log:                               # container logs on this node
    include: [/var/log/pods/*/*/*.log]
    exclude: [/var/log/pods/*/otel-agent/*.log]   # not the agent's own logs
    start_at: end
    include_file_path: true
    storage: file_storage                 # resume from saved offsets after a restart
    operators:
      - type: container                   # docker, CRI-O, containerd; adds pod metadata
      - type: json_parser                 # services log JSON with level and trace IDs
        if: 'body startsWith "{"'
        severity: {parse_from: attributes.level}
        trace:
          trace_id: {parse_from: attributes.trace_id}
          span_id: {parse_from: attributes.span_id}
extensions:
  file_storage:
    directory: /var/lib/otelcol/file_storage   # a hostPath: offsets outlive the pod
    create_directory: true
processors:
  memory_limiter: {check_interval: 1s, limit_mib: 400, spike_limit_mib: 100}
  filter/drop_debug:                      # drop DEBUG and TRACE; keep unparsed (0)
    log_conditions:
      - >-
        log.severity_number > SEVERITY_NUMBER_UNSPECIFIED and
        log.severity_number < SEVERITY_NUMBER_INFO
exporters:
  load_balancing/by_trace:                # tail sampling: whole traces on one replica
    routing_key: traceID
    protocol:
      otlp:
        tls:                              # endpoints are pod IPs: verify the DNS name
          ca_file: /tls/ca.crt
          cert_file: /tls/tls.crt
          key_file: /tls/tls.key
          server_name_override: otel-gateway.observability.svc.cluster.local
    resolver:
      dns: {hostname: otel-gateway-headless.observability.svc.cluster.local, port: 4317}
  load_balancing/by_service:              # span metrics: each service on one replica
    routing_key: service
    protocol:
      otlp:
        tls:
          ca_file: /tls/ca.crt
          cert_file: /tls/tls.crt
          key_file: /tls/tls.key
          server_name_override: otel-gateway.observability.svc.cluster.local
    resolver:
      dns: {hostname: otel-gateway-headless.observability.svc.cluster.local, port: 4319}
  otlp_grpc/gateway:                      # metrics and logs: any replica
    endpoint: otel-gateway.observability.svc.cluster.local:4317
    tls: {ca_file: /tls/ca.crt, cert_file: /tls/tls.crt, key_file: /tls/tls.key}
    sending_queue: {batch: {}}
service:
  extensions: [file_storage]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [load_balancing/by_trace, load_balancing/by_service]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp_grpc/gateway]
    logs:
      receivers: [otlp, file_log]
      processors: [memory_limiter, filter/drop_debug]
      exporters: [otlp_grpc/gateway]
```

```yaml
# gateway.yaml: Deployment with two or more replicas behind a headless Service
receivers:
  otlp:                                   # spans routed by trace ID; metrics; logs
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317
        tls:                              # mutual TLS: clients need a platform-CA cert
          cert_file: /tls/tls.crt
          key_file: /tls/tls.key
          client_ca_file: /tls/ca.crt
  otlp/by_service:                        # spans routed by service, for span metrics only
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4319
        tls:
          cert_file: /tls/tls.crt
          key_file: /tls/tls.key
          client_ca_file: /tls/ca.crt
processors:
  memory_limiter: {check_interval: 1s, limit_mib: 1500, spike_limit_mib: 300}
  redaction:
    allow_all_keys: true                  # keep keys, but mask risky values
    blocked_key_patterns: [".*token.*", ".*password.*", ".*secret.*"]
    # e-mail addresses, masked in attributes and in string log bodies
    blocked_values: ['[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}']
    summary: info
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - {name: errors, type: status_code, status_code: {status_codes: [ERROR]}}
      - {name: slow, type: latency, latency: {threshold_ms: 2000}}
      - {name: baseline, type: probabilistic, probabilistic: {sampling_percentage: 1}}
connectors:
  span_metrics:
    metrics_expiration: 5m                # stop a service's series after it moves away
exporters:
  otlp_grpc/backend:
    endpoint: backend.example.internal:4317
    headers:
      authorization: "Bearer ${env:BACKEND_TOKEN}"   # injected from a secret
    sending_queue:
      queue_size: 5000
      batch: {}                           # batches after sampling, inside the exporter
    retry_on_failure:
      max_elapsed_time: 10m
service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: ${env:MY_POD_IP}    # scraped only by the platform Prometheus
                port: 8888
                without_type_suffix: true
                without_units: true
  pipelines:
    traces/sampled:
      receivers: [otlp]
      processors: [memory_limiter, redaction, tail_sampling]
      exporters: [otlp_grpc/backend]
    traces/span_metrics:
      receivers: [otlp/by_service]
      processors: [memory_limiter]
      exporters: [span_metrics]
    metrics:
      receivers: [otlp, span_metrics]
      processors: [memory_limiter]
      exporters: [otlp_grpc/backend]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, redaction]
      exporters: [otlp_grpc/backend]
```

**What the routing costs.** Each span crosses the network twice. The alternative, span metrics on the trace-ID-routed replicas,
works only because the connector adds a `collector.instance.id` dimension (a UUID per instance) by default, so each replica writes
its own series,[^spanmetrics] multiplying series by the replica count. Never exclude that dimension while several replicas compute
span metrics. Service routing keeps the dimension and some churn: by default the UUID is new each time a gateway
starts,[^smfactory][^otelinternal] so every restart or rollout starts new span-metric series, and when the N replicas change about
1/N of services move too.[^lbexp] `metrics_expiration` retires the old series; dashboards still need
`sum without (collector_instance_id)`.

**Who can reach what.** The agent's receiver has no TLS or authentication and its hostPort opens it on every node, so anything
that reaches port 4317 on a node can send data. Keep that port inside the cluster (host firewall, security group) or give the agent
TLS and an authenticator too. Only clients with a platform-CA certificate reach the gateways. Agents dial gateway pod IPs;
`server_name_override` makes them verify the Service's DNS name instead, so issue gateway certificates for that
name.[^lbexp][^configtls] Port 8888 should accept only the platform's Prometheus (a network policy, not shown). Since contrib
v0.126.0 redaction also covers log bodies: `blocked_values` inside string bodies, key rules in map
bodies.[^redaction][^contribchangelog] Key rules cannot see a free-text `password=…` in a string, so code that never logs secrets remains the safer control.[^otelsensitive]

#### Artefact 2: Prometheus guardrails

_Example status: YAML syntax-checked; not executed._

```yaml
scrape_configs:
  - job_name: checkout
    sample_limit: 20000            # whole scrape fails above this, so alert well before it
    label_limit: 30
    label_value_length_limit: 200
    extra_scrape_metrics: true     # exposes scrape_sample_limit
    static_configs:
      - targets: ["checkout.example.internal:9464"]
    metric_relabel_configs:        # emergency containment: drop one offending metric
      - source_labels: [__name__]
        regex: checkout_orders_total
        action: drop
```

The pipeline alerts and their tests are in Lab 3 of the [hands-on labs](#hands-on-labs).

#### Artefact 3: telemetry budget estimator

It computes worst-case series (instances × product of label values × series per label set; a classic histogram adds `+Inf`, `_sum`
and `_count` to its buckets), applies the Prometheus disk formula at 1–2 bytes per sample,[^promstorage] and flags labels with more
than 100 values, where Prometheus suggests looking for alternatives.[^prominstr]

_Example status: executed with Python 3.8.10 on 2026-10-16; output below is real._

```python
#!/usr/bin/env python3
"""Estimate worst-case Prometheus-style active series and disk for a metric inventory,
and flag labels and teams that break the telemetry budget. Python 3.8+, standard library only."""
import json
import sys
from math import prod

SERIES_PER_LABELSET = {"counter": 1, "gauge": 1}


def series_for(metric):
    combos = prod(metric.get("labels", {}).values()) if metric.get("labels") else 1
    if metric["type"] == "histogram":  # classic histogram: buckets + "+Inf" + _sum + _count
        per_set = metric["buckets"] + 3
    else:
        per_set = SERIES_PER_LABELSET[metric["type"]]
    return metric["instances"] * combos * per_set


def main(path):
    inv = json.load(open(path))
    b = inv["budget"]
    problems = 0
    per_team = {}
    print("metric                                team         series (upper bound)")
    for m in inv["metrics"]:
        s = series_for(m)
        per_team[m["team"]] = per_team.get(m["team"], 0) + s
        print(f"{m['name']:<37} {m['team']:<12} {s:>12,}")
        for label, values in m.get("labels", {}).items():
            if values > b["max_label_values"]:
                problems += 1
                print(f"  WARNING label '{label}' has {values:,} values (> {b['max_label_values']}): "
                      "move it to a span or log attribute")
    for team, s in sorted(per_team.items()):
        status = "OK" if s <= b["active_series_per_team"] else "OVER BUDGET"
        if status != "OK":
            problems += 1
        print(f"team {team:<12} {s:>12,} of {b['active_series_per_team']:,}  {status}")
    total = sum(per_team.values())
    sps = total / b["scrape_interval_s"]
    seconds = b["retention_days"] * 86400
    low, high = seconds * sps * 1 / 1e9, seconds * sps * 2 / 1e9
    print(f"total series {total:,}; samples/s {sps:,.1f}; "
          f"disk for {b['retention_days']} d: {low:.1f}-{high:.1f} GB at 1-2 bytes/sample")
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))
```

```json
{
  "budget": {"active_series_per_team": 200000, "max_label_values": 100,
             "scrape_interval_s": 30, "retention_days": 15},
  "metrics": [
    {"name": "http_server_request_duration_seconds", "team": "checkout", "type": "histogram",
     "buckets": 14, "instances": 12,
     "labels": {"http_request_method": 4, "http_route": 30, "http_response_status_code": 6}},
    {"name": "checkout_orders_total", "team": "checkout", "type": "counter", "instances": 12,
     "labels": {"payment_provider": 3, "customer_id": 50000}},
    {"name": "queue_depth", "team": "fulfilment", "type": "gauge", "instances": 6,
     "labels": {"queue": 8}}
  ]
}
```

The 14 buckets match the boundaries OpenTelemetry advises for `http.server.request.duration`.[^semhttp] The label counts are
illustrative.

### Hands-on labs

Lab 1 was executed for this page. Labs 2 and 3 were **not executed** (no container runtime or `promtool` on the authoring host);
their expected output is derived from current documentation.

#### Lab 1 (beginner): budget a metric inventory

**Objective:** find the label that breaks a budget before production. **Prerequisites:** Python 3.8 or later; no cost or risk; 30
minutes.

1. Save Artefact 3 as `telemetry_budget.py` and its JSON as `inventory.json`.
2. Run `python3 telemetry_budget.py inventory.json; echo "exit=$?"`.
3. Delete the `customer_id` entry from `checkout_orders_total` and run again.
4. Replace the inventory with three real metrics from your own services.

**Expected output** (step 2, verbatim):

```text
metric                                team         series (upper bound)
http_server_request_duration_seconds  checkout          146,880
checkout_orders_total                 checkout        1,800,000
  WARNING label 'customer_id' has 50,000 values (> 100): move it to a span or log attribute
queue_depth                           fulfilment             48
team checkout        1,946,880 of 200,000  OVER BUDGET
team fulfilment             48 of 200,000  OK
total series 1,946,928; samples/s 64,897.6; disk for 15 d: 84.1-168.2 GB at 1-2 bytes/sample
exit=1
```

**Validation:** after step 3, `checkout` totals 146,916 series, the run prints `total series 146,964; samples/s 4,898.8; disk for
15 d: 6.3-12.7 GB` and exits 0. **Troubleshooting:** `ImportError: cannot import name 'prod'` means Python older than 3.8;
`KeyError: 'buckets'` means a histogram lacks a bucket count. The figures are upper bounds, because not every label combination
occurs.[^cfprom] **Teardown:** delete the two files.

#### Lab 2 (intermediate): redact and inspect a Collector pipeline

**Objective:** prove that the pipeline masks secrets and personal data, and read its internal metrics. **Prerequisites:** Docker on
Linux, about 1 GB of free memory, 45 minutes; local, no cost. **Status:** not executed.

::: caution Host networking
With `--network host` the container shares the machine's interfaces. The endpoints bind to `127.0.0.1`; with `0.0.0.0` they would
expose an unauthenticated OTLP receiver and the metrics port to every network the machine is on.
:::

1. Save this as `lab2.yaml`.

_Example status: YAML syntax-checked; not executed._

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
processors:
  memory_limiter: {check_interval: 1s, limit_mib: 400, spike_limit_mib: 100}
  redaction:
    allow_all_keys: true
    blocked_key_patterns: [".*token.*"]
    blocked_values: ['[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}']
    summary: info
exporters:
  debug: {verbosity: detailed}
service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: 127.0.0.1
                port: 8888
                without_type_suffix: true
                without_units: true
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, redaction]
      exporters: [debug]
```

2. Validate, start the Collector, send five traces with a fake e-mail and token, then inspect.[^otelcolcfg][^telemetrygen]

_Example status: syntax-checked with `bash -n`; not executed (requires Docker)._

```bash
#!/usr/bin/env bash
set -euo pipefail
REG=ghcr.io/open-telemetry
IMG=$REG/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.162.0
GEN=$REG/opentelemetry-collector-contrib/telemetrygen:v0.162.0
CFG=/etc/otelcol-contrib/config.yaml
docker run --rm -v "$PWD/lab2.yaml:$CFG:ro" "$IMG" validate --config=$CFG
docker run -d --name otelcol-lab --network host -v "$PWD/lab2.yaml:$CFG:ro" "$IMG"
sleep 5
docker run --rm --network host "$GEN" traces --otlp-insecure \
  --otlp-endpoint 127.0.0.1:4317 --traces 5 --service lab-checkout \
  --telemetry-attributes 'user.email="jane@example.com"' \
  --telemetry-attributes 'session_token="abc123"'
sleep 3
docker logs otelcol-lab 2>&1 | grep -E "user.email|session_token|redaction" | head
curl -s 127.0.0.1:8888/metrics | grep -E "otelcol_(receiver_accepted|exporter_sent)_spans"
```

**Expected output and validation:** `validate` exits 0. The debug log shows `user.email` and `session_token` masked with asterisks and a
`redaction.masked.count` attribute; neither `jane@example.com` nor `abc123` appears. The span counters on port 8888 are non-zero. **Troubleshooting:** "unknown type" errors mean an image older than the
snake_case renames;[^contribchangelog] on macOS or Windows, publish ports instead of using host networking. Image tags were checked
in the registry on 2026-10-16. **Teardown:** `docker rm -f otelcol-lab`.

#### Lab 3 (advanced): test alerts for the telemetry pipeline

**Objective:** alert on the three ways a Collector loses data (failures after retries, a full queue, refused input) and on targets
near their sample limit, and unit-test the rules. **Prerequisites:** `promtool` from Prometheus 3.x; 45 minutes. **Status:** not
executed; expected results reasoned from the test-file documentation.[^promunittest]

_Example status: YAML syntax-checked; not executed (requires `promtool`)._

```yaml
# pipeline-alerts.yml
groups:
  - name: telemetry-pipeline
    rules:
      - alert: TelemetryExportFailing
        # "or ... * 0" keeps the ratio defined when sent_spans has never been incremented
        expr: |
          sum by (job, instance) (rate(otelcol_exporter_send_failed_spans[5m]))
            /
          (sum by (job, instance) (rate(otelcol_exporter_send_failed_spans[5m]))
            + (sum by (job, instance) (rate(otelcol_exporter_sent_spans[5m]))
               or sum by (job, instance) (rate(otelcol_exporter_send_failed_spans[5m])) * 0))
            > 0.01
        for: 10m
        labels: {severity: page}
        annotations:
          summary: "{{ $labels.job }}/{{ $labels.instance }}: over 1% of spans fail to export"
      - alert: TelemetryQueueFull
        expr: sum by (job, instance) (rate(otelcol_exporter_enqueue_failed_spans[5m])) > 0
        for: 5m
        labels: {severity: page}
        annotations:
          summary: "{{ $labels.job }}/{{ $labels.instance }}: sending queue full, spans dropped"
      - alert: TelemetryReceiverRefusing
        expr: sum by (job, instance) (rate(otelcol_receiver_refused_spans[5m])) > 0
        for: 15m
        labels: {severity: ticket}
        annotations:
          summary: "{{ $labels.job }}/{{ $labels.instance }}: refusing spans (back-pressure)"
      - alert: ScrapeNearSampleLimit
        expr: scrape_samples_post_metric_relabeling / (scrape_sample_limit > 0) > 0.8
        for: 15m
        labels: {severity: ticket}
        annotations:
          summary: "{{ $labels.job }}/{{ $labels.instance }}: over 80% of sample_limit"
      - alert: Watchdog
        expr: vector(1)
        labels: {severity: none}
        annotations:
          summary: "Always firing; page through another channel if it stops"
```

```yaml
# pipeline-alerts.test.yml
rule_files: [pipeline-alerts.yml]
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'otelcol_exporter_sent_spans{job="otel-gw", instance="gw-1"}'
        values: '0+1000x30'
      - series: 'otelcol_exporter_send_failed_spans{job="otel-gw", instance="gw-1"}'
        values: '0+60x30'
      - series: 'otelcol_exporter_send_failed_spans{job="otel-gw", instance="gw-2"}'
        values: '0+100x30'   # back end down since gw-2 restarted: nothing ever sent
      - series: 'otelcol_exporter_enqueue_failed_spans{job="otel-gw", instance="gw-1"}'
        values: '0+50x30'
      - series: 'otelcol_receiver_refused_spans{job="otel-gw", instance="gw-1"}'
        values: '0+10x30'
      - series: 'scrape_samples_post_metric_relabeling{job="checkout", instance="app-7"}'
        values: '850x30'
      - series: 'scrape_sample_limit{job="checkout", instance="app-7"}'
        values: '1000x30'
    alert_rule_test:
      - eval_time: 5m
        alertname: TelemetryExportFailing
        exp_alerts: []
      - eval_time: 15m
        alertname: TelemetryExportFailing
        exp_alerts:
          - exp_labels: {severity: page, job: otel-gw, instance: gw-1}
            exp_annotations: {summary: "otel-gw/gw-1: over 1% of spans fail to export"}
          - exp_labels: {severity: page, job: otel-gw, instance: gw-2}
            exp_annotations: {summary: "otel-gw/gw-2: over 1% of spans fail to export"}
      - eval_time: 10m
        alertname: TelemetryQueueFull
        exp_alerts:
          - exp_labels: {severity: page, job: otel-gw, instance: gw-1}
            exp_annotations: {summary: "otel-gw/gw-1: sending queue full, spans dropped"}
      - eval_time: 20m
        alertname: TelemetryReceiverRefusing
        exp_alerts:
          - exp_labels: {severity: ticket, job: otel-gw, instance: gw-1}
            exp_annotations: {summary: "otel-gw/gw-1: refusing spans (back-pressure)"}
      - eval_time: 20m
        alertname: ScrapeNearSampleLimit
        exp_alerts:
          - exp_labels: {severity: ticket, job: checkout, instance: app-7}
            exp_annotations: {summary: "checkout/app-7: over 80% of sample_limit"}
      - eval_time: 1m
        alertname: Watchdog
        exp_alerts:
          - exp_labels: {severity: none}
            exp_annotations: {summary: "Always firing; page through another channel if it stops"}
```

1. Save both files; run `promtool check rules pipeline-alerts.yml`, then `promtool test rules pipeline-alerts.test.yml`.
2. Change the `gw-1` failed-span series to `'0+5x30'` (0.5% failures) and rerun; the 15-minute case should now fail.

**Expected output and validation:** step 1 reports valid rules and passed tests. At 5 minutes the export alert is pending
(`for: 10m`). At 15 minutes `gw-1` fails 60 / 1,060 ≈ 5.7% of spans and `gw-2`, which has sent nothing since it restarted, 100%;
without the `or … * 0` term the `gw-2` ratio would be empty and never page. Step 2 fails because 0.5% is under the threshold.
*Reading the counters:* the Collector documentation warns that send failures "do not inherently imply data loss since there could be
retries";[^otelinternal] in the current exporter helper, retries run inside the step that counts results, so `send_failed` counts
items that failed after their retries.[^exhelpercode] Spans refused by a full queue never reach the retries and count as
`enqueue_failed`.[^exhelper] **Troubleshooting:** a label mismatch usually means different metric names or labels (the `_total`
suffix depends on telemetry settings); check `/metrics` first.[^otelinternal] **Teardown:** none; commit the files. Route `Watchdog`
to a dead man's switch service, not to a person.

### Use cases and scenarios

**Case 1 (sourced): one engineer, global scale.** Mastodon, about 20 people, runs mastodon.social (about 10 million requests per
minute) with one engineer managing observability infrastructure: one Collector per Kubernetes namespace, the OpenTelemetry Operator
and Argo CD, Datadog as the back end, and tail sampling that keeps roughly 0.1% of successful traces and every error trace. Tim
Campbell, a Mastodon software engineer, said he had not run into a single Collector issue in about two years; the post itself names
getting started as the hardest part.[^otelmastodon] *Lesson:* one Collector tier is enough when a SaaS back end carries storage
and sampling controls volume. *Pitfall:* Mastodon sets no strict CPU or memory limits on Collector pods, which suits its load but is no general default.

**Case 2 (sourced): cardinality governance at scale.** In March 2023 Cloudflare ran 916 Prometheus instances holding about 4.9 billion
series. Because a breached `sample_limit` fails the whole scrape, Cloudflare patched Prometheus to cap series per server and keep
existing series beyond the limit, alerting the owning team. The result is self-service: "if CI checks are passing then we have the
capacity you need".[^cfprom] *Lesson:* limits plus CI make self-service safe. *Pitfall:* these
are Cloudflare's patches, not upstream behaviour.

**Case 3 (sourced): observability as an internal service.** The UK Ministry of Justice's Observability Platform gives teams on its
Modernisation Platform logs, metrics and alerting to PagerDuty or Slack on Amazon Managed Grafana; its team "maintains the
infrastructure, allowing teams to focus on running their applications". The offering is to be retired, date not announced.[^moj] *Lesson:* a platform needs a product's lifecycle, including retirement.

::: example Illustrative scenario: log cost without losing investigations
A payments company's log bill doubles in a year; debug logs from three services make up most of the volume and are read in fewer
than one incident a quarter. The team keeps error and warning logs hot for 14 days, sends debug logs to object storage for 30 days
and masks card-number patterns at the gateway. Success: log cost per million requests down, no incident review reporting missing
data, zero card numbers in the monthly scan.
:::

## Measure, govern, pay

### Metrics and KPIs

Measure the platform as a product: is the data delivered, used and affordable, and do alerts help? Targets are starting points,
not industry benchmarks.

| KPI | Definition or formula | Data source | Pitfall |
|---|---|---|---|
| Export success ratio (leading) | sent ÷ (sent + send failed + enqueue failed), per signal | Collector internal metrics[^otelinternal] | Blind to loss before the Collector; confirm with the canary |
| Freshness, p95 | query time − emit time of a synthetic canary | Canary job | One canary per path, not per platform |
| Alert-path availability | minutes the `Watchdog` heartbeat arrived ÷ minutes in period | Dead man's switch | Must use a separate channel |
| Page actionability (lagging) | pages labelled actionable ÷ all pages | Monthly alert review | Self-reported; sample a few to check |
| Unused-signal ratio | series used by no dashboard or rule in 90 days ÷ active series | TSDB statistics[^promapi] plus a rule and dashboard inventory | Ad hoc queries are invisible; ask first |
| Naming hygiene | items under `unknown_service` ÷ all items[^semres] | Back-end query | Only near zero is good |
| Budget adherence | teams within budget ÷ teams | Budget report | Budgets that never change are ignored |
| Telemetry unit cost | observability spend ÷ business units (for example per million requests) | Bills and traffic | Compare trends, not companies |

::: example Worked calculation (illustrative numbers)
**Export success.** In a month a gateway sent 9,950,000 spans; 40,000 failed to send and 10,000 could not be queued.
9,950,000 ÷ 10,000,000 = **99.5%**, so the platform missed a 99.9% objective. With retries enabled, failures are counted after
retries end, so all 50,000 were dropped (see Lab 3).
**Page actionability.** 42 pages, 15 actionable: 15 ÷ 42 = **35.7%**; review the other 27 for deletion or conversion to tickets.
**Unit cost.** 18,000 a month in any currency for 900 million requests: 18,000 ÷ 900 = **20 per million requests**. Track it
together with coverage, because cutting coverage also lowers unit cost.
:::

Report per team: a single platform-wide figure hides who generates the cost and noise.

### Quality, security, privacy and compliance

| Risk | Signal | Mitigation |
|---|---|---|
| Secrets or personal data in attributes and logs | Pattern scans; redaction counters | Data minimisation and review of what libraries emit (the implementer's responsibility, says OpenTelemetry);[^otelsensitive] redaction, including log bodies[^redaction] |
| Baggage and trace headers leaving the network | Outbound third-party requests with `baggage` or `traceparent` | Baggage allow-list without credentials or personal data; stop propagation at egress; new trace for untrusted callers[^otelctx][^otelbaggage] |
| Open ingest endpoints | Unknown sources; sudden series growth | Network policy, mutual TLS, per-tenant limits; random request paths in labels are an attack as well as a bug[^cfprom] |
| Over-broad query access | Audit logs of searches | Role-based access and tenancy; restricted views of sensitive services |
| Keeping data too long | Retention report | Retention per data class; EU law, for example, requires personal data to be kept "no longer than is necessary" (GDPR Article 5, unofficial text)[^gdpr5] |

Obligations differ by jurisdiction and sector: take legal advice, and coordinate with [SecDataOps](../secdataops/),
[PrivacyOps](../privacyops/) and [ComplianceOps](../complianceops/).

### Cost, capacity and sustainability

**Drivers:** active series (memory), samples per second, log bytes ingested and indexed, spans kept, query load, retention and
cross-region transfer. **Capacity:** Prometheus disk ≈ retention seconds × samples per second × 1–2 bytes;[^promstorage] for memory,
Cloudflare divides `go_memstats_alloc_bytes` by `prometheus_tsdb_head_series` as a rough bytes-per-series figure.[^cfprom] **Levers,
cheapest first:** delete unused signals; fix unbounded labels; sample traces; tier and shorten retention; downsample; raise scrape
intervals last, because cutting series is more effective.[^promstorage] **Allocation:** tag telemetry by team or `service.namespace`
([FinOps](../finops/)). **Sustainability:** less data stored means less compute and storage ([GreenOps](../greenops/)).

### Assurance: watching the watchers

The observability system can fail silently, so it needs its own loops: an independent watcher (a small separate Prometheus or an
external probe); the `Watchdog` heartbeat; a freshness canary; a weekly review of dropped data (`…send_failed_*`,
`…enqueue_failed_*`, `otelcol_receiver_refused_*`, tail-sampling drops);[^otelinternal][^tailsamp] and Git history plus CI results as
the configuration audit trail. Early warnings: queues above 80% of capacity, rising `scrape_series_added`, series growing faster than
traffic.

## When things go wrong

### Common failures and troubleshooting

| Symptom | Likely cause | Diagnosis | Remediation and prevention |
|---|---|---|---|
| Metrics database memory climbs; restarts | Cardinality explosion | TSDB statistics; `scrape_series_added` | [Runbook below](#worked-runbook-a-cardinality-explosion); budget checks in CI |
| Traces missing spans | Tail sampler overloaded, or spans split across instances | `…_sampling_trace_dropped_too_early`[^tailsamp] | Raise `num_traces` or shorten `decision_wait`; route by trace ID |
| Export failures; queue near capacity | Back end down or throttling | `otelcol_exporter_queue_size` versus `_capacity`[^otelresil] | Scale or fix the back end; persistent queue for critical data |
| One request shows up as several traces | Context lost at a proxy or queue; mixed B3 and W3C | Follow a test request hop by hop | One propagator set; header pass-through; span links for queues[^otelenv][^semmsg] |
| Gaps, jumps, "out-of-order samples" | Two writers for one metric stream | Back-end errors[^otelgw] | Unique resource identity per writer |
| Span-metric series multiply or jump after scaling | Span metrics on trace-ID-routed replicas, or `collector.instance.id` excluded | Series per `collector_instance_id`[^spanmetrics] | Route by service; `sum without (collector_instance_id)` |
| Logs not linked to traces | No trace IDs in log lines, or not parsed | Share of records with empty `trace_id` | Emit IDs; parse them at the agent[^otellogs] |
| Telemetry under `unknown_service` | SDK resource not configured | Count by `service.name`[^semres] | Set the service name in templates |
| Pages nobody acts on | Cause-based or over-sensitive rules | Alert review | Delete, merge or convert to tickets[^sremon] |
| No alerts during an outage | Broken alert path | `Watchdog` heartbeat missing | Fix routing; test the path end to end[^promalert] |
| Secrets in stored logs | Logging in code; no redaction | Pattern scan | Rotate the secret; purge; fix the code |

### Worked runbook: a cardinality explosion

::: figure Runbook for a metrics cardinality explosion
```mermaid
flowchart TB
  A["Alert: head series or memory rising fast"]
  B["Confirm: series growth by job"]
  C["Find: top metrics and labels in TSDB stats"]
  D{"Memory at risk?"}
  E["Contain: drop the offending metric for that job"]
  F["Fix at source: bounded label values, IDs as span attributes"]
  G["Recover: wait for head garbage collection"]
  H["Prevent: CI budget check, scrape limits, growth alert"]
  A --> B
  B --> C
  C --> D
  D -- "yes" --> E
  D -- "no" --> F
  E --> F
  F --> G
  G --> H
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class A,E risk
  class D decision
  class B,C,F,G,H core
```
Text equivalent: an alert on fast-rising head series or memory starts the runbook. Confirm which job adds series and find the top
metrics and labels in the TSDB statistics. If memory is at risk, drop the offending metric first; then fix the source, wait for head
garbage collection, and prevent recurrence with CI budget checks, scrape limits and a growth alert.
:::

::: example Illustrative incident
After a release, `checkout` starts labelling `checkout_orders_total` with `customer_id`. Prometheus memory doubles in an hour.
:::

1. **Confirm.** Which jobs are adding series?

```promql
sort_desc(sum by (job) (sum_over_time(scrape_series_added[1h])))
```

2. **Find.** Ask the TSDB which metrics and labels hold the most series.[^promapi]

_Example status: syntax-checked with `bash -n`; not executed (requires a Prometheus server)._

```bash
#!/usr/bin/env bash
set -euo pipefail
PROM="http://prometheus.example.internal:9090"
curl -s "$PROM/api/v1/status/tsdb?limit=5" \
  | jq '.data.headStats.numSeries, .data.seriesCountByMetricName, .data.labelValueCountByLabelName'
```

3. **Contain** if memory is at risk: drop the metric for that job with `metric_relabel_configs` (Artefact 2) and reload. Avoid
   `labeldrop`, because metrics must stay uniquely labelled, and `sample_limit`, which fails the whole scrape.[^promcfg]
4. **Fix at source.** Replace `customer_id` with a bounded value (customer tier) and record the ID as a span attribute.
5. **Recover.** Series scraped even once stay in memory for one to three hours, until head garbage collection,[^cfprom] so memory
   falls slowly; restart only if the server keeps crashing.
6. **Prevent.** Run the Lab 1 check in CI, set scrape limits, and alert on `scrape_series_added`.

**Done when** head series are back to the pre-release level and the inventory passes the budget check.

### Anti-patterns and myths

- **"Collect everything; decide later."** Cost grows without value, and the CNCF whitepaper warns against it.[^tagobs] *Instead:* name
  a consumer for every signal.
- **"Three pillars means three tools."** The whitepaper prefers "primary signals" and notes that one or two signals can be a valid
  trade-off.[^tagobs] *Instead:* choose signals per question and link them.
- **"The observability team owns everyone's alerts."** It becomes a bottleneck and works against DORA's aim of making every developer
  proficient with monitoring.[^dora] *Instead:* templates and reviews; service teams own their alerts.
- **"Self-hosting is free."** You run stateful databases, upgrades and on-call yourself.
- **"Tracing works once the SDK is installed."** Traces break wherever context is not propagated.
- **"AI will find it."** Correlation needs clean, consistent data first ([AIOps](../aiops/)).

## Deciding and adopting

### When to use and when not to use

**Invest in a deliberate ObservabilityOps function when** several teams share telemetry tools; cost or noise is a recurring
complaint; incidents are slowed by missing or untrusted data; telemetry may hold regulated data; or you self-host stateful back ends.
**Do not create a team yet when** one or two teams use a managed tool with sensible defaults, volumes are small and nobody complains
about cost or noise. One practitioner notes that
a few people part-time "with a few high leverage SaaS investments" may be enough.[^mustin] **Hidden responsibilities** either way:
on-call for the platform, upgrades, vendor management, access reviews and retention enforcement.

### Decision framework

::: figure Deciding how much ObservabilityOps you need (team counts are this Atlas's rule of thumb)
```mermaid
flowchart TB
  S("`How many teams share
  telemetry tools?`")
  H1("`Self-hosted back ends
  or regulated data?`")
  O1["`Named owner
  on call`"]
  N["`Part-time owner,
  vendor defaults`"]
  C("`Cost, noise or gaps
  a top problem?`")
  H("`Self-host stateful
  back ends?`")
  G["`Guild and
  standards`"]
  P["`Platform team
  with on-call`"]
  Q["`Small function:
  standards, budgets`"]
  S -- "one or two" --> H1
  S -- "three or more" --> C
  H1 -- "yes" --> O1
  H1 -- "no" --> N
  C -- "yes" --> H
  C -- "no" --> G
  H -- "yes" --> P
  H -- "no" --> Q
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class S,H1,C,H decision
  class O1,P,Q core
  class N,G neutral
```
Text equivalent: the team-count thresholds are this Atlas's rule of thumb. With one or two teams: if you self-host stateful back ends
or handle regulated data, name an owner on call for the stack; otherwise a part-time owner and vendor defaults suffice. With three
or more teams: if cost, noise or missing data is not a top problem, a guild with shared standards is enough; if it is, self-hosting
stateful back ends calls for a platform team with on-call, and otherwise a small function for standards and budgets.
:::

For the hosting choice, weight the criteria for your situation. Scores are this page's illustrative judgement (1 = weak, 5 = strong).

| Criterion | Weight | Self-hosted open source | SaaS back end | Hybrid: own pipeline, SaaS back end |
|---|---|---|---|---|
| Control of data and residency | 3 | 5 | 2 | 4 |
| Operating effort (higher = less effort) | 3 | 1 | 5 | 3 |
| Cost predictability at your volume | 2 | 3 | 2 | 4 |
| Freedom to switch back ends | 2 | 4 | 2 | 5 |
| **Weighted total** | | **32** | **29** | **39** |

A hybrid scores highest under these weights because a pipeline you own keeps sampling, redaction and routing portable; a team of
two might weight operating effort at 5 and get a different answer.

### Maturity model (this Atlas's model)

Not an official or industry model; Grafana Labs' survey splits practices only into centralised and siloed.[^gsurvey2026]

| Level | Observable behaviour | Exit criteria |
|---|---|---|
| 0 Ad hoc | Each team runs its own tools; nobody owns the whole | Inventory of tools, volumes and spend |
| 1 Centralised | Shared stack and a named owner; basic dashboards | Platform SLIs measured; `Watchdog` in place |
| 2 Standardised | OpenTelemetry, conventions and one propagator set; templates; limits | Most services onboarded through templates; `unknown_service` near zero |
| 3 Governed product | Showback budgets; alert reviews; platform SLOs; tested configuration in Git | Budget adherence and page actionability tracked for two quarters |
| 4 Optimised | Sampling and retention set by value; unused signals pruned automatically | Unit cost falls or holds while coverage holds or rises |

### Adoption roadmap: 30/60/90 days

| Window | Goals | Deliverables | Measures |
|---|---|---|---|
| Days 1–30 | Baseline and safety | Named owner; inventory of tools, volumes and spend; `Watchdog` and an external probe; worst three labels fixed | Baseline numbers exist |
| Days 31–60 | Standards and guardrails | Instrumentation and log standard; gateways with limits and redaction; scrape limits; showback; first alert review | Export success measured; first deletions |
| Days 61–90 | Run it as a product | Platform SLOs; tested configuration in Git; first pruning; tail sampling for the largest producer; first unit-cost report | Budget adherence, page actionability, unit cost |

**Quick wins:** delete unused dashboards and alerts, add the dead man's switch, fix the worst label. **Dependencies:** a service and
owner list, and someone with authority over the budget.

## Toolkit

### Checklists

**Quick start**
- [ ] One named owner for the observability platform
- [ ] Inventory: tools, series, log volume, spend, pages per week
- [ ] `Watchdog` routed to a dead man's switch, plus one external probe

**Service onboarding (preflight)**
- [ ] `service.name` and owner set; routes recorded as templates
- [ ] Expected series and log volume inside the team budget
- [ ] Standard RED dashboard; at least one symptom alert with a runbook
- [ ] No personal data or secrets in a sample of debug output

**Pipeline change**
- [ ] Configuration validated and alert rules unit-tested in CI
- [ ] Component names, stability levels and defaults checked against the pinned version's release notes; rollback is a Git revert

**Monthly review**
- [ ] Platform SLIs, dropped data and queue use
- [ ] Budget report per team; expired exceptions removed
- [ ] Alert review with actionability labels

**Security and privacy audit (quarterly)**
- [ ] Gateways require mutual TLS; agent ports closed to outside the cluster; baggage keys match the allow-list
- [ ] Query access matches roles; searches of sensitive logs are audited
- [ ] Retention per data class enforced; redaction rules tested

**Platform incident readiness**
- [ ] Independent watcher alive; runbooks for back end down, queue full, cardinality explosion

### Templates

_Template status: illustrative; adapt the fields._

```text
ONE-PAGE OBSERVABILITY PLATFORM CHARTER
Mission:        Every engineer can answer a new question about production within
                minutes, from trustworthy data, at an agreed cost.
Users:          <service teams, SRE, incident commanders, security (read-only)>
Services:       Ingest (OTLP, scrape, container logs), storage tiers, dashboards,
                alert routing, budget reports
Platform SLOs:  Export success >= <99.9%>/30 d; freshness p95 <= <60 s>;
                alert-path heartbeat 100%
Budgets:        Active series and log GB/day per team; exceptions expire after <90> d
Standards:      Propagators <tracecontext,baggage>; baggage keys on an allow-list;
                JSON logs with severity, trace_id, span_id
Not our job:    Other teams' SLO targets, their alerts' content, their on-call
Reviews:        Weekly health; monthly budget and alerts; quarterly pruning, standards
```

```text
TELEMETRY CHANGE REQUEST
Team / service:              <name>             Owner on call: <name>
Signal:                      metric | log stream | trace attribute | new source
Question it answers:         <who uses it, in which dashboard or alert>
Labels and value counts:     <label: values, ...>  Expected series or GB/day: <n>
Personal or secret data?     no | yes -> masking rule: <rule>
Retention class:             hot <d> / warm <d> / archive <d>
Review date:                 <date>; retire if unused for 90 days
Decision:                    approved | redesign | exception until <date>
Approved by:                 <budget owner>
```

```text
ALERT REVIEW (per page, monthly)
Alert | Pages | Actionable? (y/n) | Action taken | User-visible? (y/n)
      | Decision: keep / tune / ticket / delete | Owner
```

```text
ARCHITECTURE DECISION RECORD: trace sampling
Context:      <traces per second, cost, current back end, regulatory constraints>
Decision:     Tail sampling at gateways: keep all errors and traces > <2 s>;
              <1>% of the rest
Consequences: Spans routed by trace ID to the samplers and by service to span
              metrics (one writer per series); agent-to-gateway trace traffic doubles;
              new span-metric series after each gateway restart (aggregate them);
              monitor dropped-too-early; rare successful paths missing from samples
Alternatives: Span metrics on the sampling replicas (series x replicas, keep
              collector.instance.id); head sampling in SDKs (cheaper, loses the
              error guarantee); no sampling (cost)
Review:       <date>
```

```text
SERVICE OBSERVABILITY SCORECARD (0-2 each, 14 max)
Naming and ownership | RED dashboard        | Symptom alert with runbook
Unbroken traces      | Logs with trace IDs  | Budget adherence
No sensitive data in samples or baggage
```

### Deep comparisons

| | Head sampling | Tail sampling |
|---|---|---|
| Decision point | Trace start, from the trace ID | After `decision_wait`, seeing the trace |
| Keeps every error? | No | Yes, with a status policy |
| Cost to run | Negligible | Stateful tier; all spans of a trace to one instance[^tailsamp] |
| Best for | Uniform, high-volume traffic | Mixed traffic where rare failures matter |

### FAQ

**Is ObservabilityOps a job title I should hire for?** Rarely under that name: look for observability or platform engineers.

**We use one SaaS vendor. Do we still need OpenTelemetry?** It keeps instrumentation and pipeline policy portable, and many back
ends accept OTLP.[^otelvendors]

**Which signal should we start with?** Metrics for RED and USE views and symptom alerts; then traces across services; logs for detail.
The CNCF whitepaper suggests starting with one or a few signals.[^tagobs]

**Who pays for telemetry?** The teams that generate it: showback first, chargeback only if you charge back other costs
([FinOps](../finops/)).

**How long should we keep data?** As long as a named question or obligation needs it, per data class. Metrics downsample well;
traces do not, so keep them only as long as investigations need them.

## Going further

### Advanced topics and emerging direction

| Topic | Status (October 2026) | What it changes |
|---|---|---|
| Continuous profiling | Established technique; OpenTelemetry Profiles public alpha since March 2026[^otelprof] | Cost per code path, linked to traces |
| eBPF zero-code instrumentation | OpenTelemetry eBPF Instrumentation (OBI, formerly Grafana Beyla), first alpha November 2025[^otelobi] | Protocol-level telemetry without code changes; less detail |
| Consistent probability sampling | Specification work, with W3C Trace Context Level 2[^otelsampm] | Different sampling rates per service without broken traces |
| Events as logs | Span Event API being deprecated (March 2026)[^otelspanev] | One way to emit events, correlated with spans |
| Native histograms | Stable in Prometheus since v3.8.0[^promnh] | Cheaper, more accurate latency distributions |

Treat "AI observability" and "observability pipeline" product claims as marketing until tested on your own data.

### Learning path

- **Prerequisites:** Linux and HTTP basics, one programming language, [DevOps](../devops/) and [SRE](../sre/) vocabulary.
- **One day:** read the CNCF whitepaper's signals section[^tagobs] and the SRE book's monitoring chapter;[^sremon] run Lab 1.
- **One week:** instrument a sample service, run Lab 2, build a RED dashboard and one symptom alert.
- **30 days:** run Lab 3 in CI, route traces to a sampling tier, and produce a budget report for real services.
- **Specialise:** time-series storage, tracing, log pipelines, or cost and governance.
- **Portfolio project:** a repository with Collector configuration, tested rules, a budget checker and a one-page charter.

## Wrap-up

### Executive recap and cheat sheet

| Decide | Practise | Measure | Watch out for |
|---|---|---|---|
| Who owns the platform; hosting model; budgets per team | OpenTelemetry and conventions; bounded labels; tail sampling with RED computed first; retention tiers | Export success, freshness, alert-path heartbeat, page actionability, unit cost | Cardinality explosions; silent data loss; secrets in logs; component renames on upgrade |

### A judgement call: cut traces or cut logs?

::: decision The situation (illustrative)
Leadership asks for a 30% cut in observability spend this quarter. Traces are 45% of the bill and logs 40%. Option A: tail-sample to
1% of successful requests, keeping errors and slow traces. Option B: keep all traces for 3 days and archive debug logs cheaply.
:::

**Assumptions:** about 2,000 traces per second, mostly healthy; engineers start investigations from traces and rarely read debug
logs; one regulated service must keep its logs. **Why A looks attractive:** one configuration change gives the biggest saving, and
OpenTelemetry suggests sampling at this volume.[^otelsampling] **Why A alone can be wrong:** dashboards that count stored spans
undercount once sampling starts; the stateful sampling tier must be sized and monitored;[^tailsamp] and rare successful paths you may
want to compare against disappear. **Why B alone can be wrong:** it may not reach 30%, and three-day traces break investigations of
slow-burning problems. **A defensible answer:** A for high-volume services, with span metrics computed before sampling (Artefact 1);
100% for low-volume services; and the cheap half of B (archive debug logs, keep error logs hot). Track unit cost and "data missing"
findings for a quarter; the open question is how often engineers need a successful trace, so check five incident reviews before
going below 1%.

### Learning objectives and mastery check

After this page you can explain the two readings of ObservabilityOps; design a pipeline with routing, limits, redaction and
sampling; find where a trace breaks; estimate series and disk; test alerts for the platform; and choose an organisational model.

::: answer Why does adding a `customer_id` label to a counter create a cost problem, and where should the ID go instead?
Every label combination is a new series, so the label multiplies series by the number of customers.[^promnaming] Put the ID on
spans or logs.
:::

::: answer A metric has 4 methods, 30 routes and 6 status codes on 12 instances, as a classic histogram with 14 buckets. How many series?
4 × 30 × 6 × 12 × (14 + 3) = 146,880 as an upper bound.
:::

::: answer Why are spans routed by trace ID to tail-sampling Collectors but by service to span-metrics Collectors?
The sampler decides per trace, so it needs all spans of a trace on one instance.[^tailsamp] Span metrics aggregate per service and
each metric stream needs a single writer; routed by trace ID, every replica would write its own copy of each series.[^otelgw][^lbexp]
:::

::: answer A checkout trace shows the API span, but the payment worker's spans appear as a separate trace. Name likely causes.
A queue without context in its message headers or without consumer span links; a proxy or gateway that dropped `traceparent`;
or different propagators (B3 and W3C).[^semmsg][^otelenv] Follow one test request hop by hop to find where the context disappears.
:::

::: answer A team asks the platform team to own its alerts. What do you answer?
No: provide templates and reviews; the service team owns its alerts because it can act on them, and DORA wants every developer
to be proficient with monitoring.[^dora]
:::

::: answer When should a small company not build an ObservabilityOps function?
When one or two teams use a managed tool with sensible defaults and cost and noise are not problems: a part-time owner is enough,
unless back ends are self-hosted or data is regulated.
:::

### Conclusion

Behind the ObservabilityOps label is real work that every organisation with more than a few services ends up doing, well or
badly: deciding what telemetry to produce, moving it safely, keeping it affordable, and making sure the alerts mean something.
OpenTelemetry makes the pipeline portable; the decisions stay human: which questions matter, how many series a team may spend, which data is too sensitive to keep, which page should wake
someone. Run the system that answers them like a product, with users, SLOs, a budget and the discipline to delete what nobody uses.

## Sources and further reading

Each footnote gives its check date. Versions, statuses, defaults and survey figures are time-sensitive: re-check them against
the release notes and component READMEs below at each Collector minor release and at least quarterly. The evidence ledger is
`research/observabilityops-ledger.md`.

[^otelwhat]: [What is OpenTelemetry?](https://opentelemetry.io/docs/what-is-opentelemetry/) — OpenTelemetry, updated 2026-04-06. Checked 2026-10-16.
[^dora]: [Capabilities: Monitoring and observability](https://dora.dev/capabilities/monitoring-and-observability/) — DORA, undated. Checked 2026-10-16.
[^glteam]: [Observability Team](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/production-engineering/observability/) — The GitLab Handbook, living page. Checked 2026-10-16.
[^moj]: [What is the Observability Platform?](https://user-guide.observability-platform.service.justice.gov.uk/overview) — UK Ministry of Justice, reviewed 2026-10-01. Checked 2026-10-16.
[^nrcoe]: [Observability Center of Excellence](https://docs.newrelic.com/docs/new-relic-solutions/observability-maturity/operational-efficiency/observability-coe/) — New Relic Documentation. Checked 2026-10-16.
[^gsurvey2025]: [Observability Survey Report 2025](https://grafana.com/observability-survey/2025/) — Grafana Labs, 2025 (fieldwork 2024-09-18 to 2025-01-02). Checked 2026-10-16.
[^gsurvey2026]: [Observability Survey Report 2026](https://grafana.com/observability-survey/2026/) — Grafana Labs, 2026 (fieldwork 2025-10-01 to 2026-01-10). Checked 2026-10-17.
[^cfprom]: [How Cloudflare runs Prometheus at scale](https://blog.cloudflare.com/how-cloudflare-runs-prometheus-at-scale/) — Łukasz Mierzwa, Cloudflare, 2023-03-03. Checked 2026-10-16.
[^sremon]: [Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/) — Rob Ewaschuk, *Site Reliability Engineering* (Google, O'Reilly, 2016). Checked 2026-10-16.
[^motadata]: [ObserveOps](https://www.motadata.com/products/observeops) — Motadata, product page. Checked 2026-10-16.
[^tagobs]: [Observability Whitepaper](https://github.com/cncf/tag-observability/blob/main/whitepaper.md) — CNCF TAG Observability, v1.0 (October 2023). Checked 2026-10-16.
[^use]: [The USE Method](https://www.brendangregg.com/usemethod.html) — Brendan Gregg, 2012. Checked 2026-10-16.
[^red]: [The RED Method: How to Instrument Your Services](https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/) — Julie Dam on Tom Wilkie's talk, Grafana Labs, 2018. Checked 2026-10-16.
[^cncfprom]: [Prometheus](https://www.cncf.io/projects/prometheus/) — CNCF project page. Checked 2026-10-16.
[^cncfotel]: [OpenTelemetry](https://www.cncf.io/projects/opentelemetry/) — CNCF project page. Checked 2026-10-16.
[^cncfthanos]: [Thanos](https://www.cncf.io/projects/thanos/) — CNCF project page. Checked 2026-10-16.
[^w3ctc]: [Trace Context](https://www.w3.org/TR/trace-context/) — W3C Recommendation, 2021-11-23 (edits the 2020-02-06 Recommendation). Checked 2026-10-16.
[^prom3]: [Announcing Prometheus 3.0](https://prometheus.io/blog/2024/11/14/prometheus-3-0/) — Prometheus Team, 2024-11-14. Checked 2026-10-16.
[^cncfjaeger]: [Jaeger v2 released: OpenTelemetry in the core!](https://www.cncf.io/blog/2024/11/12/jaeger-v2-released-opentelemetry-in-the-core/) — CNCF blog, 2024-11-12. Checked 2026-10-16.
[^otelprof]: [OpenTelemetry Profiles Enters Public Alpha](https://opentelemetry.io/blog/2026/profiles-alpha/) — OpenTelemetry blog, 2026-03-26. Checked 2026-10-16.
[^otelgrad]: [OpenTelemetry Graduates](https://opentelemetry.io/blog/2026/otel-graduates/) — OpenTelemetry blog, 2026-05-21. Checked 2026-10-16.
[^mustin]: [Notes on an Observability Team](https://ericmustin.substack.com/p/notes-on-an-observability-team) — Eric Mustin, 2022-04-20. Checked 2026-10-16.
[^semconv]: [OpenTelemetry semantic conventions 1.44.0](https://opentelemetry.io/docs/specs/semconv/) — OpenTelemetry. Checked 2026-10-16.
[^otelcolcfg]: [Configuration](https://opentelemetry.io/docs/collector/configuration/) — OpenTelemetry Collector docs, updated 2026-09-10. Checked 2026-10-16.
[^otelsampling]: [Sampling](https://opentelemetry.io/docs/concepts/sampling/) — OpenTelemetry, updated 2025-10-16. Checked 2026-10-16.
[^promalert]: [Alerting](https://prometheus.io/docs/practices/alerting/) — Prometheus best practices. Checked 2026-10-16.
[^cncfplat]: [CNCF Platforms White Paper](https://tag-app-delivery.cncf.io/whitepapers/platforms/) — CNCF TAG App Delivery. Checked 2026-10-16.
[^promnaming]: [Metric and label naming](https://prometheus.io/docs/practices/naming/) — Prometheus best practices. Checked 2026-10-16.
[^otelsensitive]: [Handling sensitive data](https://opentelemetry.io/docs/security/handling-sensitive-data/) — OpenTelemetry, updated 2026-01-14. Checked 2026-10-16.
[^otelmastodon]: [How Mastodon Runs OpenTelemetry Collectors in Production](https://opentelemetry.io/blog/2026/devex-mastodon/) — OpenTelemetry blog, 2026-03-18. Checked 2026-10-17.
[^colchangelog]: [opentelemetry-collector CHANGELOG](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CHANGELOG.md) — OpenTelemetry. Checked 2026-10-16.
[^contribchangelog]: [opentelemetry-collector-contrib CHANGELOG](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CHANGELOG.md) — OpenTelemetry. Checked 2026-10-17.
[^promunittest]: [Unit testing for rules](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/) — Prometheus docs. Checked 2026-10-16.
[^pint]: [pint, a Prometheus rule linter](https://github.com/cloudflare/pint) — Cloudflare (v0.89.2, 2026-10-08). Checked 2026-10-16.
[^semres]: [Service resource conventions](https://opentelemetry.io/docs/specs/semconv/resource/service/) — OpenTelemetry. Checked 2026-10-16.
[^memlim]: [Memory Limiter Processor](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md) — OpenTelemetry Collector. Checked 2026-10-16.
[^otelgw]: [Gateway deployment pattern](https://opentelemetry.io/docs/collector/deployment/gateway/) — OpenTelemetry, updated 2026-09-10. Checked 2026-10-16.
[^otelresil]: [Resiliency](https://opentelemetry.io/docs/collector/resiliency/) — OpenTelemetry Collector docs. Checked 2026-10-16.
[^tailsamp]: [Tail Sampling Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md) — OpenTelemetry contrib. Checked 2026-10-16.
[^promstorage]: [Storage](https://prometheus.io/docs/prometheus/latest/storage/) — Prometheus docs. Checked 2026-10-16.
[^semhttp]: [Semantic conventions for HTTP metrics](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/) — OpenTelemetry. Checked 2026-10-16.
[^otelspanev]: [Deprecating Span Events API](https://opentelemetry.io/blog/2026/deprecating-span-events/) — OpenTelemetry blog, 2026-03-17. Checked 2026-10-16.
[^prominstr]: [Instrumentation](https://prometheus.io/docs/practices/instrumentation/) — Prometheus best practices. Checked 2026-10-16.
[^lokibp]: [Label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/) — Grafana Loki docs. Checked 2026-10-16.
[^promcfg]: [Configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) — Prometheus docs. Checked 2026-10-16.
[^promjobs]: [Jobs and instances](https://prometheus.io/docs/concepts/jobs_instances/) — Prometheus docs. Checked 2026-10-16.
[^honeyhc]: [High Cardinality](https://docs.honeycomb.io/get-started/observability/concepts/high-cardinality) — Honeycomb docs. Checked 2026-10-16.
[^srewbslo]: [Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/) — *The Site Reliability Workbook* (Google, O'Reilly, 2018). Checked 2026-10-16.
[^otelstatus]: [Status](https://opentelemetry.io/status/) — OpenTelemetry. Checked 2026-10-16.
[^promnh]: [Native Histograms](https://prometheus.io/docs/specs/native_histograms/) — Prometheus specification. Checked 2026-10-16.
[^relotel]: [opentelemetry-collector-releases v0.163.1](https://github.com/open-telemetry/opentelemetry-collector-releases/releases/tag/v0.163.1) — GitHub, 2026-10-16. Checked 2026-10-16.
[^alloy]: [Introduction to Grafana Alloy](https://grafana.com/docs/alloy/latest/introduction/) — Grafana Labs. Checked 2026-10-16.
[^fluentbit]: [What is Fluent Bit?](https://docs.fluentbit.io/manual/about/what-is-fluent-bit) — Fluent Bit docs. Checked 2026-10-16.
[^vector]: [Vector](https://vector.dev/) — Vector project site. Checked 2026-10-16.
[^relprom]: [Prometheus v3.15.0](https://github.com/prometheus/prometheus/releases/tag/v3.15.0) — GitHub, 2026-09-25. Checked 2026-10-16.
[^relloki]: [Loki v3.7.8](https://github.com/grafana/loki/releases/tag/v3.7.8) — GitHub, 2026-09-17. Checked 2026-10-16.
[^reles]: [Elasticsearch v9.5.5](https://github.com/elastic/elasticsearch/releases/tag/v9.5.5) — GitHub, 2026-10-06. Checked 2026-10-16.
[^reljaeger]: [Jaeger v2.22.0](https://github.com/jaegertracing/jaeger/releases/tag/v2.22.0) — GitHub, 2026-10-06. Checked 2026-10-16.
[^reltempo]: [Tempo v3.1.0](https://github.com/grafana/tempo/releases/tag/v3.1.0) — GitHub, 2026-09-29. Checked 2026-10-16.
[^jaegerarch]: [Architecture](https://www.jaegertracing.io/docs/latest/architecture/) — Jaeger docs (v2.22), updated 2026-10-08. Checked 2026-10-16.
[^relgrafana]: [Grafana v13.2.4](https://github.com/grafana/grafana/releases/tag/v13.2.4) — GitHub, 2026-10-13. Checked 2026-10-16.
[^otelvendors]: [Vendors](https://opentelemetry.io/ecosystem/vendors/) — OpenTelemetry, updated 2026-10-09. Checked 2026-10-16.
[^grafanaagpl]: [Grafana, Loki, and Tempo will be relicensed to AGPLv3](https://grafana.com/blog/2021/04/20/grafana-loki-tempo-relicensing-to-agplv3/) — Raj Dutt, Grafana Labs, 2021. Checked 2026-10-16.
[^elasticagpl]: [Elasticsearch is Open Source, Again](https://www.elastic.co/blog/elasticsearch-is-open-source-again) — Shay Banon, Elastic, 2024-08-29. Checked 2026-10-16.
[^spanmetrics]: [Span Metrics Connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/README.md) — OpenTelemetry contrib. Checked 2026-10-17.
[^redaction]: [Redaction Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/redactionprocessor/README.md) — OpenTelemetry contrib. Checked 2026-10-16.
[^telemetrygen]: [telemetrygen](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/telemetrygen/README.md) — OpenTelemetry contrib. Checked 2026-10-16.
[^otelinternal]: [Internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/) — OpenTelemetry Collector docs. Checked 2026-10-17.
[^promapi]: [HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/) — Prometheus docs. Checked 2026-10-16.
[^gdpr5]: [Art. 5 GDPR: Principles relating to processing of personal data](https://gdpr-info.eu/art-5-gdpr/) — unofficial consolidated text, intersoft consulting (the official EUR-Lex text could not be fetched automatically). Checked 2026-10-16.
[^otelobi]: [OpenTelemetry eBPF Instrumentation Marks the First Release](https://opentelemetry.io/blog/2025/obi-announcing-first-release/) — OpenTelemetry blog, 2025-11-03. Checked 2026-10-16.
[^otelsampm]: [OpenTelemetry Sampling update](https://opentelemetry.io/blog/2025/sampling-milestones/) — OpenTelemetry blog, 2025-10-15. Checked 2026-10-16.
[^oebook]: [Observability Engineering, 2nd Edition](https://www.honeycomb.io/observability-engineering-oreilly-book) — Honeycomb book page (Majors, Fong-Jones, Miranda, Parker; O'Reilly; no publication date shown). Checked 2026-10-16.
[^promcl]: [Prometheus CHANGELOG](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) — Prometheus (entry "2.47.0 / 2023-09-06"). Checked 2026-10-16.
[^lbexp]: [Load Balancing Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/loadbalancingexporter/README.md) — OpenTelemetry contrib. Checked 2026-10-17.
[^exhelper]: [Exporter Helper](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md) — OpenTelemetry Collector. Checked 2026-10-16.
[^exhelpercode]: [base_exporter.go](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/internal/base_exporter.go) — OpenTelemetry Collector source. Checked 2026-10-16.
[^otelctx]: [Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/) — OpenTelemetry, updated 2026-08-10. Checked 2026-10-16.
[^otelbaggage]: [Baggage](https://opentelemetry.io/docs/concepts/signals/baggage/) — OpenTelemetry, updated 2026-02-22. Checked 2026-10-16.
[^otelenv]: [Environment Variable Specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/) — OpenTelemetry specification 1.61.0. Checked 2026-10-16.
[^semmsg]: [Semantic conventions for messaging spans](https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/) — OpenTelemetry semantic conventions 1.44.0 (status: Development). Checked 2026-10-16.
[^otellogs]: [Logs](https://opentelemetry.io/docs/concepts/signals/logs/) — OpenTelemetry, updated 2026-08-07. Checked 2026-10-16.
[^k8slog]: [Logging Architecture](https://kubernetes.io/docs/concepts/cluster-administration/logging/) — Kubernetes documentation, updated 2024-10-17. Checked 2026-10-16.
[^otelk8s]: [Important Components for Kubernetes](https://opentelemetry.io/docs/platforms/kubernetes/collector/components/) — OpenTelemetry, updated 2026-09-10. Checked 2026-10-17.
[^container]: [container operator](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/container.md) — OpenTelemetry contrib. Checked 2026-10-16.
[^filelog]: [Filelog Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/filelogreceiver/README.md) — OpenTelemetry contrib. Checked 2026-10-17.
[^recombine]: [recombine operator](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/recombine.md) — OpenTelemetry contrib. Checked 2026-10-16.
[^filter]: [Filter Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md) — OpenTelemetry contrib. Checked 2026-10-16.
[^ottllog]: [OTTL log context](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottllog/README.md) — OpenTelemetry contrib. Checked 2026-10-16.
[^smfactory]: [Span metrics connector factory.go](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/factory.go) — OpenTelemetry contrib source. Checked 2026-10-17.
[^configtls]: [TLS Configuration Settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md) — OpenTelemetry Collector. Checked 2026-10-17.
[^filestorage]: [File Storage Extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/storage/filestorage/README.md) — OpenTelemetry contrib. Checked 2026-10-17.

**Further reading:** [The Site Reliability Workbook: Monitoring](https://sre.google/workbook/monitoring/) (Google, 2018);
*Observability Engineering* ([first edition, O'Reilly, 2022](https://bookshare.org/browse/book/6423904)).

**Changelog.** 2026-10-16: first version (research, ledger, three artefacts, three labs of which Lab 1 was executed, six diagrams);
term status confirmed as ambiguous; awaiting independent review.
2026-10-16 (r1 revision after independent review): span metrics routed by service (Artefact 1, figure 3, ADR); context propagation,
baggage and log pipelines added; log-body redaction, Trace Context date, Prometheus OTLP history and Mastodon attribution corrected;
loss alerts and least-privilege bindings added; four diagrams redrawn; trimmed.
2026-10-17, post-score polish: span-metric churn under service routing; `server_name_override`, agent exposure and hostPath
offsets in Artefact 1; exact survey label and Mastodon attribution; re-check cadence.