ObservabilityOps

ObservabilityOps has no standard definition: searches on 2026-10-16 found no definition, standards body, certification or vendor programme for the label (see term status). In this Atlas it means running observability as an internal product: operating the telemetry pipeline and back ends, setting instrumentation standards, keeping cardinality and cost within budgets and keeping alerts useful, so that any engineer can answer a new question about production quickly, from trustworthy data, at a predictable cost.

Observability itself is well defined. OpenTelemetry describes it as “the ability to understand the internal state of a system by examining its outputs”, typically traces, metrics and logs.^[1] DORA separates monitoring, which watches predefined metrics or logs, from observability, which explores properties and patterns “not defined in advance”.^[2] ObservabilityOps is the operational work that keeps that ability available and affordable.

For engineers who run the stack, SREs and service owners who depend on it, and leaders who pay for it.

When nobody owns the observability system, three failures recur.

1. Cost grows faster than value. Telemetry volume follows traffic and code, not the questions anyone asks. In Grafana Labs’ 2025 survey (1,255 self-selected respondents, so indicative at best) mean reported observability spend was 17% of compute infrastructure spend (median 10%), and 37% named “Costs too much” as a concern.^[6]
2. Noise drowns signal. In the same vendor’s 2026 survey, the top three concerns were complexity/overhead (38%), signal-to-noise challenges (34%) and cost (31%).^[7] The human side of alert noise is covered in AIOps and IncidentOps.
3. The tools fail when they are needed most. One label filled with user-supplied values can create millions of series. Cloudflare describes how such a “cardinality explosion” can make Prometheus run out of memory, so that “you lose all observability as a result”.^[8]

The label could mean either of two things:

1. Operating with observability: running production by looking at telemetry, which duplicates SRE, IncidentOps and AIOps.
2. Operating the observability system: treating telemetry as an internal product with users, SLOs, a budget and governance. This page takes this reading.

• Not the product “ObserveOps”. Motadata sells an observability platform with that name.^[11]
• Not SRE. SRE sets reliability targets and policy; ObservabilityOps runs the measurement system they depend on.
• Not monitoring under a new name. Exploring what was “not defined in advance”^[2] needs high-dimensional data, which is where most cost and governance problems sit.
• Not a team that does observability for everyone. DORA writes: “Monitoring systems should not be confined to a single individual or team within an organization.”^[2] A central team can run those systems, provided every team can query and alert on them itself. Service teams still instrument their code and own their alerts.

No source dates the first observability teams; one practitioner wrote in 2022 that the concept was “relatively new”.^[23]

One slow request, four signals (illustrative). A checkout request takes 4 seconds instead of 300 ms.

Mental model: a telemetry supply chain. Producers emit signals; a pipeline moves, enriches, redacts and samples them; back ends store them; consumers (dashboards, alert rules, queries, AIOps) turn them into decisions. ObservabilityOps owns the middle. Its unit of value is a question answered quickly; its unit of cost is the bytes and series kept.

flowchart TB
  subgraph prod["Producers"]
    A["`AppOps:
    service code`"]
    K["`KubernetesOps, CloudOps:
    infrastructure`"]
  end
  F["`FinOps:
  cost allocation`"]
  O["`ObservabilityOps: pipeline,
  storage, standards, budgets`"]
  SD["`SecDataOps:
  security telemetry`"]
  subgraph cons["Consumers"]
    S["`SRE:
    SLOs`"]
    I["`IncidentOps:
    response`"]
    AI["`AIOps:
    correlation`"]
  end
  A --> O
  K --> O
  F -. "budgets" .-> O
  O -. "shared tools" .- SD
  O --> S
  O --> I
  O --> AI
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class O core
  class A,K,S,I,AI,F,SD neutral

No official ObservabilityOps manifesto exists. Mission: every engineer can answer a new question about production within minutes, from trustworthy data, at an agreed cost.

1. Collect for a consumer. Each signal serves a dashboard, an alert or a known debugging question. Signals used by no prebaked dashboard and no alert “are candidates for removal” (Google SRE book).^[9]
2. Open standards at the edges. Instrument with OpenTelemetry so the back end can change: “You own the data that you generate.”^[1]
3. Bounded dimensions in metrics, rich context in events. Unbounded identifiers go into traces and logs, never metric labels.^[29][12]
4. Symptoms page; causes explain. Page on what users feel and use cause-level data to diagnose.^[9][30]
5. The platform has SLOs and is monitored end to end.^[30]
6. Cost is visible and allocated to the teams and units of work that generate it.
7. Enable, do not gatekeep. Self-service within guardrails; the platform team is not the only team that can query or alert.^[2]
8. Minimise and redact at the edge. Personal data and secrets do not enter the pipeline unless needed.^[31]

Invariants: dropped data is counted and alerted on; no secrets in telemetry; every page has an owner and a runbook. Non-goals: owning other teams’ SLOs, being on call for their services, reading every log.

• Observability platform team: a product owner and engineers who run the pipeline and back ends. In a small organisation this may be one person, as at Mastodon (Case 1).^[32]
• Community structure: New Relic’s guidance proposes a core team (typically one to three people), a council that ratifies standards and a volunteer guild.^[5] It is one vendor’s model, useful when a full team is not justified.

R = responsible, A = accountable, C = consulted, I = informed. Adjust to your organisation, but leave no row blank.

flowchart TB
  P["Propose: owner, question it answers, expected series or volume"]
  R{"Meets standard and budget?"}
  X["Redesign: move IDs to span attributes, or request a time-limited exception"]
  I["Instrument and ship through the pipeline"]
  U["Use: dashboards, alerts, ad hoc queries"]
  Q{"Used in 90 days?"}
  T["Downsample, move to a cheaper tier, or retire"]
  P --> R
  R -- "yes" --> I
  R -- "no" --> X
  X --> P
  I --> U
  U --> Q
  Q -- "yes" --> U
  Q -- "no" --> T
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class P,I,U core
  class R,Q decision
  class X,T risk

• Day 0 (design): choose standards, architecture, tools, budget rules and retention per data class.
• Day 1 (onboarding): templates give each service OpenTelemetry defaults, a RED dashboard and a symptom alert.
• Day 2 (operations): capacity, upgrades, reviews and pruning. Collector releases are frequent and rename components (see the tooling landscape).^[33][34] Mastodon upgrades “within a day or two of each release” and sees breaking changes as a sign of healthy development, provided you stay current.^[32]

Change control: pipeline and rule configuration lives in Git, is validated in CI (otelcol validate, promtool, a linter such as pint)^[25][35][36] and is deployed by reconciliation, as Mastodon does with the OpenTelemetry Operator and Argo CD.^[32] Escalation: a platform outage is an incident like any other (IncidentOps); budget exceptions end with whoever owns the cost.

flowchart TB
  subgraph src["1. Producers"]
    APP["`Services with
    OpenTelemetry SDKs`"]
    INF["`Hosts, clusters,
    container logs`"]
  end
  subgraph agt["2. Agents"]
    AG["`Agent per node: metadata,
    memory limit, logs, routing`"]
  end
  subgraph gwt["3. Gateway replicas, mTLS"]
    TS["`Tail sampling:
    spans by trace ID`"]
    SM["`Span metrics:
    spans by service`"]
    PL["`Metrics, logs, profiles:
    redact, filter`"]
  end
  T[("`Trace
  store`")]
  M[("`Metric
  store`")]
  L[("`Log
  store`")]
  PR[("`Profile
  store`")]
  D["`Dashboards
  and queries`"]
  R["`Alert rules
  and routing`"]
  W["`Meta-monitoring:
  independent watcher`"]
  APP --> AG
  INF --> AG
  AG -- "by trace ID" --> TS
  AG -- "by service" --> SM
  AG -- "any replica" --> PL
  TS --> T
  SM --> M
  PL --> M
  PL --> L
  PL --> PR
  M --> D
  L --> D
  T --> D
  PR --> D
  M --> R
  W -.-> PL
  W -.-> R
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class AG,TS,SM,PL core
  class APP,INF,M,L,T,PR,D,R neutral
  class W risk

Alternatives. Sending from SDKs straight to a back end is simplest but leaves no place for central policy; a single Collector tier, as at Mastodon (Case 1), avoids routing altogether. A gateway tier buys central credentials and policy at the price of “one more thing to maintain and that can fail”, latency and resources. It must also respect the single-writer rule: every metric stream needs “a single writer and a globally unique identity”, or the series shows gaps, jumps or out-of-order errors.^[39] That rule is why span metrics are routed by service.

Trust boundaries. Anything that can reach an OTLP endpoint can create series and cost. The Collector binds to localhost by default; documentation examples use 0.0.0.0 only for convenience.^[25] Artefact 1 states who can reach each endpoint. Logs and traces can contain personal data, so query access needs roles and tenancy.

sequenceDiagram
  participant BO as Budget owner
  participant ST as Team
  participant OB as ObsOps
  participant CI as CI
  ST->>OB: Request: signals, size
  OB->>OB: Check standard, PII, budget
  alt Within budget
    OB->>CI: Merge routes, limits
    CI->>CI: Validate, test rules
    CI-->>OB: Deployed
    OB-->>ST: Dashboard, alerts live
  else Over budget or risky
    OB-->>ST: Redesign or seek exception
    ST->>BO: Cost and expiry date
    BO-->>ST: Approve or reject
    ST->>OB: Approved exception
    OB->>CI: Merge with expiry date
  end
  OB->>ST: Review after 30 days

Acceptance criteria for “onboarded”: telemetry arrives under the right service.name; the standard dashboard renders; at least one symptom alert has a runbook and an owner; series and log volume are inside budget; a test request yields one unbroken trace; nothing in the debug output matches the redaction patterns.

Instrumentation standard. Use OpenTelemetry SDKs and semantic conventions. service.name is required; when it is missing, SDKs fall back to unknown_service: plus the executable name,^[37] so its share is a cheap hygiene metric. Record HTTP routes as templates: the conventions say “the URI path can NOT substitute” http.route.^[45] Emit events as logs correlated with the current span; OpenTelemetry is deprecating the Span Event API.^[46]

Cardinality control. Prometheus advises that “every unique combination of key-value label pairs represents a new time series” and that labels must not hold “user IDs, email addresses, or other unbounded sets of values”.^[29] Keep a metric’s cardinality below about 10 and investigate anything above 100.^[47] Loki asks the same of log stream labels: dynamic labels “ideally limited to tens of values”, and no trace or order IDs.^[48] Scrape limits (sample_limit, label_limit) fail the whole scrape when exceeded,^[49] so watch scrape_series_added and scrape_samples_post_metric_relabeling.^[50] High-cardinality context is not bad in itself, only in metric labels: event stores treat user or order IDs as the key to isolating a problem.^[51]

Sampling. OpenTelemetry suggests sampling from about 1,000 traces per second, and avoiding it at tens of small traces per second or where regulation forbids dropping data. Head sampling is cheap but cannot guarantee that every error trace is kept; tail sampling can, but it is “difficult to operate”.^[26] The Collector’s tail sampler needs every span of a trace on one instance, and its sampling_trace_dropped_too_early metric shows traces lost before a decision.^[41] Derive request rates from all spans before the sampler drops any, since rates counted from sampled spans fall by the sampling factor (Artefact 1). Mastodon’s published configuration uses the same order: trace statistics from every span first, then tail sampling.^[32]

Context propagation. A trace survives only if every hop passes its context on. OpenTelemetry SDKs default to the W3C tracecontext and baggage propagators (OTEL_PROPAGATORS); B3 is an option.^[52] Fix the list in the standard, and while some services still use B3 configure both (tracecontext,baggage,b3multi). Traces break in three places:

• Queues and batches. A span has one parent, so OpenTelemetry’s messaging conventions (still in development) link consumer spans to each message’s creation context with span links.^[53] The queue client must carry context in message headers.
• Intermediaries. W3C Trace Context requires a vendor that receives traceparent and tracestate to send them on;^[17] a proxy, API gateway or mesh that drops them starts a new trace behind it.
• Mixed instrumentation. A service on another propagator produces fragments that look like separate traces.

Trust and baggage. OpenTelemetry advises caution with incoming context from external sources (forged headers can manipulate tracing data) and with sending context to external endpoints (IDs and baggage reveal internal details).^[27] Instrumentation adds baggage to most outgoing requests, so it can reach third-party APIs, and nothing checks its integrity.^[54] Keep credentials, API keys and personal data out of it,^[27] although OpenTelemetry’s baggage page lists user IDs among typical uses:^[54] allow-list baggage keys. A workable policy starts a new trace at the public edge, which W3C Trace Context permits,^[17] and strips baggage at egress.

Log pipelines. Four decisions keep logs useful and affordable.

1. Structure. OpenTelemetry defines a structured log by its stable schema, not by valid JSON alone; its log records carry TraceId, SpanId and severity as top-level fields.^[55] Require them in every log line: SDKs can inject the IDs;^[27] services that print to stdout must write them as fields.
2. Collection. Kubernetes describes a node-level logging agent, run as a DaemonSet, that reads container log files (by default under /var/log/pods).^[56] For the Collector, OpenTelemetry calls the file_log receiver “the de facto solution”; its container parser reads docker, CRI-O and containerd formats and adds pod metadata.^[57][58] Fluent Bit is an alternative agent.^[59] Without a storage extension, offsets live only in memory and a restarted agent cannot resume.^[60]
3. Parsing and multiline. Parse JSON lines into severity and trace fields at the agent (Artefact 1). Join stack traces with the recombine operator after the container parser,^[61] or better, log each exception as one record.
4. Routing by level. The filter processor drops records matching a condition such as log.severity_number < SEVERITY_NUMBER_WARN.^[62] Records whose severity was never parsed have severity number 0 (SEVERITY_NUMBER_UNSPECIFIED) and match it too,^[63] so exclude 0 or parse severity first.

Retention tiers. Prometheus keeps 15 days by default on single-node local storage.^[44] A typical design keeps raw data hot briefly, downsampled metrics warm for months and compressed logs in object storage as policy requires; security retention belongs to SecDataOps.

Dashboards and alerts. Give every request-driven service a RED view and every resource a USE view, as Wilkie recommends.^[14][13] Design the drill-down: alert, dashboard, exemplar trace, that trace’s logs.^[12] Page on symptoms, use tickets for slow problems, and test each page against the SRE book’s questions (urgent, actionable, user-visible).^[9] Burn-rate thresholds belong to the SLO owner (SRE).^[64]

Meta-monitoring. Prometheus recommends alerts that the monitoring works, and notes that one black-box test of the alert path “is better than individual alerts on each” component.^[30] The usual implementation is a dead man’s switch: an always-firing alert whose absence pages through another channel.

None is mandatory; what binds comes from contracts and law, such as privacy rules on personal data in logs.

Selection criteria. OTLP at the edge, so you can switch back ends; the cost model (per series, per GB, per host); query languages your engineers can learn; licence (Grafana, Loki and Tempo moved from Apache 2.0 to AGPLv3 in 2021;^[80] Elastic added AGPL next to ELv2 and SSPL in 2024^[81]); and the burden of running stateful databases. Version churn is real: the Collector renamed otlp to otlp_grpc (v0.144.0) and many contrib components to snake_case,^[33][34] and native histograms went from experimental in Prometheus 3.0 to stable in 3.8.^[19][67] Pin versions. Defaults move too: current Collector documentation batches in the exporter’s sending queue (sending_queue → batch), not in the beta batch processor, which holds data in memory outside any persistent queue.^[25][82]

Automate governance first: configuration validation, rule unit tests, cardinality checks in CI, budget reports and lists of unused signals. Anomaly detection and event correlation, and when simple rules beat them, are the subject of AIOps. Natural-language query assistants need their generated queries checked, and anything that reads telemetry follows the same redaction rules. Observability of LLM applications is covered in LLMOps.

Assumptions: services on Kubernetes, instrumented with OpenTelemetry; Collector contrib 0.162 or later; Prometheus 3.x; an OTLP-capable back end at the placeholder backend.example.internal; certificates from a platform CA. Three artefacts follow; Lab 1 runs the third.

Two files implement the reference architecture. Agents send each span twice: by trace ID to the tail sampler and by service to a second receiver for span metrics;^[39][42] metrics and logs go to any replica. The agent also collects container logs, parses JSON lines and drops DEBUG and TRACE records. Names follow the 2026 snake_case renames.^[34] span_metrics and filter are alpha, and the span-metrics duration unit is changing from milliseconds to seconds;^[43][62] redaction is beta for traces, alpha for logs and metrics.^[83] MY_POD_IP comes from the downward API. The DaemonSet mounts /var/log/pods read-only^[57] and a writable hostPath for file_storage, so log offsets survive a replaced pod.^[84]

Example status: YAML syntax-checked; not executed (requires Kubernetes, certificates from a platform CA and a back end).

# agent.yaml: DaemonSet, one Collector per node
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317   # hostPort; no TLS or auth (see below)
  file_log:                               # container logs on this node
    include: [/var/log/pods/*/*/*.log]
    exclude: [/var/log/pods/*/otel-agent/*.log]   # not the agent's own logs
    start_at: end
    include_file_path: true
    storage: file_storage                 # resume from saved offsets after a restart
    operators:
      - type: container                   # docker, CRI-O, containerd; adds pod metadata
      - type: json_parser                 # services log JSON with level and trace IDs
        if: 'body startsWith "{"'
        severity: {parse_from: attributes.level}
        trace:
          trace_id: {parse_from: attributes.trace_id}
          span_id: {parse_from: attributes.span_id}
extensions:
  file_storage:
    directory: /var/lib/otelcol/file_storage   # a hostPath: offsets outlive the pod
    create_directory: true
processors:
  memory_limiter: {check_interval: 1s, limit_mib: 400, spike_limit_mib: 100}
  filter/drop_debug:                      # drop DEBUG and TRACE; keep unparsed (0)
    log_conditions:
      - >-
        log.severity_number > SEVERITY_NUMBER_UNSPECIFIED and
        log.severity_number < SEVERITY_NUMBER_INFO
exporters:
  load_balancing/by_trace:                # tail sampling: whole traces on one replica
    routing_key: traceID
    protocol:
      otlp:
        tls:                              # endpoints are pod IPs: verify the DNS name
          ca_file: /tls/ca.crt
          cert_file: /tls/tls.crt
          key_file: /tls/tls.key
          server_name_override: otel-gateway.observability.svc.cluster.local
    resolver:
      dns: {hostname: otel-gateway-headless.observability.svc.cluster.local, port: 4317}
  load_balancing/by_service:              # span metrics: each service on one replica
    routing_key: service
    protocol:
      otlp:
        tls:
          ca_file: /tls/ca.crt
          cert_file: /tls/tls.crt
          key_file: /tls/tls.key
          server_name_override: otel-gateway.observability.svc.cluster.local
    resolver:
      dns: {hostname: otel-gateway-headless.observability.svc.cluster.local, port: 4319}
  otlp_grpc/gateway:                      # metrics and logs: any replica
    endpoint: otel-gateway.observability.svc.cluster.local:4317
    tls: {ca_file: /tls/ca.crt, cert_file: /tls/tls.crt, key_file: /tls/tls.key}
    sending_queue: {batch: {}}
service:
  extensions: [file_storage]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [load_balancing/by_trace, load_balancing/by_service]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp_grpc/gateway]
    logs:
      receivers: [otlp, file_log]
      processors: [memory_limiter, filter/drop_debug]
      exporters: [otlp_grpc/gateway]

# gateway.yaml: Deployment with two or more replicas behind a headless Service
receivers:
  otlp:                                   # spans routed by trace ID; metrics; logs
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317
        tls:                              # mutual TLS: clients need a platform-CA cert
          cert_file: /tls/tls.crt
          key_file: /tls/tls.key
          client_ca_file: /tls/ca.crt
  otlp/by_service:                        # spans routed by service, for span metrics only
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4319
        tls:
          cert_file: /tls/tls.crt
          key_file: /tls/tls.key
          client_ca_file: /tls/ca.crt
processors:
  memory_limiter: {check_interval: 1s, limit_mib: 1500, spike_limit_mib: 300}
  redaction:
    allow_all_keys: true                  # keep keys, but mask risky values
    blocked_key_patterns: [".*token.*", ".*password.*", ".*secret.*"]
    # e-mail addresses, masked in attributes and in string log bodies
    blocked_values: ['[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}']
    summary: info
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - {name: errors, type: status_code, status_code: {status_codes: [ERROR]}}
      - {name: slow, type: latency, latency: {threshold_ms: 2000}}
      - {name: baseline, type: probabilistic, probabilistic: {sampling_percentage: 1}}
connectors:
  span_metrics:
    metrics_expiration: 5m                # stop a service's series after it moves away
exporters:
  otlp_grpc/backend:
    endpoint: backend.example.internal:4317
    headers:
      authorization: "Bearer ${env:BACKEND_TOKEN}"   # injected from a secret
    sending_queue:
      queue_size: 5000
      batch: {}                           # batches after sampling, inside the exporter
    retry_on_failure:
      max_elapsed_time: 10m
service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: ${env:MY_POD_IP}    # scraped only by the platform Prometheus
                port: 8888
                without_type_suffix: true
                without_units: true
  pipelines:
    traces/sampled:
      receivers: [otlp]
      processors: [memory_limiter, redaction, tail_sampling]
      exporters: [otlp_grpc/backend]
    traces/span_metrics:
      receivers: [otlp/by_service]
      processors: [memory_limiter]
      exporters: [span_metrics]
    metrics:
      receivers: [otlp, span_metrics]
      processors: [memory_limiter]
      exporters: [otlp_grpc/backend]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, redaction]
      exporters: [otlp_grpc/backend]

What the routing costs. Each span crosses the network twice. The alternative, span metrics on the trace-ID-routed replicas, works only because the connector adds a collector.instance.id dimension (a UUID per instance) by default, so each replica writes its own series,^[43] multiplying series by the replica count. Never exclude that dimension while several replicas compute span metrics. Service routing keeps the dimension and some churn: by default the UUID is new each time a gateway starts,^[85][86] so every restart or rollout starts new span-metric series, and when the N replicas change about 1/N of services move too.^[42] metrics_expiration retires the old series; dashboards still need sum without (collector_instance_id).

Who can reach what. The agent’s receiver has no TLS or authentication and its hostPort opens it on every node, so anything that reaches port 4317 on a node can send data. Keep that port inside the cluster (host firewall, security group) or give the agent TLS and an authenticator too. Only clients with a platform-CA certificate reach the gateways. Agents dial gateway pod IPs; server_name_override makes them verify the Service’s DNS name instead, so issue gateway certificates for that name.^[42][87] Port 8888 should accept only the platform’s Prometheus (a network policy, not shown). Since contrib v0.126.0 redaction also covers log bodies: blocked_values inside string bodies, key rules in map bodies.^[83][34] Key rules cannot see a free-text password=… in a string, so code that never logs secrets remains the safer control.^[31]

Example status: YAML syntax-checked; not executed.

scrape_configs:
  - job_name: checkout
    sample_limit: 20000            # whole scrape fails above this, so alert well before it
    label_limit: 30
    label_value_length_limit: 200
    extra_scrape_metrics: true     # exposes scrape_sample_limit
    static_configs:
      - targets: ["checkout.example.internal:9464"]
    metric_relabel_configs:        # emergency containment: drop one offending metric
      - source_labels: [__name__]
        regex: checkout_orders_total
        action: drop

The pipeline alerts and their tests are in Lab 3 of the hands-on labs.

It computes worst-case series (instances × product of label values × series per label set; a classic histogram adds +Inf, _sum and _count to its buckets), applies the Prometheus disk formula at 1–2 bytes per sample,^[44] and flags labels with more than 100 values, where Prometheus suggests looking for alternatives.^[47]

Example status: executed with Python 3.8.10 on 2026-10-16; output below is real.

#!/usr/bin/env python3
"""Estimate worst-case Prometheus-style active series and disk for a metric inventory,
and flag labels and teams that break the telemetry budget. Python 3.8+, standard library only."""
import json
import sys
from math import prod

SERIES_PER_LABELSET = {"counter": 1, "gauge": 1}


def series_for(metric):
    combos = prod(metric.get("labels", {}).values()) if metric.get("labels") else 1
    if metric["type"] == "histogram":  # classic histogram: buckets + "+Inf" + _sum + _count
        per_set = metric["buckets"] + 3
    else:
        per_set = SERIES_PER_LABELSET[metric["type"]]
    return metric["instances"] * combos * per_set


def main(path):
    inv = json.load(open(path))
    b = inv["budget"]
    problems = 0
    per_team = {}
    print("metric                                team         series (upper bound)")
    for m in inv["metrics"]:
        s = series_for(m)
        per_team[m["team"]] = per_team.get(m["team"], 0) + s
        print(f"{m['name']:<37} {m['team']:<12} {s:>12,}")
        for label, values in m.get("labels", {}).items():
            if values > b["max_label_values"]:
                problems += 1
                print(f"  WARNING label '{label}' has {values:,} values (> {b['max_label_values']}): "
                      "move it to a span or log attribute")
    for team, s in sorted(per_team.items()):
        status = "OK" if s <= b["active_series_per_team"] else "OVER BUDGET"
        if status != "OK":
            problems += 1
        print(f"team {team:<12} {s:>12,} of {b['active_series_per_team']:,}  {status}")
    total = sum(per_team.values())
    sps = total / b["scrape_interval_s"]
    seconds = b["retention_days"] * 86400
    low, high = seconds * sps * 1 / 1e9, seconds * sps * 2 / 1e9
    print(f"total series {total:,}; samples/s {sps:,.1f}; "
          f"disk for {b['retention_days']} d: {low:.1f}-{high:.1f} GB at 1-2 bytes/sample")
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

{
  "budget": {"active_series_per_team": 200000, "max_label_values": 100,
             "scrape_interval_s": 30, "retention_days": 15},
  "metrics": [
    {"name": "http_server_request_duration_seconds", "team": "checkout", "type": "histogram",
     "buckets": 14, "instances": 12,
     "labels": {"http_request_method": 4, "http_route": 30, "http_response_status_code": 6}},
    {"name": "checkout_orders_total", "team": "checkout", "type": "counter", "instances": 12,
     "labels": {"payment_provider": 3, "customer_id": 50000}},
    {"name": "queue_depth", "team": "fulfilment", "type": "gauge", "instances": 6,
     "labels": {"queue": 8}}
  ]
}

The 14 buckets match the boundaries OpenTelemetry advises for http.server.request.duration.^[45] The label counts are illustrative.

Lab 1 was executed for this page. Labs 2 and 3 were not executed (no container runtime or promtool on the authoring host); their expected output is derived from current documentation.

Objective: find the label that breaks a budget before production. Prerequisites: Python 3.8 or later; no cost or risk; 30 minutes.

1. Save Artefact 3 as telemetry_budget.py and its JSON as inventory.json.
2. Run python3 telemetry_budget.py inventory.json; echo "exit=$?".
3. Delete the customer_id entry from checkout_orders_total and run again.
4. Replace the inventory with three real metrics from your own services.

Expected output (step 2, verbatim):

metric                                team         series (upper bound)
http_server_request_duration_seconds  checkout          146,880
checkout_orders_total                 checkout        1,800,000
  WARNING label 'customer_id' has 50,000 values (> 100): move it to a span or log attribute
queue_depth                           fulfilment             48
team checkout        1,946,880 of 200,000  OVER BUDGET
team fulfilment             48 of 200,000  OK
total series 1,946,928; samples/s 64,897.6; disk for 15 d: 84.1-168.2 GB at 1-2 bytes/sample
exit=1

Validation: after step 3, checkout totals 146,916 series, the run prints total series 146,964; samples/s 4,898.8; disk for 15 d: 6.3-12.7 GB and exits 0. Troubleshooting: ImportError: cannot import name 'prod' means Python older than 3.8; KeyError: 'buckets' means a histogram lacks a bucket count. The figures are upper bounds, because not every label combination occurs.^[8] Teardown: delete the two files.

Objective: prove that the pipeline masks secrets and personal data, and read its internal metrics. Prerequisites: Docker on Linux, about 1 GB of free memory, 45 minutes; local, no cost. Status: not executed.

1. Save this as lab2.yaml.

Example status: YAML syntax-checked; not executed.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
processors:
  memory_limiter: {check_interval: 1s, limit_mib: 400, spike_limit_mib: 100}
  redaction:
    allow_all_keys: true
    blocked_key_patterns: [".*token.*"]
    blocked_values: ['[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}']
    summary: info
exporters:
  debug: {verbosity: detailed}
service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: 127.0.0.1
                port: 8888
                without_type_suffix: true
                without_units: true
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, redaction]
      exporters: [debug]

2. Validate, start the Collector, send five traces with a fake e-mail and token, then inspect.^[25][88]

Example status: syntax-checked with bash -n; not executed (requires Docker).

#!/usr/bin/env bash
set -euo pipefail
REG=ghcr.io/open-telemetry
IMG=$REG/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.162.0
GEN=$REG/opentelemetry-collector-contrib/telemetrygen:v0.162.0
CFG=/etc/otelcol-contrib/config.yaml
docker run --rm -v "$PWD/lab2.yaml:$CFG:ro" "$IMG" validate --config=$CFG
docker run -d --name otelcol-lab --network host -v "$PWD/lab2.yaml:$CFG:ro" "$IMG"
sleep 5
docker run --rm --network host "$GEN" traces --otlp-insecure \
  --otlp-endpoint 127.0.0.1:4317 --traces 5 --service lab-checkout \
  --telemetry-attributes 'user.email="jane@example.com"' \
  --telemetry-attributes 'session_token="abc123"'
sleep 3
docker logs otelcol-lab 2>&1 | grep -E "user.email|session_token|redaction" | head
curl -s 127.0.0.1:8888/metrics | grep -E "otelcol_(receiver_accepted|exporter_sent)_spans"

Expected output and validation: validate exits 0. The debug log shows user.email and session_token masked with asterisks and a redaction.masked.count attribute; neither jane@example.com nor abc123 appears. The span counters on port 8888 are non-zero. Troubleshooting: “unknown type” errors mean an image older than the snake_case renames;^[34] on macOS or Windows, publish ports instead of using host networking. Image tags were checked in the registry on 2026-10-16. Teardown: docker rm -f otelcol-lab.

Objective: alert on the three ways a Collector loses data (failures after retries, a full queue, refused input) and on targets near their sample limit, and unit-test the rules. Prerequisites: promtool from Prometheus 3.x; 45 minutes. Status: not executed; expected results reasoned from the test-file documentation.^[35]

Example status: YAML syntax-checked; not executed (requires promtool).

# pipeline-alerts.yml
groups:
  - name: telemetry-pipeline
    rules:
      - alert: TelemetryExportFailing
        # "or ... * 0" keeps the ratio defined when sent_spans has never been incremented
        expr: |
          sum by (job, instance) (rate(otelcol_exporter_send_failed_spans[5m]))
            /
          (sum by (job, instance) (rate(otelcol_exporter_send_failed_spans[5m]))
            + (sum by (job, instance) (rate(otelcol_exporter_sent_spans[5m]))
               or sum by (job, instance) (rate(otelcol_exporter_send_failed_spans[5m])) * 0))
            > 0.01
        for: 10m
        labels: {severity: page}
        annotations:
          summary: "{{ $labels.job }}/{{ $labels.instance }}: over 1% of spans fail to export"
      - alert: TelemetryQueueFull
        expr: sum by (job, instance) (rate(otelcol_exporter_enqueue_failed_spans[5m])) > 0
        for: 5m
        labels: {severity: page}
        annotations:
          summary: "{{ $labels.job }}/{{ $labels.instance }}: sending queue full, spans dropped"
      - alert: TelemetryReceiverRefusing
        expr: sum by (job, instance) (rate(otelcol_receiver_refused_spans[5m])) > 0
        for: 15m
        labels: {severity: ticket}
        annotations:
          summary: "{{ $labels.job }}/{{ $labels.instance }}: refusing spans (back-pressure)"
      - alert: ScrapeNearSampleLimit
        expr: scrape_samples_post_metric_relabeling / (scrape_sample_limit > 0) > 0.8
        for: 15m
        labels: {severity: ticket}
        annotations:
          summary: "{{ $labels.job }}/{{ $labels.instance }}: over 80% of sample_limit"
      - alert: Watchdog
        expr: vector(1)
        labels: {severity: none}
        annotations:
          summary: "Always firing; page through another channel if it stops"

# pipeline-alerts.test.yml
rule_files: [pipeline-alerts.yml]
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'otelcol_exporter_sent_spans{job="otel-gw", instance="gw-1"}'
        values: '0+1000x30'
      - series: 'otelcol_exporter_send_failed_spans{job="otel-gw", instance="gw-1"}'
        values: '0+60x30'
      - series: 'otelcol_exporter_send_failed_spans{job="otel-gw", instance="gw-2"}'
        values: '0+100x30'   # back end down since gw-2 restarted: nothing ever sent
      - series: 'otelcol_exporter_enqueue_failed_spans{job="otel-gw", instance="gw-1"}'
        values: '0+50x30'
      - series: 'otelcol_receiver_refused_spans{job="otel-gw", instance="gw-1"}'
        values: '0+10x30'
      - series: 'scrape_samples_post_metric_relabeling{job="checkout", instance="app-7"}'
        values: '850x30'
      - series: 'scrape_sample_limit{job="checkout", instance="app-7"}'
        values: '1000x30'
    alert_rule_test:
      - eval_time: 5m
        alertname: TelemetryExportFailing
        exp_alerts: []
      - eval_time: 15m
        alertname: TelemetryExportFailing
        exp_alerts:
          - exp_labels: {severity: page, job: otel-gw, instance: gw-1}
            exp_annotations: {summary: "otel-gw/gw-1: over 1% of spans fail to export"}
          - exp_labels: {severity: page, job: otel-gw, instance: gw-2}
            exp_annotations: {summary: "otel-gw/gw-2: over 1% of spans fail to export"}
      - eval_time: 10m
        alertname: TelemetryQueueFull
        exp_alerts:
          - exp_labels: {severity: page, job: otel-gw, instance: gw-1}
            exp_annotations: {summary: "otel-gw/gw-1: sending queue full, spans dropped"}
      - eval_time: 20m
        alertname: TelemetryReceiverRefusing
        exp_alerts:
          - exp_labels: {severity: ticket, job: otel-gw, instance: gw-1}
            exp_annotations: {summary: "otel-gw/gw-1: refusing spans (back-pressure)"}
      - eval_time: 20m
        alertname: ScrapeNearSampleLimit
        exp_alerts:
          - exp_labels: {severity: ticket, job: checkout, instance: app-7}
            exp_annotations: {summary: "checkout/app-7: over 80% of sample_limit"}
      - eval_time: 1m
        alertname: Watchdog
        exp_alerts:
          - exp_labels: {severity: none}
            exp_annotations: {summary: "Always firing; page through another channel if it stops"}

1. Save both files; run promtool check rules pipeline-alerts.yml, then promtool test rules pipeline-alerts.test.yml.
2. Change the gw-1 failed-span series to '0+5x30' (0.5% failures) and rerun; the 15-minute case should now fail.

Expected output and validation: step 1 reports valid rules and passed tests. At 5 minutes the export alert is pending (for: 10m). At 15 minutes gw-1 fails 60 / 1,060 ≈ 5.7% of spans and gw-2, which has sent nothing since it restarted, 100%; without the or … * 0 term the gw-2 ratio would be empty and never page. Step 2 fails because 0.5% is under the threshold. Reading the counters: the Collector documentation warns that send failures “do not inherently imply data loss since there could be retries”;^[86] in the current exporter helper, retries run inside the step that counts results, so send_failed counts items that failed after their retries.^[89] Spans refused by a full queue never reach the retries and count as enqueue_failed.^[82] Troubleshooting: a label mismatch usually means different metric names or labels (the _total suffix depends on telemetry settings); check /metrics first.^[86] Teardown: none; commit the files. Route Watchdog to a dead man’s switch service, not to a person.

Case 1 (sourced): one engineer, global scale. Mastodon, about 20 people, runs mastodon.social (about 10 million requests per minute) with one engineer managing observability infrastructure: one Collector per Kubernetes namespace, the OpenTelemetry Operator and Argo CD, Datadog as the back end, and tail sampling that keeps roughly 0.1% of successful traces and every error trace. Tim Campbell, a Mastodon software engineer, said he had not run into a single Collector issue in about two years; the post itself names getting started as the hardest part.^[32] Lesson: one Collector tier is enough when a SaaS back end carries storage and sampling controls volume. Pitfall: Mastodon sets no strict CPU or memory limits on Collector pods, which suits its load but is no general default.

Case 2 (sourced): cardinality governance at scale. In March 2023 Cloudflare ran 916 Prometheus instances holding about 4.9 billion series. Because a breached sample_limit fails the whole scrape, Cloudflare patched Prometheus to cap series per server and keep existing series beyond the limit, alerting the owning team. The result is self-service: “if CI checks are passing then we have the capacity you need”.^[8] Lesson: limits plus CI make self-service safe. Pitfall: these are Cloudflare’s patches, not upstream behaviour.

Case 3 (sourced): observability as an internal service. The UK Ministry of Justice’s Observability Platform gives teams on its Modernisation Platform logs, metrics and alerting to PagerDuty or Slack on Amazon Managed Grafana; its team “maintains the infrastructure, allowing teams to focus on running their applications”. The offering is to be retired, date not announced.^[4] Lesson: a platform needs a product’s lifecycle, including retirement.

Measure the platform as a product: is the data delivered, used and affordable, and do alerts help? Targets are starting points, not industry benchmarks.

Report per team: a single platform-wide figure hides who generates the cost and noise.

Obligations differ by jurisdiction and sector: take legal advice, and coordinate with SecDataOps, PrivacyOps and ComplianceOps.

Drivers: active series (memory), samples per second, log bytes ingested and indexed, spans kept, query load, retention and cross-region transfer. Capacity: Prometheus disk ≈ retention seconds × samples per second × 1–2 bytes;^[44] for memory, Cloudflare divides go_memstats_alloc_bytes by prometheus_tsdb_head_series as a rough bytes-per-series figure.^[8] Levers, cheapest first: delete unused signals; fix unbounded labels; sample traces; tier and shorten retention; downsample; raise scrape intervals last, because cutting series is more effective.^[44] Allocation: tag telemetry by team or service.namespace (FinOps). Sustainability: less data stored means less compute and storage (GreenOps).

The observability system can fail silently, so it needs its own loops: an independent watcher (a small separate Prometheus or an external probe); the Watchdog heartbeat; a freshness canary; a weekly review of dropped data (…send_failed_*, …enqueue_failed_*, otelcol_receiver_refused_*, tail-sampling drops);^[86][41] and Git history plus CI results as the configuration audit trail. Early warnings: queues above 80% of capacity, rising scrape_series_added, series growing faster than traffic.

flowchart TB
  A["Alert: head series or memory rising fast"]
  B["Confirm: series growth by job"]
  C["Find: top metrics and labels in TSDB stats"]
  D{"Memory at risk?"}
  E["Contain: drop the offending metric for that job"]
  F["Fix at source: bounded label values, IDs as span attributes"]
  G["Recover: wait for head garbage collection"]
  H["Prevent: CI budget check, scrape limits, growth alert"]
  A --> B
  B --> C
  C --> D
  D -- "yes" --> E
  D -- "no" --> F
  E --> F
  F --> G
  G --> H
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class A,E risk
  class D decision
  class B,C,F,G,H core

1. Confirm. Which jobs are adding series?

sort_desc(sum by (job) (sum_over_time(scrape_series_added[1h])))

2. Find. Ask the TSDB which metrics and labels hold the most series.^[90]

Example status: syntax-checked with bash -n; not executed (requires a Prometheus server).

#!/usr/bin/env bash
set -euo pipefail
PROM="http://prometheus.example.internal:9090"
curl -s "$PROM/api/v1/status/tsdb?limit=5" \
  | jq '.data.headStats.numSeries, .data.seriesCountByMetricName, .data.labelValueCountByLabelName'

3. Contain if memory is at risk: drop the metric for that job with metric_relabel_configs (Artefact 2) and reload. Avoid labeldrop, because metrics must stay uniquely labelled, and sample_limit, which fails the whole scrape.^[49]
4. Fix at source. Replace customer_id with a bounded value (customer tier) and record the ID as a span attribute.
5. Recover. Series scraped even once stay in memory for one to three hours, until head garbage collection,^[8] so memory falls slowly; restart only if the server keeps crashing.
6. Prevent. Run the Lab 1 check in CI, set scrape limits, and alert on scrape_series_added.

Done when head series are back to the pre-release level and the inventory passes the budget check.

• “Collect everything; decide later.” Cost grows without value, and the CNCF whitepaper warns against it.^[12] Instead: name a consumer for every signal.
• “Three pillars means three tools.” The whitepaper prefers “primary signals” and notes that one or two signals can be a valid trade-off.^[12] Instead: choose signals per question and link them.
• “The observability team owns everyone’s alerts.” It becomes a bottleneck and works against DORA’s aim of making every developer proficient with monitoring.^[2] Instead: templates and reviews; service teams own their alerts.
• “Self-hosting is free.” You run stateful databases, upgrades and on-call yourself.
• “Tracing works once the SDK is installed.” Traces break wherever context is not propagated.
• “AI will find it.” Correlation needs clean, consistent data first (AIOps).

Invest in a deliberate ObservabilityOps function when several teams share telemetry tools; cost or noise is a recurring complaint; incidents are slowed by missing or untrusted data; telemetry may hold regulated data; or you self-host stateful back ends. Do not create a team yet when one or two teams use a managed tool with sensible defaults, volumes are small and nobody complains about cost or noise. One practitioner notes that a few people part-time “with a few high leverage SaaS investments” may be enough.^[23] Hidden responsibilities either way: on-call for the platform, upgrades, vendor management, access reviews and retention enforcement.

flowchart TB
  S("`How many teams share
  telemetry tools?`")
  H1("`Self-hosted back ends
  or regulated data?`")
  O1["`Named owner
  on call`"]
  N["`Part-time owner,
  vendor defaults`"]
  C("`Cost, noise or gaps
  a top problem?`")
  H("`Self-host stateful
  back ends?`")
  G["`Guild and
  standards`"]
  P["`Platform team
  with on-call`"]
  Q["`Small function:
  standards, budgets`"]
  S -- "one or two" --> H1
  S -- "three or more" --> C
  H1 -- "yes" --> O1
  H1 -- "no" --> N
  C -- "yes" --> H
  C -- "no" --> G
  H -- "yes" --> P
  H -- "no" --> Q
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class S,H1,C,H decision
  class O1,P,Q core
  class N,G neutral

For the hosting choice, weight the criteria for your situation. Scores are this page’s illustrative judgement (1 = weak, 5 = strong).

A hybrid scores highest under these weights because a pipeline you own keeps sampling, redaction and routing portable; a team of two might weight operating effort at 5 and get a different answer.

Not an official or industry model; Grafana Labs’ survey splits practices only into centralised and siloed.^[7]

Quick wins: delete unused dashboards and alerts, add the dead man’s switch, fix the worst label. Dependencies: a service and owner list, and someone with authority over the budget.

Quick start

• One named owner for the observability platform
• Inventory: tools, series, log volume, spend, pages per week
• Watchdog routed to a dead man’s switch, plus one external probe

Service onboarding (preflight)

• service.name and owner set; routes recorded as templates
• Expected series and log volume inside the team budget
• Standard RED dashboard; at least one symptom alert with a runbook
• No personal data or secrets in a sample of debug output

Pipeline change

• Configuration validated and alert rules unit-tested in CI
• Component names, stability levels and defaults checked against the pinned version’s release notes; rollback is a Git revert

Monthly review

• Platform SLIs, dropped data and queue use
• Budget report per team; expired exceptions removed
• Alert review with actionability labels

Security and privacy audit (quarterly)

• Gateways require mutual TLS; agent ports closed to outside the cluster; baggage keys match the allow-list
• Query access matches roles; searches of sensitive logs are audited
• Retention per data class enforced; redaction rules tested

Platform incident readiness

• Independent watcher alive; runbooks for back end down, queue full, cardinality explosion

Template status: illustrative; adapt the fields.

ONE-PAGE OBSERVABILITY PLATFORM CHARTER
Mission:        Every engineer can answer a new question about production within
                minutes, from trustworthy data, at an agreed cost.
Users:          <service teams, SRE, incident commanders, security (read-only)>
Services:       Ingest (OTLP, scrape, container logs), storage tiers, dashboards,
                alert routing, budget reports
Platform SLOs:  Export success >= <99.9%>/30 d; freshness p95 <= <60 s>;
                alert-path heartbeat 100%
Budgets:        Active series and log GB/day per team; exceptions expire after <90> d
Standards:      Propagators <tracecontext,baggage>; baggage keys on an allow-list;
                JSON logs with severity, trace_id, span_id
Not our job:    Other teams' SLO targets, their alerts' content, their on-call
Reviews:        Weekly health; monthly budget and alerts; quarterly pruning, standards

TELEMETRY CHANGE REQUEST
Team / service:              <name>             Owner on call: <name>
Signal:                      metric | log stream | trace attribute | new source
Question it answers:         <who uses it, in which dashboard or alert>
Labels and value counts:     <label: values, ...>  Expected series or GB/day: <n>
Personal or secret data?     no | yes -> masking rule: <rule>
Retention class:             hot <d> / warm <d> / archive <d>
Review date:                 <date>; retire if unused for 90 days
Decision:                    approved | redesign | exception until <date>
Approved by:                 <budget owner>

ALERT REVIEW (per page, monthly)
Alert | Pages | Actionable? (y/n) | Action taken | User-visible? (y/n)
      | Decision: keep / tune / ticket / delete | Owner

ARCHITECTURE DECISION RECORD: trace sampling
Context:      <traces per second, cost, current back end, regulatory constraints>
Decision:     Tail sampling at gateways: keep all errors and traces > <2 s>;
              <1>% of the rest
Consequences: Spans routed by trace ID to the samplers and by service to span
              metrics (one writer per series); agent-to-gateway trace traffic doubles;
              new span-metric series after each gateway restart (aggregate them);
              monitor dropped-too-early; rare successful paths missing from samples
Alternatives: Span metrics on the sampling replicas (series x replicas, keep
              collector.instance.id); head sampling in SDKs (cheaper, loses the
              error guarantee); no sampling (cost)
Review:       <date>

SERVICE OBSERVABILITY SCORECARD (0-2 each, 14 max)
Naming and ownership | RED dashboard        | Symptom alert with runbook
Unbroken traces      | Logs with trace IDs  | Budget adherence
No sensitive data in samples or baggage

Is ObservabilityOps a job title I should hire for? Rarely under that name: look for observability or platform engineers.

We use one SaaS vendor. Do we still need OpenTelemetry? It keeps instrumentation and pipeline policy portable, and many back ends accept OTLP.^[79]

Which signal should we start with? Metrics for RED and USE views and symptom alerts; then traces across services; logs for detail. The CNCF whitepaper suggests starting with one or a few signals.^[12]

Who pays for telemetry? The teams that generate it: showback first, chargeback only if you charge back other costs (FinOps).

How long should we keep data? As long as a named question or obligation needs it, per data class. Metrics downsample well; traces do not, so keep them only as long as investigations need them.

Treat “AI observability” and “observability pipeline” product claims as marketing until tested on your own data.

• Prerequisites: Linux and HTTP basics, one programming language, DevOps and SRE vocabulary.
• One day: read the CNCF whitepaper’s signals section^[12] and the SRE book’s monitoring chapter;^[9] run Lab 1.
• One week: instrument a sample service, run Lab 2, build a RED dashboard and one symptom alert.
• 30 days: run Lab 3 in CI, route traces to a sampling tier, and produce a budget report for real services.
• Specialise: time-series storage, tracing, log pipelines, or cost and governance.
• Portfolio project: a repository with Collector configuration, tested rules, a budget checker and a one-page charter.

Assumptions: about 2,000 traces per second, mostly healthy; engineers start investigations from traces and rarely read debug logs; one regulated service must keep its logs. Why A looks attractive: one configuration change gives the biggest saving, and OpenTelemetry suggests sampling at this volume.^[26] Why A alone can be wrong: dashboards that count stored spans undercount once sampling starts; the stateful sampling tier must be sized and monitored;^[41] and rare successful paths you may want to compare against disappear. Why B alone can be wrong: it may not reach 30%, and three-day traces break investigations of slow-burning problems. A defensible answer: A for high-volume services, with span metrics computed before sampling (Artefact 1); 100% for low-volume services; and the cheap half of B (archive debug logs, keep error logs hot). Track unit cost and “data missing” findings for a quarter; the open question is how often engineers need a successful trace, so check five incident reviews before going below 1%.

After this page you can explain the two readings of ObservabilityOps; design a pipeline with routing, limits, redaction and sampling; find where a trace breaks; estimate series and disk; test alerts for the platform; and choose an organisational model.

Behind the ObservabilityOps label is real work that every organisation with more than a few services ends up doing, well or badly: deciding what telemetry to produce, moving it safely, keeping it affordable, and making sure the alerts mean something. OpenTelemetry makes the pipeline portable; the decisions stay human: which questions matter, how many series a team may spend, which data is too sensitive to keep, which page should wake someone. Run the system that answers them like a product, with users, SLOs, a budget and the discipline to delete what nobody uses.

Each footnote gives its check date. Versions, statuses, defaults and survey figures are time-sensitive: re-check them against the release notes and component READMEs below at each Collector minor release and at least quarterly. The evidence ledger is research/observabilityops-ledger.md.

Further reading: The Site Reliability Workbook: Monitoring (Google, 2018); Observability Engineering (first edition, O’Reilly, 2022).

Changelog. 2026-10-16: first version (research, ledger, three artefacts, three labs of which Lab 1 was executed, six diagrams); term status confirmed as ambiguous; awaiting independent review. 2026-10-16 (r1 revision after independent review): span metrics routed by service (Artefact 1, figure 3, ADR); context propagation, baggage and log pipelines added; log-body redaction, Trace Context date, Prometheus OTLP history and Mastodon attribution corrected; loss alerts and least-privilege bindings added; four diagrams redrawn; trimmed. 2026-10-17, post-score polish: span-metric churn under service routing; server_name_override, agent exposure and hostPath offsets in Artefact 1; exact survey label and Mastodon attribution; re-check cadence.