--- title: "DataOps" slug: dataops category: data term_status: emerging evidence_status: verified summary: "DataOps is the practice of delivering and running data pipelines with software discipline: changes are versioned and tested in CI, each load is checked before consumers see it, and data is monitored after, so correct data arrives on time. The label is real but not standardised." audience: [Beginners, Data and analytics engineers, Data platform leads, Architects, Data and analytics leaders] last_verified: 2026-10-17 key_takeaways: - "No standard defines DataOps. The label dates from 2014–2015, a practitioner manifesto was online by 2017, and Gartner still called the tools market emerging in 2025. This page uses it for delivering and operating data pipelines." - "Treat every published dataset as a release: change it through version control and CI, audit each load before consumers see it, and monitor it after." - "Write-audit-publish is the central control: a bad load stays invisible instead of being rolled back in public." - "Contracts at producer boundaries prevent silent upstream changes; tests and statistical monitors catch the rest." - "Measure what consumers feel: freshness against an agreed time, correctness, and the share of incidents that consumers found before your monitors did." - "Baselines must respect seasonality: in this page's lab, 3-sigma limits over all days missed a 19% weekday drop that a weekday-only baseline caught." - "Access policy (DataSecOps), metric definitions (AnalyticsOps), engine internals (WorkflowOps) and model lifecycles (MLOps) are linked from here, not re-taught." related: [datasecops, analyticsops, mlops, workflowops, dbops, secdataops, researchdataops, devops] --- ## What DataOps is ### In 30 seconds **DataOps is the practice of delivering and running data pipelines with the discipline good teams apply to software.** Every change to a pipeline is versioned, tested in continuous integration (CI) and released deliberately. Every load of data is checked before consumers can see it, and monitored after. The aim: the data people act on arrives correct and on time, and when it does not, the data team knows before its consumers do. **The label is real but not standardised.** Lenny Liebmann named and described it in 2014, and Andy Palmer of Tamr defined it in 2015.[^liebmann][^palmer] A manifesto stewarded by the vendor DataKitchen lists 18 principles, plus a later "+1".[^manifesto] Gartner's October 2025 Market Guide defines DataOps and still calls the tools market "emerging".[^gartner-mg] An academic study found "no general accepted definition of DataOps yet",[^ereth] and in April 2026 English Wikipedia merged its DataOps article into its DevOps article.[^wiki-hist] This page uses the label for one coherent practice: data delivery run as a production line, with quality gates, statistical monitoring and producer-consumer contracts. It is for data and analytics engineers, platform leads, architects and the leaders who pay for trustworthy data. ### At a glance | Item | Summary | |---|---| | Goal | Correct, complete data on an agreed schedule; failures caught before consumers see them; safe pipeline changes | | Primary users | Data and analytics engineers, platform teams, data product owners, and the people and services consuming data | | Scope (this page) | Version control and CI/CD for pipelines, environments for data, data tests, data observability (freshness, volume, schema, distribution, lineage), orchestration as used, data contracts, write-audit-publish, data incidents | | Out of scope (linked) | Who may see which data ([DataSecOps](../datasecops/)); security telemetry ([SecDataOps](../secdataops/)); research data stewardship ([ResearchDataOps](../researchdataops/)); metric definitions ([AnalyticsOps](../analyticsops/)); engine internals ([WorkflowOps](../workflowops/)); databases as running systems ([DBOps](../dbops/)); models ([MLOps](../mlops/)) | | Prerequisites | SQL, Git and pull requests, what CI does; [DevOps](../devops/) helps | | Key outputs | Tested pipelines, Slim CI, contracts for critical inputs, monitors, an incident process, a weekly scorecard | | Term status | **Emerging.** In use since about 2015, with a manifesto and a tools market, but no standard definition or standards body | ### The problem DataOps solves Applications usually fail loudly; data pipelines usually fail quietly. A job reports success after loading half the rows. A producer renames a column or changes what a status code means. A late file makes yesterday's partition (the day's slice of a table) look complete. Nothing crashes: a dashboard shows a plausible wrong number and someone acts on it. Google's SRE Workbook warns that every stage of a pipeline can report that all is well while data is lost between stages.[^sre-wb] Without deliberate practice, the first monitor is a consumer. In 2019 an internal survey at Airbnb found that data scientists had trouble identifying which sources met their quality bar,[^airbnb] and at GoCardless engineers changed services without knowing that dropping a field could break dashboards downstream.[^gocardless] | Without DataOps | With DataOps | |---|---| | Pipeline code edited in production; no review | Every change goes through a pull request, CI and review | | A load that finished is assumed correct | Each load is audited before it is published | | Upstream teams change schemas without warning | Critical inputs have contracts with owners and versions | | Consumers report errors; fixes overwrite history | Monitors alarm first; loads are replayable; incidents name affected tables and periods | ### What DataOps is not: naming and boundaries - **Not "DevOps for databases".** Running database engines is [DBOps](../dbops/). - **Not a product.** Gartner reports DataOps features moving into larger data management platforms, so stand-alone DataOps solutions "risk being viewed as redundant".[^gartner-mg] A tool does not create the practice. - **Not governance or security** ([DataSecOps](../datasecops/), [PrivacyOps](../privacyops/)). - **Not only the Manifesto's analytics-team process** ("Working analytics over comprehensive documentation").[^manifesto] Centring this page on the pipeline, and the split of the data family below, are an **editorial distinction** of this Atlas, not a standard. | Page | Object it manages | Question it answers | |---|---|---| | DataOps (this page) | The data pipeline | Is the data arriving correct and on time? | | [DataSecOps](../datasecops/) | Business data as a protected asset | Who may see which data, and can we prove it? | | [SecDataOps](../secdataops/) | Security telemetry | Do defenders have complete, affordable evidence? | | [ResearchDataOps](../researchdataops/) | Research data over decades | Can others find, cite and reuse it? | | [AnalyticsOps](../analyticsops/) | Metrics and dashboards | Is there one trusted definition per metric? | ## Where DataOps comes from and where it sits ### Origins and evolution of the label | When | Milestone | |---|---| | June 2014 | Liebmann (IBM blog): best practices that improve coordination between data science and operations[^liebmann] | | May 2015 | Palmer: "DataOps is a data management method that emphasizes communication, collaboration, integration, automation and measurement of cooperation between data engineers, data scientists and other data professionals."[^palmer] | | By August 2017 | The DataOps Manifesto is online with 18 principles (earliest Internet Archive capture)[^manifesto-2017] | | 2018 | Ereth's review and expert interviews find no accepted definition[^ereth] | | 2021 | A research review calls DataOps "an emerging approach advocated by practitioners"[^mainali] | | October 2025 | Gartner Market Guide: five capability sets, features converging into platforms[^gartner-mg] | | 2026 | Wikipedia redirects "DataOps" to its DevOps article (April);[^wiki-hist] dbt v2.0 and ODCS v3.2.0 ship (September)[^dbt-versions][^odcs] | The practices are older than the name: the Manifesto cites Agile, DevOps, Deming and lean manufacturing and asks for monitoring that detects "unexpected variation",[^manifesto] the job of the manufacturing control chart.[^nist-spc] Two readings coexist: an **analytics-team reading** (the Manifesto, Palmer) about collaborating to deliver insight, and a **pipeline-operations reading** (Gartner's capability sets) about orchestrating, testing, deploying and observing data flows.[^gartner-mg] Vendor definitions blend them; IBM's stresses collaboration, speed of delivery and quality.[^ibm] This page teaches the pipeline reading. ### Glossary and mental model **Mental model: two loops** (this page's teaching device). In the **change loop** someone edits pipeline code, and CI and review protect it, as in [DevOps](../devops/). In the **run loop** the code stays the same but the data changes every hour or day: fewer rows, a new category value, a field whose meaning shifted. Data can break with no deploy at all, so DataOps checks every load and monitors every published table. | Term | Meaning | Example | |---|---|---| | Partition; backfill | A slice of a table, usually one day, loaded and replaced as a unit; a backfill reruns past partitions | Rebuild 1–15 October after a fix | | CDC (change data capture) | Copying each insert, update and delete from a source database's log | Checkout's order changes stream to the raw zone | | SLA, SLO, error budget | SLA: a promise to consumers; SLO: the internal target; error budget: the misses an SLO allows ([SRE](../sre/)) | SLO 07:00 on 95% of business days: 3 late days a quarter | | dbt model; materialisation | A file holding one `select`; how dbt persists it (table, view, incremental…)[^dbt-models][^dbt-mat] | `fct_orders.sql` built as a table | | Manifest, state, defer | dbt's `manifest.json` describes a run; `--state` points at a saved one; `--defer` reads unbuilt parents from it[^dbt-defer] | `--state prod-artifacts --defer` | | Dag | Airflow's workflow: tasks and the order they run in | `orders_daily`: write → audit → publish | | Data test | An assertion that returns failing rows; zero rows means pass[^dbt-tests] | No duplicate `order_id` today | | Unit test (for data) | Checks transformation logic on small fixed inputs[^dbt-unit] | A refund yields a negative amount | | Data contract | Bitol: "an agreement between a data producer and its consumers"; schema, quality rules, service levels, owners[^bitol] | Checkout guarantees unique `order_id`, loaded by 06:00 | | Freshness (latency) | Age of the newest data | Newest order is 1.2 hours old | | Distribution drift | Values change while the schema does not | Share of `returned` jumps from 3% to 21% | | Lineage | Which jobs read and write which datasets[^openlineage] | `fct_orders` feeds three dashboards and a model | | Write-audit-publish (WAP) | Write to a hidden place, check, then expose[^aws-wap] | Iceberg audit branch fast-forwarded to `main`[^iceberg] | | Idempotent load | A rerun for a period gives the same result, never duplicates[^af-best] | Replace the 2026-10-15 partition | | Slim CI | Build and test only changed models and their children, reusing production for the rest[^dbt-state] | `dbt build --select state:modified+ --defer --state prod-artifacts` | | Data downtime | Vendor term: "periods of time when data is partial, erroneous, missing, or otherwise inaccurate"[^mc-pillars] | 6 hours on 15 October | ### How DataOps relates to other Ops ::: figure Where DataOps sits among neighbouring Atlas pages ```mermaid flowchart TB DEV["DevOps: CI/CD habits"]:::neutral WF["WorkflowOps: engines"]:::neutral DB["DBOps: source databases"]:::neutral D["DataOps: correct, on time"]:::core AN["AnalyticsOps: metrics on tested tables"]:::neutral ML["MLOps: training data onward"]:::neutral DS["DataSecOps: access policy, using lineage"]:::neutral SD["SecDataOps: same tools, security data"]:::neutral RD["ResearchDataOps: after publication"]:::neutral DEV -->|"practices"| D WF -->|"retries"| D DB -->|"CDC, schemas"| D D --> AN D --> ML D --> DS AN ~~~ SD ML ~~~ RD D -.-> SD D -.-> RD classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: DevOps supplies CI/CD habits, WorkflowOps engine behaviour (retries, backfills) and DBOps the source databases and their change data capture (CDC) feeds. DataOps hands tested tables to AnalyticsOps, training data to MLOps, and lineage and owners to DataSecOps, which sets access policy. Dotted links, drawn one row lower, are looser: SecDataOps uses the same tools for security data; ResearchDataOps takes over research data at publication. ::: | Neighbour | What the neighbour owns | Handoff | |---|---|---| | [DevOps](../devops/) | General delivery practice, DORA metrics | Applied to pipelines, plus data checks | | [WorkflowOps](../workflowops/) | Engine semantics: retries, durable execution, backfill mechanics | DataOps uses Airflow or Dagster; engine tuning goes there | | [DBOps](../dbops/) | Engine operations, migrations, backups | Producer schema changes arrive through contracts | | [AnalyticsOps](../analyticsops/) | Semantic layer, metric definitions, dashboard certification | DataOps guarantees the tables beneath | | [MLOps](../mlops/) | Features, training, serving, skew and drift | Upstream training-data quality stays here | | [DataSecOps](../datasecops/) | Classification, masking, row and column policy | DataOps keeps personal data out of test outputs | | [ObservabilityOps](../observabilityops/) | Application telemetry pipelines | Data observability stays here | | [IncidentOps](../incidentops/) | Severity, roles, communication | DataOps adds data diagnosis and backfill | ## Principles and operating model ### Charter and principles The Manifesto addresses analytics teams.[^manifesto] This charter is **this Atlas's proposal** for pipeline-centred DataOps. **Mission:** data that people rely on is correct, complete and on time, and every change to it is deliberate and reversible. 1. **Every published dataset is a release:** reviewed code, audited data. 2. **Consumers never see unaudited data for critical assets** (write, audit, then publish). 3. **Checks fail closed where wrong data is expensive, and warn elsewhere.** Severity is a decision, not a default. 4. **Producers own the shape of what they emit; consumers own their expectations.** Contracts capture the first, tests the second. 5. **Every load is idempotent and replayable by period.** 6. **Watch distributions, not only rules,** to find failures nobody wrote a rule for. 7. **Measure what consumers feel:** freshness, correctness, and who found the problem first. 8. **Environments for data are cheap and disposable** (the Manifesto's principle 12). **Non-goals:** perfect data; a test on every column; one central team owning every pipeline; replacing governance. ### Fundamentals: the building blocks Six building blocks recur in every implementation: **version control and CI/CD**; **environments for data** (CI schemas, table or lake branches); **tests** of logic, of each load and at boundaries; **orchestration** on time or on data; **observability and lineage**; and **incident and change management** (owners, severity, backfills, deprecation). The [practices](#core-practices-and-playbooks) and [tooling](#tooling-landscape) tables below fill each one in. ### Roles, ownership and RACI Airbnb gives every certified data model an engineering owner and a data science owner, and invites consuming teams to review designs.[^airbnb] The RACI below generalises that (R responsible, A accountable, C consulted, I informed). As charter principle 4 says, the producer owns the **input** contract it emits; the data product owner owns the **published** table's contract. | Activity | Producer team | Pipeline owner | Platform team | Data product owner | Consumers | |---|---|---|---|---|---| | Define, change or deprecate an input contract | A/R | C | I | C | I | | Define, change or deprecate the published-table contract | I | R | C | A | C | | Maintain tests on a published table | I | R | C | A | C | | Provide CI, environments, orchestrator, monitors | I | C | A/R | I | I | | Respond to a tier-1 data incident | C | R | C | A | I | | Run a backfill after a fix | I | R | C | A | I | Skills: SQL, Git, an orchestrator, a test framework, basic statistics and incident handling; platform teams add infrastructure as code and cost awareness ([FinOps](../finops/)). ### Lifecycle: Day 0, Day 1, Day 2 - **Day 0, design:** tier, owners, consumers and deadline; contracts for critical inputs; a design spec sized to the tier (Airbnb's specs cover metrics, pipeline, service levels, backfill, schemas and checks; some complex ones exceed 20 pages).[^airbnb] - **Day 1, release:** build in CI, backfill history, run old and new side by side for critical assets, publish behind write-audit-publish, announce the service level. - **Day 2, operate:** monitor every load, handle incidents, review contract changes, tune thresholds, retire unused assets. ### Operating model and governance Tiers keep cost proportional to harm: | Tier | Example | Required controls | |---|---|---| | 1, critical | Finance close, regulatory returns, training sets, customer-facing data | Input contracts, write-audit-publish with blocking checks, freshness SLO, on-call, incident and change review | | 2, important | Team dashboards, weekly reporting | Key tests, freshness and volume monitors, owner, visible warnings | | 3, exploratory | Sandboxes, one-off analysis | Version control; no service level; labelled unsupported | Ceremonies: a **weekly reliability review** of the [scorecard](#metrics-and-kpis), a **contract change review**, and a **blameless review** of each tier-1 incident ([IncidentOps](../incidentops/)). ## Architecture and workflow ### Reference architecture: data plane and control plane The **data plane** moves data; the **control plane** decides what runs, checks the output and alerts people. Many failures are a missing link between them, such as checks that cannot stop a publish. ::: figure Reference architecture: a data plane governed by a control plane ```mermaid flowchart TB S["Sources: apps, SaaS, files"]:::neutral I["Ingest: CDC, batch, stream"]:::neutral R["Raw zone, append-only"]:::neutral T["Transform: SQL models"]:::core A["Audit area, hidden"]:::core P["Published tables"]:::core C["Consumers: BI, ML, services"]:::neutral G["Git and CI"]:::decision O["Orchestrator"]:::decision K["Contracts"]:::decision M["Monitors"]:::decision L["Lineage, metadata"]:::decision N["On-call, incidents"]:::risk S --> I --> R --> T --> A -->|"checks pass"| P --> C G -.->|"deploys"| O O -.->|"runs"| T K -.->|"rules"| A M -.->|"watches"| P M -.->|"alerts"| N L -.->|"impact"| N classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: the data plane runs top to bottom: sources, ingestion, an append-only raw zone, SQL transformations, a hidden audit area and, only after checks pass, published tables that consumers read. Control-plane components (amber, dotted links): Git and CI deploy to the orchestrator, which runs transformations; contracts supply audit rules; monitors watch published tables and alert the on-call owner (red); lineage shows which assets an incident affects. ::: - **Keep raw data immutable** so any period can be replayed after a fix. - **Separate audit from publish:** a staging schema plus an atomic swap, an Apache Iceberg audit branch fast-forwarded to `main` after validation,[^iceberg] or a lakeFS branch merged after checks.[^lakefs] - **Trigger on data, not only on time.** Airflow marks an asset as updated "only if the task completes successfully", so a failed publish does not trigger consumers.[^af-assets] - **Emit lineage in a standard form.** OpenLineage models jobs, runs and datasets, extended by "facets".[^openlineage] ### End-to-end workflow: from pull request to published table ::: figure The change loop: from pull request to published data, with the exception paths ```mermaid flowchart TB PR["Pull request opened"]:::neutral CI["CI builds changed models and their children in a PR schema"]:::core T{"Unit, data and contract checks pass?"}:::decision BR{"Breaking change to a published contract?"}:::decision FIX["Author fixes; CI reruns"]:::risk VER["New contract version, deprecation window, consumer review"]:::risk REV["Owner review and merge"]:::neutral DEP["Deploy to the orchestrator"]:::core RUN["Next run: write, audit, publish"]:::core PR --> CI --> T T -->|"no"| FIX --> CI T -->|"yes"| BR BR -->|"yes"| VER --> REV BR -->|"no"| REV REV --> DEP --> RUN classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: a pull request triggers CI, which builds only the changed models and their children in a pull-request schema. Failed checks return the change to the author. If checks pass but the change breaks a published contract, a new contract version with a deprecation window goes to consumers for review. The owner merges, the change is deployed, and the next run writes, audits and publishes. ::: Plan for the exception paths: 1. **CI is red because data changed, not code.** Keep logic checks (unit tests on fixtures, which dbt Labs recommends running only in development and CI)[^dbt-unit] separate from checks on today's data. 2. **A breaking change is unavoidable.** Publish the new shape beside the old with a dated window. GoCardless made changing a contract "intentionally heavy".[^gocardless] 3. **Tonight's run fails its audit.** Consumers keep yesterday's data; see the [runbook](#worked-runbook-the-revenue-dashboard-dropped-overnight). ### Write-audit-publish, step by step ::: figure Write-audit-publish for one daily load ```mermaid sequenceDiagram participant Q as Checks participant O as Orchestrator participant A as Audit area participant P as Published O->>A: write partition (hidden) O->>Q: run checks on the partition alt blocking checks pass Q-->>O: pass O->>A: publish A->>P: atomic swap or fast-forward Note over O: asset marked updated else a blocking check fails Q-->>O: fail + failing rows O->>A: keep as quarantine Note over A,P: last good data stays, owner paged end ``` Text equivalent: the orchestrator's load job writes one day's partition into a hidden audit area, then asks the checks to run on it. If every blocking check passes, the orchestrator publishes: the audit area replaces the published data in one atomic step (a table swap or a branch fast-forward), and the asset is marked updated so consumers can run. If one fails, the checks return the failing rows, the audit area is kept as a quarantine, the last good data stays published and the owner is paged. ::: A subtlety: `dbt build` skips **downstream** models when a test fails,[^dbt-build] but the tested model has already been rebuilt in its target schema, so anyone querying it directly sees the bad version (an inference from the documented order of operations). If consumers query the model itself, build into an audit schema and publish separately, as in [Lab 2](#hands-on-labs). ## Practices, frameworks and tools ### Core practices and playbooks | Practice | Goal | Trigger | Owner | Verification | |---|---|---|---|---| | Slim CI | Catch logic and contract breaks before merge | Pull request | Pipeline owner | Escaped defects fall | | Unit tests | Prove tricky logic[^dbt-unit] | New logic, past bugs | Pipeline owner | Fail when logic is broken on purpose | | Data tests | Keys, nulls, accepted values, relationships | Every load | Pipeline owner | Thresholds documented | | Contracts | Stop silent upstream changes | New producer, breaking change | Producer | Checked in producer CI | | Write-audit-publish | No unaudited tier-1 data | Every tier-1 load | Pipeline owner | Zero unaudited publishes | | Monitors | Late, partial or shifted data | Every load, schedule | Platform team | Coverage, precision | | Idempotent loads | Safe reruns and backfills | Retries, fixes | Pipeline owner | Reruns reproduce counts | | Incident handling | Contain, fix, inform | Alert, consumer report | On-call owner | Detect and restore times fall | | Late-data handling | Late rows land in their own period | Every incremental run | Pipeline owner | Restated totals reconcile with source | **Late and out-of-order data.** A load can be on time while its data is not. Reconcile each partition with the source's own count; reprocess a lookback window (dbt's incremental example re-reads the last three days for late-arriving records[^dbt-micro]); and tell consumers when a period is final. Stream processors use event-time watermarks, a declaration that events up to a timestamp should have arrived; later events need an explicit policy.[^flink-time] ### Methods, standards and frameworks None is mandatory in general; regulation may make parts so ([compliance](#quality-security-privacy-and-compliance)). | Item | Kind | What it gives DataOps | |---|---|---| | DataOps Manifesto | Vendor-stewarded manifesto | 18 principles plus a later "+1", including reproducibility and monitoring for unexpected variation[^manifesto] | | Open Data Contract Standard v3.2.0 | Open standard (Bitol, Linux Foundation AI & Data) | YAML for schema, quality rules, service levels, roles[^odcs][^bitol] | | OpenLineage | Open standard | Run, job and dataset events; Marquez is the reference implementation[^openlineage] | | ISO/IEC 25012 | International standard | 15 data quality characteristics, such as accuracy and completeness[^iso25012] | | SRE guidance for pipelines | Engineering practice | Freshness and correctness SLOs; end-to-end measurement[^sre-wb] | | Statistical process control | Statistical method | Control charts, usually with 3-sigma limits[^nist-spc] | | Data mesh | Organisational approach | Domain ownership, data as a product, self-serve platform, federated governance[^mesh] | ### Tooling landscape Gartner groups DataOps tools into generalists, orchestrators and specialists, across five capability sets: pipeline orchestration, observability, test automation, deployment automation and environment management.[^gartner-mg] Choose by capability first: | Capability | Examples checked for this page (October 2026) | Watch out for | |---|---|---| | Transformation, tests, contracts | dbt v1.12 and v2.0: data tests, unit tests, model contracts, freshness[^dbt-versions][^dbt-contracts] | Contracts cover only some materialisations; tests cost warehouse compute | | Validation frameworks | Great Expectations (GX Core 1.24), Soda Core v4 contracts, Deequ on Spark[^gx][^soda][^deequ] | Another runtime to operate; Python version floors | | Orchestration | Airflow 3.3 assets and Deadline Alerts; Dagster asset checks and freshness policies[^af-deadline][^dg-checks][^dg-fresh] | Airflow Deadline Alerts are experimental (since 3.1) and Dagster freshness policies in preview; engine semantics belong to [WorkflowOps](../workflowops/) | | Environments, versioned data | Iceberg branches and tags; lakeFS branches[^iceberg][^lakefs] | Branch retention and clean-up | | Observability and lineage | OpenLineage with Marquez; commercial data observability platforms[^openlineage][^mc-pillars] | Heavy overlap with platforms' built-in features[^gartner-mg] | | Streaming contracts | Schema Registry compatibility rules (Confluent default: BACKWARD);[^confluent] Confluent data contracts add metadata, plus CEL rules on paid tiers (7.4+)[^confluent-dc] | Compatibility checks are structural, not semantic | Ask: can it **block** a publish? Where does it store failing rows? What does each check cost? Can it emit OpenLineage? How hard is it to leave? Gartner advises checking whether your main platform's embedded features are "good enough" first.[^gartner-mg] ### Automation and AI The useful AI here is mostly statistics. Uber's Data Quality Monitor bundles table metrics into principal components ("the top-ranked PC series explain more than 90 percent of the variation") and flags deviations from one-step-ahead forecasts.[^uber] Deequ uses machine learning to enhance constraint suggestions and to detect anomalies in quality time series,[^deequ] and Google's data validation in TFX checks data fed to ML pipelines against a schema[^tfdv] (its training/serving skew checks belong to [MLOps](../mlops/)). Generative assistants now draft tests (dbt documents one).[^dbt-tests] A generated test describes the data as it **is**, not as it **should be**, so someone who knows the business must review it. ## DataOps in practice ### Reference implementation **Assumptions:** a warehouse supported by dbt, Git with CI, Airflow 3.x, and one tier-1 asset, `fct_orders`, fed by the checkout team's `orders` table (versions as of October 2026: dbt Core 1.12, Airflow 3.3, ODCS v3.2.0). ```text data-platform/ contracts/orders.odcs.yaml # producer-owned contract (ODCS v3.2.0) models/staging/_sources.yml # sources, freshness, source tests models/staging/stg_orders.sql models/marts/_marts.yml # contract, data tests, unit tests models/marts/fct_orders.sql dags/orders_daily.py # write -> audit -> publish .github/workflows/dbt-ci.yml # Slim CI on pull requests ``` **1. The contract,** owned by the producer and reviewed by consumers. Metrics such as `nullValues`, SLA properties such as `latency` (which ODCS prefers to "freshness") and the `{object}` and `{property}` placeholders in SQL rules come from the standard.[^odcs] _Example status: validated on 2026-10-17 against the ODCS v3.2.0 JSON Schema with Ajv 8.17.1 in Draft 2019-09 mode (rejects unknown keys); executed in Lab 2._ ```yaml apiVersion: v3.2.0 kind: DataContract id: 7f1c2a4e-0b8d-4d55-9a51-2f3c8e0d9b11 name: orders version: 1.2.0 status: active domain: checkout description: purpose: One row per customer order, loaded daily for finance and analytics. limitations: Amounts are in the order currency; no FX conversion. usage: Join to customers on customer_id. Do not use for real-time stock decisions. schema: - name: orders physicalName: orders logicalType: object physicalType: table quality: - id: orders_daily_volume metric: rowCount mustBeGreaterThan: 500 dimension: completeness description: A normal day has 900 to 1,100 orders; fewer than 500 means a broken load. properties: - name: order_id logicalType: string physicalType: varchar(36) primaryKey: true primaryKeyPosition: 1 required: true quality: - id: order_id_unique metric: duplicateValues mustBe: 0 dimension: uniqueness - name: customer_id logicalType: string physicalType: varchar(36) required: true quality: - id: customer_id_not_null metric: nullValues mustBe: 0 dimension: completeness - name: status logicalType: string physicalType: varchar(16) required: true quality: - id: status_accepted metric: invalidValues arguments: validValues: [placed, shipped, completed, returned] mustBe: 0 dimension: conformity - name: amount logicalType: number physicalType: decimal(12,2) required: true quality: - id: amount_not_negative type: sql query: SELECT COUNT(*) FROM {object} WHERE {property} < 0 mustBe: 0 dimension: accuracy - name: loaded_at logicalType: timestamp physicalType: timestamp required: true team: name: checkout-platform members: - username: owner@example.com role: owner slaProperties: - id: orders_latency property: latency value: 6 unit: h element: orders.loaded_at - id: orders_frequency property: frequency value: 1 unit: d element: orders.loaded_at - id: orders_time_to_detect property: timeToDetect value: 30 unit: min ``` **2. dbt sources and marts:** freshness on the input; an enforced contract, data tests with explicit severities and a unit test on the output.[^dbt-fresh][^dbt-contracts][^dbt-severity] The `arguments:` key needs dbt 1.10.5 or later.[^dbt-tests] _Example status: YAML syntax-checked; written for dbt Core 1.12 against the documentation; not executed here (see Lab 3)._ ```yaml # models/staging/_sources.yml sources: - name: checkout schema: raw_checkout config: freshness: warn_after: {count: 6, period: hour} error_after: {count: 12, period: hour} loaded_at_field: loaded_at tables: - name: orders columns: - name: order_id data_tests: [unique, not_null] --- # models/marts/_marts.yml models: - name: fct_orders config: contract: {enforced: true} columns: - name: order_id data_type: varchar constraints: [{type: not_null}] data_tests: [unique] - name: customer_id data_type: varchar data_tests: - relationships: arguments: {to: ref('dim_customers'), field: customer_id} config: {severity: warn} - name: status data_type: varchar data_tests: - accepted_values: arguments: {values: [placed, shipped, completed, returned]} config: {severity: error, error_if: ">10", warn_if: ">0"} - name: net_amount data_type: decimal(12,2) unit_tests: - name: returned_orders_have_negative_net_amount model: fct_orders given: - input: ref('stg_orders') rows: - {order_id: o1, customer_id: c1, status: returned, amount: 40.00} - {order_id: o2, customer_id: c1, status: completed, amount: 15.50} expect: rows: - {order_id: o1, net_amount: -40.00} - {order_id: o2, net_amount: 15.50} ``` **3. The orchestration:** write, audit and publish as separate tasks; an asset so consumer Dags run only after a successful publish;[^af-assets] and a Deadline Alert (SLAs were removed in Airflow 3[^af-upgrade]) that pages the owner if the run has not finished an hour after it was scheduled to start. Deadline Alerts "are new in Airflow 3.1 and should be considered experimental" (as of October 2026),[^af-deadline] so tier 1 also needs a monitor outside Airflow that pages if `fct_orders` has no new partition by 06:00; it also catches a stalled scheduler. Credentials stay out of the Dag: `profiles.yml` reads them with dbt's `env_var()` (values of `DBT_ENV_SECRET` variables are scrubbed from logs),[^dbt-env] set by the worker's deployment or secrets backend, and `BashOperator` passes the worker's environment on by default.[^af-bash] _Example status: Python syntax-checked; imports and parameters taken from the Airflow 3.3 documentation; not executed (requires an Airflow deployment and a warehouse)._ ```python """orders_daily.py: write -> audit -> publish for fct_orders (Airflow 3.x).""" from datetime import datetime, timedelta from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier from airflow.providers.standard.operators.bash import BashOperator from airflow.sdk import DAG, Asset, AsyncCallback, DeadlineAlert, DeadlineReference FCT_ORDERS = Asset("warehouse://analytics/fct_orders") DBT = "cd /opt/data-platform && dbt" with DAG( dag_id="orders_daily", schedule="0 5 * * *", start_date=datetime(2026, 10, 1), catchup=False, deadline=DeadlineAlert( reference=DeadlineReference.DAGRUN_LOGICAL_DATE, interval=timedelta(hours=1), callback=AsyncCallback( SlackWebhookNotifier, kwargs={"text": "{{ dag_run.dag_id }} missed its deadline " "({{ deadline.deadline_time }})"}, ), ), ): # No credentials here: profiles.yml uses env_var("DBT_ENV_SECRET_..."); BashOperator # (env=None) inherits the worker environment set by the deployment or secrets backend. write = BashOperator( task_id="write_to_audit_schema", bash_command=f"{DBT} source freshness --select source:checkout " f"&& {DBT} build --select +fct_orders --exclude test_type:unit --target audit", ) audit = BashOperator( task_id="audit_contract", bash_command="python /opt/data-platform/ci/verify_contract.py " "contracts/orders.odcs.yaml --target audit --date {{ ds }}", ) publish = BashOperator( task_id="publish", bash_command="python /opt/data-platform/ci/publish_partition.py fct_orders --date {{ ds }}", outlets=[FCT_ORDERS], ) write >> audit >> publish ``` `verify_contract.py` and `publish_partition.py` are your own small scripts (Lab 2 shows the logic); the second replaces the day's partition in one transaction, so a retry never duplicates rows, as Airflow advises for tasks.[^af-best] ### Hands-on labs Labs 1 and 2 need only Python and were executed for this page; Lab 3 needs Python 3.10 or later and was not executed. **Lab 1 (beginner): checks and a control chart on one table.** Objective: write the checks DataOps runs on every load and see why a baseline must respect seasonality. Prerequisites: Python 3.8+, no packages; about 20 minutes; local files only, no cost. _Example status: executed on Python 3.8.10 with SQLite 3.31.1 on 2026-10-16 and re-run on 2026-10-17; the output below is verbatim._ 1. Save the generator: 28 daily loads (about 1,000 orders on weekdays, 820 at weekends); the last load has five defects. ```python """make_orders.py: build shop.db with 28 daily loads of an orders table; day 28 carries five defects.""" import random import sqlite3 from datetime import datetime, timedelta random.seed(42) db = sqlite3.connect("shop.db") db.executescript(""" DROP TABLE IF EXISTS customers; DROP TABLE IF EXISTS orders; CREATE TABLE customers (customer_id TEXT PRIMARY KEY); CREATE TABLE orders (order_id TEXT, customer_id TEXT, status TEXT, amount REAL, load_date TEXT, loaded_at TEXT); """) db.executemany("INSERT INTO customers VALUES (?)", [(f"c{n:04d}",) for n in range(2000)]) start = datetime(2026, 9, 18, 5, 0) statuses = ["placed", "shipped", "completed", "returned"] seq = 0 for day in range(28): load = start + timedelta(days=day) weekend = load.weekday() >= 5 n = int(random.gauss(820 if weekend else 1000, 30)) if day == 27: n = 610 # defect 1: a partial load rows = [] for _ in range(n): seq += 1 rows.append([f"o{seq:06d}", f"c{random.randrange(2000):04d}", random.choice(statuses), round(random.uniform(5, 400), 2), load.date().isoformat(), (load + timedelta(minutes=random.randrange(50))).isoformat(" ")]) if day == 27: rows[0][1] = None # defect 2: missing customer_id rows[1][1] = None rows[2][2] = "refunded_partial" # defect 3: a status nobody agreed on rows[3][1] = "c9999" # defect 4: orphan customer rows.append(list(rows[4])) # defect 5: a duplicated order db.executemany("INSERT INTO orders VALUES (?,?,?,?,?,?)", rows) db.commit() print(db.execute("SELECT COUNT(*), MIN(load_date), MAX(load_date) FROM orders").fetchone()) ``` 2. Save the checker. Errors block publication. The volume checks compare today's count with the mean ± 3 standard deviations of history,[^nist-spc] over all days and over weekdays only. ```python """check_orders.py: run DataOps checks on the latest load of shop.db; exit 1 if an error-level check fails.""" import sqlite3 import statistics import sys from datetime import datetime NOW = datetime(2026, 10, 15, 7, 0) # pin "now" so the lab is repeatable db = sqlite3.connect("shop.db") latest = db.execute("SELECT MAX(load_date) FROM orders").fetchone()[0] day = f"load_date = '{latest}'" # internal constant, not user input def scalar(sql): return db.execute(sql).fetchone()[0] checks = [] # (name, severity, failures, detail) age_h = (NOW - datetime.fromisoformat(scalar("SELECT MAX(loaded_at) FROM orders"))).total_seconds() / 3600 checks.append(("freshness <= 6 h", "error", int(age_h > 6), f"newest row is {age_h:.1f} h old")) counts = [r[0] for r in db.execute( "SELECT COUNT(*) FROM orders GROUP BY load_date ORDER BY load_date")] history, today = counts[:-1], counts[-1] mean, sd = statistics.mean(history), statistics.stdev(history) low, high = mean - 3 * sd, mean + 3 * sd checks.append(("volume within mean +/- 3 sd", "error", int(not low <= today <= high), f"today {today}, limits {low:.0f}..{high:.0f} from {len(history)} days")) # Same check against weekdays only: seasonality makes the all-days limits too wide. wk = [c for (d, c) in db.execute("SELECT load_date, COUNT(*) FROM orders GROUP BY load_date") if d != latest and datetime.fromisoformat(d).weekday() < 5] wl, wh = statistics.mean(wk) - 3 * statistics.stdev(wk), statistics.mean(wk) + 3 * statistics.stdev(wk) checks.append(("volume vs weekdays only", "error", int(not wl <= today <= wh), f"limits {wl:.0f}..{wh:.0f} from {len(wk)} weekdays")) checks.append(("customer_id not null", "error", scalar(f"SELECT COUNT(*) FROM orders WHERE {day} AND customer_id IS NULL"), "rows")) checks.append(("order_id unique", "error", scalar( f"SELECT COUNT(*) FROM (SELECT order_id FROM orders WHERE {day} " "GROUP BY order_id HAVING COUNT(*) > 1)"), "duplicated ids")) checks.append(("status in accepted set", "warn", scalar( f"SELECT COUNT(*) FROM orders WHERE {day} AND status NOT IN " "('placed','shipped','completed','returned')"), "rows")) checks.append(("customer_id exists in customers", "warn", scalar( f"SELECT COUNT(*) FROM orders o LEFT JOIN customers c USING (customer_id) " f"WHERE o.{day} AND o.customer_id IS NOT NULL AND c.customer_id IS NULL"), "orphans")) print(f"checks for load_date {latest}") worst = 0 for name, sev, fails, detail in checks: status = "PASS" if fails == 0 else sev.upper() worst = max(worst, 2 if status == "ERROR" else 1 if status == "WARN" else 0) print(f" {status:5} {name:34} {fails:>4} {detail}") print("verdict:", ["publish", "publish with warning", "BLOCK: do not publish"][worst]) sys.exit(1 if worst == 2 else 0) ``` 3. Run `python3 make_orders.py && python3 check_orders.py; echo "exit code: $?"`. Expected output: ```text (26453, '2026-09-18', '2026-10-15') checks for load_date 2026-10-15 PASS freshness <= 6 h 0 newest row is 1.2 h old ERROR volume within mean +/- 3 sd 1 today 611, limits 695..1219 from 27 days ERROR volume vs weekdays only 1 limits 913..1106 from 19 weekdays ERROR customer_id not null 2 rows ERROR order_id unique 1 duplicated ids WARN status in accepted set 1 rows WARN customer_id exists in customers 1 orphans verdict: BLOCK: do not publish exit code: 1 ``` 4. Make the partial load subtler: change `n = 610` to `n = 820` and run both scripts again. Executed result (first lines of the checker's output): weekend history widens the all-days limits, so only the weekday baseline catches a load about 19% below a normal weekday. ```text checks for load_date 2026-10-15 PASS freshness <= 6 h 0 newest row is 1.2 h old PASS volume within mean +/- 3 sd 0 today 821, limits 695..1219 from 27 days ERROR volume vs weekdays only 1 limits 913..1106 from 19 weekdays ``` **Validation:** run 1 exits with code 1; run 2 shows the volume checks disagreeing. **Troubleshooting:** other counts mean the seed was changed. **Teardown:** delete `shop.db`. **Take-away:** mean ± 3 sd assumes one stable process; a weekly cycle is two. Compare like with like, or forecast as Uber does.[^uber] **Lab 2 (intermediate): contract-driven write-audit-publish.** Objective: audit a batch against the ODCS contract from the [reference implementation](#reference-implementation), publish or quarantine it, and prove a rerun does not duplicate data. Prerequisites: Python 3.8+ with PyYAML; the contract saved as `orders.odcs.yaml`; about 20 minutes; local files. _Example status: executed twice on Python 3.8.10 with PyYAML 5.3.1 on 2026-10-16 and again on 2026-10-17; the output below is verbatim._ ```python """wap.py: write-audit-publish for one daily batch, audited against an ODCS v3.2.0 contract (subset).""" import random import sqlite3 import sys import yaml # PyYAML contract = yaml.safe_load(open("orders.odcs.yaml")) obj = contract["schema"][0] TABLE, AUDIT = obj["physicalName"], obj["physicalName"] + "__audit" COLS = [p["name"] for p in obj["properties"]] db = sqlite3.connect("warehouse.db") db.execute(f"CREATE TABLE IF NOT EXISTS {TABLE} ({', '.join(COLS)})") db.execute("CREATE TABLE IF NOT EXISTS publish_log (load_date, outcome, detail)") def metric_sql(rule, column): """Translate ODCS library metrics (and sql rules) into a failure-count query on the audit table.""" m = rule.get("metric") if rule.get("type") == "sql": # ODCS: {object} = table, {property} = column sql = rule["query"].replace("{object}", AUDIT) return sql.replace("{property}", column) if column else sql if m == "rowCount": return f"SELECT COUNT(*) FROM {AUDIT}" if m == "nullValues": return f"SELECT COUNT(*) FROM {AUDIT} WHERE {column} IS NULL" if m == "duplicateValues": return f"SELECT COUNT(*) - COUNT(DISTINCT {column}) FROM {AUDIT}" if m == "invalidValues": allowed = ", ".join("'" + v.replace("'", "''") + "'" for v in rule["arguments"]["validValues"]) return f"SELECT COUNT(*) FROM {AUDIT} WHERE {column} NOT IN ({allowed})" raise ValueError(f"unsupported rule {rule}") def passes(rule, value): ops = {"mustBe": lambda v, t: v == t, "mustBeGreaterThan": lambda v, t: v > t, "mustBeLessThan": lambda v, t: v < t} return all(fn(value, rule[op]) for op, fn in ops.items() if op in rule) def audit(): rules = [(r, None) for r in obj.get("quality", [])] rules += [(r, p["name"]) for p in obj["properties"] for r in p.get("quality", [])] failures = [] for rule, column in rules: value = db.execute(metric_sql(rule, column)).fetchone()[0] if not passes(rule, value): failures.append(f"{rule['id']}={value}") return failures def run(load_date, rows): db.execute(f"DROP TABLE IF EXISTS {AUDIT}") db.execute(f"CREATE TABLE {AUDIT} AS SELECT * FROM {TABLE} WHERE 0") # WRITE (hidden) db.executemany(f"INSERT INTO {AUDIT} VALUES ({', '.join('?' * len(COLS))})", rows) failures = audit() # AUDIT with db: # one transaction if failures: quarantine = f"{TABLE}__quarantine_{load_date.replace('-', '')}" db.execute(f"DROP TABLE IF EXISTS {quarantine}") db.execute(f"ALTER TABLE {AUDIT} RENAME TO {quarantine}") db.execute("INSERT INTO publish_log VALUES (?, 'quarantined', ?)", (load_date, ", ".join(failures))) else: # Idempotent publish: replace the whole partition, never append to it. db.execute(f"DELETE FROM {TABLE} WHERE substr(loaded_at, 1, 10) = ?", (load_date,)) db.execute(f"INSERT INTO {TABLE} SELECT * FROM {AUDIT}") # PUBLISH db.execute(f"DROP TABLE {AUDIT}") db.execute("INSERT INTO publish_log VALUES (?, 'published', ?)", (load_date, f"{len(rows)} rows")) print(f"{load_date}: {'QUARANTINED ' + ', '.join(failures) if failures else 'published'}") def batch(load_date, n, broken=False): random.seed(load_date) rows = [[f"{load_date}-{i:05d}", f"c{random.randrange(2000):04d}", random.choice(["placed", "shipped", "completed", "returned"]), round(random.uniform(5, 400), 2), f"{load_date} 05:{random.randrange(60):02d}:00"] for i in range(n)] if broken: rows[0][1] = None # customer_id missing rows[1][2] = "on_hold" # undeclared status rows[2][3] = -12.5 # negative amount rows.append(list(rows[3])) # duplicate order_id return rows run("2026-10-14", batch("2026-10-14", 1000)) run("2026-10-15", batch("2026-10-15", 980, broken=True)) print("published rows:", db.execute(f"SELECT COUNT(*) FROM {TABLE}").fetchone()[0]) print("tables:", [r[0] for r in db.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name")]) sys.exit(1 if db.execute("SELECT COUNT(*) FROM publish_log WHERE outcome='quarantined'").fetchone()[0] else 0) ``` Run it twice. Expected output, identical both times (exit code 1, because a batch was quarantined): ```text 2026-10-14: published 2026-10-15: QUARANTINED order_id_unique=1, customer_id_not_null=1, status_accepted=1, amount_not_negative=1 published rows: 1000 tables: ['orders', 'orders__quarantine_20261015', 'publish_log'] ``` **Validation:** after two runs the published table still holds 1,000 rows and the quarantine table holds the 981 rejected rows. **Troubleshooting:** `unsupported rule` means the contract uses a metric this subset lacks (such as `missingValues`). **Teardown:** delete `warehouse.db`. **Caution:** the script runs SQL from the contract, so review contracts like code and never load them from untrusted sources. **Lab 3 (advanced): Slim CI with dbt and DuckDB.** Objective: change one model and let CI build and test only what changed, deferring unchanged parents to production.[^dbt-state][^dbt-defer] Prerequisites: Python 3.10+ (required by dbt Core 1.12.6 and dbt-duckdb 1.11.0 as of October 2026[^dbt-duckdb]); about 45 minutes; no cloud cost. _Example status: YAML and bash syntax-checked; not executed (this environment has Python 3.8). Expected results are derived from the dbt documentation, not observed._ 1. Create the project. Both targets share one DuckDB file, so CI can defer to "production". ```yaml # profiles.yml dataops_lab: target: prod outputs: prod: {type: duckdb, path: lab.duckdb, schema: prod} ci: {type: duckdb, path: lab.duckdb, schema: ci} --- # dbt_project.yml name: dataops_lab version: "1.0.0" profile: dataops_lab ``` ```text # seeds/raw_orders.csv order_id,customer_id,status,amount,loaded_at o1,c1,completed,15.50,2026-10-15 05:10:00 o2,c2,returned,40.00,2026-10-15 05:12:00 o3,c1,shipped,22.00,2026-10-15 05:20:00 ``` ```sql -- models/stg_orders.sql select order_id, customer_id, lower(status) as status, cast(amount as decimal(12, 2)) as amount, cast(loaded_at as timestamp) as loaded_at from {{ ref('raw_orders') }} -- models/fct_orders.sql select order_id, customer_id, status, case when status = 'returned' then -amount else amount end as net_amount from {{ ref('stg_orders') }} ``` Add `models/_models.yml` from the reference implementation, minus the `relationships` test (no `dim_customers` here). 2. Build production once and keep its manifest. ```bash python3 -m venv .venv && . .venv/bin/activate pip install "dbt-core==1.12.6" "dbt-duckdb==1.11.0" dbt build --profiles-dir . --target prod mkdir -p prod-artifacts && cp target/manifest.json prod-artifacts/ ``` 3. Introduce a bug: change `-amount` to `amount` in `fct_orders.sql` (a lost sign), then run the Slim CI command. ```bash dbt build --profiles-dir . --target ci --select "state:modified+" --defer --state prod-artifacts ``` **Expected (not observed):** only `fct_orders` and its tests are selected; `ref('stg_orders')` resolves to `prod` because it was not selected and does not exist in `ci`;[^dbt-defer] the unit test fails. **Validation:** revert and rerun; the build passes. **Troubleshooting:** if dbt selects nothing, the saved manifest already contains the change. If the contract check reports a type mismatch, write the type your platform reports (DuckDB's names are `DECIMAL` and `VARCHAR`); dbt ignores size, precision and scale when it compares.[^dbt-contract-cfg] **Teardown:** delete `lab.duckdb`, `target/` and `prod-artifacts/`. 4. In a team, CI runs the same command against the shared warehouse on every pull request. This sketch is not runnable as is: replace `dbt-` and the login step with your warehouse's; `fetch_prod_manifest.sh` downloads the last production manifest. Actions are pinned to commit SHAs; GitHub ranks a workflow lockfile, where adoptable, above SHA pinning.[^gh-secure] ```yaml # .github/workflows/dbt-ci.yml (sketch: placeholders in ) name: dbt-ci on: pull_request: branches: [main] permissions: contents: read id-token: write # lets the job request an OIDC token; grants no repository write access jobs: slim-ci: runs-on: ubuntu-24.04 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.12" - run: pip install "dbt-core==1.12.6" "dbt-==" - name: Exchange the OIDC token for short-lived warehouse credentials run: ./ci/warehouse_login.sh # - name: Fetch the manifest saved by the last production run run: ./ci/fetch_prod_manifest.sh prod-artifacts/ - name: Build and test only what changed run: dbt build --target ci --select "state:modified+" --defer --state prod-artifacts ``` **Who can reach the warehouse through this job.** - **Every branch pull request.** Without an environment, the default OIDC subject of a `pull_request` job is `repo:ORG/REPO:pull_request` (plus owner and repository IDs for repositories created after 15 July 2026 or opted in; not on GitHub Enterprise Server),[^gh-oidc-ref][^gh-oidc] whoever wrote the pull request, so a trust policy on it admits unreviewed code. - **Not forks.** Fork pull requests get no secrets except `GITHUB_TOKEN`,[^gh-usesecrets] which is read-only for them;[^gh-events] never run pull-request code under `pull_request_target`.[^gh-secure] - **Production reads are exposed.** Deferral reads production tables, and a new model is arbitrary SQL, so a pull request could copy production data, including personal data, into a CI schema or the job log. Grant only masked or non-sensitive views, defer to a sanitised clone, or put the job in a GitHub environment whose required reviewers approve each run (prevent self-review, disallow administrator bypass; on Free, Pro and Team plans, public repositories only); the subject becomes `repo:ORG/REPO:environment:NAME`.[^gh-env][^gh-oidc-ref] - **Writes and deploys.** CI writes only to CI schemas; deployment uses a separate role behind a protected environment.[^gh-secrets] ### Use cases and scenarios These are the organisations' own accounts, not independent audits. Contexts and methods come from the sources; the pitfalls marked "ours" and the last column are this page's suggestions, not reported results. | Scenario | Context | Method | Trade-off or pitfall | What to measure (suggested) | |---|---|---|---|---| | Certified data (Airbnb, 2020) | Growing warehouse; scientists unsure which sources to trust[^airbnb] | Owners, design spec, validation, four reviews, landing-time SLAs, badges | Certified models are much slower to build | Key metrics on certified models; SLA attainment | | Contracts at the service boundary (GoCardless) | Critical data copied by CDC; upstream changes broke dashboards[^gocardless] | Self-service contracts in Git that provision the resources | Speed versus sprawl | Upstream-caused incidents; adoption | | Statistical monitoring (Uber, 2020) | Tens of thousands of tables; per-metric alerts too noisy[^uber] | Principal components, one-step-ahead forecasts, table-level scores | Scores are harder to explain than rules; a learned baseline can absorb a slow decline (ours) | Alert precision | | Pipeline SLOs (Spotify, in Google's SRE Workbook) | Event delivery feeding many jobs[^sre-wb] | Timeliness, completeness, skewness SLOs measured by independent systems | Reprocessing when completeness or skew breaks; a monitor sharing the pipeline's failure modes goes silent with it (ours) | SLO attainment by tier | | Training-data validation (Google) | Data into ML pipelines for hundreds of teams[^tfdv] | Schema-based validation of the data in TFX | Schemas must evolve with data; an inferred schema accepts today's errors unless someone reviews it (ours) | Errors caught before training | Two lessons travel. Airbnb's certified data has "landing time SLAs, backed by a central incident management process", and building it is "much slower than building uncertified data",[^airbnb] so certify only the few assets that carry decisions. At GoCardless, merging a Jsonnet contract provisions BigQuery and Pub/Sub resources; after six months about 30 contracts powered roughly 60% of asynchronous inter-service events, with the best adoption where teams wanted new data streams.[^gocardless] Producers adopt contracts when they get something back. ## Measure, govern, pay ### Metrics and KPIs Measure what consumers feel, then the machinery. Google's SRE Workbook's freshness objectives include "The oldest data is no older than Y".[^sre-wb] Delivery metrics borrow from DORA.[^dora-metrics] | KPI | Definition and formula | Pitfall | Cadence | |---|---|---|---| | Freshness SLO attainment | Loads published by the agreed time ÷ all loads, per tier-1 asset | Count audited publishes, not finished jobs | Weekly | | Correctness incidents | Periods when consumers could see wrong or missing data | Unlogged small incidents | Monthly | | Consumer-detected share | Incidents first reported by a consumer ÷ all incidents (this page's proposed headline indicator) | Needs an honest "who found it" field | Monthly | | Time to detect and restore | Visible → detected; detected → corrected (p50, p90) | Averages hide tails | Monthly | | Data downtime | Sum of (time to detect + time to restore), a vendor-coined measure[^mc-pillars] | Weight by tier | Monthly | | Check coverage | Tier-1 assets with freshness, volume, key tests and contracts ÷ tier-1 assets | Test counts reward noise | Monthly | | Alert precision | Alerts that led to action ÷ all alerts | Unlabelled alerts | Monthly | | Change fail rate (pipelines) | Deployments needing immediate intervention ÷ deployments[^dora-metrics] | Data-only incidents are not deployment failures | Monthly | | Cost per tier-1 load | Compute for build and checks ÷ loads ([FinOps](../finops/)) | Moves with prices | Monthly | **Worked calculation 1: a freshness error budget.** SLO: the finance mart is published by 07:00 on 95% of business days in a quarter. Over 63 business days the budget is 63 × 5% = 3.15, so **3 late days**. Two late days by week 6 use two-thirds of the budget at half-time and should trigger the pre-agreed policy (say, no feature work on that pipeline until the cause is fixed); as in [SRE](../sre/), a budget without consequences is only a chart. **Worked calculation 2: downtime is mostly detection.** Four tier-1 incidents in a month: | Incident | Time to detect (h) | Time to restore (h) | Found by | |---|---|---|---| | A | 0.5 | 2 | Monitor | | B | 6 | 3 | Consumer | | C | 0.25 | 1 | Monitor | | D | 26 | 5 | Consumer | Data downtime = 2.5 + 9 + 1.25 + 31 = **43.75 hours**, of which detection is 32.75 hours (75%). Consumer-detected share = 2 ÷ 4 = **50%**. The best investment is the monitor that would have caught D, not a faster fix. **Worked calculation 3: the false-alarm bill of 3-sigma limits.** For a normally distributed statistic in control, a point falls outside ±3σ with probability 0.0027.[^nist-spc] With 200 tables × 3 statistics checked daily, that is 600 × 0.0027 ≈ **1.6 false alarms a day**, about 49 a month. Real data is rarely normal or independent, so measure precision; page only on tier-1 checks and send the rest to a daily digest. ### Quality, security, privacy and compliance - **Least privilege.** Production jobs write only to their own schemas. CI identities write only to CI schemas and read only what unreviewed pull-request code may see, because every branch pull request can assume them (see [Lab 3](#hands-on-labs)). General access policy belongs to [IAMOps](../iamops/), data-level policy to [DataSecOps](../datasecops/). - **Test outputs copy data.** dbt's `store_failures` saves failing records to an audit schema,[^dbt-store] and GX publishes validation results in Data Docs.[^gx] If failing rows hold personal data, apply the source's access rules and retention, or store only keys. - **Development data** copied from production keeps its legal obligations: masking rules come from [DataSecOps](../datasecops/) and [PrivacyOps](../privacyops/); DataOps applies them in every environment build. - **Regulation (jurisdiction-specific, not legal advice).** For high-risk AI systems, Article 10 of the EU AI Act requires training, validation and testing data to be "relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose".[^aiact] As of October 2026 this applies from 2 December 2027 (Annex III systems) or 2 August 2028 (Annex I), dates set by Regulation (EU) 2026/1744.[^ai-omnibus] Banks look to BCBS 239.[^bcbs239] Model governance belongs to [ModelOps](../modelops/) and [ComplianceOps](../complianceops/). - **Safe repairs.** A backfill rewrites history people have used: for tier-1 assets require approval, announce the period, and keep the previous version (for example an Iceberg tag).[^iceberg] ### Cost, capacity and sustainability Checks are queries, and queries cost money. Under BigQuery's on-demand model, for example, you pay for bytes processed; `LIMIT` does not reduce them, while partitioning and clustering do.[^bq-pricing] Levers: scope checks to the new partition (dbt's freshness `filter` limits the data scanned[^dbt-fresh]); build only what changed in CI and run unit tests only in CI;[^dbt-state][^dbt-unit] use `dbt build --empty` for structural dry runs;[^dbt-build] reserve write-audit-publish for tier 1: it adds latency and orchestration,[^aws-wap] and doubles writes when publishing copies data (as in Lab 2) rather than swapping metadata. ::: example Illustrative scenario: what a uniqueness test costs `fct_orders` holds three years of daily partitions; its `order_id` column is 90 GiB. A daily full-column uniqueness test reads about 90 GiB × 30 ≈ 2.6 TiB a month. Scoped to the new partition it reads about 90 ÷ 1,095 ≈ 0.08 GiB a day, roughly 2.5 GiB a month. A partition-scoped test misses duplicates across days, so add a weekly full-history run (about 0.35 TiB a month): still about seven times cheaper. Prices change, so multiply by your own rate. ::: Capacity is mostly time: slack between the slowest load on the heaviest day and consumers' deadlines; deadline alerts and an independent freshness monitor page on overruns. Fewer redundant scans also save energy ([GreenOps](../greenops/)). ### Assurance of the operating model - **Monitor the monitors:** alert when an asset has produced no check results for longer than its schedule allows. A silent monitor looks exactly like a healthy one. - **Log overrides:** every manual publish that bypassed an audit is justified and reviewed weekly. - **Close the loop:** for each incident, add the check that would have caught it earlier and watch the consumer-detected share. - **Sample audits:** each quarter re-derive a few tier-1 figures from source, or compare against "golden data" with known expected output, as the SRE Workbook suggests.[^sre-wb] ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Remediation and prevention | |---|---|---|---| | Load "succeeded" with far fewer rows | Partial or late extract | Volume against a seasonal baseline; source counts | Quarantine and rerun; volume checks plus write-audit-publish | | Numbers jump, no code change | Upstream change in meaning | Distribution monitors; lineage; producer deploys | Pin consumers to the last good version; contracts with semantics | | Duplicates after a retry | Non-idempotent append | Count by key per partition | Replace the partition; Airflow advises UPSERT over INSERT[^af-best] | | Build fails after a producer deploy | Renamed column or changed type | Schema diff against the contract | Versioned change; contract checks in producer CI; registry rules[^confluent] | | Late every Monday | Weekend backlog, contention | Duration and queue time by weekday | Capacity or earlier start; deadline alerts | | Published totals change days later | Late-arriving records | Published count against source count a day later | Lookback window; mark periods final; completeness checks | | Tests green, consumers still find errors | Checks test the wrong things | Which check would have caught it? | Add it; track consumer-detected share | | Alerts ignored | Static thresholds on seasonal data | Precision per check | Seasonal baselines; fewer, sharper checks | ### Worked runbook: the revenue dashboard dropped overnight ::: figure Data incident runbook: from alarm to repaired history ```mermaid flowchart TB A["Alert or consumer report"]:::risk B{"Real-world change or data fault?"}:::decision C["Confirm with the business owner; annotate the dashboard"]:::neutral D["Contain: stop downstream publishes; pin to last good data"]:::core E["Diagnose: freshness, volume, schema, tests, lineage upstream"]:::core F{"Code change or data change?"}:::decision G["Revert the commit"]:::core H["Fix with the producer or add a guard"]:::core I["Backfill affected partitions idempotently"]:::core J["Verify with consumers; publish; review"]:::neutral A --> B B -->|"real"| C B -->|"fault or unsure"| D --> E --> F F -->|"code"| G --> I F -->|"data"| H --> I I --> J classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: an alert or consumer report starts the runbook. If the world really changed, confirm with the business owner and annotate the dashboard. If the data is wrong, or nobody is sure, contain (stop downstream publishes, pin consumers to the last good data), then diagnose through freshness, volume, schema, tests and upstream lineage. Revert a code change; fix a data change with the producer or a new guard. Backfill affected partitions idempotently, verify with consumers, publish and review. ::: ::: example Illustrative scenario: revenue down 38% at 08:10 **08:10.** Finance reports yesterday's revenue 38% below the day before. No monitor fired. The on-call engineer opens a tier-1 incident ([IncidentOps](../incidentops/) roles apply). 1. **Real or fault? (08:15)** No promotion ended; the checkout service's own order count is normal. Treat it as a fault. 2. **Contain (08:20).** Pause Dags that read `fct_orders`, put a banner on the dashboard, tell consumers when to expect an update. 3. **Diagnose (08:25–09:10).** Freshness, volume, schema and tests pass, but the `returned` share jumped from about 3% to 21%. Lineage leads to the source: since a 23:40 deploy, checkout writes `returned` for *partially* refunded orders, and the model makes every `returned` order negative. 4. **Fix (09:10–10:30).** A data change, so nothing to revert. The producer will emit `partially_returned`; meanwhile the pipeline uses the refund-amount column. 5. **Backfill (10:30–11:00).** Rebuild partitions since 23:40 in the audit area, re-run checks, publish atomically, re-enable Dags. 6. **Close (11:15).** Finance confirms the figure. Detection took about 8.5 hours from the deploy, restoration about 3. 7. **Review.** A distribution monitor on status shares now blocks tier-1 publishes, the contract defines what `returned` means, and the producer's CI checks the contract before deploying. ::: ### Anti-patterns and myths - **"The job succeeded, so the data is fine."** Success means the code ran; only checks on the output speak about the data. - **Testing everything, warning forever.** Thousands of warn-only tests hide the few that matter; a warning nobody reads is a decision to publish bad data. - **One central team owns every pipeline.** It becomes the bottleneck GoCardless removed by letting producers own contracts;[^gocardless] data mesh argues the same.[^mesh] Central teams should run the platform. - **Fixing history in place without a snapshot,** or copying unmasked production data into sandboxes. - **Myth: "DataOps is DevOps with SQL."** DevOps also runs and monitors production; DataOps adds the run loop: checks on data that changes with no deploy. ## Deciding and adopting ### When to use and when not to use **Use DataOps practices** when decisions, money, customers or models depend on regularly refreshed data, when several teams produce or consume it, and when consumers have found errors before. **Do not adopt the full practice** for one-off analyses, exploratory notebooks, throwaway prototypes, small datasets whose owner is the only consumer, or reference data that changes once a year; version control and a few sanity checks are enough. The full practice has hidden costs: on-call duty, contract negotiation, compute for checks and slower delivery of certified assets.[^airbnb] ### Decision framework ::: figure Decision tree: how much DataOps, and what to do first ```mermaid flowchart TB Q1{"Do decisions, money or models depend on this regularly refreshed data?"}:::decision N1["Light hygiene: Git, a few checks, no service level"]:::neutral Q2{"Where did recent errors start?"}:::decision P1["Contracts first"]:::core P2["Freshness, volume, WAP"]:::core P3["CI with unit and data tests"]:::core P4["Start an incident log"]:::risk Q3{"Tier-1 consumers with a deadline?"}:::decision S1["Add SLOs, on-call and error budgets"]:::core S2["Owners, monitors, weekly review"]:::neutral Q1 -->|"no"| N1 Q1 -->|"yes"| Q2 Q2 -->|"upstream"| P1 --> Q3 Q2 -->|"loads"| P2 --> Q3 Q2 -->|"our code"| P3 --> Q3 Q2 -->|"don't know"| P4 --> Q3 Q3 -->|"yes"| S1 Q3 -->|"no"| S2 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: if decisions, money or models do not depend on regularly refreshed data, light hygiene is enough. If they do, ask where recent errors started: upstream producers call for contracts first; late or partial loads for freshness and volume checks with write-audit-publish; the team's own changes for CI with tests; unknown, for an incident log. Tier-1 consumers with a deadline then need SLOs, on-call and error budgets; otherwise owners, monitors and a weekly review suffice. ::: Assumption: the cheapest control aims at where failures start, and you cannot aim without an incident log. ### Maturity model This is **this Atlas's model**, not an industry standard. Move up when the exit criteria hold for tier-1 assets. | Level | Observable behaviour | Exit criteria | |---|---|---| | 0, Ad hoc | Scripts edited in place; consumers find most errors | Tier-1 pipeline code in Git with review | | 1, Versioned | Git, an orchestrator, a few tests; informal incidents | Owners, freshness and volume checks on tier 1; incidents logged with "who found it" | | 2, Tested and monitored | CI with data tests on every pull request; tier-1 monitors | Slim CI in use; consumer-detected share measured for a quarter | | 3, Contracted and audited | Input contracts; write-audit-publish, SLOs and lineage for tier 1 | No unaudited tier-1 publish for a quarter; error-budget policy used | | 4, Optimised | Seasonal monitors with tracked precision; cost per load managed; self-service platform | Consumer-detected share low and falling | ### Adoption roadmap: 30/60/90 days | Period | Actions | Milestone and metric | |---|---|---| | Days 1–30 | List up to ten tier-1 assets with owners and deadlines; Git with review; freshness, volume and key tests; incident log | Baselines for consumer-detected share and freshness | | Days 31–60 | Slim CI; idempotent loads; write-audit-publish for the three most critical assets; first contract with the most damaging producer; weekly review | No duplicate-on-retry incidents; first contract merged | | Days 61–90 | Tier-1 SLOs and error budgets; lineage; seasonal monitors; two more contracts; first sample audit and cost-per-load figure | Level 2 exit criteria met; SLO attainment and alert precision reported | Quick wins: freshness checks on the five most-used tables, a dashboard banner procedure, a publish log. Dependencies: a platform team for CI schemas and credentials; producers willing to review contracts. ## Toolkit ### Checklists **Quick start for a new pipeline** - [ ] Tier, owner, consumers and their deadline written down - [ ] Code in Git; changes by pull request only - [ ] Idempotent load (replace a partition, never append twice) - [ ] Freshness, volume and key tests in place - [ ] Failures route to a named person **Before merging a change** - [ ] Slim CI passed for changed models and their children - [ ] Unit tests cover new date, window or case logic - [ ] Breaking contract changes have a new version and a deprecation date - [ ] Backfill plan written if history must change **Tier-1 launch readiness** - [ ] Input contracts merged and checked in the producers' CI - [ ] Write-audit-publish with blocking checks agreed with consumers - [ ] Freshness SLO and error-budget policy agreed - [ ] Lineage emitted; runbook and on-call rota exist; cost per load estimated - [ ] Test outputs that copy personal data are access-controlled **Weekly reliability review** - [ ] Error budgets, new incidents and who found them; low-precision alerts, audit overrides, silent monitors **During a data incident** - [ ] Real-world change ruled out; downstream publishes paused; consumers informed - [ ] Affected assets and period listed from lineage; fix verified in the audit area; review booked ### Templates **Data contract, one page (the human summary of an ODCS file)** ```text Contract: orders (version 1.2.0, status: active) Producer: checkout-platform team Purpose: one row per customer order for finance and analytics Schema: order_id (string, unique, required) | customer_id (string, required) | status (placed|shipped|completed|returned) | amount (decimal 12,2; >= 0; order currency) | loaded_at (timestamp) Semantics: "returned" = whole order refunded. Partial refunds are not "returned". Service levels: latency <= 6 h; frequency daily; time to detect <= 30 min Breaking change process: new major version, 30-day overlap, consumers review the PR Consumers: finance mart (tier 1), forecasting features (tier 1), ops dashboard (tier 2) ``` **Data incident record** ```text Incident ID: Asset(s) and tier: Visible from / to: Detected at: Restored at: Found by: monitor | consumer (name) Severity: What consumers saw: Root cause (code change | data change | infrastructure): Containment taken: Backfill: partitions, approval, verification Check that would have caught it earlier (added? yes/no): Actions, owners, due dates: ``` ### Deep comparisons **DataOps versus DevOps** | Dimension | [DevOps](../devops/) | DataOps | |---|---|---| | Unit of change | Code and configuration | Also the data itself, on every run | | Rollback | Redeploy the previous version | Revert code **and** rebuild data; consumers may have used bad data | | Environments | Containers, infrastructure as code | Plus data isolation: CI schemas, clones, branches | | Headline metrics | DORA delivery metrics | Freshness, correctness, consumer-detected incidents; DORA for pipeline changes | **Checks you write versus a data observability platform** | Factor | dbt tests, GX, Soda | Observability platform | |---|---|---| | Finds | What you anticipated | Also unanticipated anomalies, from learned baselines | | Blocks a publish? | Yes, inside the pipeline | Often monitors after publication; check the integration | | Cost and risk | Engineering time; gaps nobody wrote a check for | Licence, alert tuning, lock-in, overlap with platform features[^gartner-mg] | Mature setups use both: rules as gates, statistical monitors as a net. ### FAQ **Is DataOps a job title or a team?** Both appear. DataOps engineers usually build the platform that pipeline owners use; a team owning everyone's data quality becomes a bottleneck. **How is DataOps different from data governance?** Governance sets rules (classification, access, retention, accountability); DataOps runs pipelines that follow them and produces the evidence. **Should every check block publication?** No. Block where failure means consumers would act on wrong data (keys, totals, volume collapses, contract breaks on tier-1 assets); warn on the rest and review warnings weekly. **How is a data contract different from a schema registry?** A registry enforces compatibility between schema versions (Confluent's default is BACKWARD).[^confluent] Some registries now attach metadata and rules as well (Confluent's rules need a paid tier, as of October 2026),[^confluent-dc] but compatibility checks stay structural. A contract adds meaning, quality rules, service levels and owners,[^bitol] which is why the runbook's `returned` failure passed every schema check. **Where does DataOps end and MLOps begin?** At the training or feature dataset: DataOps makes it correct and on time; features, training, serving and model drift are [MLOps](../mlops/); retrieval corpora for LLM applications are [LLMOps](../llmops/). ## Going further ### Advanced topics and emerging direction - **Established:** Slim CI;[^dbt-state] asset-aware orchestration;[^af-assets] table-format branching for write-audit-publish;[^iceberg] open standards for contracts and lineage.[^odcs][^openlineage] - **Newer, still settling (October 2026):** dbt v2 "checks" that enforce project standards before warehouse work;[^dbt-checks] column-level lineage in dbt's information schema;[^dbt-build] Dagster freshness policies, "under active development";[^dg-fresh] Bitol's Open Data Product Standard (v1.0 in September 2025; v1.1 released with ODCS v3.2.0, which shares five RFCs with it).[^odps][^odcs] - **Vendor direction, not settled practice:** AI agents that build and operate pipelines (Gartner reports AI-assisted and agentic tools emerging;[^gartner-mg] lakeFS markets branches for agents behind human review[^lakefs]). Treat autonomous repair as unproven. ### Learning path | Stage | What to do | Outcome | |---|---|---| | Prerequisites | SQL joins and aggregates; Git pull requests; what CI does | Can review a pipeline change | | 1-day primer | Read the Manifesto and this page's charter; run Labs 1 and 2 | Can explain checks, severity, write-audit-publish | | 1 week | Small dbt project on DuckDB with freshness, tests, a unit test and a contract; Lab 3 | Can set up Slim CI | | 30 days | Write-audit-publish and one producer contract on a real tier-1 asset; incident log and scorecard | Can run the weekly review | | Advanced | Seasonal and multivariate monitoring;[^uber] OpenLineage; streaming contracts; cost with [FinOps](../finops/) | Can design a platform for many teams | ## Wrap-up ### Executive recap and cheat sheet - **Decide** which assets are tier 1, who owns them, and which failures block publication. - **Practise** CI for every pipeline, idempotent loads, write-audit-publish for tier 1, contracts with the producers that hurt you most, seasonal monitors. - **Measure** freshness against an error budget, consumer-detected share, detect and restore times, alert precision, cost per load. - **Watch out for** checks that cannot block, warn-only sprawl, personal data in failing-row tables, CI roles that read production. - **Next:** list tier-1 assets this week, add freshness and volume checks, start the incident log with "who found it". ### A judgement call: block the 07:00 finance load or publish with a warning? ::: decision The dilemma At 06:45 the finance mart's audit fails one blocking check: today's order volume is 12% below the weekday baseline's lower control limit. Everything else passes. The board pack is generated at 07:00. Option A: block, so the board sees yesterday's figures labelled "not updated". Option B: publish with a warning banner. ::: **Assumptions.** A false alarm is possible (a holiday, a genuinely bad day); the board pack is copied into slides and e-mails; yesterday's figures are correct but a day old. **Why B tempts, and why it is usually wrong for tier 1.** It keeps the schedule, but the banner stays on the dashboard while the numbers travel into slides without it. Stale-but-labelled data fails safe; wrong-but-plausible data does not. **Deciding in fifteen minutes.** Check the cheapest discriminating evidence: the source system's own order count, the holiday calendar, the producer's deploy log. If the source agrees with the low number, the world changed: publish and state the cause. If it shows more orders than were loaded, it is a partial load: block, page the producer, rerun. If you cannot tell by 07:00, block; one late morning costs little next to a board decision on a wrong number. **What would change the answer.** For a staffing dashboard used within the hour, freshness may matter more than a 12% count error: publish with the warning. Either way, **agree the rule in advance** and write it into the contract or runbook; alert precision tells you how often such alarms are false. ### Learning objectives and mastery check You should now be able to **explain** DataOps and its contested label, **design** write-audit-publish with blocking checks, **write** checks with a seasonal baseline, **specify** a contract, **measure** error budgets and downtime, and **run** a data incident. ::: answer 1. A load succeeded and all schema checks passed, yet dashboard revenue fell 38%. Which checks could have caught it, and why not schema checks? A distribution monitor on status shares, and accepted values whose meanings are written in the contract. The schema did not change; the meaning of `returned` did, which column names and types cannot show. See the [runbook](#worked-runbook-the-revenue-dashboard-dropped-overnight). ::: ::: answer 2. Why does `dbt build` not protect consumers who query the model whose test failed? It skips downstream resources, but the tested model has already been rebuilt in its target schema.[^dbt-build] Build into an audit schema and publish separately. ::: ::: answer 3. A 28-day mean ± 3 sd volume check missed a partial Thursday load. Why, and what would you change? Weekend days in the baseline inflate the standard deviation. In Lab 1 the all-days limits (695 to 1,219) accepted a load about 19% below a normal weekday; weekday-only limits (913 to 1,106) caught it. Compare like with like, or forecast seasonality. ::: ::: answer 4. A retry doubled yesterday's rows. Which principle broke, and what is the fix? Idempotency: the task appended. Replace the partition (or merge on a key) in one transaction, as Airflow advises and Lab 2 does.[^af-best] ::: ::: answer 5. Which identity should a pull-request CI job use to reach the warehouse, and what can it expose? A dedicated CI identity with short-lived OpenID Connect credentials and no deployment rights. With GitHub's default subject, a pull-request job presents `repo:ORG/REPO:pull_request` (with IDs in newer repositories), which every branch pull request shares,[^gh-oidc-ref] so any contributor's unreviewed SQL runs with it. Let it write only to CI schemas and read only masked or non-sensitive views (or a sanitised clone), or require an environment approval before each run (private repositories: GitHub Enterprise only).[^gh-env] ::: ::: answer 6. Yesterday's order total grew 4% after today's run, with no code change. What happened, and what is missing? Late-arriving records: the incremental run reprocessed its lookback window and added them to yesterday's period, which is correct. What is missing is a rule, agreed with consumers, for when a period becomes final, and a check of published counts against the source. ::: ### Conclusion DataOps is a young label on an old idea: treat the data you deliver as the output of a production line and inspect it before it leaves the factory. The label is contested, but the problem it names is not: pipelines fail quietly, and without deliberate practice the first monitor is the person who trusted the number. What works is narrower than the hype: test every change, make loads idempotent, audit tier-1 data before anyone sees it, write contracts where producers can break you, use baselines that know what day it is, and judge yourself by who finds problems first. ## Sources and further reading [^liebmann]: [3 reasons why DataOps is essential for big data success](https://web.archive.org/web/20180810105123/http://www.ibmbigdatahub.com/blog/3-reasons-why-dataops-essential-big-data-success) — L. Liebmann, IBM Big Data & Analytics Hub, 2014-06-19 (Internet Archive copy). Checked 2026-10-16. [^palmer]: [From DevOps to DataOps](https://web.archive.org/web/20180712103947/https://www.tamr.com/from-devops-to-dataops-by-andy-palmer/) — A. Palmer, Tamr, 2015-05-07 (Internet Archive copy). Checked 2026-10-16. [^manifesto]: [The DataOps Manifesto](https://dataopsmanifesto.org/en/) — stewarded by DataKitchen, living page. Checked 2026-10-16. [^manifesto-2017]: [The DataOps Manifesto, capture of 24 August 2017](https://web.archive.org/web/20170824093708/http://dataopsmanifesto.org/) — Internet Archive, 2017. Checked 2026-10-16. [^ereth]: [DataOps – Towards a Definition](https://ceur-ws.org/Vol-2191/paper13.pdf) — J. Ereth, LWDA 2018, CEUR-WS Vol-2191, 2018. Checked 2026-10-16. [^mainali]: [Discovering DataOps: A Comprehensive Review of Definitions, Use Cases, and Tools](https://personales.upv.es/thinkmind/DATA_ANALYTICS/DATA_ANALYTICS_2021/data_analytics_2021_3_50_60037.html) — K. Mainali et al., DATA ANALYTICS 2021, IARIA. Checked 2026-10-16. [^gartner-mg]: [Market Guide for DataOps Tools](https://www.astronomer.io/uploads/Market_Guide_for_Dat_836742_ndx.pdf) — M. Simone, S. Menon, R. Thanaraj, Gartner, 2025-10-24 (ID G00836742; licensed reprint hosted by Astronomer). Checked 2026-10-16. [^wiki-hist]: [DataOps: revision history](https://en.wikipedia.org/w/index.php?title=DataOps&action=history) — Wikipedia, redirect created 2026-04-08. Checked 2026-10-16. [^ibm]: [What is DataOps?](https://www.ibm.com/think/topics/dataops) — IBM Think, living page. Checked 2026-10-16. [^bitol]: [Bitol: open standards for data contracts and data products](https://bitol.io/) — Bitol (Linux Foundation AI & Data). Checked 2026-10-16. [^odcs]: [Open Data Contract Standard v3.2.0](https://bitol-io.github.io/open-data-contract-standard/latest/) — Bitol, released 2026-09-08 ([release history](https://github.com/bitol-io/open-data-contract-standard/releases)). Checked 2026-10-16. [^odps]: [Open Data Product Standard: releases](https://github.com/bitol-io/open-data-product-standard/releases) — Bitol, GitHub (v1.0.0 2025-09-24; v1.1.0 2026-09-08). Checked 2026-10-16. [^openlineage]: [About OpenLineage](https://openlineage.io/docs/) — OpenLineage project, docs v1.54.0; [object model](https://openlineage.io/docs/spec/object-model/). Checked 2026-10-16. [^iso25012]: [ISO/IEC 25012](https://iso25000.com/en/iso-25000-standards/iso-25012) — iso25000.com (informational site; iso.org blocked automated access). Checked 2026-10-16. [^nist-spc]: [What are Control Charts?](https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc31.htm) — NIST/SEMATECH e-Handbook of Statistical Methods, section 6.3.1. Checked 2026-10-16. [^mesh]: [Data Mesh Principles and Logical Architecture](https://martinfowler.com/articles/data-mesh-principles.html) — Z. Dehghani, martinfowler.com, 2020-12-03. Checked 2026-10-16. [^sre-wb]: [Data Processing Pipelines](https://sre.google/workbook/data-processing/) — R. Sodt and I. Maravić, The Site Reliability Workbook, chapter 13, Google, 2018. Checked 2026-10-16. [^dora-metrics]: [DORA's software delivery performance metrics](https://dora.dev/guides/dora-metrics/) — DORA. Checked 2026-10-16. [^dbt-versions]: [About dbt versions](https://docs.getdbt.com/docs/dbt-versions) — dbt Labs (v2.0 released 2026-09-14; v1.12 released 2026-07-16). Checked 2026-10-16. [^dbt-tests]: [Add data tests to your DAG](https://docs.getdbt.com/docs/build/data-tests) — dbt Labs. Checked 2026-10-16. [^dbt-unit]: [Unit tests](https://docs.getdbt.com/docs/build/unit-tests) — dbt Labs. Checked 2026-10-16. [^dbt-fresh]: [freshness](https://docs.getdbt.com/reference/resource-configs/freshness) — dbt Labs. Checked 2026-10-16. [^dbt-contracts]: [Model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts) — dbt Labs. Checked 2026-10-16. [^dbt-state]: [About local state in dbt](https://docs.getdbt.com/reference/node-selection/state-selection) — dbt Labs. Checked 2026-10-16. [^dbt-defer]: [Defer](https://docs.getdbt.com/reference/node-selection/defer) — dbt Labs. Checked 2026-10-16. [^dbt-build]: [About dbt build command](https://docs.getdbt.com/reference/commands/build) — dbt Labs. Checked 2026-10-16. [^dbt-severity]: [severity, error_if, and warn_if](https://docs.getdbt.com/reference/resource-configs/severity) — dbt Labs. Checked 2026-10-16. [^dbt-store]: [store_failures](https://docs.getdbt.com/reference/resource-configs/store_failures) — dbt Labs. Checked 2026-10-16. [^dbt-checks]: [Checks](https://docs.getdbt.com/docs/build/checks) — dbt Labs (dbt v2). Checked 2026-10-16. [^dbt-models]: [About dbt models](https://docs.getdbt.com/docs/build/models) — dbt Labs. Checked 2026-10-16. [^dbt-mat]: [Materializations](https://docs.getdbt.com/docs/build/materializations) — dbt Labs. Checked 2026-10-16. [^dbt-contract-cfg]: [contract](https://docs.getdbt.com/reference/resource-configs/contract) — dbt Labs. Checked 2026-10-16. [^dbt-env]: [About env_var function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var) — dbt Labs. Checked 2026-10-16. [^dbt-micro]: [About microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch) — dbt Labs. Checked 2026-10-16. [^dbt-duckdb]: [dbt-duckdb](https://github.com/duckdb/dbt-duckdb) — DuckDB project, README; version 1.11.0 on PyPI (2026-08-07). Checked 2026-10-16. [^gx]: [GX Core overview](https://docs.greatexpectations.io/docs/core/introduction/gx_overview/) — Great Expectations, GX Core 1.24.0. Checked 2026-10-16. [^soda]: [Soda Core](https://github.com/sodadata/soda-core) — Soda, README (v4). Checked 2026-10-16. [^deequ]: [Automating large-scale data quality verification](https://www.amazon.science/publications/automating-large-scale-data-quality-verification) — S. Schelter et al., VLDB 2018. Checked 2026-10-16. [^tfdv]: [Data Validation for Machine Learning](https://mlsys.org/Conferences/2019/doc/2019/167.pdf) — E. Breck et al., SysML 2019. Checked 2026-10-16. [^af-assets]: [Asset-Aware Scheduling](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/asset-scheduling.html) — Apache Airflow docs, version 3.3.2. Checked 2026-10-16. [^af-deadline]: [Deadline Alerts](https://airflow.apache.org/docs/apache-airflow/stable/howto/deadline-alerts.html) — Apache Airflow docs. Checked 2026-10-16. [^af-upgrade]: [Upgrading to Airflow 3](https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html) — Apache Airflow docs. Checked 2026-10-16. [^af-best]: [Best Practices](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html) — Apache Airflow docs. Checked 2026-10-16. [^af-bash]: [airflow.providers.standard.operators.bash](https://airflow.apache.org/docs/apache-airflow-providers-standard/stable/_api/airflow/providers/standard/operators/bash/index.html) — Apache Airflow standard provider. Checked 2026-10-16. [^dg-checks]: [Testing assets with asset checks](https://dagster.io/docs/guides/test/asset-checks) — Dagster docs. Checked 2026-10-16. [^dg-fresh]: [Asset freshness policies](https://dagster.io/docs/guides/observe/asset-freshness-policies) — Dagster docs. Checked 2026-10-16. [^iceberg]: [Branching and Tagging](https://iceberg.apache.org/docs/latest/branching/) — Apache Iceberg docs. Checked 2026-10-16. [^aws-wap]: [Build Write-Audit-Publish pattern with Apache Iceberg branching and AWS Glue Data Quality](https://aws.amazon.com/blogs/big-data/build-write-audit-publish-pattern-with-apache-iceberg-branching-and-aws-glue-data-quality/) — AWS Big Data Blog, 2024-12-09. Checked 2026-10-16. [^lakefs]: [Welcome to lakeFS](https://docs.lakefs.io/) — lakeFS documentation. Checked 2026-10-16. [^flink-time]: [Timely Stream Processing](https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/) — Apache Flink docs (stable). Checked 2026-10-16. [^confluent]: [Schema Evolution and Compatibility](https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html) — Confluent docs. Checked 2026-10-16. [^confluent-dc]: [Data Contracts for Schema Registry on Confluent Platform](https://docs.confluent.io/platform/current/schema-registry/fundamentals/data-contracts.html) — Confluent docs. Checked 2026-10-17. [^mc-pillars]: [What Is Data Observability? 5 Key Pillars To Know](https://montecarlo.ai/blog-what-is-data-observability) — B. Moses, Monte Carlo (vendor), updated 2026-10-08. Checked 2026-10-16. [^airbnb]: [Data Quality at Airbnb, Part 2 — A New Gold Standard](https://medium.com/airbnb-engineering/data-quality-at-airbnb-870d03080469) — V. Quoss et al., The Airbnb Tech Blog, 2020-11-24. Checked 2026-10-17. [^gocardless]: [7 Lessons From GoCardless' Implementation of Data Contracts](https://montecarlo.ai/blog-data-contracts) — A. Jones, Monte Carlo blog (guest post), updated 2026-03-30. Checked 2026-10-16. [^uber]: [Monitoring Data Quality at Scale with Statistical Modeling](https://www.uber.com/blog/monitoring-data-quality-at-scale/) — S. Shanmugam et al., Uber Engineering, 2020-05-07. Checked 2026-10-16. [^bq-pricing]: [BigQuery pricing](https://cloud.google.com/bigquery/pricing) — Google Cloud (terms as of October 2026). Checked 2026-10-16. [^gh-usesecrets]: [Using secrets in GitHub Actions](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets) — GitHub Docs. Checked 2026-10-16. [^gh-secrets]: [Secrets](https://docs.github.com/en/actions/concepts/security/secrets) — GitHub Docs. Checked 2026-10-16. [^gh-oidc]: [OpenID Connect](https://docs.github.com/en/actions/concepts/security/openid-connect) — GitHub Docs. Checked 2026-10-16. [^gh-secure]: [Secure use reference](https://docs.github.com/en/actions/reference/security/secure-use) — GitHub Docs. Checked 2026-10-16. [^gh-oidc-ref]: [OpenID Connect reference](https://docs.github.com/en/actions/reference/security/oidc) — GitHub Docs. Checked 2026-10-17. [^gh-events]: [Events that trigger workflows](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows) — GitHub Docs. Checked 2026-10-16. [^gh-env]: [Deployments and environments](https://docs.github.com/en/actions/reference/workflows-and-actions/deployments-and-environments) — GitHub Docs. Checked 2026-10-17. [^aiact]: [Article 10: Data and Data Governance](https://artificialintelligenceact.eu/article/10/) — EU Artificial Intelligence Act explorer (Future of Life Institute; not the official text). Checked 2026-10-16. [^ai-omnibus]: [Regulation (EU) 2026/1744 (Digital Omnibus on AI)](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=OJ:L_202601744) — European Parliament and Council, 8 July 2026, OJ L, 2026-07-24 (Article 1, point 40). Checked 2026-10-17. [^bcbs239]: [Principles for effective risk data aggregation and risk reporting](https://www.bis.org/publications/201301-guidelines-principles-effective-risk-data-aggregation-and-risk-reporting) — Basel Committee on Banking Supervision, 2013-01-09. Checked 2026-10-16. Changelog: 2026-10-16, first version: researched, drafted, Labs 1 and 2 executed, Lab 3 not executed, by the managing author; evidence ledger in `research/dataops-ledger.md`. 2026-10-17, r1 revision (`reviews/dataops-r1-response.md`): CI security side effects, experimental Deadline Alerts, glossary, late-arriving data, ODCS placeholders, RACI, AI Act dates, URLs; Labs 1 and 2 re-run. 2026-10-17, post-score polish (`reviews/dataops-polish.md`): GitHub plan and Enterprise Server limits, Confluent rule tiers, glossary SLO, DevOps myth, Airbnb spec length; five sources re-checked, repetition trimmed.