MLOps

MLOps is the engineering practice of getting trained machine-learning models into production and keeping them correct there. A model is produced by code, data, configuration and a software environment. MLOps versions all four, automates the pipeline that turns them into a model, releases the model through tests and gates, watches it in production, and retrains or rolls it back when the world it predicts has changed.

The label is used widely and the core practices are stable, but published definitions stress different things:

• Google Cloud calls MLOps “an ML engineering culture and practice that aims at unifying ML system development (Dev)” with ML system operation, and stresses automation and monitoring at every step.^[1]
• Google’s practitioner guide calls it “a methodology for ML engineering that unifies ML system development (the ML element) with ML system operations (the Ops element)”, and says it provides “a set of standardized processes and technology capabilities”.^[2]
• A peer-reviewed study by Kreuzberger, Kühl and Hirschl defines it as “a paradigm, including aspects like best practices, sets of concepts, as well as a development culture” that draws on machine learning, software engineering and data engineering.^[3]
• The CD Foundation’s MLOps special-interest group, in its current (2024) roadmap, defines it as “the extension of the DevOps methodology to include Machine Learning and Data Science assets as part of a Continuous Delivery process”; its 2020–2022 editions ended “as first class citizens within the DevOps ecology”.^[4]

Who this page is for. Data scientists whose models must leave the notebook, ML and platform engineers, architects, and leaders deciding how much automation a model portfolio justifies. Beginners: this part and the glossary. Practitioners: skew and drift, labs, runbook. Leaders: Deciding and adopting and the cheat sheet.

In 2015 a Google team described ML systems as a fast way to build up technical debt: “it is dangerous to think of these quick wins as coming for free”.^[9] Their Figure 1: “Only a small fraction of real-world ML systems is composed of the ML code”, and “The required surrounding infrastructure is vast and complex”. They also named the CACE principle, “Changing Anything Changes Everything”: changing one feature, hyperparameter or sampling rule can change everything else.^[9]

Microsoft researchers who studied their own teams found three ways in which ML differs from earlier software: data is harder to discover, manage and version; customising and reusing models needs different skills; and ML components are harder to keep as separate modules.^[10] The usual starting point is what Google Cloud calls “level 0”: data scientists work by hand in notebooks and hand a model file to engineers who rebuild the feature code. The hand-off itself can cause training-serving skew, and “models often break when they are deployed in the real world”.^[1]

Without an MLOps practice, five failures recur:

1. The model cannot be rebuilt. Nobody recorded which data snapshot, code commit, parameters and library versions produced the artifact in production. scikit-learn, for example, states that “there are no supported ways to load a model trained with a different version of scikit-learn”.^[11]
2. Training and serving disagree. Features are computed one way in a notebook and another in production.^[12]
3. Decay is silent. A model that has gone stale throws no exception. Google’s Rules of ML describe a table used by Google Play that was six months out of date; refreshing it alone raised the install rate by 2%.^[12]
4. Every model is a one-off project. Before Uber built its Michelangelo platform, it reported that “there was no established path to deploying a model into production”.^[13]
5. There is no safe way back. Without a registry and versioned serving, rolling back means finding an old file and hoping it still loads.

flowchart TB
  subgraph BEFORE["Before: level 0, manual hand-off"]
    direction TB
    N["Notebook on a laptop"] --> F["Model file sent by email or share"]
    F --> R["Engineer rewrites feature code"]
    R --> S["Production service"]
    S -.->|"no monitoring, no lineage"| Q["Silent decay"]
  end
  subgraph AFTER["After: versioned pipeline and closed loop"]
    direction TB
    P["Versioned training pipeline"] --> G["Registry with lineage and gates"]
    G --> V["Serving with canary and rollback"]
    V --> M["Monitoring: skew, drift, quality"]
    M -->|"trigger or alert"| P
  end
  BEFORE ~~~ AFTER
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class N,F,R,S,Q risk
  class P,G,V,M core

Two misconceptions to correct early:

• “MLOps means putting a model behind an API.” Deployment is one of seven processes in Google’s lifecycle (lifecycle).^[2]
• “Every model needs full automation.” Google’s first rule of ML is “Don’t be afraid to launch a product without machine learning”.^[12] How much MLOps a model needs depends on how fast its world changes and what an error costs (decision framework).

No reliable source records who first used the word “MLOps”, so this page does not attribute it. The practices have a traceable history:

Mental model: the model is a build artifact whose source code includes data. In ML, the “source” is code + data + configuration + environment, and the “compiler” is a training run that may not even be deterministic: PyTorch warns that “Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms”.^[23] CD4ML names three axes of change: code, model and data.^[19] MLOps is therefore a build system for models (versioned inputs, pipelines, lineage), a release process (gates, canaries, rollback) and a control loop (monitoring that decides when to rebuild).

“Skew” is used in two ways. Rules of ML define training-serving skew as “a difference between performance during training and performance during serving”, with three causes: different data handling in the two pipelines, data changing between training and serving, and feedback loops.^[12] Google Cloud’s model monitoring uses the word for a distribution difference between production and training features.^[30] Because the fixes differ, this page separates skew (your two pipelines disagree), an upstream data defect (a source, schema or join change before your pipelines; one form of the second cause) and drift (the world changed). That split is an editorial convention, not a standard.

MLOps sits between the data that feeds models and the decisions that consume them. The boundaries below are this Atlas’s editorial distinctions; vendors draw them differently, and some use MLOps and ModelOps as synonyms.

Other links: AnalyticsOps owns the metric definitions business KPIs reuse; ResearchDataOps long-term stewardship of research datasets; PrivacyOps the legal basis for training on personal data; DevSecOps build-time supply-chain controls; GreenOps carbon accounting.

flowchart TB
  subgraph UP["Upstream"]
    direction LR
    DO["DataOps: trusted data"]
    PR["PrivacyOps: lawful use"]
  end
  ML["MLOps: build, release and monitor trained models"]
  subgraph SIDE["Alongside"]
    direction LR
    MO["ModelOps: governs the portfolio"]
    FA["FinOps for AI: AI economics"]
    KO["KubernetesOps: GPU clusters"]
  end
  subgraph DOWN["Downstream"]
    direction LR
    DE["DecisionOps: decision logic"]
    LL["LLMOps: LLM applications"]
    SR["SRE: service SLOs"]
  end
  UP --> ML
  ML <--> SIDE
  ML --> DOWN
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class ML core
  class DO,PR,MO,FA,KO,DE,LL,SR neutral

Two sets of principles should not be confused. The first is published: Kreuzberger et al. derived nine principles from literature, a tool review and expert interviews, from CI/CD automation, orchestration, reproducibility and versioning of data, model and code to continuous training, metadata tracking, monitoring and feedback loops.^[3] From interviews with 18 ML engineers, Shankar et al. identified three variables that govern the success of a production deployment: “Velocity, Validation, and Versioning”.^[21]

The second is this Atlas’s proposed charter: invariants a team can test, not an industry standard.

Kreuzberger et al. list nine technical components;^[3] Google’s guide lists similar capabilities.^[2] The table merges them; tool examples are in the tooling landscape.

Kreuzberger et al. describe seven roles: business stakeholder, solution architect, data scientist, data engineer, software engineer, DevOps engineer, and ML engineer or MLOps engineer, a cross-domain role that combines aspects of the others and builds and operates the ML infrastructure.^[3] Job titles vary; the responsibilities do not. Either product teams own their models end to end on a self-service ML platform (the pattern in PlatformOps), or a central data-science team builds models for many product teams, which works only if each production model’s owner is written down.

R = responsible, A = accountable, C = consulted, I = informed. An Atlas template: adapt the columns, but keep one “A” per row.

Google’s guide describes the MLOps lifecycle as “seven integrated and iterative processes”: ML development, training operationalisation, continuous training, model deployment, prediction serving, continuous monitoring, and data and model management.^[2] This page’s Figure 3 draws them as a loop (Google shows them in its own Figure 2).

flowchart TB
  D["1. ML development: experiments, training procedure"]
  O["2. Training operationalisation: pipeline built and tested"]
  C["3. Continuous training: pipeline runs on triggers"]
  P["4. Model deployment: package, test, canary"]
  S["5. Prediction serving: batch, online, streaming"]
  M["6. Continuous monitoring: skew, drift, quality"]
  G["7. Data and model management: features, registry, lineage"]
  D --> O --> C --> P --> S --> M
  M -->|"retraining trigger"| C
  M -->|"new idea or concept change"| D
  G -.-> D
  G -.-> C
  G -.-> P
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class D,O,C,P,S,M core
  class G neutral

Retirement deserves explicit work. Sculley et al. warn about undeclared consumers that silently use a model’s output as input.^[9] Keep a consumer list in the registry and access control on prediction outputs, or a “retired” model will be impossible to switch off.

• Ceremonies. A weekly model health review (checklist); a monthly or quarterly model review (slices, staleness, cost, consumers); a post-incident review for every user-visible model failure, via IncidentOps.
• Policies as code. Promotion thresholds, retraining triggers and validation rules live next to the pipeline, so a threshold change is reviewed like code (templates).
• The registry is the system of record. Google’s guide defines model governance as “registering, reviewing, validating, and approving models for deployment”.^[2] MLOps builds the registry and enforces gates; ModelOps decides which risk tiers need which approvals, and tags such as MLflow’s validation_status:approved example carry that decision into the pipeline.^[24]
• Escalation. A model incident is a service incident. Severity follows business impact, not the size of the metric change.

Figure 4 is a vendor-neutral reference architecture for predictive models.

flowchart TB
  %% Training zone declared first: the layout then keeps the orchestrator above the job
  subgraph TRAIN["Training zone"]
    ORC["Pipeline orchestrator"]
    SNAP["Versioned training set"]
    ORC --> TR["Training job"]
    SNAP --> TR
    TR --> EV["Evaluation and gate"]
  end
  subgraph DATA["Data zone: restricted access"]
    SRC["Source tables and events"] --> VAL["Data validation"]
    VAL --> FS["Feature store: offline and online"]
  end
  subgraph REC["Records"]
    META["Metadata and experiments"]
    REG["Model registry with aliases"]
  end
  subgraph SERVE["Serving zone"]
    BA["Batch scoring job"]
    DEP["Deploy step: pins a version"]
    ON["Online service"]
  end
  MON["Monitoring: logs, statistics, labels"]
  FS -->|"point-in-time join"| SNAP
  TR -.->|"lineage"| META
  EV -->|"new version"| REG
  REG -->|"loads by alias"| BA
  REG -->|"resolves alias"| DEP
  DEP --> ON
  FS -.->|"online features"| ON
  BA --> MON
  ON --> MON
  MON -.->|"retraining trigger"| ORC
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class SRC,VAL,FS,SNAP,ORC,TR,ON,BA,DEP core
  class META,REG neutral
  class EV,MON decision

Trust boundaries. Training data can be a company’s most sensitive data, so the data and training zones have tighter access than serving. Treat the artifact crossing from training to serving as a supply-chain boundary (security).

Choose by four questions: How fast must an answer come? How fresh must features be? Are all entities known in advance? How often will the model change relative to the application? Batch is the cheapest and simplest to monitor; go online only when the decision genuinely happens at request time.

Training runs are batch jobs lasting minutes or days, often on shared or preemptible GPUs. KubernetesOps runs the cluster (node pools, device plugins, scheduling), FinOps for AI decides what GPU hours are worth, and MLOps owns how a training job survives on that capacity:

1. Checkpoint and resume. Save model and optimiser state, step, data position and random state at a fixed interval, and make a restarted job continue from the latest checkpoint. Do it on a schedule, not only on a shutdown signal: AWS gives a two-minute warning before reclaiming a Spot Instance, emitted “on a best effort basis”.^[31] Test resume as you test rollback. For distributed training, PyTorch’s Distributed Checkpoint “handles load-time resharding”, so a job can resume on a different number of GPUs.^[32]
2. Queue jobs against quotas. On shared GPUs a job should wait for capacity, not fail or starve serving. Kueue, for example, “manages quotas and how jobs consume them” and decides when a job waits, starts or is preempted; it supports batch Jobs, JobSets, Ray and Kubeflow’s TrainJob (alpha), as of October 2026.^[33] Rank retraining below serving and above experiments.
3. Read utilisation as a signal. Idle accelerators often mean the job is waiting for data. PyTorch’s DataLoader, for example, loads data in the main process by default (num_workers=0), so training waits for every batch.^[34] Record step time, data-wait time and peak memory with each run.
4. Route failures into the retraining policy. A preempted job resumes; a job that fails twice for the same reason (out of memory, a corrupt shard) stops and alerts the owner. The champion keeps serving, but its freshness clock keeps running.

Figure 5 follows one retraining cycle and its three exception paths. Each step names the identity that performs it, because that separation is a security control (registry permissions).

flowchart TB
  T["Trigger: schedule, new labels or drift alert"] --> V{"Pipeline: input data valid?"}
  V -->|"No"| S["Stop before training, alert the owner"]
  V -->|"Yes"| TR["Pipeline: train, score challenger and champion, log lineage"]
  TR --> G{"Gate passes?"}
  G -->|"No"| GR["Run stays in tracking, reasons to the owner"]
  G -->|"Yes"| RC["Pipeline: register in candidates model as challenger"]
  RC --> AP["Owner approves, promotion identity copies it to production"]
  AP --> CN["Deploy step: resolve alias, canary 10%"]
  CN --> H{"Canary healthy for 48 hours?"}
  H -->|"No"| RB["Deploy step: canary to 0%, open an incident"]
  H -->|"Yes"| CH["Promotion identity: move champion alias"]
  CH --> FL["Deploy step: resolve champion, all traffic"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class TR,RC,CN,FL core
  class S,GR,RB risk
  class V,G,H decision
  class T,AP,CH neutral

Invalid data stops the pipeline before training, the cheapest place to stop; Google Cloud recommends stopping on schema skew and treating value skew as a reason to retrain.^[1]

Practice 1: a reproducible training run. Record the commit, an immutable data reference (a content hash such as DVC’s .dvc metafile,^[35] or a table snapshot ID), the parameters and the environment (lockfile or container digest); scikit-learn advises keeping such metadata next to a saved model.^[11] Set seeds, but define reproducibility as metrics within a tolerance (mental model).

Practice 2: train like you serve. Rules of ML: “Re-use code between your training pipeline and your serving pipeline whenever possible” (#32); log serving features and train on them (#29); remember that a joined table may change in between (#31); and measure skew, because the same example should “give you exactly the same result” in both paths (#37).^[12] The reference implementation’s parity test does that. A feature store serves one definition to both paths and builds training sets with point-in-time joins; by default Feast can still return a value backfilled after the event, and filter_by_created_timestamp=True excludes it.^[26]

Practice 3: gate the challenger. Compare challenger and champion on the same frozen test set, split by time (train on the past, test on a later period) so that temporal leakage cannot flatter the result. Kapoor and Narayanan’s survey found 22 papers from 17 fields reporting leakage-related errors that collectively affected 294 papers.^[27] Google’s guide recommends comparing the two on evaluation metrics and on business KPIs from online experiments.^[2] Check slices (Lab 2), latency, the input schema and any fairness criteria your ModelOps policy requires.

Practice 4: progressive delivery. In order of risk: a shadow deployment gets a copy of the requests, and “Only the responses of the production variant are returned”;^[36] a canary sends a small share of real traffic to the candidate (in KServe, canaryTrafficPercent);^[37] an online controlled experiment measures the business effect. Booking.com reported that “increasing the performance of a model does not necessarily translate into a gain in [business] value” (as quoted by Adrian Colyer).^[38][39]

Practice 5: monitor at three levels. Service: latency, errors, saturation, owned with SRE. Data: schema violations, missing values, shifts in inputs and outputs, skew. Quality: model metrics once labels arrive; Google’s guide describes evaluating effectiveness “in hindsight”.^[2] Until then, use proxies: the prediction distribution; prediction bias, where predicted and observed label distributions should usually match (before labels arrive, compare with the latest matured base rate, and remember that a gap can be prior shift, not model error); and action limits on unusual numbers of real-world actions.^[9]

Practice 6: a retraining policy. Google Cloud lists five triggers: on demand, on a schedule, on new training data, on performance degradation and on a significant distribution change.^[1] Choose the schedule from a staleness test (ML Test Score: know “The impact of model staleness”):^[16] train up to a date and measure how quality falls on each later week. Write down when retraining must not run (template).

Practice 7: rollback and fallback. Roll back by moving a traffic split, or an alias plus a redeploy, to the previous version; never by retraining. Keep a non-ML fallback (a rule, a default, the last batch score) for when the model service itself is down, and agree in advance which of the two on-call uses.

A model stays useful only if you can tell kinds of change apart, which one generic “drift dashboard” cannot do. Every investigation starts with: did my system change, did the data feeding it change, or did the world change?

flowchart TB
  Q0["Quality or proxy signal drops"] --> Q1{"Same features in training and serving?"}
  Q1 -->|"No"| SK["Skew: fix the pipeline, do not retrain"]
  Q1 -->|"Yes"| QU{"Upstream data or its meaning changed?"}
  QU -->|"Yes"| UD["Data defect: fix at source, rescore, do not retrain"]
  QU -->|"No"| DR["The world changed: check all three; if none moved, tune the alert"]
  DR -->|"P(X) moved"| DD["Data drift: retrain if labels allow"]
  DR -->|"P(y) moved"| PS["Prior shift: recalibrate thresholds"]
  DR -->|"Worse, same inputs"| CD["Concept drift: new labels, features or redesign"]
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class Q1,QU decision
  class SK,UD,CD risk
  class DR,DD,PS core
  class Q0 core

The tree rules out the cheapest, most actionable causes first, and checks all three kinds of drift because they can co-occur (P(X) and P(y|X) can change together).^[28]

How an upstream data defect arises (the data, before your pipelines). A source system renames a category, changes units or null handling, or a join starts dropping or duplicating rows. Both pipelines faithfully process the wrong data, so a parity test passes. Google Cloud’s advice to stop the pipeline on schema skew applies,^[1] and the fix belongs with the data’s owner (DataOps).

How skew arises (your system). Two implementations of one feature; joins against tables that change between training and serving;^[12] different handling of missing values or unseen categories; features using information not available at prediction time;^[27] different library versions of preprocessing code.^[11] Skew is a defect, so tests find it: replay logged requests through the offline code, validate the schema at both ends, and compare training and serving statistics, as TFDV does with a per-feature “skew comparator”.^[40]

How drift arises (the world). Lu et al. define concept drift as “unforeseeable changes in the underlying distribution of streaming data over time”: the joint distribution P(X, y) changes, through P(X) alone (“virtual drift”), P(y|X), or both, and it can be sudden, gradual, incremental or reoccurring.^[28] Input monitors see P(X) change; seeing P(y|X) change needs labels or outcome proxies.

The distance measures in common use.

Five rules, this Atlas’s synthesis of the sources above, make these measures useful rather than noisy:

1. Calibrate thresholds on your own history. Compute the measure between pairs of “normal” past windows and alert well above that variation; a vendor default is a starting point, not a property of your data.
2. Compare like with like. Use a reference window from the same season or weekday mix, or every holiday becomes an incident.
3. Weight by consequence. A large shift in an unimportant feature may not matter; a small shift in the top feature may. The prediction distribution summarises what the model actually does.
4. Do not let a dataset-level rule hide one critical column. Evidently, for example, flags dataset drift by default only when the share of drifting columns exceeds 0.5.^[42]
5. Treat label-free performance estimates as estimates. NannyML’s confidence-based performance estimation can estimate classification quality without labels if probabilities are well calibrated, but its documentation states that the method “will not work under concept drift”.^[45]

No standards body defines an MLOps process; what exists is vendor guidance, research rubrics and documentation practices. Legal obligations (for example for high-risk AI systems or bank model risk) are mapped in ModelOps, not here.

In this Atlas’s view, the ML Test Score is the most directly usable for audits. A test earns half a point if run manually with documented results and a full point if automated; the score is the minimum of the four section totals, so strength in one area cannot hide gaps in another. Its authors calibrated the interpretation against internal Google systems: 0 is “More of a research project than a productionized system”, and above 5 is “Exceptional levels of automated testing and monitoring”.^[16]

Pick by job, not by brand. Versions and project status below are as of October 2026 and change often.

Platform choices (managed, assembled or minimal) are compared in the toolkit.

Automate the mechanics: pipeline runs, data validation, gates, packaging, deployment, and automatic rollback on hard service failures. Keep people in four places: changing a threshold or gate; promoting high-risk models (a ModelOps decision); responding to concept drift, which needs judgement about what the target now means; and retraining where the model’s own actions shaped the labels. Sculley et al. describe direct feedback loops (a model influences the selection of its own training data) and hidden feedback loops between systems, and warn about fixed thresholds in dynamic systems.^[9] A nightly “retrain and auto-promote” job can turn each of these into an outage (judgement call).

Generative AI assistants can draft pipeline code and summarise drift reports; treat their output as a draft to review, never as an approval, and keep them away from production credentials. Operating LLM applications is covered in LLMOps.

Design (illustrative). A subscription business predicts which customers will cancel in the next 30 days; labels arrive 30 days later. Scores are produced nightly in batch for the retention team and online for the support tool. Two ML engineers run it on a Kubernetes cluster with KServe, an MLflow tracking server and a DVC remote in object storage. Constraints: the training data is personal data, so the data zone is restricted. A wrong score costs a wasted offer, not a safety incident, so the pipeline runs unattended up to a registered candidate, one human approval starts the canary, and the champion moves after 48 healthy hours. Three identities, which the registry’s permission model requires (security): the training pipeline writes only to churn-classifier-candidates, a promotion identity writes to churn-classifier, and the serving workload reads one storage prefix.

The repository layout keeps policy next to code:

churn/
  params.yaml           # training parameters and the time cutoff for the test split
  gate_policy.json      # promotion thresholds (Lab 2)
  dvc.yaml              # pipeline: validate -> train -> gate -> register
  src/features.py       # ONE feature definition, for training and the online service
  src/validate.py       # schema and range checks; exits non-zero on failure
  src/train.py          # trains, logs lineage, scores challenger and champion
  src/promotion_gate.py # Lab 2
  src/register.py       # runs only after the gate passes
  src/promote.py        # promotion identity: copy to production, move aliases
  src/deploy.py         # renders the InferenceService from an alias; reconciliation
  tests/test_features.py
  monitoring/alerts.yaml

Example status: syntax-checked by the validator, not executed (requires DVC, MLflow, scikit-learn and pandas).

# dvc.yaml - each stage reruns only when its deps or params change (hashes in dvc.lock)
stages:
  validate:
    cmd: python src/validate.py data/raw/customers.csv reports/validation.json
    deps:
      - src/validate.py
      - data/raw/customers.csv
    outs:
      - reports/validation.json:
          cache: false
  train:
    cmd: python src/train.py data/raw/customers.csv reports/candidates.json
    deps:
      - src/train.py
      - src/features.py
      - data/raw/customers.csv
      - reports/validation.json
    params:
      - train.learning_rate
      - train.max_depth
      - train.cutoff_date
    outs:
      - reports/candidates.json:
          cache: false
  gate:
    cmd: >-
      python src/promotion_gate.py gate_policy.json reports/candidates.json
      > reports/gate.txt
    deps:
      - src/promotion_gate.py
      - gate_policy.json
      - reports/candidates.json
    outs:
      - reports/gate.txt:
          cache: false
  register:
    cmd: python src/register.py reports/candidates.json reports/registered.json
    deps:
      - src/register.py
      - reports/candidates.json
      - reports/gate.txt
    outs:
      - reports/registered.json:
          cache: false

The stage syntax follows DVC’s pipeline documentation; DVC records hashes in dvc.lock and re-runs a stage only when a dependency or tracked parameter changes.^[54] A failing validate or gate stops the pipeline, so only a challenger that passed reaches register; rejected runs stay in experiment tracking and never become registry versions.

The single feature definition is the most important file: training and the online service import it, and unseen categories map to one explicit code instead of crashing or becoming a missing value.

Example status: syntax-checked, not executed.

"""One feature definition, imported by the training pipeline AND the online service."""
CONTRACT_TYPES = ("monthly", "one_year", "two_year")
FEATURES = ["tenure_months", "monthly_charges", "contract_code", "support_calls_90d"]


def contract_code(value):
    """Unseen or missing contract types map to one explicit 'other' code."""
    return CONTRACT_TYPES.index(value) if value in CONTRACT_TYPES else len(CONTRACT_TYPES)


def build_features(df):
    out = df.copy()
    out["contract_code"] = out["contract_type"].map(contract_code)
    out["support_calls_90d"] = out["support_calls_90d"].fillna(0).clip(upper=50)
    return out

The parity replay behind Practice 2 and the skew-rate KPI is short; the ML Test Score calls the skew test “Perhaps the most important and least implemented test”.^[16] The online service logs each raw request with the features it computed (Rules of ML #29),^[12] and the test replays a sample through the offline code.

Example status: syntax-checked, not executed (requires pandas and pytest, and a sample of logged requests).

"""Parity replay: offline feature code must reproduce what the online service served."""
import json
import pathlib

import pandas as pd

from features import FEATURES, build_features

# One JSON object per line: {"request": {raw input}, "served_features": {feature values}}
SAMPLE = pathlib.Path("tests/data/serving_log_sample.jsonl")


def test_offline_features_match_served_features():
    rows = [json.loads(line) for line in SAMPLE.read_text().splitlines() if line.strip()]
    raw = pd.DataFrame([r["request"] for r in rows])
    served = pd.DataFrame([r["served_features"] for r in rows])[FEATURES].astype(float)
    offline = build_features(raw)[FEATURES].astype(float)
    differs = ((offline - served).abs() > 1e-9) | (offline.isna() != served.isna())
    skew_rate = differs.any(axis=1).mean()
    assert skew_rate == 0, f"skew rate {skew_rate:.1%} on {len(rows)} replayed requests"


def test_unseen_and_missing_values_have_explicit_codes():
    raw = pd.DataFrame({"contract_type": ["monthly", "weekly", None],
                        "support_calls_90d": [1, None, 99]})
    out = build_features(raw)
    assert out["contract_code"].tolist() == [0, 3, 3]
    assert out["support_calls_90d"].tolist() == [1, 0, 50]

Run it in CI on every change to features.py or the online service, and daily on fresh samples; the share of differing rows is the skew rate.

The training step logs lineage, scores challenger and champion on the same time-based test set and writes the gate’s input; it registers nothing. It uses MLflow 3 APIs: log_model(..., name=...) (artifact_path is deprecated), signatures and aliases.^[55][29][56]

Example status: syntax-checked, not executed (needs an MLflow tracking server and an existing “champion” alias; Lab 3 shows how to create the first one).

"""Train, log lineage, and score challenger and champion on the same test set."""
import hashlib
import json
import statistics
import subprocess
import sys
import time

import mlflow
import pandas as pd
import yaml
from mlflow import MlflowClient
from mlflow.models import get_model_info, infer_signature
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score

from features import FEATURES, build_features

MODEL = "churn-classifier"  # the production model: this job can only read it
# skops >= 0.15 does not trust tree nodes by default; we trust our own artifact
# (if log_model names more untrusted types, review them before adding them)
TRUSTED = ["sklearn.ensemble._hist_gradient_boosting.predictor.TreePredictor"]
MIN_SLICE_ROWS = 200
SLICES = {
    "region=north": lambda d: d["region"] == "north",
    "region=south": lambda d: d["region"] == "south",
    "tenure<6m": lambda d: d["tenure_months"] < 6,
}


def git(*args):
    return subprocess.run(["git", *args], capture_output=True, text=True)


def evaluate(model, test):
    x, y = build_features(test)[FEATURES], test["churned"]
    scores = model.predict_proba(x)[:, 1]
    slices, skipped = {}, []
    for name, f in SLICES.items():
        mask = f(test).to_numpy()
        if mask.sum() < MIN_SLICE_ROWS or y[mask].nunique() < 2:
            skipped.append(name)  # AUC is undefined for one class, noisy for few rows
        else:
            slices[name] = round(roc_auc_score(y[mask], scores[mask]), 3)
    timings = []
    for i in range(200):  # single-row latency, a rough proxy for online serving
        start = time.perf_counter()
        model.predict_proba(x.iloc[[i]])
        timings.append((time.perf_counter() - start) * 1000)
    p95 = statistics.quantiles(timings, n=20)[18]
    return {"metrics": {"auc": round(roc_auc_score(y, scores), 3)}, "slices": slices,
            "skipped_slices": skipped, "p95_latency_ms": round(p95, 1)}


def main(raw_path, report_path):
    params = yaml.safe_load(open("params.yaml"))["train"]
    data = pd.read_csv(raw_path, parse_dates=["snapshot_date"])
    data_md5 = hashlib.md5(open(raw_path, "rb").read()).hexdigest()
    commit = git("rev-parse", "HEAD").stdout.strip()
    clean = git("diff", "--quiet", "HEAD", "--", "src", "params.yaml").returncode == 0
    cutoff = pd.Timestamp(params["cutoff_date"])
    train = data[data["snapshot_date"] < cutoff]
    test = data[data["snapshot_date"] >= cutoff]  # later period: no temporal leakage

    model = HistGradientBoostingClassifier(learning_rate=params["learning_rate"],
                                           max_depth=params["max_depth"], random_state=42)
    mlflow.set_experiment(MODEL)
    with mlflow.start_run() as run:
        mlflow.log_params(params)
        mlflow.set_tags({"data_md5": data_md5, "git_commit": commit})
        x_train = build_features(train)[FEATURES]
        model.fit(x_train, train["churned"])
        signature = infer_signature(x_train, model.predict_proba(x_train)[:, 1])
        challenger = evaluate(model, test)
        mlflow.log_metrics({"auc": challenger["metrics"]["auc"],
                            "p95_latency_ms": challenger["p95_latency_ms"]})
        info = mlflow.sklearn.log_model(model, name="model", signature=signature,
                                        skops_trusted_types=TRUSTED)

    client = MlflowClient()
    champ = client.get_model_version_by_alias(MODEL, "champion")
    champ_uri = f"models:/{MODEL}/{champ.version}"
    champion = evaluate(mlflow.sklearn.load_model(champ_uri), test)
    champ_signature = get_model_info(champ_uri).signature
    champ_commit = client.get_run(champ.run_id).data.tags.get("git_commit", "unknown")
    same_features = git("diff", "--quiet", champ_commit, "HEAD", "--",
                        "src/features.py").returncode == 0
    checks = {  # computed, not asserted
        "data_validation": json.load(open("reports/validation.json"))["passed"],
        "signature_match": champ_signature is not None
        and signature.to_dict() == champ_signature.to_dict(),
        "lineage_recorded": bool(commit) and clean,
    }
    challenger.update(version=f"run-{run.info.run_id[:8]}", checks=checks,
                      model_uri=info.model_uri,
                      lineage={"git_commit": commit, "data_md5": data_md5})
    report = {
        "champion": dict(champion, version=f"v{champ.version}"),
        "challenger": challenger,
        "warnings": [] if same_features else [
            "features.py changed since the champion: re-score it from its own commit"],
    }
    json.dump(report, open(report_path, "w"), indent=2)


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

Three details matter. signature_match and lineage_recorded are computed: the challenger’s input signature must equal the champion’s (callers send the old schema, so a schema change needs a deliberate release), and lineage counts only if src/ matches the recorded commit. Slices with too few rows or one class are skipped and listed. If features.py changed since the champion’s commit, the champion was scored with feature code it never saw in training, so the report warns: re-score it from its own commit (for example with git worktree) before trusting the comparison.

register.py writes to the candidates model with lineage tags; only the promotion identity runs promote.py.

Example status: syntax-checked, not executed.

"""Register a challenger that passed the gate (DVC runs this stage after the gate)."""
import json
import sys

import mlflow
from mlflow import MlflowClient

CANDIDATES = "churn-classifier-candidates"  # the training identity can write only here


def main(candidates_path, out_path):
    challenger = json.load(open(candidates_path))["challenger"]
    tags = dict(challenger["lineage"], gate="passed")
    mv = mlflow.register_model(challenger["model_uri"], CANDIDATES, tags=tags)
    MlflowClient().set_registered_model_alias(CANDIDATES, "challenger", mv.version)
    json.dump({"model": CANDIDATES, "version": mv.version}, open(out_path, "w"))


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

Example status: syntax-checked, not executed.

"""Promotion, run only by the promotion identity after approval.

  python src/promote.py canary    copy the approved candidate into production
  python src/promote.py champion  after a healthy canary, make the challenger the champion
"""
import sys

from mlflow import MlflowClient

PROD, CANDIDATES = "churn-classifier", "churn-classifier-candidates"

client = MlflowClient()
if sys.argv[1:] == ["canary"]:
    mv = client.copy_model_version(f"models:/{CANDIDATES}@challenger", PROD)
    client.set_registered_model_alias(PROD, "challenger", mv.version)
elif sys.argv[1:] == ["champion"]:
    mv = client.get_model_version_by_alias(PROD, "challenger")
    client.set_registered_model_alias(PROD, "champion", mv.version)
else:
    sys.exit(__doc__)
print(f"{PROD} v{mv.version}: alias updated")

Serving has two paths. The batch job loads models:/churn-classifier@champion, so it picks up a new champion on its next run.^[49] The online endpoint is a KServe InferenceService, which serves a storage URI, not an alias, so a deploy step resolves the alias, exports that version’s MLflow model directory to a per-version prefix, records the version in annotations, and later checks that the endpoint still matches the registry.

Example status: syntax-checked; render() and check() exercised on 2026-10-17 with stubbed MLflow, AWS CLI, kubectl and YAML (both check() outcomes reproduced); not run against real services.

"""Render the KServe InferenceService from a registry alias, and check it later.

  python src/deploy.py canary 10   serve @challenger to 10% of traffic
  python src/deploy.py full        serve @champion to all traffic
  python src/deploy.py check       fail if the live endpoint and its alias disagree
"""
import json
import subprocess
import sys
import tempfile

import mlflow
import yaml
from mlflow import MlflowClient

MODEL, NAMESPACE = "churn-classifier", "ml-serving"
PREFIX = "s3://ml-artifacts-example/serving/churn-classifier"  # serving reads only this
VERSION_KEY, ALIAS_KEY = "example.com/registry-version", "example.com/registry-alias"


def export(version):
    """Copy the registered MLflow model directory to a per-version prefix."""
    target = f"{PREFIX}/v{version}"
    with tempfile.TemporaryDirectory() as tmp:
        uri = f"models:/{MODEL}/{version}"
        local = mlflow.artifacts.download_artifacts(uri, dst_path=tmp)
        subprocess.run(["aws", "s3", "sync", local, target], check=True)
    return target


def render(mode, percent="0"):
    alias = "challenger" if mode == "canary" else "champion"
    version = MlflowClient().get_model_version_by_alias(MODEL, alias).version
    predictor = {"serviceAccountName": "churn-model-reader",
                 "model": {"modelFormat": {"name": "mlflow"}, "protocolVersion": "v2",
                           "storageUri": export(version),
                           "resources": {"requests": {"cpu": "500m", "memory": "1Gi"},
                                         "limits": {"cpu": "1", "memory": "2Gi"}}}}
    if mode == "canary":
        predictor = {"canaryTrafficPercent": int(percent), **predictor}
    metadata = {"name": MODEL, "namespace": NAMESPACE,
                "annotations": {VERSION_KEY: str(version), ALIAS_KEY: alias}}
    return {"apiVersion": "serving.kserve.io/v1beta1", "kind": "InferenceService",
            "metadata": metadata, "spec": {"predictor": predictor}}


def check():
    cmd = ["kubectl", "get", "isvc", MODEL, "-n", NAMESPACE, "-o", "json"]
    live = subprocess.run(cmd, capture_output=True, text=True, check=True).stdout
    notes = json.loads(live)["metadata"]["annotations"]
    served, alias = notes[VERSION_KEY], notes[ALIAS_KEY]
    expected = MlflowClient().get_model_version_by_alias(MODEL, alias).version
    if served != str(expected):
        sys.exit(f"endpoint serves v{served}; @{alias} is v{expected}")
    print("endpoint matches the registry")


if __name__ == "__main__":
    if sys.argv[1] == "check":
        check()
    else:
        print(yaml.safe_dump(render(*sys.argv[1:3]), sort_keys=False))

For version 13, deploy.py canary 10 writes the manifest below. modelFormat: mlflow with the v2 protocol follows KServe’s MLflow example;^[57] the predictor’s service account carries the storage credential,^[58] which should be read-only and limited to the serving prefix. Check that the runtime’s MLflow version can load the model’s format. KServe supports canaries only in serverless mode.^[37]

Example status: YAML syntax-checked; equal (keys and values) to render("canary", "10") in the stubbed run; PyYAML’s exact quoting not checked.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: churn-classifier
  namespace: ml-serving
  annotations:
    example.com/registry-version: '13'
    example.com/registry-alias: challenger
spec:
  predictor:
    canaryTrafficPercent: 10
    serviceAccountName: churn-model-reader
    model:
      modelFormat:
        name: mlflow
      protocolVersion: v2
      storageUri: s3://ml-artifacts-example/serving/churn-classifier/v13
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: '1'
          memory: 2Gi

Example status: syntax-checked, not executed.

dvc repro      # validate -> train -> gate -> register; stops at the first failing stage
dvc push       # upload data and pipeline outputs to the DVC remote
git add dvc.lock && git commit -m "Retrain churn-classifier"  # code committed before
# After approval, as the promotion identity and the deploy step:
python src/promote.py canary
python src/deploy.py canary 10 > deploy/isvc.yaml   # or commit it for a GitOps controller
kubectl apply -f deploy/isvc.yaml
kubectl get isvc churn-classifier -n ml-serving     # PREV and LATEST columns: 90/10
# Healthy after 48 h: promote.py champion, then deploy.py full.
# Regressing: deploy.py canary 0.
python src/deploy.py check                          # on a schedule

These are KServe’s documented canary controls: remove canaryTrafficPercent to promote, set it to 0 to roll back.^[37] After promotion, roll back by pointing champion at the previous version and running deploy.py full. The scheduled check catches an alias moved without a redeploy, or a hand-edited manifest. Signing is not shown; if you add it, sign in register and verify in export() before the upload.

Monitoring uses two custom metrics (names chosen for this example): a counter in the online service, model_schema_violations_total, and model_feature_psi, which the nightly statistics job pushes to a Prometheus Pushgateway. That is the documented use, “capturing the outcome of a service-level batch job”, but the Pushgateway “never forgets series pushed to it”, so a stopped job would leave its last value firing;^[59] the stale-data rule watches push_time_seconds, which it adds to every group.^[60] A for clause keeps an alert pending until its condition has held for the stated time;^[61] on a nightly value it only delays the alert, so the drift rule has none, and “two nights in a row” lives in the retraining policy.

Example status: YAML syntax-checked, not executed; the 0.2 threshold is a placeholder to be calibrated on your own history.

groups:
  - name: churn-classifier-model
    rules:
      - alert: ChurnFeatureDrift
        expr: max by (feature) (model_feature_psi{model="churn-classifier"}) > 0.2
        labels:
          severity: ticket
        annotations:
          summary: "Input drift on {{ $labels.feature }} for churn-classifier"
          runbook: "https://wiki.example.com/runbooks/churn-classifier-drift"
      - alert: ChurnStatsStale
        expr: time() - max(push_time_seconds{job="churn-stats"}) > 26 * 3600
        labels:
          severity: ticket
        annotations:
          summary: "The churn statistics job has not pushed for 26 hours"
      - alert: ChurnSchemaViolations
        expr: sum(rate(model_schema_violations_total{model="churn-classifier"}[15m])) > 0
        for: 15m
        labels:
          severity: page
        annotations:
          summary: "Serving requests violate the churn-classifier input schema"

Schema violations page once their rate has stayed above zero for 15 minutes, because they usually mean skew or an upstream data defect; drift opens a ticket because it needs analysis, not a 3 a.m. response.

Labs 1 and 2 need only Python 3.8+ and the standard library, cost nothing and were executed for this page; Lab 3 needs third-party packages, and its pipeline was not executed here.

• Objective: compute three distance measures for three features, see what they can and cannot tell you, and judge which change looks like the world and which like the data feeding your pipeline.
• Prerequisites: Python 3.8+, a terminal, 30 minutes. Cost and risk: none (illustrative data).

Steps

1. Create an empty folder and save the binned counts below as drift_input.json. Each list holds the number of rows per bin in the training reference and in one week of serving logs.

Example status: JSON syntax-checked; this exact file was used for the executed run below.

{
  "model": "churn-classifier",
  "reference_window": "training snapshot 2026-08",
  "current_window": "serving log 2026-10-01..2026-10-07",
  "features": {
    "tenure_months": {
      "type": "numeric",
      "bins": ["<3", "3-6", "6-12", "12-24", "24-36", ">36"],
      "reference": [1200, 1300, 1800, 2200, 1700, 1800],
      "current":   [2100, 1700, 1900, 1900, 1200, 1200]
    },
    "monthly_charges": {
      "type": "numeric",
      "bins": ["<20", "20-40", "40-60", "60-80", "80-100", ">100"],
      "reference": [900, 1600, 2100, 2300, 1900, 1200],
      "current":   [880, 1650, 2050, 2350, 1870, 1200]
    },
    "contract_type": {
      "type": "categorical",
      "bins": ["monthly", "one_year", "two_year", "unknown"],
      "reference": [5200, 2600, 2200, 0],
      "current":   [5000, 2500, 2100, 400]
    }
  },
  "thresholds": {"psi": 0.2, "js_distance": 0.1, "linf": 0.03, "min_rows": 1000}
}

2. Save the script as drift_check.py.

Example status: executed on 2026-10-16 with Python 3.8.10; syntax-checked by the validator.

"""Compare a serving window with the training reference, feature by feature.

Standard library only (Python 3.8+). Input: binned counts in drift_input.json.
Metrics: PSI, Jensen-Shannon distance (base 2) and L-infinity distance.
"""
import json
import math
import sys

EPS = 1e-4  # replaces empty-bin proportions so that log() stays finite


def proportions(counts):
    total = sum(counts)
    return [max(c / total, EPS) for c in counts], total


def psi(ref, cur):
    return sum((a - e) * math.log(a / e) for e, a in zip(ref, cur))


def js_distance(p, q):
    m = [(x + y) / 2 for x, y in zip(p, q)]
    kl = lambda a, b: sum(x * math.log2(x / y) for x, y in zip(a, b) if x > 0)
    return math.sqrt(0.5 * kl(p, m) + 0.5 * kl(q, m))


def linf(p, q):
    return max(abs(x - y) for x, y in zip(p, q))


def main(path):
    cfg = json.load(open(path))
    t = cfg["thresholds"]
    flagged = []
    print(f"{'feature':<16}{'rows':>7}{'PSI':>8}{'JS':>8}{'Linf':>8}  verdict")
    for name, f in cfg["features"].items():
        ref, _ = proportions(f["reference"])
        cur, n = proportions(f["current"])
        v_psi, v_js, v_linf = psi(ref, cur), js_distance(ref, cur), linf(ref, cur)
        reasons = []
        if n < t["min_rows"]:
            reasons.append("too few rows to judge")
        if v_psi >= t["psi"]:
            reasons.append("PSI")
        if v_js >= t["js_distance"]:
            reasons.append("JS")
        if f["type"] == "categorical" and v_linf >= t["linf"]:
            reasons.append("Linf")
        verdict = "DRIFT (" + ", ".join(reasons) + ")" if reasons else "ok"
        if reasons:
            flagged.append(name)
        print(f"{name:<16}{n:>7}{v_psi:>8.3f}{v_js:>8.3f}{v_linf:>8.3f}  {verdict}")
    print("flagged:", ", ".join(flagged) if flagged else "none")
    return 1 if flagged else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "drift_input.json"))

3. Run python3 drift_check.py drift_input.json; echo "exit=$?".

Expected output (verbatim from the executed run):

feature            rows     PSI      JS    Linf  verdict
tenure_months     10000   0.108   0.139   0.090  DRIFT (JS)
monthly_charges   10000   0.000   0.009   0.005  ok
contract_type     10000   0.241   0.141   0.040  DRIFT (PSI, JS, Linf)
flagged: tenure_months, contract_type
exit=1

4. Interpret. tenure_months shifted towards new customers: the “<3” bin grew from 12% to 21%. That looks like drift (for example, a sign-up campaign). PSI (0.108) and Jensen-Shannon (0.139) disagree about whether it crosses the line, so thresholds must be calibrated per measure. contract_type gained a category, unknown, never seen in training. A category that appears overnight is more often an upstream data defect (null handling, a join) than customer behaviour: ask the data owner before retraining, or the model will learn that “unknown” is a real contract type.
   • Do not let PSI decide this one. All of contract_type’s PSI comes from the empty reference bin, which the script floors at EPS. With EPS at 1e-3, 1e-4, 1e-5 and 1e-6 the PSI was 0.146, 0.241, 0.333 and 0.425 (at 1e-3 the PSI rule no longer fires), while Jensen-Shannon stayed between 0.132 and 0.142. Catch new categories with a schema or domain check, as TFDV and the runbook do.
   • No distance measure tells drift from skew or a data defect; that takes the parity replay and a look upstream.

• Validation: check one number by hand. For tenure_months, the six PSI terms are 0.0504, 0.0107, 0.0005, 0.0044, 0.0174 and 0.0243, which sum to 0.108. The exit code is 1, so a scheduler would mark the job as failed.
• Troubleshooting: ZeroDivisionError means a feature has no rows in a window; the min_rows check exists because measures on small samples swing widely. If you change the bins, every number changes: PSI depends on the number of bins and the sample sizes.^[43]
• Teardown: delete the folder. Nothing else was created.

• Objective: apply a written promotion policy to a champion and a challenger, and see why an average uplift is not enough.
• Prerequisites: Python 3.8+, 20 minutes, the sed command (Linux or macOS). Cost and risk: none.

Steps

1. Save the policy as gate_policy.json and the evaluation results as candidates.json. Both models were scored on the same frozen test set upstream; the gate only applies the policy.

Example status: JSON syntax-checked; these exact files were used for the executed run.

{
  "primary_metric": "auc",
  "min_uplift": 0.005,
  "max_slice_drop": 0.02,
  "p95_latency_ms_max": 50,
  "required_checks": ["data_validation", "signature_match", "lineage_recorded"]
}

{
  "champion": {
    "version": "v12",
    "metrics": {"auc": 0.842},
    "slices": {"region=north": 0.851, "region=south": 0.829, "tenure<6m": 0.781},
    "p95_latency_ms": 31
  },
  "challenger": {
    "version": "run-7f3a9c21",
    "metrics": {"auc": 0.851},
    "slices": {"region=north": 0.866, "region=south": 0.842, "tenure<6m": 0.752},
    "p95_latency_ms": 34,
    "checks": {"data_validation": true, "signature_match": true, "lineage_recorded": true}
  }
}

2. Save the gate as promotion_gate.py.

Example status: executed with Python 3.8.10 (first edition on 2026-10-16; this revised version on 2026-10-17); syntax-checked by the validator.

"""Decide whether a challenger model may replace the champion.

Standard library only (Python 3.8+). Evaluation results are produced upstream
on the same frozen test set for both models; this script only applies policy.
Exit code 0 = promote, 1 = reject (so a CI job can stop the pipeline).
"""
import json
import sys


def decide(policy, champion, challenger):
    reasons = []
    for check in policy["required_checks"]:
        if not challenger["checks"].get(check, False):
            reasons.append(f"required check failed: {check}")
    m = policy["primary_metric"]
    uplift = challenger["metrics"][m] - champion["metrics"][m]
    if uplift < policy["min_uplift"]:
        reasons.append(f"{m} uplift {uplift:+.3f} < {policy['min_uplift']:+.3f}")
    for name, champ_value in champion["slices"].items():
        drop = champ_value - challenger["slices"].get(name, 0.0)
        if drop > policy["max_slice_drop"]:
            reasons.append(f"slice {name}: {m} drops by {drop:.3f}")
    if challenger["p95_latency_ms"] > policy["p95_latency_ms_max"]:
        reasons.append(f"p95 latency {challenger['p95_latency_ms']} ms over budget")
    return uplift, reasons


def main(policy_path, candidates_path):
    policy = json.load(open(policy_path))
    c = json.load(open(candidates_path))
    uplift, reasons = decide(policy, c["champion"], c["challenger"])
    champ, chall = c["champion"]["version"], c["challenger"]["version"]
    print(f"champion {champ} vs challenger {chall}: "
          f"{policy['primary_metric']} uplift {uplift:+.3f}")
    if reasons:
        print("REJECT")
        for r in reasons:
            print(" -", r)
        return 1
    print("PROMOTE: register as challenger; canary after approval; champion kept")
    return 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:3]))

3. Run the gate, then simulate a retrained challenger that fixed the new-customer slice, and run it again:

Example status: executed on 2026-10-17 (GNU sed 4.7).

python3 promotion_gate.py gate_policy.json candidates.json; echo "exit=$?"
sed -i 's/"tenure<6m": 0.752/"tenure<6m": 0.779/' candidates.json
python3 promotion_gate.py gate_policy.json candidates.json; echo "exit=$?"

Expected output (verbatim):

champion v12 vs challenger run-7f3a9c21: auc uplift +0.009
REJECT
 - slice tenure<6m: auc drops by 0.029
exit=1
champion v12 vs challenger run-7f3a9c21: auc uplift +0.009
PROMOTE: register as challenger; canary after approval; champion kept
exit=0

• What it teaches: the challenger is better on average (+0.009 AUC) and in both regions, yet worse for customers in their first six months, the group the retention team targets. An average-only gate would have shipped it.
• Validation: exit code 1 then 0; in CI, the first run must fail the job.
• Troubleshooting: KeyError means a slice or check name differs between the two files; on macOS, use sed -i ''. A slice with few examples gives noisy AUC: require a minimum count per slice, as train.py does.
• Teardown: delete the folder.

• Objective: run the reference pipeline locally, register a candidate only when the gate passes, and promote it by alias.
• Prerequisites: Python 3.11 or later (scikit-learn 1.9 requires it; MLflow 3.17 requires 3.10, as of October 2026), Git, about 1 GB of disk for packages, 60–90 minutes. Cost and risk: no cloud resources; everything runs on 127.0.0.1. Status: the pipeline was not executed for this page; expected results follow the tools’ documentation.

Steps

Three files the lab needs are not shown above: params.yaml, src/validate.py, and make_data.py, which writes 12,000 synthetic customers with a change planted in April 2026 (a rival’s offer to monthly customers paying over 90).

Example status: make_data.py and validate.py executed on 2026-10-17 with Python 3.8.10: 12,000 rows, validation passed (exit 0); with one churned value set to 2 it failed (exit 1). params.yaml parsed with PyYAML.

# params.yaml - DVC re-runs `train` when one of these values changes
train:
  learning_rate: 0.1
  max_depth: 4
  cutoff_date: "2026-04-01"  # rows from this date on form the test set

"""Synthetic customers for Lab 3: 12 monthly snapshots of 1,000 rows each."""
import csv
import math
import pathlib
import random

CONTRACTS, WEIGHTS = ["monthly", "one_year", "two_year"], [6, 3, 1]
random.seed(7)
out = pathlib.Path("data/raw/customers.csv")
out.parent.mkdir(parents=True, exist_ok=True)
with out.open("w", newline="") as f:
    w = csv.writer(f, lineterminator="\n")
    w.writerow(["snapshot_date", "region", "tenure_months", "monthly_charges",
                "contract_type", "support_calls_90d", "churned"])
    for i in range(12):  # 2025-10-01 ... 2026-09-01
        year, month = 2025 + (9 + i) // 12, (9 + i) % 12 + 1
        for _ in range(1000):
            tenure = min(int(random.expovariate(1 / 24)), 72)
            contract = random.choices(CONTRACTS, WEIGHTS)[0]
            charges = round(random.uniform(20, 120), 2)
            calls = min(int(random.expovariate(1 / 1.5)), 50)
            z = (-2.5 + 1.0 * (contract == "monthly") + 0.02 * (charges - 70)
                 + 0.3 * calls - 0.04 * tenure)
            rival = (year, month) >= (2026, 4) and contract == "monthly"
            if rival and charges > 90:
                z += 1.5  # planted change: a rival's offer from April 2026
            churned = int(random.random() < 1 / (1 + math.exp(-z)))
            region = random.choice(["north", "south"])
            w.writerow([f"{year}-{month:02d}-01", region, tenure, charges,
                        contract, calls, churned])
print(f"wrote {out}")

"""Schema and range checks on the raw CSV; exits 1 if any check fails."""
import csv
import json
import pathlib
import sys

RULES = {  # column -> check on the raw string value
    "snapshot_date": lambda v: len(v) == 10 and v[4] == v[7] == "-",
    "region": lambda v: v in ("north", "south"),
    "tenure_months": lambda v: 0 <= int(v) <= 600,
    "monthly_charges": lambda v: 0 < float(v) < 1000,
    "contract_type": lambda v: v != "",  # unseen types are mapped, not rejected
    "support_calls_90d": lambda v: v == "" or int(v) >= 0,  # empty -> 0 later
    "churned": lambda v: v in ("0", "1"),
}


def main(csv_path, report_path):
    errors, rows = [], 0
    with open(csv_path, newline="") as f:
        reader = csv.DictReader(f)
        missing = sorted(set(RULES) - set(reader.fieldnames or []))
        if missing:
            errors.append(f"missing columns: {missing}")
        for rows, row in enumerate(reader, start=1):
            for col, ok in RULES.items():
                try:
                    good = ok(row.get(col) or "")
                except (ValueError, IndexError):
                    good = False
                if not good:
                    errors.append(f"row {rows}: {col}={row.get(col)!r}")
    passed = not errors and rows > 0
    pathlib.Path(report_path).parent.mkdir(parents=True, exist_ok=True)
    report = {"passed": passed, "rows": rows,
              "error_count": len(errors), "errors": errors[:10]}
    json.dump(report, open(report_path, "w"), indent=2)
    print(f"validation {'passed' if passed else 'FAILED'}: {rows} rows, "
          f"{len(errors)} errors")
    return 0 if passed else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Example status: syntax-checked, not executed.

python3 -m venv .venv && . .venv/bin/activate
pip install "mlflow>=3.17,<4" "dvc>=3.67,<4" "scikit-learn>=1.9,<1.10" \
  pandas pyyaml pytest
git init churn && cd churn && dvc init
# No authentication: this server must stay on 127.0.0.1 and never be exposed.
mlflow server --backend-store-uri sqlite:///mlflow.db \
  --default-artifact-root ./mlartifacts --host 127.0.0.1 --port 5000 &
export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
printf 'mlflow.db\nmlartifacts/\n' >> .gitignore   # keep the tracking store out of Git
# Copy src/, tests/, gate_policy.json and dvc.yaml from the reference
# implementation, save the three files above (validate.py in src/), then:
python make_data.py
dvc add data/raw/customers.csv && git add -A && git commit -m "Baseline"
dvc repro

1. The first dvc repro stops in train, because no champion exists yet. Register the model that run logged as version 1 of churn-classifier (in the MLflow UI, or with mlflow.register_model), give it the alias champion, and run dvc repro again. Expect REJECT (uplift +0.000): the challenger is retrained from the same data, parameters and seed.
2. Set train.cutoff_date to "2026-07-01", commit it (uncommitted code or parameters fail lineage_recorded), and run dvc repro: only the stages that depend on that parameter re-run. The first real comparison: the challenger has seen three months of the planted change, the champion none.
3. If the gate prints PROMOTE, run python src/promote.py canary and then python src/promote.py champion (locally you play the promotion identity and skip KServe). If it prints REJECT, read the reasons and stop.

• Expected output on PROMOTE: all four stages succeed; churn-classifier-candidates has a version tagged gate=passed, git_commit and data_md5, aliased challenger; after both promote.py calls, a new churn-classifier version holds champion; dvc status reports nothing changed.
• Expected output on REJECT: dvc repro exits non-zero at gate, reports/gate.txt holds the reasons, no registry version is created, and dvc status lists gate as changed.
• Validation: loading models:/churn-classifier@champion returns the version the alias points to; moving the alias changes what the next load returns.^[49] PYTHONPATH=src pytest -q -k unseen tests/ passes (the parity test also needs a sample of logged requests).
• Troubleshooting: a gate stage that “succeeds” with a REJECT usually had its output piped (for example through tee), so the stage reports the last command’s exit code; keep the plain redirect, or use set -o pipefail. If registry calls fail, check MLFLOW_TRACKING_URI. On a shared server, use --app-name basic-auth with separate training and promotion users.^[62]
• Teardown: stop the server (kill %1), deactivate, delete the folder and the virtual environment.

Read together (an editorial synthesis, not a finding of the sources): the platforms that worked invested in the plumbing between data, model and outcome, and the painful cases involved a world that changed faster than feedback reached the models’ owners.

Measure the system that produces and runs models, not only the models. Service metrics are defined in SRE and cost per prediction in FinOps for AI; the table lists model-specific measures.

Leading and lagging. Skew, input drift, prediction distribution and freshness are leading signals: available now, weakly related to harm. Label-based quality and business outcomes are lagging: decisive, but late. Show both, labelled. Vanity metrics: models in production, retraining runs, and offline gains never confirmed online.

• Model files can execute code. Python’s documentation states that “The pickle module is not secure” and that malicious data can “execute arbitrary code during unpickling”.^[66] scikit-learn adds: “You should never load a pickle file from an untrusted source”. Its safer options have conditions: skops.io loads only types trusted by default or by you (review what get_untrusted_types returns), and because “arbitrary computations can be represented using the ONNX format”, ONNX models should be served in a sandbox.^[11] Safetensors is a safe alternative to pickle for tensors.^[67] Load third-party models only in such formats, under those conditions, or in an isolated environment. MLflow 3 logs scikit-learn models as skops except on Databricks, but autolog() defaults to cloudpickle; other types must be listed in skops_trusted_types when the model is logged.^[55] Since skops 0.15 these include the node structures of tree models, which a crafted file can use to crash predict().^[68]
• Treat the registry as a deployment control, and check what it can enforce. Moving champion is a deployment: restrict it like production deploy rights and log every change. Open-source MLflow has no authentication unless started with --app-name basic-auth; then the default permission is READ, and “Create Model Version” and “Set Registered Model Alias” both need can_update on the registered model,^[62] so whoever can register versions can move that model’s aliases. MLflow’s answer is separate registered models per environment:^[49] in the reference implementation the training pipeline writes only to a candidates model, and a promotion identity copies approved versions into production with copy_model_version and moves aliases there.^[56] Serving gets a read-only credential for its prefix.^[58]
• Sign what you serve. Sign artifacts at registration and verify before serving; Sigstore’s model-transparency signs a model directory and verifies file hashes. With keyless signing, each signing event is recorded in Sigstore’s append-only transparency log, which signers are told to monitor for their identity; if that is unacceptable for a private model, use a private Sigstore instance or key-based signing.^[69] Build-time supply-chain controls belong to DevSecOps.
• Training data is the crown jewel. Keep the data zone separate, grant access per project, and treat prediction logs that store served features like the source tables. The legal basis for training on personal data belongs to PrivacyOps.
• Poisoned or self-made labels. Validate new training data against expectations before training, and be wary of labels that the model’s own actions produced (feedback loops).
• Evidence. Lineage, gate reports, approvals, model cards (intended use and evaluation, including across groups)^[20] and datasheets (a dataset’s motivation, composition and collection)^[46] are what an auditor will ask for. Mapping them to regulations belongs to ModelOps; collecting evidence across systems to ComplianceOps.

Cost drivers: training compute (runs × cost per run), serving (predictions × cost per prediction; does a small tabular model need a GPU at all?), storage (snapshots, features, artifacts, prediction logs) and people (reviews, on-call, labelling). The ML team controls three levers: serving pattern (batch avoids always-on capacity), retraining frequency and model size.

A simple rule for retraining frequency, proposed by this Atlas: retrain more often only while the value recovered by fresher models exceeds the cost of the extra runs and reviews. Express both sides in the same unit.

Unit economics and commitments belong to FinOps for AI, GPU cluster capacity to KubernetesOps, and carbon accounting to GreenOps; retraining frequency and model size are also energy levers.

How do you know the practice itself works, not just one model?

• Quarterly scores and audits. Score each production model with the ML Test Score and track the minimum section score;^[16] rebuild one randomly chosen model from its lineage and check its metrics land within tolerance.
• Registry hygiene. The reconciliation check passes for every endpoint; every model lists its owner and consumers.
• Monitor the monitors. List incidents that no alert preceded; each one is a missing monitor.
• Rehearse. Each quarter, run a rollback drill and a “game day” that injects a schema change in staging.

flowchart TB
  A["Drift alert on one feature"] --> B["Pause continuous training"]
  B --> C{"Parity replay: same features in both paths?"}
  C -->|"No: skew"| D["Fix the code path"]
  C -->|"Yes"| U{"Upstream data or its meaning changed?"}
  U -->|"Yes: data defect"| F["Fix upstream or guard in serving"]
  U -->|"No: drift"| E["Estimate impact with labels or proxies"]
  D --> G["Rescore affected entities"]
  F --> G
  E --> H{"Harmful?"}
  H -->|"Yes"| I["Retrain on valid data, canary, promote"]
  H -->|"No"| J["Recalibrate the alert"]
  G --> K["Verify, quarantine bad days, resume training"]
  I --> K
  J --> K
  K --> L["Post-incident review and new validation rule"]
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class A risk
  class C,U,H decision
  class B,D,E,F,G,I,J,K,L core

The response, step by step (illustrative):

1. Contain (06:15). Pause the retraining trigger: a second night above the threshold would otherwise start a retraining run on the defect (retraining policy). For the 08:00 offers, use yesterday’s scores for customers whose contract_type is unknown (the agreed fallback) and today’s scores for everyone else.
2. Classify (06:30). The parity replay passes: both paths apply the same features.py, so this is not skew. But 4% of serving rows now carry unknown, a value absent from the training reference. Upstream change log: on Sunday evening a changed join in the CRM export stopped matching some contract records, so those customers get unknown instead of their real contract type. Verdict: an upstream data defect, not skew and not customer behaviour; retraining would teach the model the defect.
3. Remediate (by 11:00). The CRM team reverts the join; the data team backfills the affected rows; the ML engineer rescores the 400 affected customers, replacing the fallback scores.
4. Verify (next nightly run). contract_type PSI near zero, parity replay passing, prediction distribution back to normal.
5. Resume. Mark Sunday and Monday as quarantined in the training-data snapshot, then resume the trigger.
6. Learn. Add a validation rule at ingestion and at serving (“share of unknown ≤ 0.5%”), agree a data contract with the CRM export owner, and record the incident in the model’s history.

Several anti-patterns are the practices above in reverse: the notebook as the production system (Practice 1), gates on the average (Practice 3 and Lab 2), nightly retrain and auto-promote (gates, canaries and pause rules; see the judgement call) and one drift dashboard with default thresholds, where everything is red, so nothing is. Four deserve a closer look:

• Platform before purpose. A year spent building a platform before any model has an owner and a metric. Rules of ML: keep the first model simple and get the infrastructure right.^[12] Instead: build the platform around the first two real models.
• Registry as a file share. Services load s3://…/model_final_v2.pkl, and nobody can say which version is live. Instead: load by alias, or let a deploy step derive the path from an alias and check it on a schedule.
• Judging a model before its labels mature. A 30-day label judged after 7 days measures the label pipeline, not the model.
• Myth: “More frequent retraining is always better.” It costs money and review time and can amplify feedback loops; measure staleness first.

Invest in MLOps when a model serves repeated decisions in production; its world changes (prices, behaviour, fraud, sensors); several models or people depend on the same data and infrastructure; errors are costly or regulated; or the model must outlive the person who built it.

Use little or none of it when a rule performs nearly as well (Rules of ML #1);^[12] the model supports a one-off analysis or an annual report; or no label will ever tell you whether it was right (then question ML itself). Even a prototype deserves versioned code and data, one tracked run and a registered artifact.

Hidden responsibilities: on-call for models; a label pipeline, often the hardest part; retention for snapshots and logs; documentation; retiring models. Alternatives: rules; managed AutoML; or a model bought as a service, whose outputs and approval you still own (ModelOps, and LLMOps for LLM providers).

flowchart TB
  A{"Does a simple rule do nearly as well?"} -->|"Yes"| R["Use the rule; revisit later"]
  A -->|"No"| B{"Repeated decisions in production?"}
  B -->|"No"| C["One-off analysis: version code and data only"]
  B -->|"Yes"| D{"Does its world change faster than you can retrain by hand?"}
  D -->|"No"| E["Level 1: reproducible runs, registry, manual retrain"]
  D -->|"Yes"| F{"Do labels arrive within that period?"}
  F -->|"No"| H["Monitor proxies, schedule retrains, invest in labels"]
  F -->|"Yes"| G["Level 2 to 3: CT pipeline with gates and canary"]
  G --> I{"Several models or teams?"}
  H --> I
  I -->|"Yes"| J["Shared platform: registry, features, serving"]
  I -->|"No"| K["Per-team stack on managed services"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A,B,D,F,I decision
  class E,G,H,J,K core
  class R,C neutral

The tree assumes that labels eventually exist, that a platform pays off only across several models, and that errors cost moderately. For high-stakes decisions, the ModelOps risk tier sets minimum gates regardless of change rate.

This is this Atlas’s model, not an official one, mapped approximately to Google’s levels 0–2 and Microsoft’s levels 0–4;^[1][5] Microsoft notes that organisations often show traits of several levels at once. Not every model should reach level 4.

Production readiness (before a model first serves traffic)

• A named owner, a success metric, guardrail metrics and a non-ML baseline exist.
• Training is reproducible from recorded lineage (commit, data hash, parameters, environment).
• Training and serving import one feature definition; a parity test passes on replayed requests.
• The test split respects time and was reviewed for leakage.
• The model is registered with a signature; serving follows an alias, and the reconciliation check passes.
• Reference statistics are stored; drift, skew and service alerts have owners.
• The label pipeline and its delay are documented.
• Rollback and the non-ML fallback have been rehearsed.

Promotion preflight (each challenger)

• Same frozen test set as the champion; slices and latency checked; required checks pass.
• Training data passed validation and contains no quarantined days.
• Shadow or canary plan, duration and abort criteria are written.
• Approvals required by the model’s risk tier are recorded in the registry; only the promotion identity can move aliases.

Weekly model health review

• Freshness within policy; skew rate at zero or explained.
• Drift alerts reviewed; alert precision updated.
• Live quality on matured labels compared with the gate’s estimate.
• Open incidents and paused pipelines have owners and dates.

Retirement

• Consumers identified from access logs and notified; replacement or fallback agreed.
• Endpoint and batch jobs removed; aliases deleted; artifacts and lineage retained per policy.

1. Model release record, stored with the registry version and reviewed at promotion.

Example status: YAML syntax-checked; fields are this Atlas’s proposal.

model: churn-classifier
version: "13"                       # production model; candidate run-7f3a9c21
owner: retention-ml@example.com
decision_supported: "Which customers receive a retention offer this week"
lineage:
  code_commit: "<git-sha>"
  data_snapshot: "data/raw/customers.csv md5 <hash>"
  environment: "registry.example.com/churn-train@sha256:<digest>"
evaluation:
  test_window: "2026-08-01..2026-08-31"
  auc: 0.851
  champion_auc: 0.842
  slices: {"region=north": 0.866, "region=south": 0.842, "tenure<6m": 0.779}
gates_passed: [data_validation, signature_match, lineage_recorded, slice_check, latency]
release:
  method: canary
  traffic_percent: 10
  duration_hours: 48
  abort_if: "AUC proxy or error rate worse than the champion"
risk_tier: "medium (ModelOps)"
approvals: [{role: product-owner, date: "2026-10-14"}]
consumers: [retention-batch, support-tool]
rollback:
  during_canary: "python src/deploy.py canary 0"
  after_promotion: "point alias champion at v12, then python src/deploy.py full"

2. Retraining policy

Example status: YAML syntax-checked; thresholds are placeholders to calibrate.

model: churn-classifier
freshness_max_days: 35          # from the staleness test
triggers:
  - schedule: weekly            # Wednesday 02:00 UTC
  - new_labels_min_rows: 20000
  - drift: {measure: psi, feature_any_above: 0.2, consecutive_nights: 2}
do_not_retrain_when:
  - data_validation_failed
  - open_data_incident
  - labels_generated_by_model_actions_above_share: 0.5
promotion: {gate: gate_policy.json, human_approval: true}
review: monthly

3. Architecture decision record: serving pattern

Title: Nightly batch scores, plus an online lookup of the latest batch score
Context: offers go out daily; support agents need a score during calls;
  features change daily.
Options: (a) batch only; (b) online model service;
  (c) batch scores served online from a key-value store.
Decision: (c). Key-value lookup latency; one model run per night; one monitoring path.
Consequences: scores are up to 24 hours old; new customers get a default until
  the next run.
Revisit when: offers become real-time, or same-day behaviour changes the score materially.

• Online and continual learning (established in research, rarer in production): models that update with each event remove retraining delay but make governance harder; the CD Foundation’s 2024 roadmap says such models sit “outside the governance constraints of current MLOps platforms”.^[4]
• Feedback loops and exploration (established problem, active research): when a model chooses what it later learns from, plain supervised retraining is biased. Sculley et al. note that bandit algorithms are the theoretically correct answer but may not scale to large action spaces.^[9] A practical mitigation (Atlas suggestion): keep a small randomised share of decisions and log the probability with which each action was chosen, so that later training can correct for the model’s own influence.
• Convergence with LLM tooling (vendor direction, not a settled practice): MLflow presents itself on PyPI as a platform for agents and LLMs as well as traditional models;^[48] KServe calls itself a “Generative and Predictive AI Inference Platform”;^[22] Microsoft says GenAIOps complements its MLOps levels rather than replacing them.^[5]

• Prerequisites: Git, Python, a CI system, basic statistics (distributions; precision and recall), and the DevOps and DataOps pages.
• One day: read the first two parts, Sculley et al. 2015^[9] and Google Cloud’s levels;^[1] run Labs 1 and 2.
• One week: do Lab 3; score one of your models with the ML Test Score;^[16] write its retraining policy.
• 30 days: take one real model from level 0 or 1 to level 2: pipeline, data validation, gate with slices, parity replay.
• Specialise: serving at scale (KServe, GPUs with KubernetesOps); training jobs on shared or preemptible capacity; monitoring and drift statistics; feature platforms; or governance with ModelOps.
• Portfolio project: a public repository that trains a model on open data with DVC and MLflow, gates it, serves it, injects drift, a skew bug and an upstream data defect, and shows each being caught and told apart.

Situation (illustrative). The churn team proposes a nightly job that retrains on the latest data and promotes automatically if overall AUC does not fall. The alternative is weekly retraining plus drift-triggered runs, with the slice gate, a 48-hour canary and one human approval.

Option A: nightly retrain and auto-promote. Attractive: always fresh, no meetings, simple to explain.

Option B: weekly plus triggered retraining, gated. Slower to react, and a person approves each promotion.

Assumptions. Labels mature after 30 days, so a nightly run mostly retrains on the same labelled data plus a few new rows; the retention team acts on scores, so offers influence future churn labels; upstream data changes happen a few times a quarter.

Evidence. Freshness needs come from measurement, not habit (Rules of ML #8; ML Test Score “Model 4”).^[12][16] Feedback loops and fixed thresholds in a changing system are documented sources of ML debt.^[9] The runbook above shows how one upstream defect would be trained into a nightly model.

Why the attractive choice can be wrong. With 30-day labels, nightly retraining adds little freshness but 30 chances a month for bad data or a feedback effect to reach production, and an overall-AUC rule would miss a slice collapse. Why B can also be wrong. If the business changes weekly (a price war), a weekly cadence plus approval may be too slow, and drift thresholds too noisy to trigger reliably.

Decision process. Run a staleness test over the last year. If decay within a week is below the gate’s minimum uplift, choose B, with an “expedite” path: a drift alert plus a confirmed business change allows same-day retraining through the same gates. Review after a quarter using freshness, alert precision and incidents.

Uncertainty. The staleness test uses history; a regime change (as in 2020^[63]) can make history a poor guide, and in that case people, not schedules, decide.

After this page you should be able to:

1. Explain why a trained model is a build artifact of code, data, configuration and environment, and what MLOps adds to DevOps.
2. Distinguish skew, upstream data defects, data drift, prior shift and concept drift, and choose the right response to each.
3. Design a training pipeline with lineage, a promotion gate with slices, and a canary release with rollback.
4. Select monitoring signals for a model with delayed labels, and calibrate drift thresholds.
5. Evaluate how much MLOps a given model needs, and defend a retraining policy with evidence.

MLOps is what turns a model from an experiment that worked once into a component that keeps working. Its centre is not a tool but a loop: versioned inputs produce a model, gates decide whether it may serve, monitors decide whether it still should, and a written policy decides when to rebuild it.

The failures on this page are rarely exotic: skew a parity test would have caught, a data defect nobody paused for, drift a calibrated alert would have shown. Build the boring safeguards first (lineage, aliases, a gate with slices, a practised rollback), automate only where your measurements show it pays, and remember that no pipeline makes an unpredictable world predictable.

Every source below was opened on the date in its footnote. The claim-by-claim ledger, including sources that could not be fetched, is research/mlops-ledger.md in the Atlas repository. Re-check quarterly (an Atlas suggestion): Kueue’s TrainJob stage, KServe’s canary mode and docs version, the Agent Platform naming and Model Monitoring v2 status, skops’ default trusted types, and the package versions in Lab 3.

Changelog.

• 2026-10-16, r0: first edition (Blueprint v2, 13 parts). Labs 1 and 2 executed locally with Python 3.8.10; Lab 3 and the reference implementation syntax-checked only.
• 2026-10-16/17, r1: fixes from the independent fact check (reviews/mlops-r1.md): deploy step and identities, data-defect branch, training jobs, parity test, alerts, attributions; Lab 2 re-executed.
• 2026-10-17, post-score polish (reviews/mlops-polish.md): skops trusted types (set at logging; TreePredictor in train.py); Lab 3 helper files, two executed, and the expected first REJECT; alert wording; Figure 4 re-routed; re-check list.