--- title: "MLOps" slug: mlops category: data term_status: established evidence_status: verified summary: "MLOps is the engineering practice of getting trained machine-learning models into production and keeping them correct there: versioned data, code and models, automated training and release pipelines, monitoring for skew and drift, and controlled retraining." audience: [Beginners, Data scientists, ML engineers, Platform engineers, Architects, Engineering leaders] last_verified: 2026-10-17 key_takeaways: - "A trained model is a build output whose inputs include data. MLOps is the build system, release process and monitoring for that artifact." - "Code, data and the model all change, so CI also tests data and models, and continuous training (CT) adds a loop that ordinary software does not have." - "Skew (your two pipelines disagree) and upstream data defects are bugs you can test for; drift is the world changing. Diagnose which one you have before you retrain." - "Promote through gates: compare the challenger with the champion on the same frozen test set and by slice, then canary or shadow it before full traffic." - "Labels may arrive weeks late. Watch input and output distributions as early signals, and measure real quality when the labels land." - "Start small: one reproducible pipeline, a registry with lineage and a rollback path beats a platform nobody uses." related: [llmops, modelops, dataops, finops-for-ai, decisionops, aiops, kubernetesops, devops] --- ## What MLOps is ### In 30 seconds **MLOps is the engineering practice of getting trained machine-learning models into production and keeping them correct there.** A model is produced by code, data, configuration and a software environment. MLOps versions all four, automates the pipeline that turns them into a model, releases the model through tests and gates, watches it in production, and retrains or rolls it back when the world it predicts has changed. The label is used widely and the core practices are stable, but published definitions stress different things: - Google Cloud calls MLOps "an ML engineering culture and practice that aims at unifying ML system development (Dev)" with ML system operation, and stresses automation and monitoring at every step.[^gc-mlops] - Google's practitioner guide calls it "a methodology for ML engineering that unifies ML system development (the ML element) with ML system operations (the Ops element)", and says it provides "a set of standardized processes and technology capabilities".[^gc-guide] - A peer-reviewed study by Kreuzberger, Kühl and Hirschl defines it as "a paradigm, including aspects like best practices, sets of concepts, as well as a development culture" that draws on machine learning, software engineering and data engineering.[^kreuzberger] - The CD Foundation's MLOps special-interest group, in its current (2024) roadmap, defines it as "the extension of the DevOps methodology to include Machine Learning and Data Science assets as part of a Continuous Delivery process"; its 2020–2022 editions ended "as first class citizens within the DevOps ecology".[^cdf-roadmap] **Who this page is for.** Data scientists whose models must leave the notebook, ML and platform engineers, architects, and leaders deciding how much automation a model portfolio justifies. Beginners: this part and the glossary. Practitioners: skew and drift, labs, runbook. Leaders: [Deciding and adopting](#deciding-and-adopting) and the [cheat sheet](#executive-recap-and-cheat-sheet). ### At a glance | Aspect | Summary | |---|---| | Goal | Models that can be reproduced, released safely, monitored, and retrained or rolled back before users notice a problem | | Unit of work | A trained *predictive* model (classification, regression, ranking, forecasting, vision) and the pipeline that produces it | | Main users | ML engineers, data scientists, ML platform and data engineers, owners of services that call the model | | In scope | Experiment tracking; versioning; feature stores; training pipelines and jobs; continuous training; CI/CD for pipelines; the model registry; batch, online and streaming serving; skew, drift and quality monitoring; retraining and rollback | | Prerequisites | [DevOps](../devops/) basics (version control, CI/CD, infrastructure as code), [DataOps](../dataops/) basics, model evaluation (train/test splits; [precision and recall](../aiops/#metrics-and-kpis), defined for the Atlas in AIOps) | | Key outputs | A versioned training pipeline, registered model versions with lineage, a serving endpoint or batch job, monitors and alerts, a retraining policy, release records | | Term status | **Established.** Peer-reviewed definitions,[^kreuzberger] vendor-maintained frameworks,[^gc-mlops][^ms-maturity][^aws-ml-lens] recognised roles and certifications[^gcp-mle-cert] and foundation-hosted tools[^kubeflow-cncf] exist, but no standards body publishes a normative definition | | Last verified | 2026-10-17 | ### The problem MLOps solves In 2015 a Google team described ML systems as a fast way to build up technical debt: "it is dangerous to think of these quick wins as coming for free".[^sculley-2015] Their Figure 1: "Only a small fraction of real-world ML systems is composed of the ML code", and "The required surrounding infrastructure is vast and complex". They also named the **CACE principle**, "Changing Anything Changes Everything": changing one feature, hyperparameter or sampling rule can change everything else.[^sculley-2015] Microsoft researchers who studied their own teams found three ways in which ML differs from earlier software: data is harder to discover, manage and version; customising and reusing models needs different skills; and ML components are harder to keep as separate modules.[^amershi] The usual starting point is what Google Cloud calls "level 0": data scientists work by hand in notebooks and hand a model file to engineers who rebuild the feature code. The hand-off itself can cause training-serving skew, and "models often break when they are deployed in the real world".[^gc-mlops] Without an MLOps practice, five failures recur: 1. **The model cannot be rebuilt.** Nobody recorded which data snapshot, code commit, parameters and library versions produced the artifact in production. scikit-learn, for example, states that "there are no supported ways to load a model trained with a different version of scikit-learn".[^sklearn-persistence] 2. **Training and serving disagree.** Features are computed one way in a notebook and another in production.[^rules-ml] 3. **Decay is silent.** A model that has gone stale throws no exception. Google's Rules of ML describe a table used by Google Play that was six months out of date; refreshing it alone raised the install rate by 2%.[^rules-ml] 4. **Every model is a one-off project.** Before Uber built its Michelangelo platform, it reported that "there was no established path to deploying a model into production".[^uber-michelangelo] 5. **There is no safe way back.** Without a registry and versioned serving, rolling back means finding an old file and hoping it still loads. ::: figure From a one-time hand-off to a continuous-training loop ```mermaid flowchart TB subgraph BEFORE["Before: level 0, manual hand-off"] direction TB N["Notebook on a laptop"] --> F["Model file sent by email or share"] F --> R["Engineer rewrites feature code"] R --> S["Production service"] S -.->|"no monitoring, no lineage"| Q["Silent decay"] end subgraph AFTER["After: versioned pipeline and closed loop"] direction TB P["Versioned training pipeline"] --> G["Registry with lineage and gates"] G --> V["Serving with canary and rollback"] V --> M["Monitoring: skew, drift, quality"] M -->|"trigger or alert"| P end BEFORE ~~~ AFTER classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 class N,F,R,S,Q risk class P,G,V,M core ``` Text equivalent: Two panels. Before: a notebook model is sent as a file, an engineer rewrites the feature code, and the model runs in production with no monitoring or lineage, so it decays silently. After: a versioned pipeline produces a model that is registered with lineage and gated; serving releases it with a canary and a rollback path; monitoring watches skew, drift and quality and triggers the pipeline again. ::: ### What MLOps is not | It is not… | Why the distinction matters | Where it belongs | |---|---|---| | Operating LLM applications | Prompts, retrieval, evaluations and guardrails; training or fine-tuning *pipelines* stay here | [LLMOps](../llmops/) | | Governing the model portfolio | Inventory, risk tiers, independent validation and regulatory mapping, across all model types | [ModelOps](../modelops/) | | AI for IT operations | AIOps applies ML *to* operations events; MLOps is operations *for* ML | [AIOps](../aiops/) | | Upstream data pipelines | Ingestion, data tests and data contracts serve every consumer, not only models | [DataOps](../dataops/) | | Data science | Choosing features and algorithms; MLOps makes the result repeatable and safe to run | — | | A product you buy | Platforms help, but a platform without ownership, gates and monitoring is level 0 with a nicer UI | [Tooling](#tooling-landscape) | Two misconceptions to correct early: - **"MLOps means putting a model behind an API."** Deployment is one of seven processes in Google's lifecycle ([lifecycle](#lifecycle-day-0-day-1-day-2)).[^gc-guide] - **"Every model needs full automation."** Google's first rule of ML is "Don't be afraid to launch a product without machine learning".[^rules-ml] How much MLOps a model needs depends on how fast its world changes and what an error costs ([decision framework](#decision-framework)). ## Where MLOps comes from and where it sits ### Origins and evolution No reliable source records who first used the word "MLOps", so this page does not attribute it. The practices have a traceable history: | Year | Milestone | Why it mattered | |---|---|---| | 2014 | Sculley et al., "Machine Learning: The High Interest Credit Card of Technical Debt" (NIPS workshop)[^sculley-2014] | Warned that it is "remarkably easy to incur massive ongoing maintenance costs at the system level" with ML | | 2015 | "Hidden Technical Debt in Machine Learning Systems", NeurIPS[^sculley-2015] | Named entanglement, hidden feedback loops, undeclared consumers and data dependencies | | 2017 | TFX paper (KDD);[^tfx-paper] Uber's Michelangelo post;[^uber-michelangelo] ML Test Score (IEEE Big Data);[^ml-test-score] Kubeflow announced[^kubeflow-2017] | Public in-house ML platforms; a production-readiness rubric | | 2018 | MLflow announced with Tracking, Projects and Models[^mlflow-launch] | An open-source tool for tracking, packaging and reproducing runs | | 2019 | Continuous Delivery for Machine Learning (CD4ML);[^cd4ml] Microsoft's software-engineering-for-ML study;[^amershi] model cards[^model-cards] | Continuous delivery for code, data and model; model documentation | | 2020–2021 | CD Foundation MLOps roadmap (first edition 2020; latest 2024);[^cdf-roadmap] Google's practitioner guide (May 2021)[^gc-guide] | Community and vendor frameworks for the whole lifecycle | | 2022–2023 | Interview study of 18 ML engineers;[^shankar] Kreuzberger et al. definition (preprint 2022, *IEEE Access* 2023)[^kreuzberger] | Empirical and definitional research on MLOps itself | | 2023–2026 | Kubeflow joins CNCF (2023) and graduates (2026-07-24);[^kubeflow-cncf] KServe accepted as a CNCF incubating project (2025-09-29);[^kserve-cncf] Microsoft's maturity model separates GenAIOps from predictive MLOps (section present by November 2025)[^ms-maturity] | Core tools move to neutral foundations; vendors split predictive MLOps from generative-AI operations | ### Glossary and mental model **Mental model: the model is a build artifact whose source code includes data.** In ML, the "source" is code + data + configuration + environment, and the "compiler" is a training run that may not even be deterministic: PyTorch warns that "Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms".[^pytorch-random] CD4ML names three axes of change: code, model and data.[^cd4ml] MLOps is therefore a **build system** for models (versioned inputs, pipelines, lineage), a **release process** (gates, canaries, rollback) and a **control loop** (monitoring that decides when to rebuild). | Term | Meaning | Example | |---|---|---| | Continuous training (CT) | Google's term for automatically retraining and serving models, "a new property, unique to ML systems"[^gc-mlops] | The pipeline re-runs when a week of new labelled data arrives | | Model registry | The catalogue of model versions with lineage, tags and aliases[^mlflow-registry] | Version 13 links to the run, data hash and commit that made it | | Alias | "a mutable, named reference to a particular version" of a registered model[^mlflow-registry] | `models:/churn-classifier@champion` | | Champion / challenger | The model in production and a candidate competing to replace it[^gc-guide] | v12 champion, v13 challenger | | Feature store | Serves the same feature definitions to training (offline store) and serving (online store)[^feast] | `days_since_last_login` | | Point-in-time correctness | Joining each training example only with feature values that existed at its timestamp[^feast-pit] | No "future" account balance in a fraud-training row | | Data leakage | Information unavailable at prediction time leaks into training or evaluation, inflating results[^kapoor] | Imputing with whole-dataset statistics before the split | | Training-serving skew | The training and serving pipelines compute different feature values for the same situation (this page's usage; see the note below)[^rules-ml] | Training maps a missing value to `NULL`, serving to `"unknown"` | | Upstream data defect | A source, schema or join change that corrupts the data *both* pipelines receive (this page's term) | A CRM export starts sending `unknown` for missing contracts | | Data drift (covariate shift) | The input distribution P(X) changes over time[^lu-drift] | A campaign brings many new, short-tenure customers | | Prior (label) shift | The share of each outcome P(y) changes[^lu-drift] | Fraud rate doubles during a holiday season | | Concept drift | Here, in the narrow sense: P(y\|X) changes, so the same inputs now mean something else. Lu et al. use the term for any change in P(X, y)[^lu-drift] | Customers who used to stay now leave after a competitor's price cut | | Staleness / freshness | How old the model's training data is relative to what it predicts; Rules of ML: "Know the freshness requirements of your system."[^rules-ml] | An ads model may need daily updates | | Label delay | Time between a prediction and the arrival of its true outcome | Churn labels arrive 30 days later | | Model signature | The declared input and output schema, validated at inference by the serving tool[^mlflow-signature] | `tenure_months: long, contract_type: string` | | Feedback loop | The model's predictions influence its own future training data[^sculley-2015] | A recommender only learns from items it chose to show | "Skew" is used in two ways. Rules of ML define training-serving skew as "a difference between performance during training and performance during serving", with three causes: different data handling in the two pipelines, data changing between training and serving, and feedback loops.[^rules-ml] Google Cloud's model monitoring uses the word for a *distribution* difference between production and training features.[^vertex-monitoring] Because the fixes differ, this page separates **skew** (your two pipelines disagree), an **upstream data defect** (a source, schema or join change before your pipelines; one form of the second cause) and **drift** (the world changed). That split is an editorial convention, not a standard. ### How MLOps relates to other Ops MLOps sits between the data that feeds models and the decisions that consume them. The boundaries below are this Atlas's **editorial distinctions**; vendors draw them differently, and some use MLOps and ModelOps as synonyms. | Neighbour | Shared ground | MLOps owns | The neighbour owns | |---|---|---|---| | [DataOps](../dataops/) | Data pipelines, data tests | Training-data snapshots, feature definitions, skew checks | Upstream ingestion, data quality for all consumers, data contracts | | [LLMOps](../llmops/) | Registry, monitoring vocabulary | Training and fine-tuning pipelines; predictive-model drift | Prompts, retrieval (RAG), LLM evaluations, guardrails, traces | | [ModelOps](../modelops/) | The registry and its lifecycle states | Building and running the registry, gates and lineage | Inventory across model types, risk tiers, independent validation, approval policy, regulation | | [FinOps for AI](../finops-for-ai/) | GPU and serving cost | Engineering for efficient training and serving | Unit economics, allocation, commitments, who decides on spend | | [DevOps](../devops/) | CI/CD, infrastructure as code, DORA-style delivery | Data and model tests, CT, model release gates | Delivery culture and the software pipeline | | [KubernetesOps](../kubernetesops/) | GPU node pools, operators | Training-job design (checkpoints, queues, priorities) and inference services as workloads | Cluster upgrades, scheduling mechanics, node pools, device plugins | | [ObservabilityOps](../observabilityops/) and [SRE](../sre/) | Telemetry and SLOs | Model-quality signals: skew, drift, calibration, label-based quality | Telemetry pipelines, service SLOs, error budgets | | [DecisionOps](../decisionops/) | Model scores used in decisions | Producing reliable scores | Decision logic that combines rules, thresholds and scores | | [AIOps](../aiops/) | The letters "AI" and "Ops" | Operations *for* ML | ML *for* IT-operations events | Other links: [AnalyticsOps](../analyticsops/) owns the metric definitions business KPIs reuse; [ResearchDataOps](../researchdataops/) long-term stewardship of research datasets; [PrivacyOps](../privacyops/) the legal basis for training on personal data; [DevSecOps](../devsecops/) build-time supply-chain controls; [GreenOps](../greenops/) carbon accounting. ::: figure Where MLOps sits among its neighbours ```mermaid flowchart TB subgraph UP["Upstream"] direction LR DO["DataOps: trusted data"] PR["PrivacyOps: lawful use"] end ML["MLOps: build, release and monitor trained models"] subgraph SIDE["Alongside"] direction LR MO["ModelOps: governs the portfolio"] FA["FinOps for AI: AI economics"] KO["KubernetesOps: GPU clusters"] end subgraph DOWN["Downstream"] direction LR DE["DecisionOps: decision logic"] LL["LLMOps: LLM applications"] SR["SRE: service SLOs"] end UP --> ML ML <--> SIDE ML --> DOWN classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class ML core class DO,PR,MO,FA,KO,DE,LL,SR neutral ``` Text equivalent: Three bands around MLOps, which builds, releases and monitors trained models. Upstream, DataOps supplies trusted data and PrivacyOps confirms lawful use of personal data. Alongside, exchanging information both ways: ModelOps governs the portfolio, FinOps for AI owns AI economics, KubernetesOps runs GPU clusters. Downstream: DecisionOps embeds scores in decision logic, LLMOps builds LLM applications that may use fine-tuned models, and SRE holds the SLOs of services that call the model. ::: ## Principles and operating model ### Charter and principles Two sets of principles should not be confused. The first is published: Kreuzberger et al. derived **nine principles** from literature, a tool review and expert interviews, from CI/CD automation, orchestration, reproducibility and versioning of data, model and code to continuous training, metadata tracking, monitoring and feedback loops.[^kreuzberger] From interviews with 18 ML engineers, Shankar et al. identified three variables that govern the success of a production deployment: "Velocity, Validation, and Versioning".[^shankar] The second is **this Atlas's proposed charter**: invariants a team can test, not an industry standard. ::: note An MLOps charter (Atlas proposal) **Mission.** Every production model can be rebuilt from recorded inputs, released and withdrawn safely, and is watched by an owner who will act on what the monitors say. **Invariants (must always hold).** 1. Every model that serves traffic maps to exactly one registry version with complete lineage: data snapshot, code commit, parameters and environment. 2. Training and serving compute each feature from one definition, and their parity is tested (Rules of ML #32 and #37; ML Test Score "Monitor 3").[^rules-ml][^ml-test-score] 3. No model reaches full traffic without a gate that compares it with the champion on the same frozen test set and by slice. 4. Rolling back to the previous version takes minutes and has been rehearsed (ML Test Score "Infra 7").[^ml-test-score] 5. Every model has a named owner, a stated freshness requirement and at least one monitored signal with a threshold someone has agreed to act on. 6. Retraining runs on a written policy, and it stops automatically when input validation fails. **Non-goals.** Choosing algorithms; approving high-risk models (that is [ModelOps](../modelops/)); automating for its own sake. **Trade-offs it accepts.** Gates slow some releases; reproducibility costs storage; one shared feature definition limits local freedom. ::: ### Fundamentals: the building blocks Kreuzberger et al. list nine technical components;[^kreuzberger] Google's guide lists similar capabilities.[^gc-guide] The table merges them; tool examples are in the [tooling landscape](#tooling-landscape). | Building block | Job | "Good" looks like | |---|---|---| | Source and CI/CD | Version and test pipeline code, feature code and serving code | Every change runs unit, data and integration tests | | Data and model versioning | Make data snapshots and artifacts addressable by hash or version | A model version points to an immutable data snapshot | | Workflow orchestration | Run the training pipeline as a graph of steps, with retries and caching | The same pipeline runs in development and production[^gc-mlops] | | Feature management | One definition per feature for training and serving | Point-in-time-correct training sets; low-latency online reads | | Training infrastructure | Compute for training, from CPU jobs to GPU clusters | Jobs are reproducible, right-sized, queued and able to resume ([training jobs](#operating-training-jobs)) | | Experiment tracking and ML metadata | Record runs, parameters, metrics, artifacts and lineage | Any number in a report traces to a run | | Model registry | Catalogue versions, aliases, tags and approvals | Every serving path traces to one registry version, chosen through an alias | | Serving | Run the model for batch, online or streaming predictions | Versioned, observable, able to canary and roll back | | Monitoring | Watch inputs, outputs, quality and the service itself | Alerts that lead to action; labels joined when they arrive | ### Roles, ownership and RACI Kreuzberger et al. describe seven roles: business stakeholder, solution architect, data scientist, data engineer, software engineer, DevOps engineer, and ML engineer or MLOps engineer, a cross-domain role that combines aspects of the others and builds and operates the ML infrastructure.[^kreuzberger] Job titles vary; the responsibilities do not. Either product teams own their models end to end on a self-service **ML platform** (the pattern in [PlatformOps](../platformops/)), or a central data-science team builds models for many product teams, which works only if each production model's owner is written down. | Activity | Product owner | Data scientist | ML engineer | Data engineer | Platform / SRE | |---|---|---|---|---|---| | Frame the problem, success metric and guardrails | **A** | R | C | C | I | | Training-data snapshot and feature definitions | I | R | C | **A** | I | | Experiments and candidate model | I | **A/R** | C | C | — | | Training pipeline, CT and lineage | I | C | **A/R** | C | C | | Promotion gate thresholds | **A** | R | R | — | C | | Independent validation of high-risk models | I | C | C | — | — (ModelOps is **A**) | | Serving, scaling and service SLOs | C | I | R | — | **A** | | Model monitoring and on-call response | I | C | **A/R** | C | R | | Retirement and consumer notification | **A** | C | R | C | C | R = responsible, A = accountable, C = consulted, I = informed. An Atlas template: adapt the columns, but keep one "A" per row. ### Lifecycle: Day 0, Day 1, Day 2 Google's guide describes the MLOps lifecycle as "seven integrated and iterative processes": ML development, training operationalisation, continuous training, model deployment, prediction serving, continuous monitoring, and data and model management.[^gc-guide] This page's Figure 3 draws them as a loop (Google shows them in its own Figure 2). ::: figure The MLOps lifecycle as a loop (after Google's seven processes) ```mermaid flowchart TB D["1. ML development: experiments, training procedure"] O["2. Training operationalisation: pipeline built and tested"] C["3. Continuous training: pipeline runs on triggers"] P["4. Model deployment: package, test, canary"] S["5. Prediction serving: batch, online, streaming"] M["6. Continuous monitoring: skew, drift, quality"] G["7. Data and model management: features, registry, lineage"] D --> O --> C --> P --> S --> M M -->|"retraining trigger"| C M -->|"new idea or concept change"| D G -.-> D G -.-> C G -.-> P classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class D,O,C,P,S,M core class G neutral ``` Text equivalent: Six processes in a chain: ML development, training operationalisation (a tested pipeline), continuous training on triggers, model deployment (package, test, canary), prediction serving, and continuous monitoring of skew, drift and quality. Monitoring loops back to continuous training when a retraining trigger fires, and to ML development when a new idea or a concept change needs human work. Data and model management (features, registry, lineage) supports development, training and deployment. ::: | Phase | What happens | Exit evidence | |---|---|---| | **Day 0: frame and build** | Define the decision the model supports, a non-ML baseline, success and guardrail metrics; check that labels exist and how late they arrive; estimate the freshness requirement; build a first reproducible pipeline | A baseline to beat; a data snapshot; one pipeline run that anyone can repeat | | **Day 1: release** | Register the first version with lineage; deploy in shadow or canary; record reference statistics for monitoring; agree alert owners | Version 1 serving through an alias; dashboards with a reference window; rollback rehearsed | | **Day 2: operate and improve** | Monitor; retrain by policy; review slices and staleness; handle incidents; retire models, notifying consumers | Model health review held; retraining decisions logged; retired models removed from serving | Retirement deserves explicit work. Sculley et al. warn about **undeclared consumers** that silently use a model's output as input.[^sculley-2015] Keep a consumer list in the registry and access control on prediction outputs, or a "retired" model will be impossible to switch off. ### Operating model and governance - **Ceremonies.** A weekly *model health review* ([checklist](#checklists)); a monthly or quarterly *model review* (slices, staleness, cost, consumers); a post-incident review for every user-visible model failure, via [IncidentOps](../incidentops/). - **Policies as code.** Promotion thresholds, retraining triggers and validation rules live next to the pipeline, so a threshold change is reviewed like code ([templates](#templates)). - **The registry is the system of record.** Google's guide defines model governance as "registering, reviewing, validating, and approving models for deployment".[^gc-guide] MLOps builds the registry and enforces gates; [ModelOps](../modelops/) decides which risk tiers need which approvals, and tags such as MLflow's `validation_status:approved` example carry that decision into the pipeline.[^mlflow-registry] - **Escalation.** A model incident is a service incident. Severity follows business impact, not the size of the metric change. ## Architecture and workflow ### Reference architecture Figure 4 is a vendor-neutral reference architecture for predictive models. ::: figure Reference architecture for training, release and monitoring ```mermaid flowchart TB %% Training zone declared first: the layout then keeps the orchestrator above the job subgraph TRAIN["Training zone"] ORC["Pipeline orchestrator"] SNAP["Versioned training set"] ORC --> TR["Training job"] SNAP --> TR TR --> EV["Evaluation and gate"] end subgraph DATA["Data zone: restricted access"] SRC["Source tables and events"] --> VAL["Data validation"] VAL --> FS["Feature store: offline and online"] end subgraph REC["Records"] META["Metadata and experiments"] REG["Model registry with aliases"] end subgraph SERVE["Serving zone"] BA["Batch scoring job"] DEP["Deploy step: pins a version"] ON["Online service"] end MON["Monitoring: logs, statistics, labels"] FS -->|"point-in-time join"| SNAP TR -.->|"lineage"| META EV -->|"new version"| REG REG -->|"loads by alias"| BA REG -->|"resolves alias"| DEP DEP --> ON FS -.->|"online features"| ON BA --> MON ON --> MON MON -.->|"retraining trigger"| ORC classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 class SRC,VAL,FS,SNAP,ORC,TR,ON,BA,DEP core class META,REG neutral class EV,MON decision ``` Text equivalent: Four zones, then monitoring. In the restricted data zone, source tables and events pass validation and feed a feature store (offline and online). A point-in-time join from the store becomes a versioned training set in the training zone, where the orchestrator runs a training job on it; its result goes to evaluation and a gate. The job writes lineage to the metadata records, and the gate writes each new version to the registry. In the serving zone, a batch job loads the version an alias points to, and a deploy step resolves an alias and pins the online service to that version; the online service also reads online features. Both serving paths send logs to monitoring, which can trigger the orchestrator. CI/CD (not drawn) builds and tests the code. ::: | Component | Responsibility | Key design choice | Trade-off | |---|---|---|---| | Data validation | Stop bad inputs before training; flag them at serving | Schema plus statistical expectations per feature (ML Test Score "Data 1")[^ml-test-score] | Too strict blocks every run; too loose trains on garbage | | Feature store | One feature definition for both paths; point-in-time joins | Adopt a store, or share a feature library | Another system to run; Feast says it is "not a general purpose data pipelining system"[^feast] | | Monitoring | Detect skew, drift and quality loss; join labels | Log the features actually served (Rules of ML #29)[^rules-ml] | Logging every request costs storage and may hold personal data | **Trust boundaries.** Training data can be a company's most sensitive data, so the data and training zones have tighter access than serving. Treat the artifact crossing from training to serving as a supply-chain boundary ([security](#quality-security-privacy-and-compliance)). ### Serving patterns: batch, online, streaming and embedded | Pattern | How it works | Good for | Watch out for | |---|---|---|---| | **Batch scoring** | A scheduled job scores all known entities and writes predictions to a table | Nightly churn scores, demand forecasts; Uber ran offline models as Spark jobs[^uber-michelangelo] | Predictions are as stale as the last run; new entities have no score | | **Online service** | A separate service answers requests over the network | Fraud checks, ranking | Tail latency, feature freshness, an extra network hop; but releases independent of the app[^cd4ml] | | **Streaming** | The model consumes an event stream and emits scores continuously | Real-time anomaly detection on sensor or payment events | Exactly-once semantics, back-pressure, state | | **Embedded** | The model ships inside the consuming application or device | Mobile or edge inference, very low latency | Model and application release together; old app versions keep old models | Choose by four questions: How fast must an answer come? How fresh must features be? Are all entities known in advance? How often will the model change relative to the application? Batch is the cheapest and simplest to monitor; go online only when the decision genuinely happens at request time. ### Operating training jobs Training runs are batch jobs lasting minutes or days, often on shared or preemptible GPUs. [KubernetesOps](../kubernetesops/) runs the cluster (node pools, device plugins, scheduling), [FinOps for AI](../finops-for-ai/) decides what GPU hours are worth, and MLOps owns how a training job survives on that capacity: 1. **Checkpoint and resume.** Save model and optimiser state, step, data position and random state at a fixed interval, and make a restarted job continue from the latest checkpoint. Do it on a schedule, not only on a shutdown signal: AWS gives a two-minute warning before reclaiming a Spot Instance, emitted "on a best effort basis".[^aws-spot] Test resume as you test rollback. For distributed training, PyTorch's Distributed Checkpoint "handles load-time resharding", so a job can resume on a different number of GPUs.[^pytorch-dcp] 2. **Queue jobs against quotas.** On shared GPUs a job should wait for capacity, not fail or starve serving. Kueue, for example, "manages quotas and how jobs consume them" and decides when a job waits, starts or is preempted; it supports batch Jobs, JobSets, Ray and Kubeflow's TrainJob (alpha), as of October 2026.[^kueue] Rank retraining below serving and above experiments. 3. **Read utilisation as a signal.** Idle accelerators often mean the job is waiting for data. PyTorch's `DataLoader`, for example, loads data in the main process by default (`num_workers=0`), so training waits for every batch.[^pytorch-tuning] Record step time, data-wait time and peak memory with each run. 4. **Route failures into the retraining policy.** A preempted job resumes; a job that fails twice for the same reason (out of memory, a corrupt shard) stops and alerts the owner. The champion keeps serving, but its freshness clock keeps running. ### End-to-end workflow: one retraining cycle Figure 5 follows one retraining cycle and its three exception paths. Each step names the identity that performs it, because that separation is a security control ([registry permissions](#quality-security-privacy-and-compliance)). ::: figure One retraining cycle: who does what, and the exception paths ```mermaid flowchart TB T["Trigger: schedule, new labels or drift alert"] --> V{"Pipeline: input data valid?"} V -->|"No"| S["Stop before training, alert the owner"] V -->|"Yes"| TR["Pipeline: train, score challenger and champion, log lineage"] TR --> G{"Gate passes?"} G -->|"No"| GR["Run stays in tracking, reasons to the owner"] G -->|"Yes"| RC["Pipeline: register in candidates model as challenger"] RC --> AP["Owner approves, promotion identity copies it to production"] AP --> CN["Deploy step: resolve alias, canary 10%"] CN --> H{"Canary healthy for 48 hours?"} H -->|"No"| RB["Deploy step: canary to 0%, open an incident"] H -->|"Yes"| CH["Promotion identity: move champion alias"] CH --> FL["Deploy step: resolve champion, all traffic"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class TR,RC,CN,FL core class S,GR,RB risk class V,G,H decision class T,AP,CH neutral ``` Text equivalent: A trigger (a schedule, new labels or a drift alert) starts the pipeline, which first checks the input data; invalid data stops it before training and alerts the owner. Otherwise it trains, scores challenger and champion on the same test set and logs lineage. A rejected challenger stays in experiment tracking and the owner gets the reasons; a passing one is registered in a candidates model as "challenger". After the owner approves, a separate promotion identity copies it into the production model, and the deploy step resolves the alias and starts a 10 percent canary. If the canary is unhealthy within 48 hours, the deploy step sets it to 0 percent and an incident is opened; if it is healthy, the promotion identity moves the "champion" alias and the deploy step sends all traffic to that version. ::: Invalid data stops the pipeline before training, the cheapest place to stop; Google Cloud recommends stopping on schema skew and treating value skew as a reason to retrain.[^gc-mlops] ## Practices, standards and tools ### Core practices and playbooks | Practice | Goal | Trigger | Owner | Output | Verification | |---|---|---|---|---|---| | 1. Reproducible training run | Rebuild any production model | Every training run | ML engineer | Run record: commit, data hash, parameters, environment, metrics | Re-run reproduces metrics within an agreed tolerance | | 2. Train like you serve | Remove training-serving skew | Any feature change | Data + ML engineer | One feature definition; logged serving features; parity test | Replayed requests give identical feature values | | 3. Gate the challenger | Promote only real improvements | New candidate | Data scientist + product owner | Gate report: overall, slices, latency, schema | Gate is code in CI; rejected runs recorded | | 4. Progressive delivery | Limit the blast radius of a bad model | Gate passed | ML engineer | Shadow, canary or online experiment | Canary metrics compared with the champion | | 5. Three-level monitoring | See service, data and quality problems | Continuous | ML engineer + SRE | Dashboards and alerts with owners | Alert precision reviewed weekly | | 6. Retraining policy | Retrain when it pays, never on bad data | Policy triggers | Model owner | Written triggers and pause rules | Each retrain links to its trigger | | 7. Rollback and fallback | Recover in minutes | Regression or incident | On-call | Alias moved back, or rule-based fallback | Rehearsed at least quarterly | **Practice 1: a reproducible training run.** Record the commit, an immutable data reference (a content hash such as DVC's `.dvc` metafile,[^dvc-start] or a table snapshot ID), the parameters and the environment (lockfile or container digest); scikit-learn advises keeping such metadata next to a saved model.[^sklearn-persistence] Set seeds, but define reproducibility as metrics within a tolerance ([mental model](#glossary-and-mental-model)). **Practice 2: train like you serve.** Rules of ML: "Re-use code between your training pipeline and your serving pipeline whenever possible" (#32); log serving features and train on them (#29); remember that a joined table may change in between (#31); and measure skew, because the same example should "give you exactly the same result" in both paths (#37).[^rules-ml] The reference implementation's parity test does that. A feature store serves one definition to both paths and builds training sets with point-in-time joins; by default Feast can still return a value backfilled after the event, and `filter_by_created_timestamp=True` excludes it.[^feast-pit] **Practice 3: gate the challenger.** Compare challenger and champion on the same frozen test set, split by time (train on the past, test on a later period) so that temporal leakage cannot flatter the result. Kapoor and Narayanan's survey found 22 papers from 17 fields reporting leakage-related errors that collectively affected 294 papers.[^kapoor] Google's guide recommends comparing the two on evaluation metrics *and* on business KPIs from online experiments.[^gc-guide] Check slices ([Lab 2](#hands-on-labs)), latency, the input schema and any fairness criteria your [ModelOps](../modelops/) policy requires. **Practice 4: progressive delivery.** In order of risk: a **shadow** deployment gets a copy of the requests, and "Only the responses of the production variant are returned";[^sagemaker-shadow] a **canary** sends a small share of real traffic to the candidate (in KServe, `canaryTrafficPercent`);[^kserve-canary] an **online controlled experiment** measures the business effect. Booking.com reported that "increasing the performance of a model does not necessarily translate into a gain in [business] value" (as quoted by Adrian Colyer).[^booking-kdd][^booking-summary] **Practice 5: monitor at three levels.** *Service*: latency, errors, saturation, owned with [SRE](../sre/). *Data*: schema violations, missing values, shifts in inputs and outputs, skew. *Quality*: model metrics once labels arrive; Google's guide describes evaluating effectiveness "in hindsight".[^gc-guide] Until then, use proxies: the prediction distribution; **prediction bias**, where predicted and observed label distributions should usually match (before labels arrive, compare with the latest *matured* base rate, and remember that a gap can be prior shift, not model error); and **action limits** on unusual numbers of real-world actions.[^sculley-2015] **Practice 6: a retraining policy.** Google Cloud lists five triggers: on demand, on a schedule, on new training data, on performance degradation and on a significant distribution change.[^gc-mlops] Choose the schedule from a staleness test (ML Test Score: know "The impact of model staleness"):[^ml-test-score] train up to a date and measure how quality falls on each later week. Write down when retraining must *not* run ([template](#templates)). **Practice 7: rollback and fallback.** Roll back by moving a traffic split, or an alias plus a redeploy, to the previous version; never by retraining. Keep a non-ML fallback (a rule, a default, the last batch score) for when the model service itself is down, and agree in advance which of the two on-call uses. ### Skew and drift: how they arise and how to detect them A model stays useful only if you can tell kinds of change apart, which one generic "drift dashboard" cannot do. Every investigation starts with: **did my system change, did the data feeding it change, or did the world change?** ::: figure Diagnosing a quality drop: skew, a data defect or drift, and which kind ```mermaid flowchart TB Q0["Quality or proxy signal drops"] --> Q1{"Same features in training and serving?"} Q1 -->|"No"| SK["Skew: fix the pipeline, do not retrain"] Q1 -->|"Yes"| QU{"Upstream data or its meaning changed?"} QU -->|"Yes"| UD["Data defect: fix at source, rescore, do not retrain"] QU -->|"No"| DR["The world changed: check all three; if none moved, tune the alert"] DR -->|"P(X) moved"| DD["Data drift: retrain if labels allow"] DR -->|"P(y) moved"| PS["Prior shift: recalibrate thresholds"] DR -->|"Worse, same inputs"| CD["Concept drift: new labels, features or redesign"] classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 class Q1,QU decision class SK,UD,CD risk class DR,DD,PS core class Q0 core ``` Text equivalent: A quality or proxy signal drops. First check: do training and serving compute the same features for the same example? If not, it is skew: fix the pipeline and do not retrain. If they match, did the upstream data or its meaning change? If so, it is an upstream data defect: fix it at the source, rescore and do not retrain. If not, the world changed, and all three signals are checked: if P(X) moved, it is data drift (retrain if labels allow); if P(y) moved, prior shift (recalibrate thresholds); if quality is worse for the same inputs, concept drift (new labels, features or a redesign). If none moved, tune the alert. ::: The tree rules out the cheapest, most actionable causes first, and checks all three kinds of drift because they can co-occur (P(X) and P(y|X) can change together).[^lu-drift] **How an upstream data defect arises (the data, before your pipelines).** A source system renames a category, changes units or null handling, or a join starts dropping or duplicating rows. Both pipelines faithfully process the wrong data, so a parity test passes. Google Cloud's advice to stop the pipeline on schema skew applies,[^gc-mlops] and the fix belongs with the data's owner ([DataOps](../dataops/)). **How skew arises (your system).** Two implementations of one feature; joins against tables that change between training and serving;[^rules-ml] different handling of missing values or unseen categories; features using information not available at prediction time;[^kapoor] different library versions of preprocessing code.[^sklearn-persistence] Skew is a defect, so tests find it: replay logged requests through the offline code, validate the schema at both ends, and compare training and serving statistics, as TFDV does with a per-feature "skew comparator".[^tfdv] **How drift arises (the world).** Lu et al. define concept drift as "unforeseeable changes in the underlying distribution of streaming data over time": the joint distribution P(X, y) changes, through P(X) alone ("virtual drift"), P(y|X), or both, and it can be sudden, gradual, incremental or reoccurring.[^lu-drift] Input monitors see P(X) change; seeing P(y|X) change needs labels or outcome proxies. **The distance measures in common use.** | Measure | Idea | Where you meet it | Caveat | |---|---|---|---| | Population stability index (PSI) | Σ (aᵢ − eᵢ) · ln(aᵢ / eᵢ) over bins, with eᵢ the reference and aᵢ the current share[^psi-matlab] | Credit-risk practice; optional in Evidently (threshold 0.1)[^evidently-drift] | The 0.10/0.25 "traffic light" bands lack academic support, and PSI's expected value and variance depend on the number of bins and the sample sizes[^psi-yurdakul] | | Jensen-Shannon divergence or distance | Symmetric, bounded difference between two distributions | Google Cloud model monitoring for numerical features;[^vertex-monitoring] TFDV for numeric and categorical;[^tfdv] Evidently's default for categorical columns above 1,000 rows (distance ≥ 0.1)[^evidently-drift] | Libraries differ: divergence versus its square root (distance), log base 2 or e. Compare numbers only within one tool | | L-infinity distance | Largest absolute difference in any category's share | TFDV and Google Cloud for categorical features (Google Cloud's v1 monitors every feature with a default threshold of 0.3)[^tfdv][^vertex-monitoring] | Ignores many small shifts that add up | | Kolmogorov–Smirnov, chi-square tests | Statistical tests of "same distribution" | Evidently defaults up to 1,000 rows (above that, distances such as normed Wasserstein)[^evidently-drift] | On large samples, classical tests tend to flag differences too small to matter, as the credit-scoring literature discusses[^psi-critical] | Five rules, this Atlas's synthesis of the sources above, make these measures useful rather than noisy: 1. **Calibrate thresholds on your own history.** Compute the measure between pairs of "normal" past windows and alert well above that variation; a vendor default is a starting point, not a property of your data. 2. **Compare like with like.** Use a reference window from the same season or weekday mix, or every holiday becomes an incident. 3. **Weight by consequence.** A large shift in an unimportant feature may not matter; a small shift in the top feature may. The prediction distribution summarises what the model actually does. 4. **Do not let a dataset-level rule hide one critical column.** Evidently, for example, flags dataset drift by default only when the share of drifting columns exceeds 0.5.[^evidently-drift] 5. **Treat label-free performance estimates as estimates.** NannyML's confidence-based performance estimation can estimate classification quality without labels if probabilities are well calibrated, but its documentation states that the method "will not work under concept drift".[^nannyml-cbpe] ### Methods, standards and frameworks No standards body defines an MLOps process; what exists is vendor guidance, research rubrics and documentation practices. Legal obligations (for example for high-risk AI systems or bank model risk) are mapped in [ModelOps](../modelops/), not here. | Framework | Publisher | What it gives you | Status | |---|---|---|---| | MLOps levels 0–2 (manual; ML pipeline automation; CI/CD pipeline automation) | Google Cloud[^gc-mlops] | A simple ladder centred on CT and pipeline CI/CD | Vendor guidance | | MLOps maturity model, levels 0–4 | Microsoft[^ms-maturity] | Behaviours across people, model creation, release and integration | Vendor guidance | | Well-Architected Machine Learning Lens | AWS (2025-11-19)[^aws-ml-lens] | Review questions and best practices for ML workloads; separate Generative AI Lens | Vendor guidance | | ML Test Score | Breck et al., IEEE Big Data 2017[^ml-test-score] | 28 tests in four groups (data, model, infrastructure, monitoring) and a score | Research rubric | | Continuous Delivery for ML (CD4ML) | Sato, Wider, Windheuser[^cd4ml] | Continuous delivery applied to code, data and models | Practitioner pattern | | Rules of Machine Learning | Google (Zinkevich)[^rules-ml] | Engineering heuristics, incl. skew and freshness | Practitioner guidance | | Model cards; datasheets for datasets | Mitchell et al.;[^model-cards] Gebru et al.[^datasheets] | Model and dataset documentation | Research proposals; usable as templates | In this Atlas's view, the ML Test Score is the most directly usable for audits. A test earns half a point if run manually with documented results and a full point if automated; the score is the **minimum** of the four section totals, so strength in one area cannot hide gaps in another. Its authors calibrated the interpretation against internal Google systems: 0 is "More of a research project than a productionized system", and above 5 is "Exceptional levels of automated testing and monitoring".[^ml-test-score] ### Tooling landscape Pick by job, not by brand. Versions and project status below are **as of October 2026** and change often. | Category | Job | Examples verified for this page | Selection criteria | Risks | |---|---|---|---|---| | Experiment tracking and registry | Runs, lineage, versions, aliases | MLflow 3.17.1 (2026-10-13), with Tracking[^mlflow-tracking] and a Model Registry;[^mlflow-pypi] stages deprecated since 2.9.0 in favour of aliases and tags[^mlflow-workflow] | Alias/tag API, lineage export, access control | The registry becomes a dependency of every release | | Data and pipeline versioning | Address data by hash; rerun only changed steps | DVC 3.67.1 (2026-03-31);[^dvc-pypi] DVC's docs suggest lakeFS for data-lake scale[^dvc-start] | Data size, storage backend, Git workflow | Many-file datasets; metafiles drifting from storage | | Pipeline orchestration | Run training graphs on a cluster | Kubeflow Pipelines (Kubeflow is a CNCF graduated project)[^kubeflow-cncf][^kfp] | Kubernetes skills, portability, caching | Platform weight; GPU scheduling belongs to [KubernetesOps](../kubernetesops/) | | Feature store | One feature definition, offline and online | Feast[^feast] | Online latency, point-in-time joins, existing warehouse | Another stateful system; not a pipeline tool | | Serving | Versioned endpoints, autoscaling, canary | KServe (CNCF incubating);[^kserve-cncf] Seldon Core 1 is under the Business Source License 1.1, which its docs say "is not an Open Source license"[^seldon-license] | Protocols, autoscaling, licence, GPU support | Licence terms for production use; serving stack lock-in | | Validation and monitoring | Schema checks, drift, label-free estimates | TFDV;[^tfdv] Evidently;[^evidently-drift] NannyML;[^nannyml-cbpe] cloud model monitoring[^vertex-monitoring] | Methods and defaults you can explain, batch versus streaming | Default thresholds mistaken for truth | | Managed platforms | End-to-end training, registry, serving, monitoring | Amazon SageMaker AI (for example shadow tests);[^sagemaker-shadow] Google Cloud's model monitoring in Gemini Enterprise Agent Platform (Vertex AI became part of it on 2026-04-22; the old name survives in URLs; monitoring v2 in Preview);[^gcp-agent-platform][^vertex-monitoring] Microsoft Azure ML[^ms-maturity] | Fit with your cloud and data platform | Feature limits per mode (SageMaker shadow tests exclude serverless, async and multi-model endpoints) and exit cost | Platform choices (managed, assembled or minimal) are compared in the [toolkit](#deep-comparisons). ### Automation and AI Automate the mechanics: pipeline runs, data validation, gates, packaging, deployment, and automatic rollback on hard service failures. Keep people in four places: changing a threshold or gate; promoting high-risk models (a [ModelOps](../modelops/) decision); responding to concept drift, which needs judgement about what the target now means; and retraining where the model's own actions shaped the labels. Sculley et al. describe **direct feedback loops** (a model influences the selection of its own training data) and **hidden feedback loops** between systems, and warn about **fixed thresholds in dynamic systems**.[^sculley-2015] A nightly "retrain and auto-promote" job can turn each of these into an outage ([judgement call](#a-judgement-call-retrain-nightly-or-only-when-the-data-says-so)). Generative AI assistants can draft pipeline code and summarise drift reports; treat their output as a draft to review, never as an approval, and keep them away from production credentials. Operating LLM applications is covered in [LLMOps](../llmops/). ## MLOps in practice ### Reference implementation: a churn model with continuous training **Design (illustrative).** A subscription business predicts which customers will cancel in the next 30 days; labels arrive 30 days later. Scores are produced nightly in batch for the retention team and online for the support tool. Two ML engineers run it on a Kubernetes cluster with KServe, an MLflow tracking server and a DVC remote in object storage. **Constraints:** the training data is personal data, so the data zone is restricted. A wrong score costs a wasted offer, not a safety incident, so the pipeline runs unattended up to a registered candidate, one human approval starts the canary, and the champion moves after 48 healthy hours. **Three identities**, which the registry's permission model requires ([security](#quality-security-privacy-and-compliance)): the training pipeline writes only to `churn-classifier-candidates`, a promotion identity writes to `churn-classifier`, and the serving workload reads one storage prefix. The repository layout keeps policy next to code: ```text churn/ params.yaml # training parameters and the time cutoff for the test split gate_policy.json # promotion thresholds (Lab 2) dvc.yaml # pipeline: validate -> train -> gate -> register src/features.py # ONE feature definition, for training and the online service src/validate.py # schema and range checks; exits non-zero on failure src/train.py # trains, logs lineage, scores challenger and champion src/promotion_gate.py # Lab 2 src/register.py # runs only after the gate passes src/promote.py # promotion identity: copy to production, move aliases src/deploy.py # renders the InferenceService from an alias; reconciliation tests/test_features.py monitoring/alerts.yaml ``` _Example status: syntax-checked by the validator, not executed (requires DVC, MLflow, scikit-learn and pandas)._ ```yaml # dvc.yaml - each stage reruns only when its deps or params change (hashes in dvc.lock) stages: validate: cmd: python src/validate.py data/raw/customers.csv reports/validation.json deps: - src/validate.py - data/raw/customers.csv outs: - reports/validation.json: cache: false train: cmd: python src/train.py data/raw/customers.csv reports/candidates.json deps: - src/train.py - src/features.py - data/raw/customers.csv - reports/validation.json params: - train.learning_rate - train.max_depth - train.cutoff_date outs: - reports/candidates.json: cache: false gate: cmd: >- python src/promotion_gate.py gate_policy.json reports/candidates.json > reports/gate.txt deps: - src/promotion_gate.py - gate_policy.json - reports/candidates.json outs: - reports/gate.txt: cache: false register: cmd: python src/register.py reports/candidates.json reports/registered.json deps: - src/register.py - reports/candidates.json - reports/gate.txt outs: - reports/registered.json: cache: false ``` The stage syntax follows DVC's pipeline documentation; DVC records hashes in `dvc.lock` and re-runs a stage only when a dependency or tracked parameter changes.[^dvc-pipelines] A failing `validate` or `gate` stops the pipeline, so only a challenger that passed reaches `register`; rejected runs stay in experiment tracking and never become registry versions. The single feature definition is the most important file: training and the online service import it, and unseen categories map to one explicit code instead of crashing or becoming a missing value. _Example status: syntax-checked, not executed._ ```python """One feature definition, imported by the training pipeline AND the online service.""" CONTRACT_TYPES = ("monthly", "one_year", "two_year") FEATURES = ["tenure_months", "monthly_charges", "contract_code", "support_calls_90d"] def contract_code(value): """Unseen or missing contract types map to one explicit 'other' code.""" return CONTRACT_TYPES.index(value) if value in CONTRACT_TYPES else len(CONTRACT_TYPES) def build_features(df): out = df.copy() out["contract_code"] = out["contract_type"].map(contract_code) out["support_calls_90d"] = out["support_calls_90d"].fillna(0).clip(upper=50) return out ``` The parity replay behind Practice 2 and the skew-rate KPI is short; the ML Test Score calls the skew test "Perhaps the most important and least implemented test".[^ml-test-score] The online service logs each raw request with the features it computed (Rules of ML #29),[^rules-ml] and the test replays a sample through the offline code. _Example status: syntax-checked, not executed (requires pandas and pytest, and a sample of logged requests)._ ```python """Parity replay: offline feature code must reproduce what the online service served.""" import json import pathlib import pandas as pd from features import FEATURES, build_features # One JSON object per line: {"request": {raw input}, "served_features": {feature values}} SAMPLE = pathlib.Path("tests/data/serving_log_sample.jsonl") def test_offline_features_match_served_features(): rows = [json.loads(line) for line in SAMPLE.read_text().splitlines() if line.strip()] raw = pd.DataFrame([r["request"] for r in rows]) served = pd.DataFrame([r["served_features"] for r in rows])[FEATURES].astype(float) offline = build_features(raw)[FEATURES].astype(float) differs = ((offline - served).abs() > 1e-9) | (offline.isna() != served.isna()) skew_rate = differs.any(axis=1).mean() assert skew_rate == 0, f"skew rate {skew_rate:.1%} on {len(rows)} replayed requests" def test_unseen_and_missing_values_have_explicit_codes(): raw = pd.DataFrame({"contract_type": ["monthly", "weekly", None], "support_calls_90d": [1, None, 99]}) out = build_features(raw) assert out["contract_code"].tolist() == [0, 3, 3] assert out["support_calls_90d"].tolist() == [1, 0, 50] ``` Run it in CI on every change to `features.py` or the online service, and daily on fresh samples; the share of differing rows is the skew rate. The training step logs lineage, scores challenger and champion on the **same** time-based test set and writes the gate's input; it registers nothing. It uses MLflow 3 APIs: `log_model(..., name=...)` (`artifact_path` is deprecated), signatures and aliases.[^mlflow-sklearn][^mlflow-signature][^mlflow-client] _Example status: syntax-checked, not executed (needs an MLflow tracking server and an existing "champion" alias; Lab 3 shows how to create the first one)._ ```python """Train, log lineage, and score challenger and champion on the same test set.""" import hashlib import json import statistics import subprocess import sys import time import mlflow import pandas as pd import yaml from mlflow import MlflowClient from mlflow.models import get_model_info, infer_signature from sklearn.ensemble import HistGradientBoostingClassifier from sklearn.metrics import roc_auc_score from features import FEATURES, build_features MODEL = "churn-classifier" # the production model: this job can only read it # skops >= 0.15 does not trust tree nodes by default; we trust our own artifact # (if log_model names more untrusted types, review them before adding them) TRUSTED = ["sklearn.ensemble._hist_gradient_boosting.predictor.TreePredictor"] MIN_SLICE_ROWS = 200 SLICES = { "region=north": lambda d: d["region"] == "north", "region=south": lambda d: d["region"] == "south", "tenure<6m": lambda d: d["tenure_months"] < 6, } def git(*args): return subprocess.run(["git", *args], capture_output=True, text=True) def evaluate(model, test): x, y = build_features(test)[FEATURES], test["churned"] scores = model.predict_proba(x)[:, 1] slices, skipped = {}, [] for name, f in SLICES.items(): mask = f(test).to_numpy() if mask.sum() < MIN_SLICE_ROWS or y[mask].nunique() < 2: skipped.append(name) # AUC is undefined for one class, noisy for few rows else: slices[name] = round(roc_auc_score(y[mask], scores[mask]), 3) timings = [] for i in range(200): # single-row latency, a rough proxy for online serving start = time.perf_counter() model.predict_proba(x.iloc[[i]]) timings.append((time.perf_counter() - start) * 1000) p95 = statistics.quantiles(timings, n=20)[18] return {"metrics": {"auc": round(roc_auc_score(y, scores), 3)}, "slices": slices, "skipped_slices": skipped, "p95_latency_ms": round(p95, 1)} def main(raw_path, report_path): params = yaml.safe_load(open("params.yaml"))["train"] data = pd.read_csv(raw_path, parse_dates=["snapshot_date"]) data_md5 = hashlib.md5(open(raw_path, "rb").read()).hexdigest() commit = git("rev-parse", "HEAD").stdout.strip() clean = git("diff", "--quiet", "HEAD", "--", "src", "params.yaml").returncode == 0 cutoff = pd.Timestamp(params["cutoff_date"]) train = data[data["snapshot_date"] < cutoff] test = data[data["snapshot_date"] >= cutoff] # later period: no temporal leakage model = HistGradientBoostingClassifier(learning_rate=params["learning_rate"], max_depth=params["max_depth"], random_state=42) mlflow.set_experiment(MODEL) with mlflow.start_run() as run: mlflow.log_params(params) mlflow.set_tags({"data_md5": data_md5, "git_commit": commit}) x_train = build_features(train)[FEATURES] model.fit(x_train, train["churned"]) signature = infer_signature(x_train, model.predict_proba(x_train)[:, 1]) challenger = evaluate(model, test) mlflow.log_metrics({"auc": challenger["metrics"]["auc"], "p95_latency_ms": challenger["p95_latency_ms"]}) info = mlflow.sklearn.log_model(model, name="model", signature=signature, skops_trusted_types=TRUSTED) client = MlflowClient() champ = client.get_model_version_by_alias(MODEL, "champion") champ_uri = f"models:/{MODEL}/{champ.version}" champion = evaluate(mlflow.sklearn.load_model(champ_uri), test) champ_signature = get_model_info(champ_uri).signature champ_commit = client.get_run(champ.run_id).data.tags.get("git_commit", "unknown") same_features = git("diff", "--quiet", champ_commit, "HEAD", "--", "src/features.py").returncode == 0 checks = { # computed, not asserted "data_validation": json.load(open("reports/validation.json"))["passed"], "signature_match": champ_signature is not None and signature.to_dict() == champ_signature.to_dict(), "lineage_recorded": bool(commit) and clean, } challenger.update(version=f"run-{run.info.run_id[:8]}", checks=checks, model_uri=info.model_uri, lineage={"git_commit": commit, "data_md5": data_md5}) report = { "champion": dict(champion, version=f"v{champ.version}"), "challenger": challenger, "warnings": [] if same_features else [ "features.py changed since the champion: re-score it from its own commit"], } json.dump(report, open(report_path, "w"), indent=2) if __name__ == "__main__": main(sys.argv[1], sys.argv[2]) ``` Three details matter. `signature_match` and `lineage_recorded` are computed: the challenger's input signature must equal the champion's (callers send the old schema, so a schema change needs a deliberate release), and lineage counts only if `src/` matches the recorded commit. Slices with too few rows or one class are skipped and listed. If `features.py` changed since the champion's commit, the champion was scored with feature code it never saw in training, so the report warns: re-score it from its own commit (for example with `git worktree`) before trusting the comparison. `register.py` writes to the candidates model with lineage tags; only the promotion identity runs `promote.py`. _Example status: syntax-checked, not executed._ ```python """Register a challenger that passed the gate (DVC runs this stage after the gate).""" import json import sys import mlflow from mlflow import MlflowClient CANDIDATES = "churn-classifier-candidates" # the training identity can write only here def main(candidates_path, out_path): challenger = json.load(open(candidates_path))["challenger"] tags = dict(challenger["lineage"], gate="passed") mv = mlflow.register_model(challenger["model_uri"], CANDIDATES, tags=tags) MlflowClient().set_registered_model_alias(CANDIDATES, "challenger", mv.version) json.dump({"model": CANDIDATES, "version": mv.version}, open(out_path, "w")) if __name__ == "__main__": main(sys.argv[1], sys.argv[2]) ``` _Example status: syntax-checked, not executed._ ```python """Promotion, run only by the promotion identity after approval. python src/promote.py canary copy the approved candidate into production python src/promote.py champion after a healthy canary, make the challenger the champion """ import sys from mlflow import MlflowClient PROD, CANDIDATES = "churn-classifier", "churn-classifier-candidates" client = MlflowClient() if sys.argv[1:] == ["canary"]: mv = client.copy_model_version(f"models:/{CANDIDATES}@challenger", PROD) client.set_registered_model_alias(PROD, "challenger", mv.version) elif sys.argv[1:] == ["champion"]: mv = client.get_model_version_by_alias(PROD, "challenger") client.set_registered_model_alias(PROD, "champion", mv.version) else: sys.exit(__doc__) print(f"{PROD} v{mv.version}: alias updated") ``` Serving has two paths. The batch job loads `models:/churn-classifier@champion`, so it picks up a new champion on its next run.[^mlflow-workflow] The online endpoint is a KServe `InferenceService`, which serves a storage URI, not an alias, so a deploy step resolves the alias, exports that version's MLflow model directory to a per-version prefix, records the version in annotations, and later checks that the endpoint still matches the registry. _Example status: syntax-checked; `render()` and `check()` exercised on 2026-10-17 with stubbed MLflow, AWS CLI, kubectl and YAML (both `check()` outcomes reproduced); not run against real services._ ```python """Render the KServe InferenceService from a registry alias, and check it later. python src/deploy.py canary 10 serve @challenger to 10% of traffic python src/deploy.py full serve @champion to all traffic python src/deploy.py check fail if the live endpoint and its alias disagree """ import json import subprocess import sys import tempfile import mlflow import yaml from mlflow import MlflowClient MODEL, NAMESPACE = "churn-classifier", "ml-serving" PREFIX = "s3://ml-artifacts-example/serving/churn-classifier" # serving reads only this VERSION_KEY, ALIAS_KEY = "example.com/registry-version", "example.com/registry-alias" def export(version): """Copy the registered MLflow model directory to a per-version prefix.""" target = f"{PREFIX}/v{version}" with tempfile.TemporaryDirectory() as tmp: uri = f"models:/{MODEL}/{version}" local = mlflow.artifacts.download_artifacts(uri, dst_path=tmp) subprocess.run(["aws", "s3", "sync", local, target], check=True) return target def render(mode, percent="0"): alias = "challenger" if mode == "canary" else "champion" version = MlflowClient().get_model_version_by_alias(MODEL, alias).version predictor = {"serviceAccountName": "churn-model-reader", "model": {"modelFormat": {"name": "mlflow"}, "protocolVersion": "v2", "storageUri": export(version), "resources": {"requests": {"cpu": "500m", "memory": "1Gi"}, "limits": {"cpu": "1", "memory": "2Gi"}}}} if mode == "canary": predictor = {"canaryTrafficPercent": int(percent), **predictor} metadata = {"name": MODEL, "namespace": NAMESPACE, "annotations": {VERSION_KEY: str(version), ALIAS_KEY: alias}} return {"apiVersion": "serving.kserve.io/v1beta1", "kind": "InferenceService", "metadata": metadata, "spec": {"predictor": predictor}} def check(): cmd = ["kubectl", "get", "isvc", MODEL, "-n", NAMESPACE, "-o", "json"] live = subprocess.run(cmd, capture_output=True, text=True, check=True).stdout notes = json.loads(live)["metadata"]["annotations"] served, alias = notes[VERSION_KEY], notes[ALIAS_KEY] expected = MlflowClient().get_model_version_by_alias(MODEL, alias).version if served != str(expected): sys.exit(f"endpoint serves v{served}; @{alias} is v{expected}") print("endpoint matches the registry") if __name__ == "__main__": if sys.argv[1] == "check": check() else: print(yaml.safe_dump(render(*sys.argv[1:3]), sort_keys=False)) ``` For version 13, `deploy.py canary 10` writes the manifest below. `modelFormat: mlflow` with the v2 protocol follows KServe's MLflow example;[^kserve-mlflow] the predictor's service account carries the storage credential,[^kserve-s3] which should be read-only and limited to the serving prefix. Check that the runtime's MLflow version can load the model's [format](#quality-security-privacy-and-compliance). KServe supports canaries only in serverless mode.[^kserve-canary] _Example status: YAML syntax-checked; equal (keys and values) to `render("canary", "10")` in the stubbed run; PyYAML's exact quoting not checked._ ```yaml apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: churn-classifier namespace: ml-serving annotations: example.com/registry-version: '13' example.com/registry-alias: challenger spec: predictor: canaryTrafficPercent: 10 serviceAccountName: churn-model-reader model: modelFormat: name: mlflow protocolVersion: v2 storageUri: s3://ml-artifacts-example/serving/churn-classifier/v13 resources: requests: cpu: 500m memory: 1Gi limits: cpu: '1' memory: 2Gi ``` _Example status: syntax-checked, not executed._ ```bash dvc repro # validate -> train -> gate -> register; stops at the first failing stage dvc push # upload data and pipeline outputs to the DVC remote git add dvc.lock && git commit -m "Retrain churn-classifier" # code committed before # After approval, as the promotion identity and the deploy step: python src/promote.py canary python src/deploy.py canary 10 > deploy/isvc.yaml # or commit it for a GitOps controller kubectl apply -f deploy/isvc.yaml kubectl get isvc churn-classifier -n ml-serving # PREV and LATEST columns: 90/10 # Healthy after 48 h: promote.py champion, then deploy.py full. # Regressing: deploy.py canary 0. python src/deploy.py check # on a schedule ``` These are KServe's documented canary controls: remove `canaryTrafficPercent` to promote, set it to 0 to roll back.[^kserve-canary] After promotion, roll back by pointing `champion` at the previous version and running `deploy.py full`. The scheduled `check` catches an alias moved without a redeploy, or a hand-edited manifest. Signing is not shown; if you add it, sign in `register` and verify in `export()` before the upload. Monitoring uses two custom metrics (names chosen for this example): a counter in the online service, `model_schema_violations_total`, and `model_feature_psi`, which the nightly statistics job pushes to a Prometheus Pushgateway. That is the documented use, "capturing the outcome of a service-level batch job", but the Pushgateway "never forgets series pushed to it", so a stopped job would leave its last value firing;[^prom-pushing] the stale-data rule watches `push_time_seconds`, which it adds to every group.[^pushgateway] A `for` clause keeps an alert pending until its condition has held for the stated time;[^prom-alerting] on a nightly value it only delays the alert, so the drift rule has none, and "two nights in a row" lives in the [retraining policy](#templates). _Example status: YAML syntax-checked, not executed; the 0.2 threshold is a placeholder to be calibrated on your own history._ ```yaml groups: - name: churn-classifier-model rules: - alert: ChurnFeatureDrift expr: max by (feature) (model_feature_psi{model="churn-classifier"}) > 0.2 labels: severity: ticket annotations: summary: "Input drift on {{ $labels.feature }} for churn-classifier" runbook: "https://wiki.example.com/runbooks/churn-classifier-drift" - alert: ChurnStatsStale expr: time() - max(push_time_seconds{job="churn-stats"}) > 26 * 3600 labels: severity: ticket annotations: summary: "The churn statistics job has not pushed for 26 hours" - alert: ChurnSchemaViolations expr: sum(rate(model_schema_violations_total{model="churn-classifier"}[15m])) > 0 for: 15m labels: severity: page annotations: summary: "Serving requests violate the churn-classifier input schema" ``` Schema violations page once their rate has stayed above zero for 15 minutes, because they usually mean skew or an upstream data defect; drift opens a ticket because it needs analysis, not a 3 a.m. response. ### Hands-on labs Labs 1 and 2 need only Python 3.8+ and the standard library, cost nothing and were **executed** for this page; Lab 3 needs third-party packages, and its pipeline was **not executed** here. #### Lab 1 (beginner): detect distribution change with PSI, Jensen-Shannon and L-infinity, then decide what kind it is - **Objective:** compute three distance measures for three features, see what they can and cannot tell you, and judge which change looks like the world and which like the data feeding your pipeline. - **Prerequisites:** Python 3.8+, a terminal, 30 minutes. **Cost and risk:** none (illustrative data). **Steps** 1. Create an empty folder and save the binned counts below as `drift_input.json`. Each list holds the number of rows per bin in the training reference and in one week of serving logs. _Example status: JSON syntax-checked; this exact file was used for the executed run below._ ```json { "model": "churn-classifier", "reference_window": "training snapshot 2026-08", "current_window": "serving log 2026-10-01..2026-10-07", "features": { "tenure_months": { "type": "numeric", "bins": ["<3", "3-6", "6-12", "12-24", "24-36", ">36"], "reference": [1200, 1300, 1800, 2200, 1700, 1800], "current": [2100, 1700, 1900, 1900, 1200, 1200] }, "monthly_charges": { "type": "numeric", "bins": ["<20", "20-40", "40-60", "60-80", "80-100", ">100"], "reference": [900, 1600, 2100, 2300, 1900, 1200], "current": [880, 1650, 2050, 2350, 1870, 1200] }, "contract_type": { "type": "categorical", "bins": ["monthly", "one_year", "two_year", "unknown"], "reference": [5200, 2600, 2200, 0], "current": [5000, 2500, 2100, 400] } }, "thresholds": {"psi": 0.2, "js_distance": 0.1, "linf": 0.03, "min_rows": 1000} } ``` 2. Save the script as `drift_check.py`. _Example status: **executed** on 2026-10-16 with Python 3.8.10; syntax-checked by the validator._ ```python """Compare a serving window with the training reference, feature by feature. Standard library only (Python 3.8+). Input: binned counts in drift_input.json. Metrics: PSI, Jensen-Shannon distance (base 2) and L-infinity distance. """ import json import math import sys EPS = 1e-4 # replaces empty-bin proportions so that log() stays finite def proportions(counts): total = sum(counts) return [max(c / total, EPS) for c in counts], total def psi(ref, cur): return sum((a - e) * math.log(a / e) for e, a in zip(ref, cur)) def js_distance(p, q): m = [(x + y) / 2 for x, y in zip(p, q)] kl = lambda a, b: sum(x * math.log2(x / y) for x, y in zip(a, b) if x > 0) return math.sqrt(0.5 * kl(p, m) + 0.5 * kl(q, m)) def linf(p, q): return max(abs(x - y) for x, y in zip(p, q)) def main(path): cfg = json.load(open(path)) t = cfg["thresholds"] flagged = [] print(f"{'feature':<16}{'rows':>7}{'PSI':>8}{'JS':>8}{'Linf':>8} verdict") for name, f in cfg["features"].items(): ref, _ = proportions(f["reference"]) cur, n = proportions(f["current"]) v_psi, v_js, v_linf = psi(ref, cur), js_distance(ref, cur), linf(ref, cur) reasons = [] if n < t["min_rows"]: reasons.append("too few rows to judge") if v_psi >= t["psi"]: reasons.append("PSI") if v_js >= t["js_distance"]: reasons.append("JS") if f["type"] == "categorical" and v_linf >= t["linf"]: reasons.append("Linf") verdict = "DRIFT (" + ", ".join(reasons) + ")" if reasons else "ok" if reasons: flagged.append(name) print(f"{name:<16}{n:>7}{v_psi:>8.3f}{v_js:>8.3f}{v_linf:>8.3f} {verdict}") print("flagged:", ", ".join(flagged) if flagged else "none") return 1 if flagged else 0 if __name__ == "__main__": sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "drift_input.json")) ``` 3. Run `python3 drift_check.py drift_input.json; echo "exit=$?"`. **Expected output** (verbatim from the executed run): ```text feature rows PSI JS Linf verdict tenure_months 10000 0.108 0.139 0.090 DRIFT (JS) monthly_charges 10000 0.000 0.009 0.005 ok contract_type 10000 0.241 0.141 0.040 DRIFT (PSI, JS, Linf) flagged: tenure_months, contract_type exit=1 ``` 4. **Interpret.** `tenure_months` shifted towards new customers: the "<3" bin grew from 12% to 21%. That looks like **drift** (for example, a sign-up campaign). PSI (0.108) and Jensen-Shannon (0.139) disagree about whether it crosses the line, so thresholds must be calibrated per measure. `contract_type` gained a category, `unknown`, never seen in training. A category that appears overnight is more often an **upstream data defect** (null handling, a join) than customer behaviour: ask the data owner before retraining, or the model will learn that "unknown" is a real contract type. - **Do not let PSI decide this one.** All of `contract_type`'s PSI comes from the empty reference bin, which the script floors at `EPS`. With `EPS` at 1e-3, 1e-4, 1e-5 and 1e-6 the PSI was 0.146, 0.241, 0.333 and 0.425 (at 1e-3 the PSI rule no longer fires), while Jensen-Shannon stayed between 0.132 and 0.142. Catch new categories with a schema or domain check, as TFDV and the [runbook](#worked-runbook-a-new-category-appears-in-production) do. - No distance measure tells drift from skew or a data defect; that takes the parity replay and a look upstream. - **Validation:** check one number by hand. For `tenure_months`, the six PSI terms are 0.0504, 0.0107, 0.0005, 0.0044, 0.0174 and 0.0243, which sum to 0.108. The exit code is 1, so a scheduler would mark the job as failed. - **Troubleshooting:** `ZeroDivisionError` means a feature has no rows in a window; the `min_rows` check exists because measures on small samples swing widely. If you change the bins, every number changes: PSI depends on the number of bins and the sample sizes.[^psi-yurdakul] - **Teardown:** delete the folder. Nothing else was created. #### Lab 2 (intermediate): a promotion gate that catches a slice regression - **Objective:** apply a written promotion policy to a champion and a challenger, and see why an average uplift is not enough. - **Prerequisites:** Python 3.8+, 20 minutes, the `sed` command (Linux or macOS). **Cost and risk:** none. **Steps** 1. Save the policy as `gate_policy.json` and the evaluation results as `candidates.json`. Both models were scored on the same frozen test set upstream; the gate only applies the policy. _Example status: JSON syntax-checked; these exact files were used for the executed run._ ```json { "primary_metric": "auc", "min_uplift": 0.005, "max_slice_drop": 0.02, "p95_latency_ms_max": 50, "required_checks": ["data_validation", "signature_match", "lineage_recorded"] } ``` ```json { "champion": { "version": "v12", "metrics": {"auc": 0.842}, "slices": {"region=north": 0.851, "region=south": 0.829, "tenure<6m": 0.781}, "p95_latency_ms": 31 }, "challenger": { "version": "run-7f3a9c21", "metrics": {"auc": 0.851}, "slices": {"region=north": 0.866, "region=south": 0.842, "tenure<6m": 0.752}, "p95_latency_ms": 34, "checks": {"data_validation": true, "signature_match": true, "lineage_recorded": true} } } ``` 2. Save the gate as `promotion_gate.py`. _Example status: **executed** with Python 3.8.10 (first edition on 2026-10-16; this revised version on 2026-10-17); syntax-checked by the validator._ ```python """Decide whether a challenger model may replace the champion. Standard library only (Python 3.8+). Evaluation results are produced upstream on the same frozen test set for both models; this script only applies policy. Exit code 0 = promote, 1 = reject (so a CI job can stop the pipeline). """ import json import sys def decide(policy, champion, challenger): reasons = [] for check in policy["required_checks"]: if not challenger["checks"].get(check, False): reasons.append(f"required check failed: {check}") m = policy["primary_metric"] uplift = challenger["metrics"][m] - champion["metrics"][m] if uplift < policy["min_uplift"]: reasons.append(f"{m} uplift {uplift:+.3f} < {policy['min_uplift']:+.3f}") for name, champ_value in champion["slices"].items(): drop = champ_value - challenger["slices"].get(name, 0.0) if drop > policy["max_slice_drop"]: reasons.append(f"slice {name}: {m} drops by {drop:.3f}") if challenger["p95_latency_ms"] > policy["p95_latency_ms_max"]: reasons.append(f"p95 latency {challenger['p95_latency_ms']} ms over budget") return uplift, reasons def main(policy_path, candidates_path): policy = json.load(open(policy_path)) c = json.load(open(candidates_path)) uplift, reasons = decide(policy, c["champion"], c["challenger"]) champ, chall = c["champion"]["version"], c["challenger"]["version"] print(f"champion {champ} vs challenger {chall}: " f"{policy['primary_metric']} uplift {uplift:+.3f}") if reasons: print("REJECT") for r in reasons: print(" -", r) return 1 print("PROMOTE: register as challenger; canary after approval; champion kept") return 0 if __name__ == "__main__": sys.exit(main(*sys.argv[1:3])) ``` 3. Run the gate, then simulate a retrained challenger that fixed the new-customer slice, and run it again: _Example status: **executed** on 2026-10-17 (GNU sed 4.7)._ ```bash python3 promotion_gate.py gate_policy.json candidates.json; echo "exit=$?" sed -i 's/"tenure<6m": 0.752/"tenure<6m": 0.779/' candidates.json python3 promotion_gate.py gate_policy.json candidates.json; echo "exit=$?" ``` **Expected output** (verbatim): ```text champion v12 vs challenger run-7f3a9c21: auc uplift +0.009 REJECT - slice tenure<6m: auc drops by 0.029 exit=1 champion v12 vs challenger run-7f3a9c21: auc uplift +0.009 PROMOTE: register as challenger; canary after approval; champion kept exit=0 ``` - **What it teaches:** the challenger is better on average (+0.009 AUC) and in both regions, yet worse for customers in their first six months, the group the retention team targets. An average-only gate would have shipped it. - **Validation:** exit code 1 then 0; in CI, the first run must fail the job. - **Troubleshooting:** `KeyError` means a slice or check name differs between the two files; on macOS, use `sed -i ''`. A slice with few examples gives noisy AUC: require a minimum count per slice, as `train.py` does. - **Teardown:** delete the folder. #### Lab 3 (advanced): a local continuous-training loop with DVC and MLflow - **Objective:** run the reference pipeline locally, register a candidate only when the gate passes, and promote it by alias. - **Prerequisites:** Python 3.11 or later (scikit-learn 1.9 requires it; MLflow 3.17 requires 3.10, as of October 2026), Git, about 1 GB of disk for packages, 60–90 minutes. **Cost and risk:** no cloud resources; everything runs on `127.0.0.1`. **Status:** the pipeline was **not executed** for this page; expected results follow the tools' documentation. **Steps** Three files the lab needs are not shown above: `params.yaml`, `src/validate.py`, and `make_data.py`, which writes 12,000 synthetic customers with a change planted in April 2026 (a rival's offer to monthly customers paying over 90). _Example status: `make_data.py` and `validate.py` **executed** on 2026-10-17 with Python 3.8.10: 12,000 rows, validation passed (exit 0); with one `churned` value set to 2 it failed (exit 1). `params.yaml` parsed with PyYAML._ ```yaml # params.yaml - DVC re-runs `train` when one of these values changes train: learning_rate: 0.1 max_depth: 4 cutoff_date: "2026-04-01" # rows from this date on form the test set ``` ```python """Synthetic customers for Lab 3: 12 monthly snapshots of 1,000 rows each.""" import csv import math import pathlib import random CONTRACTS, WEIGHTS = ["monthly", "one_year", "two_year"], [6, 3, 1] random.seed(7) out = pathlib.Path("data/raw/customers.csv") out.parent.mkdir(parents=True, exist_ok=True) with out.open("w", newline="") as f: w = csv.writer(f, lineterminator="\n") w.writerow(["snapshot_date", "region", "tenure_months", "monthly_charges", "contract_type", "support_calls_90d", "churned"]) for i in range(12): # 2025-10-01 ... 2026-09-01 year, month = 2025 + (9 + i) // 12, (9 + i) % 12 + 1 for _ in range(1000): tenure = min(int(random.expovariate(1 / 24)), 72) contract = random.choices(CONTRACTS, WEIGHTS)[0] charges = round(random.uniform(20, 120), 2) calls = min(int(random.expovariate(1 / 1.5)), 50) z = (-2.5 + 1.0 * (contract == "monthly") + 0.02 * (charges - 70) + 0.3 * calls - 0.04 * tenure) rival = (year, month) >= (2026, 4) and contract == "monthly" if rival and charges > 90: z += 1.5 # planted change: a rival's offer from April 2026 churned = int(random.random() < 1 / (1 + math.exp(-z))) region = random.choice(["north", "south"]) w.writerow([f"{year}-{month:02d}-01", region, tenure, charges, contract, calls, churned]) print(f"wrote {out}") ``` ```python """Schema and range checks on the raw CSV; exits 1 if any check fails.""" import csv import json import pathlib import sys RULES = { # column -> check on the raw string value "snapshot_date": lambda v: len(v) == 10 and v[4] == v[7] == "-", "region": lambda v: v in ("north", "south"), "tenure_months": lambda v: 0 <= int(v) <= 600, "monthly_charges": lambda v: 0 < float(v) < 1000, "contract_type": lambda v: v != "", # unseen types are mapped, not rejected "support_calls_90d": lambda v: v == "" or int(v) >= 0, # empty -> 0 later "churned": lambda v: v in ("0", "1"), } def main(csv_path, report_path): errors, rows = [], 0 with open(csv_path, newline="") as f: reader = csv.DictReader(f) missing = sorted(set(RULES) - set(reader.fieldnames or [])) if missing: errors.append(f"missing columns: {missing}") for rows, row in enumerate(reader, start=1): for col, ok in RULES.items(): try: good = ok(row.get(col) or "") except (ValueError, IndexError): good = False if not good: errors.append(f"row {rows}: {col}={row.get(col)!r}") passed = not errors and rows > 0 pathlib.Path(report_path).parent.mkdir(parents=True, exist_ok=True) report = {"passed": passed, "rows": rows, "error_count": len(errors), "errors": errors[:10]} json.dump(report, open(report_path, "w"), indent=2) print(f"validation {'passed' if passed else 'FAILED'}: {rows} rows, " f"{len(errors)} errors") return 0 if passed else 1 if __name__ == "__main__": sys.exit(main(sys.argv[1], sys.argv[2])) ``` _Example status: syntax-checked, not executed._ ```bash python3 -m venv .venv && . .venv/bin/activate pip install "mlflow>=3.17,<4" "dvc>=3.67,<4" "scikit-learn>=1.9,<1.10" \ pandas pyyaml pytest git init churn && cd churn && dvc init # No authentication: this server must stay on 127.0.0.1 and never be exposed. mlflow server --backend-store-uri sqlite:///mlflow.db \ --default-artifact-root ./mlartifacts --host 127.0.0.1 --port 5000 & export MLFLOW_TRACKING_URI=http://127.0.0.1:5000 printf 'mlflow.db\nmlartifacts/\n' >> .gitignore # keep the tracking store out of Git # Copy src/, tests/, gate_policy.json and dvc.yaml from the reference # implementation, save the three files above (validate.py in src/), then: python make_data.py dvc add data/raw/customers.csv && git add -A && git commit -m "Baseline" dvc repro ``` 1. The first `dvc repro` stops in `train`, because no champion exists yet. Register the model that run logged as version 1 of `churn-classifier` (in the MLflow UI, or with `mlflow.register_model`), give it the alias `champion`, and run `dvc repro` again. Expect `REJECT` (uplift +0.000): the challenger is retrained from the same data, parameters and seed. 2. Set `train.cutoff_date` to `"2026-07-01"`, commit it (uncommitted code or parameters fail `lineage_recorded`), and run `dvc repro`: only the stages that depend on that parameter re-run. The first real comparison: the challenger has seen three months of the planted change, the champion none. 3. If the gate prints `PROMOTE`, run `python src/promote.py canary` and then `python src/promote.py champion` (locally you play the promotion identity and skip KServe). If it prints `REJECT`, read the reasons and stop. - **Expected output on PROMOTE:** all four stages succeed; `churn-classifier-candidates` has a version tagged `gate=passed`, `git_commit` and `data_md5`, aliased `challenger`; after both `promote.py` calls, a new `churn-classifier` version holds `champion`; `dvc status` reports nothing changed. - **Expected output on REJECT:** `dvc repro` exits non-zero at `gate`, `reports/gate.txt` holds the reasons, no registry version is created, and `dvc status` lists `gate` as changed. - **Validation:** loading `models:/churn-classifier@champion` returns the version the alias points to; moving the alias changes what the next load returns.[^mlflow-workflow] `PYTHONPATH=src pytest -q -k unseen tests/` passes (the parity test also needs a sample of logged requests). - **Troubleshooting:** a gate stage that "succeeds" with a `REJECT` usually had its output piped (for example through `tee`), so the stage reports the last command's exit code; keep the plain redirect, or use `set -o pipefail`. If registry calls fail, check `MLFLOW_TRACKING_URI`. On a shared server, use `--app-name basic-auth` with separate training and promotion users.[^mlflow-auth] - **Teardown:** stop the server (`kill %1`), `deactivate`, delete the folder and the virtual environment. ### Use cases and scenarios | Scenario | Context | What was done | Trade-offs | Success measure | |---|---|---|---|---| | **One path to production** (real: Uber, 2017) | Teams were "building bespoke one-off systems to use these models in production"[^uber-michelangelo] | A shared platform for six steps (manage data, train, evaluate, deploy, predict, monitor); a shared feature store with the same feature expressions at training and prediction time; predictions joined with outcomes | A large internal platform to build and staff | Reported P95 latency under 10 ms for online models with stored features; top models served over 250,000 predictions per second (2017 figures) | | **Continuous refresh** (real: Google Play, 2017) | Models refreshed continuously as new data arrives | TFX standardised validation, training, analysis and serving; time to production fell "from the order of months to weeks"[^tfx-paper] | A standard platform constrains unusual needs | Time to production; install rate | | **A sudden world change** (real: spring 2020) | Pandemic behaviour disrupted models: an inventory forecaster "was never trained on a spike like this"; at a fraud-detection firm, surging purchases of garden and power tools meant engineers had to step in[^mittr-pandemic] | Reported responses: engineers intervening by hand; companies adjusting rankings and content filters | Retraining on a few abnormal weeks may not generalise | Time to detect; manual-review load; losses | | **Offline gains, no business gain** (real: Booking.com, 2019) | About 150 customer-facing models, validated with randomised controlled trials[^booking-kdd] | Online experiments as the final gate; per Colyer's summary, offline gains did not necessarily become business value, and extra latency had a measurable conversion cost[^booking-summary] | Experiments take traffic and time | Business metric in a controlled experiment, not offline AUC | | **The limit of pipelines** (real: Zillow Offers, 2021) | Zillow wound down home buying, citing "the unpredictability in forecasting home prices", with a $304 million inventory write-down[^zillow-pr][^zillow-letter] | — (a business decision) | Pipelines cannot make an unpredictable target predictable; exposure limits belong to [ModelOps](../modelops/) and [DecisionOps](../decisionops/) | Exposure and loss limits per decision | Read together (an editorial synthesis, not a finding of the sources): the platforms that worked invested in the plumbing between data, model and outcome, and the painful cases involved a world that changed faster than feedback reached the models' owners. ## Measure, secure and pay ### Metrics and KPIs Measure the *system* that produces and runs models, not only the models. Service metrics are defined in [SRE](../sre/) and cost per prediction in [FinOps for AI](../finops-for-ai/); the table lists model-specific measures. | KPI | Definition / formula | Data source | Pitfall | Cadence | |---|---|---|---|---| | Model freshness | Review date − training-data cutoff of the serving version | Registry tags | Meaningless without a freshness requirement from a staleness test | Weekly | | Retrain-to-serve lead time | Time the champion alias moves − time of the trigger | Pipeline and registry events | Hides a long manual approval; split it into stages | Per retrain | | Gate rejection rate | Rejected challengers ÷ evaluated challengers | Gate reports | 0% may mean a toothless gate; 100% may mean upstream trouble | Monthly | | Skew rate | Replayed requests whose offline features differ ÷ replayed requests | Parity test on logged requests | Sampling only "easy" traffic | Daily | | Drift-alert precision | Alerts that led to an action ÷ all alerts | Alert log and tickets | Optimising it by never alerting; also track incidents with no prior alert | Weekly | | Label coverage | Predictions with a matured label ÷ predictions whose label window has closed | Outcome joins | Measuring quality before labels mature | Monthly | | Live quality and offline gap | Metric on the labelled window; gap = offline estimate − live value | Outcome joins, gate report | A large gap points to skew or leakage, not "bad luck" | Monthly | | Rollback time | Time to serve the previous version after the decision | Drills and incidents | Never measured until the first real incident | Quarterly drill | | Lineage coverage | Serving versions with complete lineage ÷ serving versions | Registry audit | "Complete" must be defined: data hash, commit, parameters, environment | Quarterly | **Leading and lagging.** Skew, input drift, prediction distribution and freshness are *leading* signals: available now, weakly related to harm. Label-based quality and business outcomes are *lagging*: decisive, but late. Show both, labelled. **Vanity metrics:** models in production, retraining runs, and offline gains never confirmed online. ::: example Worked calculation: a monthly review of the churn model (illustrative numbers) - **Freshness:** the serving version was trained with a cutoff of 2026-09-01; on 2026-10-16 it is **45 days** old. A staleness test showed quality holding for about five weeks and then falling, so the policy maximum is 35 days: **breach**. - **Skew rate:** 46 of 2,000 replayed requests produced different features offline, **2.3%** against a target of 0%. All 46 involved `support_calls_90d`, which a recent change to the online service had started computing over 60 days. - **Drift-alert precision:** 3 of 12 alerts led to action, **25%**: the thresholds need recalibrating. - **Label coverage:** of 180,000 predictions whose 30-day window has closed, 162,000 have a label, **90%**. - **Offline gap:** offline AUC 0.842, live AUC on the labelled window 0.821, gap **0.021**: large enough to look for skew. - **Lead time:** trigger Monday 09:00, alias moved Wednesday 15:30: **54.5 hours**, of which 48 hours were the canary. **Reading:** fix the skew first (it also explains part of the offline gap), then retrain, because retraining now would learn the defect. Recalibrate the drift thresholds against the last six "normal" weeks. ::: ### Quality, security, privacy and compliance - **Model files can execute code.** Python's documentation states that "The pickle module is not secure" and that malicious data can "execute arbitrary code during unpickling".[^python-pickle] scikit-learn adds: "You should never load a pickle file from an untrusted source". Its safer options have conditions: `skops.io` loads only types trusted by default or by you (review what `get_untrusted_types` returns), and because "arbitrary computations can be represented using the ONNX format", ONNX models should be served in a sandbox.[^sklearn-persistence] Safetensors is a safe alternative to pickle for tensors.[^safetensors] Load third-party models only in such formats, under those conditions, or in an isolated environment. MLflow 3 logs scikit-learn models as skops except on Databricks, but `autolog()` defaults to cloudpickle; other types must be listed in `skops_trusted_types` when the model is logged.[^mlflow-sklearn] Since skops 0.15 these include the node structures of tree models, which a crafted file can use to crash `predict()`.[^skops-changes] - **Treat the registry as a deployment control, and check what it can enforce.** Moving `champion` is a deployment: restrict it like production deploy rights and log every change. Open-source MLflow has no authentication unless started with `--app-name basic-auth`; then the default permission is READ, and "Create Model Version" and "Set Registered Model Alias" both need `can_update` on the registered model,[^mlflow-auth] so whoever can register versions can move that model's aliases. MLflow's answer is separate registered models per environment:[^mlflow-workflow] in the reference implementation the training pipeline writes only to a candidates model, and a promotion identity copies approved versions into production with `copy_model_version` and moves aliases there.[^mlflow-client] Serving gets a read-only credential for its prefix.[^kserve-s3] - **Sign what you serve.** Sign artifacts at registration and verify before serving; Sigstore's `model-transparency` signs a model directory and verifies file hashes. With keyless signing, each signing event is recorded in Sigstore's append-only transparency log, which signers are told to monitor for their identity; if that is unacceptable for a private model, use a private Sigstore instance or key-based signing.[^model-signing] Build-time supply-chain controls belong to [DevSecOps](../devsecops/). - **Training data is the crown jewel.** Keep the data zone separate, grant access per project, and treat prediction logs that store served features like the source tables. The legal basis for training on personal data belongs to [PrivacyOps](../privacyops/). - **Poisoned or self-made labels.** Validate new training data against expectations before training, and be wary of labels that the model's own actions produced ([feedback loops](#advanced-topics-and-emerging-direction)). - **Evidence.** Lineage, gate reports, approvals, model cards (intended use and evaluation, including across groups)[^model-cards] and datasheets (a dataset's motivation, composition and collection)[^datasheets] are what an auditor will ask for. Mapping them to regulations belongs to [ModelOps](../modelops/); collecting evidence across systems to [ComplianceOps](../complianceops/). ### Cost, capacity and sustainability Cost drivers: training compute (runs × cost per run), serving (predictions × cost per prediction; does a small tabular model need a GPU at all?), storage (snapshots, features, artifacts, prediction logs) and people (reviews, on-call, labelling). The ML team controls three levers: **serving pattern** (batch avoids always-on capacity), **retraining frequency** and **model size**. A simple rule for retraining frequency, proposed by this Atlas: retrain more often only while the value recovered by fresher models exceeds the cost of the extra runs and reviews. Express both sides in the same unit. ::: example Retraining frequency, in money (illustrative numbers) A staleness test shows that a weekly-retrained model keeps about 40 more at-risk customers a month than a monthly-retrained one, and the business values a retained customer at 50 (currency units). Extra value: 40 × 50 = 2,000 a month. Moving from monthly to weekly adds three runs a month at 120 of compute each, plus two hours of review per run at 60 an hour: 3 × (120 + 120) = 720. Weekly retraining pays (2,000 > 720). Daily retraining would add about 26 more runs (6,240 a month) for little extra freshness, so it does not. ::: Unit economics and commitments belong to [FinOps for AI](../finops-for-ai/), GPU cluster capacity to [KubernetesOps](../kubernetesops/), and carbon accounting to [GreenOps](../greenops/); retraining frequency and model size are also energy levers. ### Assurance of the operating model How do you know the practice itself works, not just one model? - **Quarterly scores and audits.** Score each production model with the ML Test Score and track the minimum section score;[^ml-test-score] rebuild one randomly chosen model from its lineage and check its metrics land within tolerance. - **Registry hygiene.** The reconciliation check passes for every endpoint; every model lists its owner and consumers. - **Monitor the monitors.** List incidents that no alert preceded; each one is a missing monitor. - **Rehearse.** Each quarter, run a rollback drill and a "game day" that injects a schema change in staging. ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Remediation | Prevention | |---|---|---|---|---| | Good offline, poor online from day one | Skew or leakage | Replay logged requests through offline features; re-check the split for future information | Fix the feature path; re-evaluate with a time-based split | One feature definition; leakage review in the gate | | Quality declines slowly over weeks | Data drift or staleness | Drift trend per feature; staleness curve | Retrain on recent data; check whether features still carry signal | Freshness requirement and policy | | Prediction distribution jumps overnight | Upstream data defect: new category, units, nulls, a changed join | Parity replay (passes); diff schemas and statistics against yesterday; read upstream change logs | Fix upstream or add a serving-side guard; pause CT; rescore | Data contracts with producers ([DataOps](../dataops/)) | | Reloaded model gives different results or fails to load | Library or environment mismatch | Compare library versions in lineage with the serving image | Rebuild the image from the recorded environment | Pin versions; record them in the registry[^sklearn-persistence] | | Every retrained challenger loses to the champion | Label problems, a feedback loop, or a test set that no longer represents traffic | Inspect label distributions and the test window | Fix labels; refresh the test window; consider exploration | Track gate rejection rate | | Drift alerts fire daily and nobody acts | Uncalibrated thresholds; seasonality | Alert precision; compare with last year's same period | Recalibrate on history; seasonal references | Monthly alert review | | Latency spikes after a promotion | Larger model, slower feature lookups, cold starts | Compare canary and champion latency | Roll back; optimise or resize | Latency budget in the gate | | A "retired" model cannot be switched off | Undeclared consumers | Access logs on prediction outputs | Notify and migrate consumers | Consumer list and access control on outputs[^sculley-2015] | | A training job dies hours in, or never starts | Preemption without checkpoints; out of memory; no quota | Exit reason (for example `OOMKilled`); checkpoint age; queue status | Resume from the last checkpoint; smaller batches; request quota | Scheduled checkpoints, tested resume, quotas ([training jobs](#operating-training-jobs)) | | The endpoint serves another version than its alias | Alias moved without a redeploy; hand-edited manifest | Reconciliation check; registry audit log | Re-render from the alias, or move it back | Deploy only through the deploy step | ### Worked runbook: a new category appears in production ::: example Illustrative incident, continuing Lab 1 **Monday 06:05.** The nightly statistics job raises `ChurnFeatureDrift` for `contract_type`. The batch scores feed retention offers sent at 08:00. ::: ::: figure Runbook: from drift alert to recovery, pausing training first ```mermaid flowchart TB A["Drift alert on one feature"] --> B["Pause continuous training"] B --> C{"Parity replay: same features in both paths?"} C -->|"No: skew"| D["Fix the code path"] C -->|"Yes"| U{"Upstream data or its meaning changed?"} U -->|"Yes: data defect"| F["Fix upstream or guard in serving"] U -->|"No: drift"| E["Estimate impact with labels or proxies"] D --> G["Rescore affected entities"] F --> G E --> H{"Harmful?"} H -->|"Yes"| I["Retrain on valid data, canary, promote"] H -->|"No"| J["Recalibrate the alert"] G --> K["Verify, quarantine bad days, resume training"] I --> K J --> K K --> L["Post-incident review and new validation rule"] classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 class A risk class C,U,H decision class B,D,E,F,G,I,J,K,L core ``` Text equivalent: A drift alert on one feature arrives. The first action is to pause continuous training. Then run the parity replay: if training and serving produce different features, it is skew, so fix the code path and rescore the affected entities. If they match, ask whether the upstream data or its meaning changed. If so, it is an upstream data defect: fix it upstream or add a guard in serving, then rescore. If not, it is drift: estimate the impact with labels or proxies; if harmful, retrain on valid data, canary and promote; if not, recalibrate the alert. All paths end with verification, quarantining the bad days from training data, resuming training, and a post-incident review that adds a new validation rule. ::: The response, step by step (illustrative): 1. **Contain (06:15).** Pause the retraining trigger: a second night above the threshold would otherwise start a retraining run on the defect ([retraining policy](#templates)). For the 08:00 offers, use yesterday's scores for customers whose `contract_type` is `unknown` (the agreed fallback) and today's scores for everyone else. 2. **Classify (06:30).** The parity replay passes: both paths apply the same `features.py`, so this is not skew. But 4% of serving rows now carry `unknown`, a value absent from the training reference. Upstream change log: on Sunday evening a changed join in the CRM export stopped matching some contract records, so those customers get `unknown` instead of their real contract type. Verdict: an **upstream data defect**, not skew and not customer behaviour; retraining would teach the model the defect. 3. **Remediate (by 11:00).** The CRM team reverts the join; the data team backfills the affected rows; the ML engineer rescores the 400 affected customers, replacing the fallback scores. 4. **Verify (next nightly run).** `contract_type` PSI near zero, parity replay passing, prediction distribution back to normal. 5. **Resume.** Mark Sunday and Monday as quarantined in the training-data snapshot, then resume the trigger. 6. **Learn.** Add a validation rule at ingestion and at serving ("share of `unknown` ≤ 0.5%"), agree a data contract with the CRM export owner, and record the incident in the model's history. ### Anti-patterns and myths Several anti-patterns are the practices above in reverse: **the notebook as the production system** (Practice 1), **gates on the average** (Practice 3 and [Lab 2](#hands-on-labs)), **nightly retrain and auto-promote** (gates, canaries and pause rules; see the [judgement call](#a-judgement-call-retrain-nightly-or-only-when-the-data-says-so)) and **one drift dashboard with default thresholds**, where everything is red, so nothing is. Four deserve a closer look: - **Platform before purpose.** A year spent building a platform before any model has an owner and a metric. Rules of ML: keep the first model simple and get the infrastructure right.[^rules-ml] Instead: build the platform around the first two real models. - **Registry as a file share.** Services load `s3://…/model_final_v2.pkl`, and nobody can say which version is live. Instead: load by alias, or let a deploy step derive the path from an alias and check it on a schedule. - **Judging a model before its labels mature.** A 30-day label judged after 7 days measures the label pipeline, not the model. - **Myth: "More frequent retraining is always better."** It costs money and review time and can amplify feedback loops; measure staleness first. ## Deciding and adopting ### When to use and when not to use **Invest in MLOps when** a model serves repeated decisions in production; its world changes (prices, behaviour, fraud, sensors); several models or people depend on the same data and infrastructure; errors are costly or regulated; or the model must outlive the person who built it. **Use little or none of it when** a rule performs nearly as well (Rules of ML #1);[^rules-ml] the model supports a one-off analysis or an annual report; or no label will ever tell you whether it was right (then question ML itself). Even a prototype deserves versioned code and data, one tracked run and a registered artifact. **Hidden responsibilities:** on-call for models; a label pipeline, often the hardest part; retention for snapshots and logs; documentation; retiring models. **Alternatives:** rules; managed AutoML; or a model bought as a service, whose outputs and approval you still own ([ModelOps](../modelops/), and [LLMOps](../llmops/) for LLM providers). ### Decision framework ::: figure How much MLOps does this model need? ```mermaid flowchart TB A{"Does a simple rule do nearly as well?"} -->|"Yes"| R["Use the rule; revisit later"] A -->|"No"| B{"Repeated decisions in production?"} B -->|"No"| C["One-off analysis: version code and data only"] B -->|"Yes"| D{"Does its world change faster than you can retrain by hand?"} D -->|"No"| E["Level 1: reproducible runs, registry, manual retrain"] D -->|"Yes"| F{"Do labels arrive within that period?"} F -->|"No"| H["Monitor proxies, schedule retrains, invest in labels"] F -->|"Yes"| G["Level 2 to 3: CT pipeline with gates and canary"] G --> I{"Several models or teams?"} H --> I I -->|"Yes"| J["Shared platform: registry, features, serving"] I -->|"No"| K["Per-team stack on managed services"] classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class A,B,D,F,I decision class E,G,H,J,K core class R,C neutral ``` Text equivalent: If a simple rule does nearly as well, use it and revisit later. If the model makes no repeated production decisions, treat it as a one-off analysis: version code and data only. If its world changes no faster than you can retrain by hand, aim for level 1 (reproducible runs, a registry, manual retraining). If it changes faster and labels arrive within that period, build a continuous-training pipeline with gates and a canary (levels 2 to 3); if labels come later, monitor proxies, schedule retraining and invest in labels. With several models or teams, build a shared platform for registry, features and serving; otherwise keep a per-team stack on managed services. ::: The tree assumes that labels eventually exist, that a platform pays off only across several models, and that errors cost moderately. For high-stakes decisions, the [ModelOps](../modelops/) risk tier sets minimum gates regardless of change rate. ### Maturity model This is **this Atlas's model**, not an official one, mapped approximately to Google's levels 0–2 and Microsoft's levels 0–4;[^gc-mlops][^ms-maturity] Microsoft notes that organisations often show traits of several levels at once. Not every model should reach level 4. | Level | Observable behaviour | Exit evidence | Approx. Google / Microsoft | |---|---|---|---| | **0 Ad hoc** | Notebooks, model files handed over, no monitoring | — | G0 / M0–M1 | | **1 Reproducible** | Versioned code, data snapshots and environment; tracked runs; registry with lineage; retraining manual but repeatable | A second person rebuilds the model within tolerance | G0 / M2 (partly) | | **2 Automated training** | Pipeline runs on triggers; data validation stops bad runs; metadata recorded automatically | Three consecutive triggered runs without manual steps | G1 / M2 | | **3 Gated delivery** | Pipeline code under CI/CD; promotion gates as code; shadow or canary; rehearsed rollback | A rejected challenger and a rollback drill, both logged | G2 / M3 | | **4 Closed loop** | Label joins; calibrated alerts; policy-driven retraining with pause rules; ML Test Score reviewed; retirement routine | Alert precision tracked; staleness within policy for a quarter | — / M4 | ### Adoption roadmap: the first 90 days | Window | Actions | Owner | Exit metric | |---|---|---|---| | **Days 1–30** | Pick two production models with clear owners. Record lineage for the current versions; put training code and a data snapshot under version control; register both models and make serving follow an alias (directly or through a deploy step); write the RACI | ML engineering lead | Both models rebuildable by a second person; every serving path traces to an alias | | **Days 31–60** | Turn one training procedure into a pipeline with data validation; add the promotion gate with slice checks; log served features and run a daily parity replay; record reference statistics | ML engineer + data engineer | Gate in CI; skew rate measured; first drift dashboard with owners | | **Days 61–90** | Add canary or shadow release; rehearse a rollback; write the retraining policy from a staleness test; hold the first monthly model review; score both models with the ML Test Score | Model owners + platform/SRE | Rollback under 15 minutes; documented retraining policy; baseline ML Test Score | ## Toolkit ### Checklists **Production readiness (before a model first serves traffic)** - [ ] A named owner, a success metric, guardrail metrics and a non-ML baseline exist. - [ ] Training is reproducible from recorded lineage (commit, data hash, parameters, environment). - [ ] Training and serving import one feature definition; a parity test passes on replayed requests. - [ ] The test split respects time and was reviewed for leakage. - [ ] The model is registered with a signature; serving follows an alias, and the reconciliation check passes. - [ ] Reference statistics are stored; drift, skew and service alerts have owners. - [ ] The label pipeline and its delay are documented. - [ ] Rollback and the non-ML fallback have been rehearsed. **Promotion preflight (each challenger)** - [ ] Same frozen test set as the champion; slices and latency checked; required checks pass. - [ ] Training data passed validation and contains no quarantined days. - [ ] Shadow or canary plan, duration and abort criteria are written. - [ ] Approvals required by the model's risk tier are recorded in the registry; only the promotion identity can move aliases. **Weekly model health review** - [ ] Freshness within policy; skew rate at zero or explained. - [ ] Drift alerts reviewed; alert precision updated. - [ ] Live quality on matured labels compared with the gate's estimate. - [ ] Open incidents and paused pipelines have owners and dates. **Retirement** - [ ] Consumers identified from access logs and notified; replacement or fallback agreed. - [ ] Endpoint and batch jobs removed; aliases deleted; artifacts and lineage retained per policy. ### Templates **1. Model release record**, stored with the registry version and reviewed at promotion. _Example status: YAML syntax-checked; fields are this Atlas's proposal._ ```yaml model: churn-classifier version: "13" # production model; candidate run-7f3a9c21 owner: retention-ml@example.com decision_supported: "Which customers receive a retention offer this week" lineage: code_commit: "" data_snapshot: "data/raw/customers.csv md5 " environment: "registry.example.com/churn-train@sha256:" evaluation: test_window: "2026-08-01..2026-08-31" auc: 0.851 champion_auc: 0.842 slices: {"region=north": 0.866, "region=south": 0.842, "tenure<6m": 0.779} gates_passed: [data_validation, signature_match, lineage_recorded, slice_check, latency] release: method: canary traffic_percent: 10 duration_hours: 48 abort_if: "AUC proxy or error rate worse than the champion" risk_tier: "medium (ModelOps)" approvals: [{role: product-owner, date: "2026-10-14"}] consumers: [retention-batch, support-tool] rollback: during_canary: "python src/deploy.py canary 0" after_promotion: "point alias champion at v12, then python src/deploy.py full" ``` **2. Retraining policy** _Example status: YAML syntax-checked; thresholds are placeholders to calibrate._ ```yaml model: churn-classifier freshness_max_days: 35 # from the staleness test triggers: - schedule: weekly # Wednesday 02:00 UTC - new_labels_min_rows: 20000 - drift: {measure: psi, feature_any_above: 0.2, consecutive_nights: 2} do_not_retrain_when: - data_validation_failed - open_data_incident - labels_generated_by_model_actions_above_share: 0.5 promotion: {gate: gate_policy.json, human_approval: true} review: monthly ``` **3. Architecture decision record: serving pattern** ```text Title: Nightly batch scores, plus an online lookup of the latest batch score Context: offers go out daily; support agents need a score during calls; features change daily. Options: (a) batch only; (b) online model service; (c) batch scores served online from a key-value store. Decision: (c). Key-value lookup latency; one model run per night; one monitoring path. Consequences: scores are up to 24 hours old; new customers get a default until the next run. Revisit when: offers become real-time, or same-day behaviour changes the score materially. ``` ### Deep comparisons | Criterion | Managed cloud platform | Assembled open-source stack | Minimal stack | |---|---|---|---| | Typical parts | Provider's pipelines, registry, endpoints, monitoring | Orchestrator, MLflow-style registry, feature store, KServe-style serving, drift jobs | Git, DVC or table snapshots, one tracking server, batch scoring job | | Time to first value | Fast if data is in that cloud | Slow: needs a platform team | Fast | | Portability | Low to medium | High | High | | Operating effort | Lowest | Highest | Low | | Main risk | Lock-in, mode-specific limits[^sagemaker-shadow] | Upgrade treadmill, licences[^seldon-license] | Outgrown when models multiply | | Fits | Few ML engineers, one cloud | Many models, Kubernetes skills, multi-cloud | One to three models, early stage | ### FAQ ::: answer Is MLOps just DevOps applied to machine learning? It reuses DevOps practices, but adds things ordinary software lacks: testing data and models, continuous training, and monitoring for skew and drift. ::: ::: answer How is MLOps different from LLMOps and ModelOps? In this Atlas, MLOps engineers the lifecycle of trained predictive models, [LLMOps](../llmops/) operates LLM applications, and [ModelOps](../modelops/) governs the whole portfolio. These are editorial boundaries ([what MLOps is not](#what-mlops-is-not)). ::: ::: answer Do we need a feature store? Not at first. You need one feature definition for both paths and point-in-time-correct training sets; a shared library and careful joins do that for a few models. A store pays off when many models share features or need low-latency online reads.[^feast] ::: ::: answer How often should we retrain? As often as a staleness test says quality falls by more than you can accept, and only when the economics work ([cost](#cost-capacity-and-sustainability)). ::: ::: answer We fine-tune an LLM on our own data. Is that MLOps or LLMOps? Both, split by object. The fine-tuning pipeline (data snapshot, training run, lineage, registry, gates) is MLOps. Whether to fine-tune, use retrieval or change the prompt, and how to evaluate the application's answers, belongs to [LLMOps](../llmops/). ::: ## Going further ### Advanced topics and emerging direction - **Online and continual learning** (established in research, rarer in production): models that update with each event remove retraining delay but make governance harder; the CD Foundation's 2024 roadmap says such models sit "outside the governance constraints of current MLOps platforms".[^cdf-roadmap] - **Feedback loops and exploration** (established problem, active research): when a model chooses what it later learns from, plain supervised retraining is biased. Sculley et al. note that bandit algorithms are the theoretically correct answer but may not scale to large action spaces.[^sculley-2015] A practical mitigation (Atlas suggestion): keep a small randomised share of decisions and log the probability with which each action was chosen, so that later training can correct for the model's own influence. - **Convergence with LLM tooling** (vendor direction, not a settled practice): MLflow presents itself on PyPI as a platform for agents and LLMs as well as traditional models;[^mlflow-pypi] KServe calls itself a "Generative and Predictive AI Inference Platform";[^kserve-cncf] Microsoft says GenAIOps complements its MLOps levels rather than replacing them.[^ms-maturity] ### Learning path - **Prerequisites:** Git, Python, a CI system, basic statistics (distributions; [precision and recall](../aiops/#metrics-and-kpis)), and the [DevOps](../devops/) and [DataOps](../dataops/) pages. - **One day:** read the first two parts, Sculley et al. 2015[^sculley-2015] and Google Cloud's levels;[^gc-mlops] run Labs 1 and 2. - **One week:** do Lab 3; score one of your models with the ML Test Score;[^ml-test-score] write its retraining policy. - **30 days:** take one real model from level 0 or 1 to level 2: pipeline, data validation, gate with slices, parity replay. - **Specialise:** serving at scale (KServe, GPUs with [KubernetesOps](../kubernetesops/)); training jobs on shared or preemptible capacity; monitoring and drift statistics; feature platforms; or governance with [ModelOps](../modelops/). - **Portfolio project:** a public repository that trains a model on open data with DVC and MLflow, gates it, serves it, injects drift, a skew bug and an upstream data defect, and shows each being caught and told apart. ## Wrap-up ### Executive recap and cheat sheet | Topic | Remember | |---|---| | Decide | Use ML only if a rule cannot do nearly as well; match automation to how fast the model's world changes and what errors cost | | Essential practices | Reproducible runs with lineage; one feature definition; gates with slices; canary or shadow; three-level monitoring; written retraining and rollback policies | | Measure | Freshness, skew rate, drift-alert precision, label coverage, live quality and offline gap, rollback time, lineage coverage | | Tool classes | Tracking and registry; data versioning; orchestration; feature store; serving; validation and monitoring | | Watch-outs | Retraining on bad data; average-only gates; uncalibrated drift alerts; feedback loops; untrusted pickles | | Next actions | Aliases instead of paths; a parity replay; a staleness test; a gate in CI; a rollback drill | ### A judgement call: retrain nightly, or only when the data says so? **Situation (illustrative).** The churn team proposes a nightly job that retrains on the latest data and promotes automatically if overall AUC does not fall. The alternative is weekly retraining plus drift-triggered runs, with the slice gate, a 48-hour canary and one human approval. **Option A: nightly retrain and auto-promote.** Attractive: always fresh, no meetings, simple to explain. **Option B: weekly plus triggered retraining, gated.** Slower to react, and a person approves each promotion. **Assumptions.** Labels mature after 30 days, so a nightly run mostly retrains on the same labelled data plus a few new rows; the retention team acts on scores, so offers influence future churn labels; upstream data changes happen a few times a quarter. **Evidence.** Freshness needs come from measurement, not habit (Rules of ML #8; ML Test Score "Model 4").[^rules-ml][^ml-test-score] Feedback loops and fixed thresholds in a changing system are documented sources of ML debt.[^sculley-2015] The runbook above shows how one upstream defect would be trained into a nightly model. **Why the attractive choice can be wrong.** With 30-day labels, nightly retraining adds little freshness but 30 chances a month for bad data or a feedback effect to reach production, and an overall-AUC rule would miss a slice collapse. **Why B can also be wrong.** If the business changes weekly (a price war), a weekly cadence plus approval may be too slow, and drift thresholds too noisy to trigger reliably. **Decision process.** Run a staleness test over the last year. If decay within a week is below the gate's minimum uplift, choose B, with an "expedite" path: a drift alert plus a confirmed business change allows same-day retraining through the same gates. Review after a quarter using freshness, alert precision and incidents. **Uncertainty.** The staleness test uses history; a regime change (as in 2020[^mittr-pandemic]) can make history a poor guide, and in that case people, not schedules, decide. ### Learning objectives and mastery check After this page you should be able to: 1. **Explain** why a trained model is a build artifact of code, data, configuration and environment, and what MLOps adds to DevOps. 2. **Distinguish** skew, upstream data defects, data drift, prior shift and concept drift, and choose the right response to each. 3. **Design** a training pipeline with lineage, a promotion gate with slices, and a canary release with rollback. 4. **Select** monitoring signals for a model with delayed labels, and calibrate drift thresholds. 5. **Evaluate** how much MLOps a given model needs, and defend a retraining policy with evidence. ::: answer 1. A model scores AUC 0.86 offline and 0.74 in its first week online. What do you check first, and why not retrain? Check for skew and leakage first: replay logged requests through the offline feature code and re-check that the test split respects time. A gap that appears on day one is rarely drift. Retraining would rebuild the same defect.[^rules-ml][^kapoor] ::: ::: answer 2. PSI on a feature is 0.15 and Jensen-Shannon distance is 0.12. Is that drift? It depends on thresholds calibrated on your own history. The PSI bands 0.1/0.25 are a convention without academic support, and both measures depend on binning and sample size. Look at the feature's importance and the prediction distribution before acting.[^psi-yurdakul] ::: ::: answer 3. A new category appears overnight in a categorical feature. Name the likely cause and the first action. Treat it first as a likely upstream data defect rather than customer behaviour. First action: pause continuous training so the defect is not learned; then run the parity replay and confirm with the data owner ([runbook](#worked-runbook-a-new-category-appears-in-production)). ::: ::: answer 4. Labels arrive after 60 days. How do you know the model still works next month? Monitor leading signals (input drift, prediction distribution, prediction bias against the latest matured base rate, action limits), use label-free estimates with their assumptions in mind, and evaluate on matured labels as they arrive. Concept drift will be detected late.[^sculley-2015][^nannyml-cbpe] ::: ::: answer 5. Why should serving follow models:/churn-classifier@champion rather than a file path? An alias decouples deployment from code: moving it changes what the next load returns, rollback is one call, and the registry records the change.[^mlflow-workflow] An endpoint that pins a storage path needs a deploy step that derives the path from the alias, and a check that the two agree; otherwise there are two sources of truth. ::: ::: answer 6. Your ML Test Score sections score 4, 3.5, 1 and 3. What is the overall score and what does it tell you? 1: the score is the minimum of the four sections, which signals serious gaps in one area despite strength elsewhere.[^ml-test-score] ::: ::: answer 7. A vendor model arrives as a pickle file. What is the risk and the safer route? Unpickling can execute arbitrary code. Load it only if you trust the source, preferably in an isolated environment, or ask for a format that does not execute arbitrary Python on load: skops with reviewed trusted types, ONNX served in a sandbox, or safetensors for tensors.[^python-pickle][^sklearn-persistence] ::: ### Conclusion MLOps is what turns a model from an experiment that worked once into a component that keeps working. Its centre is not a tool but a loop: versioned inputs produce a model, gates decide whether it may serve, monitors decide whether it still should, and a written policy decides when to rebuild it. The failures on this page are rarely exotic: skew a parity test would have caught, a data defect nobody paused for, drift a calibrated alert would have shown. Build the boring safeguards first (lineage, aliases, a gate with slices, a practised rollback), automate only where your measurements show it pays, and remember that no pipeline makes an unpredictable world predictable. ## Sources and further reading Every source below was opened on the date in its footnote. The claim-by-claim ledger, including sources that could not be fetched, is `research/mlops-ledger.md` in the Atlas repository. **Re-check quarterly** (an Atlas suggestion): Kueue's TrainJob stage, KServe's canary mode and docs version, the Agent Platform naming and Model Monitoring v2 status, skops' default trusted types, and the package versions in Lab 3. [^gc-mlops]: [MLOps: Continuous delivery and automation pipelines in machine learning](https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning) — Google Cloud Architecture Center, updated 2024-08-28. Checked 2026-10-16. [^gc-guide]: [Practitioners guide to MLOps: A framework for continuous delivery and automation of machine learning](https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf) — Google Cloud white paper (K. Salama, J. Kazmierczak, D. Schut), May 2021. Checked 2026-10-16. [^kreuzberger]: [Machine Learning Operations (MLOps): Overview, Definition, and Architecture](https://arxiv.org/abs/2205.02302) — D. Kreuzberger, N. Kühl, S. Hirschl; arXiv 2022 (v3), published in *IEEE Access* 11 (2023) 31866–31879. Checked 2026-10-16. [^cdf-roadmap]: [MLOps Roadmap 2024](https://github.com/cdfoundation/sig-mlops/blob/main/roadmap/2024/MLOpsRoadmap2024.md) — CD Foundation SIG MLOps, published 2024-11-30 (first edition 2020; the 2020–2022 editions are marked outdated). Checked 2026-10-16. [^sculley-2015]: [Hidden Technical Debt in Machine Learning Systems](https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html) — D. Sculley et al., NeurIPS 2015. Checked 2026-10-16. [^amershi]: [Software Engineering for Machine Learning: A Case Study](https://www.microsoft.com/en-us/research/publication/software-engineering-for-machine-learning-a-case-study/) — S. Amershi et al., Microsoft Research, ICSE-SEIP 2019. Checked 2026-10-16. [^sklearn-persistence]: [Model persistence](https://scikit-learn.org/stable/model_persistence.html) — scikit-learn 1.9.1 documentation. Checked 2026-10-16. [^rules-ml]: [Rules of Machine Learning](https://developers.google.com/machine-learning/guides/rules-of-ml) — M. Zinkevich, Google for Developers, updated 2025-08-25. Checked 2026-10-16. [^uber-michelangelo]: [Meet Michelangelo: Uber's Machine Learning Platform](https://www.uber.com/blog/michelangelo-machine-learning-platform/) — M. Del Balso, J. Hermann, Uber Engineering, 2017-09-05. Checked 2026-10-16. [^sculley-2014]: [Machine Learning: The High Interest Credit Card of Technical Debt](https://research.google/pubs/machine-learning-the-high-interest-credit-card-of-technical-debt/) — D. Sculley et al., SE4ML workshop at NIPS 2014. Checked 2026-10-16. [^tfx-paper]: [TFX: A TensorFlow-Based Production-Scale Machine Learning Platform](https://research.google/pubs/tfx-a-tensorflow-based-production-scale-machine-learning-platform/) — Google, KDD 2017. Checked 2026-10-16. [^ml-test-score]: [The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction](https://research.google.com/pubs/archive/aad9f93b86b7addfea4c419b9100c6cdd26cacea.pdf) — E. Breck, S. Cai, E. Nielsen, M. Salib, D. Sculley; IEEE Big Data 2017. Checked 2026-10-16. [^kubeflow-2017]: [Introducing Kubeflow - A Composable, Portable, Scalable ML Stack Built for Kubernetes](https://kubernetes.io/blog/2017/12/introducing-kubeflow-composable/) — Kubernetes blog (J. Lewi, D. Aronchick), 2017-12-21. Checked 2026-10-16. [^mlflow-launch]: [Introducing MLflow: an Open Source Machine Learning Platform](https://www.databricks.com/blog/2018/06/05/introducing-mlflow-an-open-source-machine-learning-platform.html) — M. Zaharia, C. Simeone, Databricks blog, 2018-06-05. Checked 2026-10-16. [^cd4ml]: [Continuous Delivery for Machine Learning](https://martinfowler.com/articles/cd4ml.html) — D. Sato, A. Wider, C. Windheuser, martinfowler.com, 2019-09-19. Checked 2026-10-16. [^model-cards]: [Model Cards for Model Reporting](https://arxiv.org/abs/1810.03993) — M. Mitchell et al., FAT* 2019. Checked 2026-10-16. [^shankar]: [Operationalizing Machine Learning: An Interview Study](https://arxiv.org/abs/2209.09125) — S. Shankar, R. Garcia, J. M. Hellerstein, A. G. Parameswaran, arXiv 2022-09-16. Checked 2026-10-16. [^kubeflow-cncf]: [Kubeflow](https://www.cncf.io/projects/kubeflow/) — CNCF project page (incubating 2023-07-25, graduated 2026-07-24). Checked 2026-10-16. [^kserve-cncf]: [KServe](https://www.cncf.io/projects/kserve/) — CNCF project page (incubating, accepted 2025-09-29). Checked 2026-10-16. [^ms-maturity]: [MLOps maturity model](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/mlops-maturity-model) — Microsoft Azure Architecture Center, updated 2026-06-03. Checked 2026-10-16. [^pytorch-random]: [Reproducibility](https://docs.pytorch.org/docs/2.14/notes/randomness.html) — PyTorch 2.14 documentation. Checked 2026-10-16. [^mlflow-tracking]: [MLflow Tracking](https://mlflow.org/docs/latest/ml/tracking/) — MLflow documentation (MLflow 3). Checked 2026-10-16. [^mlflow-registry]: [MLflow Model Registry](https://mlflow.org/docs/latest/ml/model-registry/) — MLflow documentation. Checked 2026-10-16. [^feast]: [Introduction](https://docs.feast.dev/) — Feast documentation. Checked 2026-10-16. [^feast-pit]: [Point-in-time joins](https://docs.feast.dev/getting-started/concepts/point-in-time-joins) — Feast documentation. Checked 2026-10-16. [^kapoor]: [Leakage and the reproducibility crisis in machine-learning-based science](https://pmc.ncbi.nlm.nih.gov/articles/PMC10499856/) — S. Kapoor, A. Narayanan, *Patterns* 4(9) 100804, 2023. Checked 2026-10-16. [^lu-drift]: [Learning under Concept Drift: A Review](https://arxiv.org/abs/2004.05785) — J. Lu et al., IEEE Transactions on Knowledge and Data Engineering 31(12), 2018 (arXiv 2020). Checked 2026-10-16. [^sagemaker-shadow]: [Shadow tests](https://docs.aws.amazon.com/sagemaker/latest/dg/shadow-tests.html) — Amazon SageMaker AI Developer Guide. Checked 2026-10-16. [^kserve-canary]: [Canary rollout example](https://kserve.github.io/website/docs/model-serving/predictive-inference/rollout-strategies/canary-example) — KServe documentation. Checked 2026-10-17. [^mlflow-signature]: [Model Signatures and Input Examples](https://mlflow.org/docs/latest/ml/model/signatures/) — MLflow documentation. Checked 2026-10-16. [^vertex-monitoring]: [Introduction to Model Monitoring](https://docs.cloud.google.com/vertex-ai/docs/model-monitoring/overview) — Google Cloud documentation, updated 2026-10-15. Checked 2026-10-17. [^dvc-start]: [Get Started with DVC](https://doc.dvc.org/start) — DVC documentation. Checked 2026-10-16. [^kfp]: [Kubeflow Pipelines overview](https://www.kubeflow.org/docs/components/pipelines/overview/) — Kubeflow documentation. Checked 2026-10-16. [^dvc-pipelines]: [Defining Pipelines](https://doc.dvc.org/user-guide/pipelines/defining-pipelines) — DVC documentation. Checked 2026-10-16. [^prom-alerting]: [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) — Prometheus documentation. Checked 2026-10-17. [^booking-kdd]: [150 successful Machine Learning models: 6 lessons learned at Booking.com](https://www.kdd.org/kdd2019/accepted-papers/view/150-successful-machine-learning-models-6-lessons-learned-at-booking.com) — L. Bernardi, T. Mavridis, P. Estevez, KDD 2019 (abstract page). Checked 2026-10-16. [^booking-summary]: [150 successful machine learning models: 6 lessons learned at Booking.com](https://blog.acolyer.org/2019/10/07/150-successful-machine-learning-models/) — A. Colyer, the morning paper, 2019-10-07 (summary of the KDD paper). Checked 2026-10-16. [^tfdv]: [Get started with TensorFlow Data Validation](https://www.tensorflow.org/tfx/data_validation/get_started) — TensorFlow documentation. Checked 2026-10-16. [^psi-matlab]: [populationStabilityIndex](https://www.mathworks.com/help/risk/risk.validation.populationstabilityindex.html) — MathWorks Risk Management Toolbox documentation (R2026a). Checked 2026-10-16. [^evidently-drift]: [Customize Data Drift](https://docs.evidentlyai.com/metrics/customize_data_drift) — Evidently AI documentation. Checked 2026-10-16. [^psi-yurdakul]: [Statistical properties of the population stability index](https://www.risk.net/journal-of-risk-model-validation/7725371/statistical-properties-of-the-population-stability-index) — B. Yurdakul, J. Naranjo, Journal of Risk Model Validation 14(4), 2020. Checked 2026-10-16. [^psi-critical]: [A critical review of existing and new population stability testing procedures in credit risk scoring](https://arxiv.org/abs/2303.01227) — J. du Pisanie et al., arXiv 2023. Checked 2026-10-16. [^nannyml-cbpe]: [Performance estimation deep dive](https://nannyml.readthedocs.io/en/stable/how_it_works/performance_estimation.html) — NannyML documentation. Checked 2026-10-16. [^aws-ml-lens]: [Machine Learning Lens](https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/machine-learning-lens.html) — AWS Well-Architected Framework, 2025-11-19. Checked 2026-10-16. [^gcp-mle-cert]: [Professional Machine Learning Engineer](https://cloud.google.com/learn/certification/machine-learning-engineer) — Google Cloud certification page. Checked 2026-10-16. [^datasheets]: [Datasheets for Datasets](https://arxiv.org/abs/1803.09010) — T. Gebru et al., arXiv (v8 2021); Communications of the ACM, December 2021. Checked 2026-10-16. [^mlflow-pypi]: [mlflow](https://pypi.org/project/mlflow/) — PyPI project page (3.17.1, released 2026-10-13). Checked 2026-10-17. [^mlflow-workflow]: [Model Registry workflows](https://mlflow.org/docs/latest/ml/model-registry/workflow/) — MLflow documentation. Checked 2026-10-16. [^dvc-pypi]: [dvc](https://pypi.org/project/dvc/) — PyPI project page (3.67.1, released 2026-03-31). Checked 2026-10-17. [^seldon-license]: [License](https://docs.seldon.ai/seldon-core-1/getting-started/license) — Seldon Core 1 documentation. Checked 2026-10-16. [^mlflow-sklearn]: [mlflow.sklearn](https://mlflow.org/docs/latest/api_reference/python_api/mlflow.sklearn.html) — MLflow 3.17.1 Python API reference. Checked 2026-10-17. [^skops-changes]: [skops Changelog](https://skops.readthedocs.io/en/stable/changes.html) — skops 0.16 documentation, v0.15 entry (0.15.0 released 2026-09-16 on PyPI). Checked 2026-10-17. [^mlflow-client]: [mlflow.client](https://mlflow.org/docs/latest/api_reference/python_api/mlflow.client.html) — MLflow 3.17.1 Python API reference. Checked 2026-10-16. [^mittr-pandemic]: [Our weird behavior during the pandemic is messing with AI models](https://www.technologyreview.com/2020/05/11/1001563/covid-pandemic-broken-ai-machine-learning-amazon-retail-fraud-humans-in-the-loop/) — W. D. Heaven, MIT Technology Review, 2020-05-11. Checked 2026-10-16. [^zillow-pr]: [Zillow Group Reports Third-Quarter 2021 Financial Results & Shares Plan to Wind Down Zillow Offers Operations](https://www.sec.gov/Archives/edgar/data/1617640/000161764021000085/q32021991.htm) — Zillow Group, Form 8-K Exhibit 99.1, SEC EDGAR, 2021-11-02. Checked 2026-10-16. [^zillow-letter]: [Q3 2021 Shareholder Letter](https://www.sec.gov/Archives/edgar/data/1617640/000161764021000085/exhibit993.htm) — Zillow Group, Form 8-K Exhibit 99.3, SEC EDGAR, 2021-11-02. Checked 2026-10-16. [^python-pickle]: [pickle — Python object serialization](https://docs.python.org/3/library/pickle.html) — Python documentation. Checked 2026-10-16. [^safetensors]: [Safetensors](https://huggingface.co/docs/safetensors/index) — Hugging Face documentation. Checked 2026-10-16. [^model-signing]: [model-transparency](https://github.com/sigstore/model-transparency) — Sigstore project on GitHub. Checked 2026-10-16. [^mlflow-auth]: [Authentication with Username and Password](https://mlflow.org/docs/latest/self-hosting/security/basic-http-auth/) — MLflow documentation. Checked 2026-10-16. [^kserve-mlflow]: [MLflow](https://kserve.github.io/website/docs/model-serving/predictive-inference/frameworks/mlflow) — KServe documentation (v0.20). Checked 2026-10-17. [^kserve-s3]: [S3](https://kserve.github.io/website/docs/model-serving/storage/providers/s3) — KServe documentation, storage providers. Checked 2026-10-16. [^prom-pushing]: [When to use the Pushgateway](https://prometheus.io/docs/practices/pushing/) — Prometheus documentation. Checked 2026-10-16. [^pushgateway]: [Prometheus Pushgateway README](https://github.com/prometheus/pushgateway) — Prometheus project on GitHub. Checked 2026-10-16. [^kueue]: [Overview](https://kueue.sigs.k8s.io/docs/overview/) — Kueue documentation, Kubernetes SIGs, last modified 2026-10-09. Checked 2026-10-17. [^pytorch-dcp]: [Distributed Checkpoint](https://docs.pytorch.org/docs/2.14/distributed.checkpoint.html) — PyTorch 2.14 documentation. Checked 2026-10-16. [^pytorch-tuning]: [Performance Tuning Guide](https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html) — PyTorch tutorials, updated 2025-07-09. Checked 2026-10-16. [^aws-spot]: [Spot Instance interruption notices](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html) — Amazon EC2 User Guide. Checked 2026-10-16. [^gcp-agent-platform]: [Gemini Enterprise Agent Platform release notes](https://docs.cloud.google.com/gemini-enterprise-agent-platform/release-notes) — Google Cloud, entry of 2026-04-22. Checked 2026-10-17. **Changelog.** - 2026-10-16, r0: first edition (Blueprint v2, 13 parts). Labs 1 and 2 executed locally with Python 3.8.10; Lab 3 and the reference implementation syntax-checked only. - 2026-10-16/17, r1: fixes from the independent fact check (reviews/mlops-r1.md): deploy step and identities, data-defect branch, training jobs, parity test, alerts, attributions; Lab 2 re-executed. - 2026-10-17, post-score polish (reviews/mlops-polish.md): skops trusted types (set at logging; `TreePredictor` in `train.py`); Lab 3 helper files, two executed, and the expected first `REJECT`; alert wording; Figure 4 re-routed; re-check list.