--- title: "ModelOps" slug: modelops category: data term_status: emerging evidence_status: verified summary: "ModelOps is the governance and lifecycle management of every model an organisation relies on (machine learning, rules, optimisation, LLM applications and agents): each one inventoried, risk-tiered, validated, approved for a use, monitored and retired." audience: [Model risk managers, AI governance leads, ML platform owners, Engineers, Auditors, Leaders] last_verified: 2026-10-18 key_takeaways: - "The label is unsettled: IBM researchers used it in 2019 for lifecycle management of AI pipelines, Gartner defines it as the governance and life cycle management of a wide range of AI and decision models, and a ModelOp co-founder notes that it is often used interchangeably with MLOps. This page uses the governance meaning." - "The core artefact is the inventory: every model in use, with its owner, purpose, risk tier, approved uses, limitations, last review and next review date." - "Rigour follows risk. US, UK and Canadian financial supervisors all call for scrutiny proportional to model risk, so independent validation and effective challenge concentrate on the models that matter most." - "Approvals need teeth: a release gate that reads the tier and approval from records developers cannot write, and a rule for which changes need a new approval." - "The map moved in 2026: SR 26-2 replaced SR 11-7 in April and leaves generative and agentic AI out of scope, and the EU AI Act's Annex III high-risk obligations now apply from 2 December 2027." - "Fairness and explainability reports are evidence for a decision, not verdicts: fairness metrics can conflict, and the four-fifths rule is a US employment screen, not a universal test." - "Bought and API models need the same governance with less visibility: validate your own use, monitor outcomes, and track the provider's retirement dates." related: [mlops, llmops, decisionops, riskops, complianceops, finops-for-ai, privacyops, dataops] --- ## What ModelOps is ### In 30 seconds **ModelOps is the governance and lifecycle management of every model an organisation relies on**: machine-learning models, scoring rules, optimisation models, applications built on large language models (LLMs) and agents, built or bought. For each model it answers: *what is it for, how risky is it, who allowed its use, and is it still fit for that use?* The working parts: an inventory, a tiering rubric, independent validation where it matters, approval with conditions, review and retirement. Gartner defines ModelOps as "focused primarily on the governance and life cycle management of a wide range of operationalized artificial intelligence (AI) and decision models, including machine learning, knowledge graphs, rules, optimization, linguistic and agent-based models".[^gartner] Banks run the governance core under an older name, **model risk management (MRM)**, which US supervisors set out in supervisory guidance in 2011 and replaced in April 2026.[^sr117][^sr262] ::: note Three meanings in circulation 1. **Engineering lifecycle (IBM Research, 2019):** a cloud platform for end-to-end development and lifecycle management of AI applications;[^ibm2019] today mostly [MLOps](../mlops/). 2. **Governance and life cycle management of a wide range of AI and decision models (Gartner):** this page's meaning.[^gartner] 3. **A synonym for MLOps:** a ModelOp co-founder, arguing in 2021 that the two differ, noted that "the terms are often used interchangeably".[^modelop2021] ::: ### At a glance | Item | Summary | |---|---| | Term status | **Emerging.** No standards body defines ModelOps; vendors now sell the capability as "AI governance" (IBM cites Gartner's "first-ever Magic Quadrant for AI Governance Platforms", which ModelOp dates to 2026).[^wxgov][^modelophome] The practice is established in banking as MRM. | | In scope | Inventory, tiering, validation, approval, lifecycle states, change classes, review, exceptions, fairness and explainability reporting, bought models, agents, regulatory mapping, retirement. | | Out of scope | Pipelines, registries and drift detection → [MLOps](../mlops/); LLM evaluations and guardrails → [LLMOps](../llmops/); decision logic → [DecisionOps](../decisionops/); risk appetite method → [RiskOps](../riskops/); audit evidence → [ComplianceOps](../complianceops/); personal data → [PrivacyOps](../privacyops/); AI spend → [FinOps for AI](../finops-for-ai/). | | Prerequisites | [MLOps](../mlops/) basics; risk vocabulary ([RiskOps](../riskops/)). | | Key outputs | Inventory with owners and tiers; validation reports; approval records; exceptions register; portfolio reports. | ### The problem ModelOps solves Organisations accumulate models faster than they track them: credit scores, fraud rules, vendor anti-money-laundering scenarios, an LLM support assistant, a refund agent. Nobody can list them all or say who approved each use. US guidance warns that "even a fundamentally sound model producing accurate outputs consistent with the model's design objective can exhibit high model risk if it is misapplied or misused".[^sr262a] Engineering quality does not remove that risk; deciding where a model may be used, and checking, does. | Operating reality | Without ModelOps | With ModelOps | |---|---|---| | Which models exist | Found in incidents or audits | One inventory, reconciled with discovery | | Scrutiny | Same for all, or none | Proportional to a documented tier | | Who approved use | The developer | Independent review for tiers 1–2 | | Vendor and API models | Procurement only | Tiered, validated for your use | | Old models | Run until something breaks | Reviewed on schedule; retired deliberately | ### What ModelOps is not - **Not MLOps with a new name.** This Atlas's **editorial distinction**: MLOps *engineers* the lifecycle (pipelines, registry, drift detection); ModelOps *governs* the portfolio (which uses are allowed, at what risk, on whose authority). Gartner's own "core capabilities" include engineering items such as CI/CD integration and rollback,[^gartner] so no definition enforces this. - **Not a committee that approves everything**, nor legal advice: audit evidence belongs to [ComplianceOps](../complianceops/), and interpreting the law to lawyers. - **Not only for banks:** hospitals, airlines and home buyers run consequential models ([use cases](#use-cases-and-scenarios)). - **"Model" means different things.** US guidance excludes "deterministic rule-based processes and software where there are no statistical, economic, or financial theories underpinning their design or use" and leaves generative and agentic AI out of scope;[^sr262a] the UK PRA asks banks to consider applying MRM to material, complex rules;[^prapdf] Canada's OSFI includes AI/ML methods.[^osfi] Your governance scope can be wider than any regulator's; record both. ## Where ModelOps comes from and where it sits ### Origins and evolution The practice is older than the name: US guidance SR 11-7 (2011) asked banks for "effective challenge" of models.[^sr117] The label appeared in IBM's 2019 paper with an engineering meaning,[^ibm2019] alongside model cards and datasheets;[^modelcards][^datasheets] since 2023 AI regulation has carried governance beyond banking ([methods and regulation](#methods-standards-and-regulation)). The date of Gartner's first definition is unknown (cited snapshot: May 2025).[^gartner] ### Glossary and mental model | Term | Meaning on this page | |---|---| | **Model** | Any method whose output informs a decision and could be wrong; regulators' narrower definitions are recorded per model. | | **Model risk** | US banking: "the potential for adverse financial consequences associated with models, which may result from decisions made based on model output".[^sr262a] That is risk to the firm; this page also counts what the EU AI Act calls "harm to the health, safety or fundamental rights of natural persons".[^euaiact] | | **Materiality** | How much rides on the output. US guidance: purpose "together with model exposure, determines model materiality".[^sr262a] | | **Risk tier** | The class that sets a model's scrutiny; OSFI's "model risk rating".[^osfi] | | **Effective challenge** | "the critical analysis conducted by objective experts" with independence and "the organizational standing and influence to effect any change".[^sr262a] | | **Material change** | A change that needs re-tiering and a new approval ([change classes](#playbook-model-changes-and-materiality)). | | **Exception** | Use outside approval: unapproved or unvalidated, outside its purpose, or with persistent performance breaches (PRA's definition, paraphrased).[^prapdf] | | **Overlay** | A documented manual correction to model output while a problem is fixed (the PRA's "post-model adjustment"). | **Mental model: a model holds a licence, not a launch date.** The licence names a holder (owner), a scope (approved uses), conditions, an expiry and an issuer who did not build the model. ModelOps issues, renews, restricts and revokes licences. ### How ModelOps relates to other Ops | Neighbour | Shared ground | The neighbour owns | ModelOps owns | |---|---|---|---| | [MLOps](../mlops/) | Registry, lifecycle states, monitoring | Pipelines, registry engineering, gates as code, drift | Approvals per tier; change classes; review; retirement | | [LLMOps](../llmops/) | Foundation models | Evaluations, guardrails, traces, provider regressions | Tier and approval of each LLM use | | [DecisionOps](../decisionops/) | Rules and scores in decisions | Decision logic and its tests | Governing the models and rules it uses | | [RiskOps](../riskops/) | Risk appetite | Enterprise risk method | Applying appetite to models | | [ComplianceOps](../complianceops/) | Audits | Collecting evidence | Model evidence; obligations mapped to models | | [PrivacyOps](../privacyops/) | Personal data in models | Lawful basis, impact assessments | Requiring them before approval | | [FinOps for AI](../finops-for-ai/) | Vendor and API models | Unit economics | Their risk tier and approval | | [DataOps](../dataops/) | Input data quality | Pipelines, data contracts | Data limitations per model | Also: [FinCrimeOps](../fincrimeops/) runs AML scenarios governed here; [ProcurementOps](../procurementops/) negotiates vendor terms; [AIOps](../aiops/) is AI *for* IT operations. ::: figure Where ModelOps sits: it licenses what neighbours build and run ```mermaid flowchart TB ML["MLOps: ML pipelines"] LL["LLMOps: LLM apps"] DE["DecisionOps: decision logic"] J1(["`**Build and run** send candidates, results receive approvals, conditions`"]) MO["ModelOps: inventory, tier, validate, approve, review, retire"] J2(["`**Set limits and prove** send appetite, obligations receive inventory, findings`"]) RI["RiskOps: appetite"] PR["PrivacyOps: lawful data"] CO["ComplianceOps: evidence"] ML & LL & DE <--> J1 J1 <--> MO MO <--> J2 J2 <--> RI & PR & CO classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef flow fill:#ffffff,stroke:#5b7a87,stroke-dasharray:4 3,color:#122029 class MO core class ML,LL,DE,RI,PR,CO neutral class J1,J2 flow ``` Text equivalent: ModelOps sits in the middle. The builders and runners above it (MLOps, LLMOps, DecisionOps) send candidates and monitoring results and receive approvals and conditions; the functions below it that set limits and prove compliance (RiskOps, PrivacyOps, ComplianceOps) send appetite and obligations and receive the inventory and findings. ::: ## Principles and operating model ### A proposed charter and principles No official ModelOps manifesto exists; this charter is **this Atlas's proposal** ([one-page version](#templates)). **Mission:** every model we rely on is known, owned, used within approved limits, and retired on purpose. 1. **Inventory before policy.** You cannot govern what you have not listed. 2. **Rigour follows risk.** US, UK and Canadian supervisors all ask for scrutiny proportional to a model's risk.[^sr262a][^prapdf][^osfi] 3. **Independence where it matters.** Validators "should therefore not be part of any model development activities, nor have a stake in whether or not a model is approved" (PRA).[^prapdf] Tier 3 may need only a peer review. 4. **Approve a use, not a model.** "Using a model beyond its intended purpose introduces additional uncertainty and risk."[^sr262a] Approvals name uses, limits and an expiry date. 5. **Evidence lives with the model**, in records that release automation reads and developers cannot edit. 6. **Bought is not exempt** ([playbook](#playbook-models-you-did-not-build)). 7. **Retire on purpose**, with notice and retained records. **Non-goals:** approving every spreadsheet; replacing developers' testing; legal advice. ### Building blocks The minimum viable set: an **inventory** with OSFI's minimum fields (ID, name and use, risk rating, owner, developer, origin);[^osfi] a one-page **tiering rubric**; a **validation checklist** per tier and model type; **approval records** with uses, conditions and expiry ([templates](#templates)); **change classes**; review dates fed by MLOps or LLMOps **monitoring**; an **exceptions register**; and **model cards** with intended use, evaluation by group and limitations.[^modelcards] ### Roles, ownership and RACI OSFI names the core roles: the **owner** coordinates development, deployment and monitoring; the **developer** designs the method; the **reviewer** assesses "conceptual soundness, inputs, methodology, assumptions, and performance"; the **approver** has "the authority to approve the use of a model"; the **user** relies on outputs.[^osfi] Large firms add a **model risk (MRM) function**;[^prapdf] the **ML platform team** implements gates. | Activity | Owner | Developer | Validator | MRM function | Approver | |---|---|---|---|---|---| | Register the model and propose a tier | **A/R** | C | — | C | — | | Confirm the tier | C | I | C | **A/R** | I | | Development testing and documentation | A | **R** | I | I | — | | Independent validation (tiers 1–2) | C | C | **R** | **A** | I | | Approve use and conditions | C | I | C | R | **A** | | Classify a change | R | C | C | **A** | I | | Monitoring and breach escalation | **A** | R | I | C | I | | Periodic review | A | C | **R** | R | I | | Exceptions and overlays | R | C | C | R | **A** | | Retirement and consumer notice | **A/R** | C | I | C | I | R = responsible, A = accountable, C = consulted, I = informed (an Atlas template). One person may hold the MRM and approver roles, never the developer and validator roles for a tier-1 model. Consultants can validate; US guidance calls for "maintaining proper oversight and integrating that work".[^sr262a] ### Lifecycle: states and transitions OSFI's lifecycle (design, review, deployment, monitoring, decommission) is "not necessarily sequential";[^osfi] release automation needs explicit **states** in the inventory. ::: figure Model lifecycle states, including provisional use and restriction ```mermaid stateDiagram-v2 %% transitions listed in the order that lays the diagram out without crossing edges [*] --> Proposed Proposed --> InValidation: tiered Suspended --> InValidation: redeveloped Approved --> UnderReview: due Proposed --> Rejected InValidation --> Provisional: urgent need Restricted --> Approved: fixed InValidation --> Rejected: unacceptable Approved --> Restricted: breach Approved --> Retired: replaced InValidation --> Approved: accepted Provisional --> Approved: validated Suspended --> Retired Restricted --> Suspended: not fixed Provisional --> Suspended: expired UnderReview --> Approved: fit UnderReview --> Restricted: narrowed Retired --> [*] ``` Text equivalent: A Proposed model is tiered and enters InValidation, or is Rejected. Validation ends in Approved, Rejected or, for an urgent need, Provisional, which becomes Approved once validated or Suspended when it expires. Approved models go to UnderReview when due and return Approved or Restricted; a breach also restricts them. Restricted ends Approved (fixed) or Suspended. Suspended models are redeveloped (back to InValidation) or Retired; Approved models are retired when replaced. ::: **Provisional** exists because US guidance accepts urgent use before validation completes, with controls (see the [judgement call](#a-judgement-call-launch-the-refreshed-fraud-model-before-validation-finishes)).[^sr262a] **Retired** is not a deletion: OSFI asks for retained models and documents, stakeholder alerts and monitoring of "downstream effects".[^osfi] ### Operating model and governance - **Intake triage** (weekly): a temporary tier within days, as OSFI allows ("a temporary risk rating that is confirmed in the model review or model approval stage").[^osfi] - **Model risk committee** (monthly): tier-1 approvals, exceptions, retirements. - **Portfolio report** (quarterly): tiers, overdue reviews, exceptions, findings and aggregate risk from "reliance on common assumptions, data, or methodologies".[^sr262a] - **Annual self-assessment**, which the PRA expects banks to update "at least annually".[^prapdf] ## Architecture and workflow ### Reference architecture The architecture is a **control plane around existing delivery systems**: it records, decides and enforces; pipelines, registries and monitoring stay with MLOps and LLMOps. ::: figure Reference architecture: the inventory as system of record, with enforcement at release ```mermaid flowchart TB subgraph SRC["Discovery sources"] direction LR REG["Model registries"] CLD["Cloud AI and API usage"] PRO["Procurement and vendors"] end INT["Intake form"] --> INV[("Inventory: owner, tier, uses, dates")] SRC -->|"reconcile weekly"| INV INV --> VAL["Validation workspace: reports, findings"] VAL --> APR["Approval record: uses, conditions, expiry"] APR --> GATE{"Release gate checks tier and approval record"} GATE -->|"pass"| PROD["Production serving"] GATE -->|"fail"| BLK["Blocked, owner notified"] PROD -->|"monitoring results"| TRG["Review triggers"] TRG --> INV INV --> RPT["Portfolio reports to committee"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class INT,INV,VAL,APR,TRG,RPT core class GATE decision class BLK risk class REG,CLD,PRO,PROD neutral ``` Text equivalent: An intake form feeds the inventory, reconciled weekly with registries, cloud AI usage and procurement. Models pass to validation and an approval record. A release gate checks tier and approval: pass reaches production, fail is blocked and the owner notified. Monitoring results raise review triggers in the inventory, which feeds portfolio reports. ::: **Trust boundaries.** Whoever can move a production alias, or write the evidence a gate reads, can bypass every approval. In MLflow, creating a version, setting a version tag and setting an alias all need `can_update` on a registered model (from 3.17, a model-version grant overrides it for versions), and admins bypass all permissions (MLflow 3.17.1).[^mlflowauth][^mlflowrbac] So the boundaries are structural, as in the [MLOps](../mlops/) reference implementation: developers register into a candidates model; one promotion identity edits the production model; the gate reads tier and approval from the inventory, which only the MRM function and approvers change; admin accounts are break-glass. **Alternative:** a GRC or AI governance platform as the inventory ([criteria](#tooling-landscape)). ### End-to-end workflow ::: figure Approval workflow for a tier-2 model, with the provisional-use exception path ```mermaid %%{init: {"sequence": {"actorMargin": 30, "width": 120}}}%% sequenceDiagram participant M as MRM function participant O as Owner participant V as Validator participant A as Approver participant P as Release gate O->>M: Intake form M-->>O: Tier 2 (temporary) O->>V: Evidence, access V-->>O: Findings alt Urgent, validation open V->>A: Interim view, exception A->>P: Provisional, expiry else Normal path O->>V: Fixes V->>A: Final report A->>P: Approval, conditions end Note over A,P: Gate checks the record Note over M,O: Next review date set ``` Text equivalent: The owner sends an intake form to the MRM function, which returns a temporary tier 2. The owner gives the validator evidence and receives findings. If the need is urgent while validation is open, the validator passes its interim view and the owner's exception request to the approver, who records a provisional approval expiring within 14 days (this Atlas's policy). Otherwise the owner sends fixes, the validator a final report, and the approver records an approval with conditions. The release gate reads that record; the MRM function sets the next review date. ::: ## Practices, frameworks and tools ### Core practices and playbooks #### Playbook: risk tiering The PRA asks for "a risk-based materiality and complexity rating" per model; complexity can include "interpretability, explainability, transparency, and the potential for designer or data bias".[^prapdf] OSFI asks for "clear, measurable criteria for each risk dimension".[^osfi] This Atlas's starting rubric, used in the [reference implementation](#reference-implementation): | Input | High | Medium | Low | |---|---|---|---| | Materiality | Drives regulatory figures, credit, prices or safety for many people, or large sums | Supports a decision with human review | Internal convenience | | Complexity / inherent risk | Opaque method, unstructured or third-party data, autonomy to act | Standard ML or optimisation | Simple, transparent logic | | Override | Affects individuals' access to credit, jobs, services or care: tier 2 or stricter | — | — | A matrix of the two ratings gives the tier ([decision tree](#decision-framework)); tier 4 means "inventory only". **Acceptance:** two reviewers tier the same ten models independently and agree on at least eight. #### Playbook: independent validation Validation covers **conceptual soundness** (is the design fit for purpose?), **outcomes analysis** (do outputs match reality?) and **ongoing monitoring**.[^sr262a] By model type: | Model type | Conceptual soundness asks | Outcomes evidence | |---|---|---| | Statistical or ML | Is the target the right proxy? Are features stable and lawful? | Back-tests; performance by segment and group | | Rules | Do thresholds follow from data or policy? Do rules conflict? | Hit rates; false-positive review | | Optimisation | Do objective and constraints encode intent and legal limits? | Scenario runs; comparison with human plans | | LLM application | Is the task suited to a generative model? What must it never say or do? | [LLMOps](../llmops/) evaluation results; red-team findings | | Agent | Which tools, data, limits and checkpoints? | Task success; unauthorised-action tests | | Vendor model | Is the design and development data documented enough? | Outcomes on your population | **Depth by tier.** The PRA expects validation rigour and intensity "commensurate with the associated model tier"; OSFI's rating drives the "frequency, intensity, and scope of model review".[^prapdf][^osfi] This Atlas's starting scopes: | Tier | Reviewer | Scope | |---|---|---| | 1 | Independent validator | Full report: replicate key results; sensitivity analysis (outputs as key assumptions change); a challenger model or benchmark; outcomes by group | | 2 | Independent validator | Focused report: review developers' tests and sensitivity evidence; outcomes on recent data; a benchmark if one exists | | 3 | Peer outside the team | Checklist: purpose, data, sampled outcomes | | 4 | None | Inventory record; owner attests the use | Each finding gets a severity, owner and due date; high-severity findings block approval or become conditions. #### Playbook: model changes and materiality The PRA expects policy to set procedures for "when a material change is made to a model" and "the materiality criteria to be used to identify the potential impact of prospective model changes", with testing of "material changes over a period of time in dynamic models" that compares outputs before and after with actual outcomes.[^prapdf] OSFI re-rates models after "a material change in the model's use, data, or infrastructure".[^osfi] This Atlas's starting classes: | Class | Typical changes | Evidence | Decision | |---|---|---|---| | Material | New use, population, method, target or features; another provider model; an agent gains a tool or permission | Re-tier; before/after outcomes; validation | New approval | | Bounded | Cut-off change; prompt or retrieval change in a tier 1–2 LLM application; recalibration | Before/after on a reference set; [LLMOps](../llmops/) evaluations | Validator reviews the delta; approver signs | | In envelope | Scheduled retraining within agreed limits | Pipeline test report | Recorded automatically; owner notified | | Record only | Documentation; infrastructure with identical outputs | Reference-set check | Logged | A **pre-approved change envelope** (this Atlas's proposal) is approved once, like a use: fixed features, method and population, limits on score shift and segment performance, a retraining cadence. A version inside it reaches the [Lab 3](#hands-on-labs) gate with an approval entry written by the MRM function's automation, not the developer. Re-check the envelope at each periodic review: small steps add up to the drift the PRA warns about. #### Playbook: explainability and fairness reporting **Explanations serve audiences.** US applicants refused credit must get reasons that are "specific and indicate the principal reason(s) for the adverse action" (Regulation B); the CFPB withdrew its 2022 circular on complex algorithms on 12 May 2025, but the regulation stands (as of October 2026).[^regb][^cfpbwithdraw] SHAP assigns "each feature an importance value for a particular prediction";[^shap] Cynthia Rudin argues that "explainable black boxes should be avoided in high-stakes decisions".[^rudin] Make the choice between explaining and an interpretable model explicit at tier 1. **Fairness is measured by group, then judged.** Model cards propose evaluation "across different cultural, demographic, or phenotypic groups";[^modelcards] Fairlearn's `MetricFrame` computes such metrics.[^fairlearn] Three cautions: 1. **Metrics conflict.** Kleinberg and colleagues show that calibration within groups and balance for the positive and negative classes cannot all hold: "except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously";[^kleinberg] Chouldechova shows one criterion "may lead to considerable disparate impact when recidivism prevalence differs across groups".[^chouldechova] Policy names the leading metric per use. 2. **The four-fifths rule is a screen.** Under US employment-selection guidelines, a group selection rate below four-fifths of the highest "will generally be regarded by the Federal enforcement agencies as evidence of adverse impact"; smaller differences can still matter, larger ones may not with small numbers.[^uggsp] Elsewhere it is, in this page's view, a heuristic. 3. **Measurement needs group data**, often sensitive. The EU's new Article 4a lets providers of high-risk systems "exceptionally process special categories of personal data" for bias detection under strict safeguards, and extends this to deployers and to other AI systems, but "does not create any obligation" to do so.[^omnibus] The lawful basis belongs to [PrivacyOps](../privacyops/). New York City bars automated employment decision tools unless the tool "has been subject to a bias audit within one year of the use of the tool".[^nyc144] #### Playbook: models you did not build - **Vendor models.** US guidance expects validation "either by internal or outside parties", plus monitoring and outcomes analysis;[^sr262a] OSFI rates external models "on a standalone basis".[^osfi] Contract for change notices ([ProcurementOps](../procurementops/)). - **Foundation models behind an API.** Approve the *use* and record the model version. Providers retire models on their own schedule: OpenAI promises at least six months' notice for generally available models (barring safety or compliance concerns), less for previews;[^oaidepr] Anthropic gives "at least 60 days' notice" for publicly released models, and partner platforms "set their own retirement schedules" (as of October 2026).[^anthdepr] A replacement is a material change. - **Agents.** Singapore's framework (January 2026) asks for "placing limits on agents' powers" and checkpoints "at which human approval is required".[^mgfagentic] Autonomy raises the tier; tool permissions become approval conditions. ### Methods, standards and regulation Separate what binds you from what helps you. Status as of October 2026; this is orientation, not legal advice. | Instrument | Who it applies to | What it asks of model governance | Status | |---|---|---|---| | SR 26-2 / OCC Bulletin 2026-13 | US banks; "most relevant" above $30 billion in assets | Risk-based MRM incl. vendor products; GenAI and agentic AI out of scope | Guidance of 17 April 2026 that "does not set forth enforceable standards"; replaced SR 11-7[^sr262][^sr262a][^occ] | | PRA SS1/23 | UK banks with internal-model approval | Five principles, from tiering to validation and risk mitigants | Effective 17 May 2024; updated April 2026[^pra] | | OSFI E-23 | Canadian federally regulated financial institutions | Enterprise-wide MRM incl. AI/ML | Effective 1 May 2027[^osfi] | | EU AI Act | Providers and deployers of AI systems in the EU | High-risk duties (risk management, data, logs, oversight) | In force 1 Aug 2024; Annex III duties from 2 Dec 2027, Annex I from 2 Aug 2028[^ecaiact][^omnibus][^euaiact] | | NIST AI RMF 1.0 | Anyone (voluntary) | GOVERN 1.6 inventory "resourced according to organizational risk priorities"; GOVERN 1.7 decommissioning | Being revised (no date given)[^nistrmf][^nist100] | | ISO/IEC 42001:2023 | Organisations providing or using AI | Requirements for an AI management system | Published 2023-12-18[^iso42001] | | Singapore MGF for Agentic AI | Organisations deploying agents | Bound risks, human checkpoints, lifecycle controls | Government guidance, "a living document", January 2026[^mgfagentic] | | OMB M-25-21 | US federal agencies | Inventory AI use cases at least annually; publish a public version | April 2025; not rescinded as of October 2026[^m2521] | **EU details.** Prohibitions apply from 2 February 2025 (those added by the Digital Omnibus, Regulation (EU) 2026/1744, from 2 December 2026), GPAI rules from 2 August 2025, transparency rules from August 2026.[^ecaiact][^omnibus] Credit scoring of natural persons is an Annex III use, "with the exception of AI systems used for the purpose of detecting financial fraud"; profiling Annex III systems are always high-risk.[^euaiact] ### Tooling landscape Start with jobs, then products (as of October 2026): | Job | Typical tools | What to check | |---|---|---| | Inventory and workflow | A shared table at first; GRC suites; AI governance platforms such as IBM watsonx.governance and ModelOp[^wxgov][^modelophome] | OSFI's inventory fields;[^osfi] an API for the gate; tier and approval edits by MRM roles only, logged; discovery; full-history export | | Approval state at release | MLflow tags and aliases (the docs' example: `validation_status:approved`);[^mlflowreg] SageMaker approval statuses, where `Approved` "can initiate CI/CD deployment"[^smapprove] | Who may write versions, tags and status (in MLflow, `can_update` on the model or a model-version grant)[^mlflowauth][^mlflowrbac] | | Documentation | Model cards; SageMaker Model Cards (intended use, risk rating, a new version for every edit except approval status)[^smcards] | Immutability; export | | Fairness and explainability | Fairlearn (`MetricFrame`);[^fairlearn] SHAP[^shap] | Metric definitions match policy; group data handling | ### Automation and AI Automate the mechanical: reconciliation with discovery sources, linting records ([Lab 1](#hands-on-labs)), review reminders, refusing releases without an approval record ([Lab 3](#hands-on-labs)), report drafts. Keep people for tier overrides, validation judgements, approvals and exceptions. An LLM can draft documentation but cannot provide effective challenge. ## ModelOps in practice ### Reference implementation The core artefact is an **inventory with a policy and a linter**: one JSON file holds the tiering policy and a record per model with OSFI's six minimum fields ([building blocks](#building-blocks)) plus dates;[^osfi] a standard-library script computes tiers, review dates and findings. **Assumptions (illustrative):** a financial company with eight models of six types; owners are named people (fake IDs); review intervals of 12, 24 and 36 months are this Atlas's starting values (the PRA asks for a frequency "consistent with the model tier").[^prapdf] _Example status: verified by execution (Python 3.8.10, 2026-10-17) with the script below._ ```json { "as_of": "2026-10-17", "policy": { "tier_matrix": {"H": {"H": 1, "M": 1, "L": 2}, "M": {"H": 2, "M": 2, "L": 3}, "L": {"H": 3, "M": 3, "L": 4}}, "people_impact_floor": 2, "review_months": {"1": 12, "2": 24, "3": 36}, "warn_days": 60, "provider_warn_days": 180 }, "models": [ {"id": "credit-pd-retail", "type": "ml", "origin": "internal", "use": "default risk for retail loan approval", "owner": "a.rossi", "developer": "credit-analytics", "materiality": "H", "complexity": "H", "people_impact": true, "status": "production", "recorded_tier": 1, "approved": "2025-11-03", "last_review": "2025-10-20", "monitoring": true}, {"id": "fraud-rules-cards", "type": "rules", "origin": "internal", "use": "block suspicious card payments", "owner": "k.mensah", "developer": "fraud-strategy", "materiality": "H", "complexity": "L", "people_impact": false, "status": "production", "recorded_tier": 3, "approved": "2024-06-10", "last_review": "2024-06-01", "monitoring": true}, {"id": "markdown-optimiser", "type": "optimisation", "origin": "internal", "use": "set clearance discounts", "owner": "l.chen", "developer": "pricing-science", "materiality": "M", "complexity": "M", "people_impact": false, "status": "production", "recorded_tier": 2, "approved": "2025-03-15", "last_review": "2025-03-01", "monitoring": true}, {"id": "aml-vendor-scenarios", "type": "ml", "origin": "vendor", "use": "flag transactions for AML review", "owner": "p.novak", "developer": "vendor (example-vendor)", "materiality": "M", "complexity": "H", "people_impact": true, "status": "production", "recorded_tier": 2, "approved": "2026-04-01", "conditions_until": "2026-09-30", "last_review": "2026-03-20", "monitoring": true}, {"id": "support-assistant", "type": "llm-app", "origin": "third-party-api", "use": "answer customer questions", "owner": "s.haddad", "developer": "customer-ai", "materiality": "M", "complexity": "H", "people_impact": false, "status": "production", "recorded_tier": 2, "approved": "2026-05-12", "last_review": "2026-05-01", "monitoring": true, "provider_retirement": "2027-01-14"}, {"id": "refund-agent", "type": "agent", "origin": "internal", "use": "issue refunds up to a limit", "owner": "s.haddad", "developer": "customer-ai", "materiality": "M", "complexity": "H", "people_impact": true, "status": "production", "recorded_tier": 2, "approved": null, "last_review": null, "monitoring": false}, {"id": "churn-propensity", "type": "ml", "origin": "internal", "use": "target retention offers", "owner": "", "developer": "marketing-analytics", "materiality": "L", "complexity": "M", "people_impact": false, "status": "production", "recorded_tier": 3, "approved": "2024-02-01", "last_review": "2024-01-15", "monitoring": true}, {"id": "branch-forecast-sheet", "type": "spreadsheet", "origin": "internal", "use": "forecast branch staffing", "owner": "j.berg", "developer": "finance-planning", "materiality": "L", "complexity": "L", "people_impact": false, "status": "production", "recorded_tier": 4, "approved": null, "last_review": null, "monitoring": false} ] } ``` _Example status: verified by execution (Python 3.8.10, 2026-10-17); the output below is the actual output._ ```python """Lint a model inventory: compute risk tiers, review due dates and blocking findings. Usage: python3 modelops_inventory.py model-inventory.json Standard library only. The tier matrix, review intervals and thresholds live in the inventory's "policy" block (this Atlas's illustrative policy, not a regulatory rule). Exit code 1 if any ERROR (unowned, unapproved or out-of-conditions use in production). """ import calendar import json import sys from datetime import date def parse(value): return date.fromisoformat(value) if value else None def add_months(d, months): m = d.month - 1 + months year, month = d.year + m // 12, m % 12 + 1 return date(year, month, min(d.day, calendar.monthrange(year, month)[1])) def tier_of(model, policy): tier = policy["tier_matrix"][model["materiality"]][model["complexity"]] if model.get("people_impact"): tier = min(tier, policy["people_impact_floor"]) return tier def check(model, policy, today): findings = [] tier = tier_of(model, policy) live = model["status"] == "production" if not model.get("owner"): findings.append(("ERROR", "no accountable owner")) for field in ("use", "developer"): if not model.get(field): findings.append(("WARN", f"missing inventory field {field}")) if tier != model.get("recorded_tier"): findings.append(("WARN", f"recorded tier {model.get('recorded_tier')}, " f"rubric says {tier}: re-tier")) if tier == 4: return tier, None, findings # inventory only: owner and attestation if live and not model.get("approved"): findings.append(("ERROR", "in production without approval")) until = parse(model.get("conditions_until")) if live and until and until < today: findings.append(("ERROR", f"approval conditions expired {until}")) due = None last = parse(model.get("last_review")) if last: due = add_months(last, policy["review_months"][str(tier)]) if due < today: findings.append(("WARN", f"review overdue since {due}")) elif (due - today).days <= policy["warn_days"]: findings.append(("INFO", f"review due {due}")) elif live: findings.append(("WARN", "never reviewed")) if live and not model.get("monitoring"): findings.append(("WARN", "no monitoring plan")) retire = parse(model.get("provider_retirement")) if retire and (retire - today).days <= policy["provider_warn_days"]: findings.append(("WARN", f"provider retires the model {retire}: " "plan and approve a replacement")) return tier, due, findings def main(path): with open(path, encoding="utf-8") as fh: inv = json.load(fh) today, policy = parse(inv["as_of"]), inv["policy"] counts = {"ERROR": 0, "WARN": 0, "INFO": 0} tiers = {} print(f"Model inventory check as of {today}") for model in inv["models"]: tier, due, findings = check(model, policy, today) tiers[tier] = tiers.get(tier, 0) + 1 print(f"{model['id']:<22} {model['type']:<12} tier {tier} " f"next review {due or '-'}") for level, text in findings: counts[level] += 1 print(f" {level:<5} {text}") summary = ", ".join(f"tier {t}: {n}" for t, n in sorted(tiers.items())) print(f"{len(inv['models'])} models ({summary})") print(f"findings: {counts['ERROR']} ERROR, {counts['WARN']} WARN, " f"{counts['INFO']} INFO") return 1 if counts["ERROR"] else 0 if __name__ == "__main__": if len(sys.argv) != 2: sys.exit("usage: python3 modelops_inventory.py ") sys.exit(main(sys.argv[1])) ``` ```text $ python3 modelops_inventory.py model-inventory.json Model inventory check as of 2026-10-17 credit-pd-retail ml tier 1 next review 2026-10-20 INFO review due 2026-10-20 fraud-rules-cards rules tier 2 next review 2026-06-01 WARN recorded tier 3, rubric says 2: re-tier WARN review overdue since 2026-06-01 markdown-optimiser optimisation tier 2 next review 2027-03-01 aml-vendor-scenarios ml tier 2 next review 2028-03-20 ERROR approval conditions expired 2026-09-30 support-assistant llm-app tier 2 next review 2028-05-01 WARN provider retires the model 2027-01-14: plan and approve a replacement refund-agent agent tier 2 next review - ERROR in production without approval WARN never reviewed WARN no monitoring plan churn-propensity ml tier 3 next review 2027-01-15 ERROR no accountable owner branch-forecast-sheet spreadsheet tier 4 next review - 8 models (tier 1: 1, tier 2: 5, tier 3: 1, tier 4: 1) findings: 3 ERROR, 5 WARN, 1 INFO ``` **How to read it.** The errors matter most: an agent live without approval, a vendor model used after its conditional approval lapsed, an unowned model. The fraud rules block card payments at scale, so the rubric moves them to tier 2. The assistant's provider retirement (illustrative date) starts re-approval planning. The spreadsheet stays as tier 4, although US guidance excludes "simple arithmetic calculations, such as those found within spreadsheets":[^sr262a] governance scope is your choice. ### Hands-on labs #### Lab 1 (beginner): run and fix an inventory **Objective:** find and resolve blocking findings. **Prerequisites:** Python 3.8+; no account or cost; 45 minutes. 1. Save the [reference](#reference-implementation) JSON as `model-inventory.json` and the script as `modelops_inventory.py`. 2. Run `python3 modelops_inventory.py model-inventory.json; echo $?`. 3. Record four decisions: give `churn-propensity` a named owner; set `refund-agent` to `"suspended"` until approved; extend the AML vendor's `conditions_until` to `"2026-12-31"` *only* if your approver signs an exception; set `fraud-rules-cards` to `recorded_tier: 2`. 4. Run again, then add five of your own models. **Expected output** (checked by execution). Step 2: the reference output, exit code `1`. Step 3: `0 ERROR, 2 WARN, 1 INFO` (the overdue fraud review and the provider retirement remain), exit code `0`. **Validation:** two colleagues agree with every tier. **Troubleshooting:** `KeyError` means a rating other than H, M or L; `ValueError: Invalid isoformat string`, a date not in `YYYY-MM-DD`. **Teardown:** delete the folder, or commit the inventory. #### Lab 2 (intermediate): a disaggregated outcome report **Objective:** produce the fairness evidence a validator reviews, and see two metrics disagree. **Prerequisites:** Python 3.8+; synthetic counts, no personal data; 45 minutes. Real credit data lacks outcomes for declined applicants; the synthetic file assumes them known. _Example status: verified by execution (Python 3.8.10, 2026-10-17); the output below is the actual output._ ```json { "model": "credit-pd-retail v7", "window": "synthetic backtest; outcomes of declined applicants assumed known", "min_group_size": 100, "thresholds": {"impact_ratio": 0.8, "tpr_gap": 0.05}, "groups": { "A": {"applicants": 2400, "approved": 1440, "good": 1900, "good_approved": 1330}, "B": {"applicants": 1100, "approved": 495, "good": 820, "good_approved": 455}, "C": {"applicants": 60, "approved": 33, "good": 45, "good_approved": 31} } } ``` ```python """Disaggregated outcome report for a binary approve/decline model. Usage: python3 fairness_report.py outcomes.json Per group: approval rate, impact ratio against the highest-rate group, true positive rate (approval rate among applicants who later repaid) and default rate among the approved. Flags are prompts for investigation, not verdicts of (un)fairness. """ import json import sys def main(path): with open(path, encoding="utf-8") as fh: data = json.load(fh) limit, th = data["min_group_size"], data["thresholds"] rows = {} for name, g in data["groups"].items(): rows[name] = { "n": g["applicants"], "rate": g["approved"] / g["applicants"], "tpr": g["good_approved"] / g["good"], "bad": (g["approved"] - g["good_approved"]) / g["approved"], } usable = {k: v for k, v in rows.items() if v["n"] >= limit} top_rate = max(v["rate"] for v in usable.values()) top_tpr = max(v["tpr"] for v in usable.values()) print(f"{data['model']} | {data['window']}") print(f"{'group':<6}{'n':>6}{'approval':>10}{'impact':>8}{'TPR':>7}" f"{'defaults':>10} flags") for name, r in rows.items(): flags = [] if r["n"] < limit: flags.append(f"n<{limit}: too small to conclude") impact = "-" else: ratio = r["rate"] / top_rate impact = f"{ratio:.2f}" if ratio < th["impact_ratio"]: flags.append("impact ratio below screen") if top_tpr - r["tpr"] > th["tpr_gap"]: flags.append(f"TPR gap {top_tpr - r['tpr']:.3f}") print(f"{name:<6}{r['n']:>6}{r['rate']:>10.1%}{impact:>8}{r['tpr']:>7.1%}" f"{r['bad']:>10.1%} {'; '.join(flags) or 'none'}") return 0 if __name__ == "__main__": if len(sys.argv) != 2: sys.exit("usage: python3 fairness_report.py ") sys.exit(main(sys.argv[1])) ``` ```text $ python3 fairness_report.py outcomes.json credit-pd-retail v7 | synthetic backtest; outcomes of declined applicants assumed known group n approval impact TPR defaults flags A 2400 60.0% 1.00 70.0% 7.6% none B 1100 45.0% 0.75 55.5% 8.1% impact ratio below screen; TPR gap 0.145 C 60 55.0% - 68.9% 6.1% n<100: too small to conclude ``` **Steps:** run the script; set group B to `"approved": 560, "good_approved": 510` (a lower cut-off) and rerun. **Expected output** (checked by execution): B's impact ratio becomes 0.85 and passes the screen, its true-positive-rate gap falls to 0.078 but still exceeds 0.05, and its default rate among the approved rises from 8.1% to 8.9%. **Validation:** you can name the policy's primary metric for this use and the cost in defaults. **Troubleshooting:** `ZeroDivisionError` means a group with no approvals or no good outcomes. **Teardown:** delete the files. #### Lab 3 (advanced): enforce approval at release **Objective:** promote a version only when the inventory holds its tier's evidence, and make sure nobody else can. **Prerequisites** (the control depends on them): - An MLflow server you administer, with authentication on (`--app-name basic-auth`, default permission READ). Written against the MLflow 3.17.1 docs; permissions changed in 3.13 (roles) and 3.17 (DENY; version-level grants).[^mlflowauth][^mlflowrbac] - `credit-pd-retail-candidates`, editable by data scientists, and `credit-pd-retail`, created first by the promotion identity, the only non-admin with edit rights on it (creating a registered model makes the creator its manager).[^mlflowauth] No data scientist is an MLflow or workspace admin, or holds EDIT or MANAGE on `registered_model:*` or `registered_model_version:*` (with default READ, the version grant alone edits the production model's versions).[^mlflowrbac] - Only the MRM function and approvers can change tiers and `approvals` in the inventory (a GRC workflow, or a repository whose approval files need their review). The lab uses a local copy. The gate uses the stricter of the recorded and rubric tiers, refuses tiers outside the policy, and accepts a provisional approval only with an exception ID, approver, monitoring plan and an expiry at most 14 days out, renewed at most once. Method names follow the MLflow client reference.[^mlflowclient] _Example status: syntax-checked; logic executed with a stub client; not run against MLflow._ ```python """Promote an approved candidate version into the production registered model. Usage: python3 promotion_gate.py [alias] The tier and the approval come from the inventory (changed only by the MRM function and approvers), never from registry tags that developers can write. On a pass, version of "-candidates" is copied into "", the approval is recorded as tags on the copy and the alias (default "champion") is moved. Run it as the promotion identity: the only identity with EDIT on the production model. """ import json import sys from datetime import date REQUIRED = { # this Atlas's illustrative policy: approval evidence per risk tier 1: ["validation_report", "committee_approval", "monitoring_plan"], 2: ["validation_report", "approver", "monitoring_plan"], 3: ["approver"], 4: [], # inventory only } PROVISIONAL = ["exception_id", "approver", "monitoring_plan", "date", "conditions_until"] MAX_PROVISIONAL_DAYS, MAX_RENEWALS = 14, 1 def effective_tier(record, policy): """The stricter of the recorded tier and the rubric's tier (1 is strictest).""" tier = policy["tier_matrix"][record["materiality"]][record["complexity"]] if record.get("people_impact"): tier = min(tier, policy["people_impact_floor"]) return min(tier, record.get("recorded_tier") or tier) def missing_evidence(record, version, tier, today): """Return a list of problems; an empty list means the gate passes.""" if tier not in REQUIRED: return [f"tier {tier} is not in the policy"] if tier == 4: return [] approval = record.get("approvals", {}).get(str(version)) if not approval: return [f"no approval record for version {version}"] decision = approval.get("decision") if decision == "provisional": needed = PROVISIONAL elif decision in ("approved", "approved_with_conditions"): needed = REQUIRED[tier] else: return [f"decision is {decision}"] problems = [f"missing {key}" for key in needed if not approval.get(key)] until = approval.get("conditions_until") if decision == "approved_with_conditions" and not until: problems.append("conditional approval without conditions_until") if until and date.fromisoformat(until) < today: problems.append(f"conditions expired {until}") if decision == "provisional" and until and approval.get("date"): days = (date.fromisoformat(until) - date.fromisoformat(approval["date"])).days if days > MAX_PROVISIONAL_DAYS: problems.append(f"provisional for {days} days (max {MAX_PROVISIONAL_DAYS})") if approval.get("renewals", 0) > MAX_RENEWALS: problems.append(f"renewals {approval['renewals']} (max {MAX_RENEWALS})") return problems def main(inventory_path, model_id, version, alias="champion"): with open(inventory_path, encoding="utf-8") as fh: inv = json.load(fh) record = {m["id"]: m for m in inv["models"]}.get(model_id) if record is None: print(f"REFUSED {model_id}: not in the inventory") return 1 tier = effective_tier(record, inv["policy"]) problems = missing_evidence(record, version, tier, date.today()) if problems: print(f"REFUSED {model_id} v{version} (tier {tier}): " + "; ".join(problems)) return 1 from mlflow import MlflowClient # imported late: the logic is testable without MLflow client = MlflowClient() copy = client.copy_model_version(f"models:/{model_id}-candidates/{version}", model_id) for key, value in record.get("approvals", {}).get(str(version), {}).items(): client.set_model_version_tag(model_id, copy.version, key, str(value)) client.set_registered_model_alias(model_id, alias, copy.version) print(f"PROMOTED {model_id}-candidates v{version} -> {model_id} " f"v{copy.version} @{alias} (tier {tier})") return 0 if __name__ == "__main__": if len(sys.argv) not in (4, 5): sys.exit("usage: promotion_gate.py [alias]") sys.exit(main(*sys.argv[1:])) ``` Merge this `approvals` object into the `credit-pd-retail` record of `model-inventory.json`; give version 8 today's `date` and a `conditions_until` 30 days later. _Example status: syntax-checked; used in the stub run._ ```json { "approvals": { "7": {"decision": "approved", "date": "2026-10-09", "validation_report": "VR-2026-031", "committee_approval": "MRC-2026-10-08", "monitoring_plan": "MP-credit-pd-v7"}, "8": {"decision": "provisional", "date": "2026-10-17", "conditions_until": "2026-11-16", "exception_id": "EX-2026-044", "approver": "cro-delegate", "monitoring_plan": "MP-credit-pd-v8", "renewals": 0}, "9": {"decision": "pending", "validation_report": "VR-2026-037"} } } ``` **Steps:** as the promotion identity, run `python3 promotion_gate.py model-inventory.json credit-pd-retail 7`, then versions 8 and 9; set version 8's `conditions_until` to ten days after its `date` and rerun it. **Expected output** (stub run, 2026-10-17): v7 `PROMOTED credit-pd-retail-candidates v7 -> credit-pd-retail v1 @champion (tier 1)`, exit 0; v8 `REFUSED ... provisional for 30 days (max 14)` and v9 `REFUSED ... decision is pending`, exit 1; v8 after the fix `PROMOTED ... v2`, exit 0. **Validation:** (1) a data scientist's attempt to create a version in, tag or move an alias of `credit-pd-retail` must fail, or the gate is advisory; (2) with `recorded_tier` 3 and no `committee_approval` on v7, the gate still applies tier 1 and refuses. **Troubleshooting:** a permission error on the copy means the promotion identity lacks rights on one of the two models. **Teardown:** delete the test versions, alias and `approvals` block. ### Use cases and scenarios #### Case 1: an external validation of a bought clinical model (sourced) The Epic Sepsis Model, "implemented at hundreds of US hospitals", was validated externally at Michigan Medicine on 38,455 hospitalisations: an area under the ROC curve of 0.63 (0.5 is chance), 67% of sepsis patients missed, and alerts for 18% of all hospitalisations; the authors concluded that it "has poor discrimination and calibration".[^epic] **Lesson:** validate a vendor model *on your population*; alert burden is part of its risk. #### Case 2: a convenient proxy becomes a group harm (sourced) Obermeyer and colleagues found that a widely used care-management algorithm predicted health-care **costs** rather than illness, so "At a given risk score, Black patients are considerably sicker than White patients"; remedying this would raise the share of Black patients receiving extra help from 17.7% to 46.5%.[^obermeyer] **Lesson:** conceptual soundness asks whether the target is the right proxy. #### Case 3: model risk appetite, not model quality (sourced) In November 2021 Zillow wound down Zillow Offers; its CEO said "the unpredictability in forecasting home prices far exceeds what we anticipated" and that continuing "would result in too much earnings and balance-sheet volatility".[^zillow8k] It reported a $304 million inventory write-down "as a result of purchasing homes at higher prices than our current estimates of future selling prices".[^zillowletter] **Lesson** (this page's reading, not Zillow's): when a model drives large exposure, approval conditions need exposure caps, enforced in [DecisionOps](../decisionops/). #### Case 4: the chatbot's answer is the company's answer (sourced) In *Moffatt v. Air Canada* (British Columbia Civil Resolution Tribunal, 14 February 2024), the airline's chatbot said bereavement fares could be claimed retroactively; they could not. The tribunal called the suggestion that the chatbot is "a separate legal entity that is responsible for its own actions" "a remarkable submission", held: "It makes no difference whether the information comes from a static page or a chatbot", and ordered Air Canada to pay C$812.02.[^moffatt] **Lesson:** customer-facing LLM applications need an owner and approved uses. #### Scenario 5: approving the support assistant (illustrative) ::: example Illustrative scenario: tier, approve and re-approve an LLM application **Context.** The reference inventory's `support-assistant` runs on a provider's model that retires on 2027-01-14. **Method.** The MRM function confirms tier 2; [LLMOps](../llmops/) evaluations on 500 recent questions, including refund and fee questions (Case 4's risk), show policy answers wrong 4% of the time. **Decision.** Approved with conditions: refund, fee and legal-rights questions get the policy link and a hand-off to a person; monthly sampled review; expiry at the retirement date. **Trade-off:** fewer automated answers. **Success criteria:** policy-answer error below 1% in the monthly sample, no off-topic answers, replacement approved 60 days before retirement. ::: ## Measure, govern, pay ### Metrics and KPIs | KPI | Definition / formula | Pitfall | |---|---|---| | Inventory coverage | Discovered models in the inventory ÷ discovered models | Self-reported models only | | Owned share | Models with a named person as owner ÷ inventory | A team name with nobody behind it | | Validated before use | Tier 1–2 models first used after validation ÷ those put into use | Counting provisional use | | Review currency | Tier 1–3 production models within review interval ÷ tier 1–3 production models | Re-tiering to a laxer tier to hide overdue reviews | | Unapproved or lapsed use | Production models with no approval or expired conditions (target 0) | Approvals with no expiry | | Exception age | Median days open of exceptions; count past expiry | Renewing exceptions indefinitely | | Finding closure | High-severity findings closed by due date ÷ due | Closing by downgrading severity | | Time to decision | Median days from intake to approval, per tier | Speed bought by skipping review | | Monitoring coverage | Tier 1–3 production models with an active monitoring plan ÷ tier 1–3 production models | Plans alerting nobody | Owned share, review currency and exception age lead; incidents and audit findings lag. Review monthly in the MRM function, quarterly with senior management. #### Worked calculation From the [reference implementation](#reference-implementation) (eight models, all in production): - **Owned share:** 7 ÷ 8 = **87.5%** (`churn-propensity` has no owner). - **Review currency:** of seven tier 1–3 models, `fraud-rules-cards` is overdue and `refund-agent` never reviewed: 5 ÷ 7 = **71%**. - **Unapproved or lapsed use:** **2** (the agent; the AML vendor model). - **Monitoring coverage** for tiers 1–3: 6 ÷ 7 = **86%**. After the Lab 1 fixes, owned share is 100% and unapproved or lapsed use 0. Review currency rises to 5 ÷ 6 = 83% only because the suspended agent left the denominator; the fraud rules are still overdue. **Vanity traps:** "models approved this quarter", "share of models with a model card" (empty cards count). ### Quality, security, privacy and compliance | Risk | Signal | Mitigation | Owner | |---|---|---|---| | Shadow models outside intake | AI services in cloud bills; procurement lines | Weekly reconciliation with discovery | MRM function | | Approval bypass | Alias, version or tag change not made by the promotion identity | Separate candidate and production models; evidence read from the inventory ([Lab 3](#hands-on-labs)) | Platform team | | Vendor opacity | No method or data documentation | Contract terms; outcome monitoring; overlays | Owner, procurement | | Provider changes the model behind an API | Version change; retirement notice | Pin versions; material change; [LLMOps](../llmops/) regressions | Owner | | Bias measurement needs sensitive data | No fairness evidence for tier 1 | Lawful basis ([PrivacyOps](../privacyops/)) | Privacy, owner | | Evidence edited or lost | Changed approvals | Immutable versions (as SageMaker Model Cards keep[^smcards]) | MRM function | | EU high-risk logs kept too briefly | Retention under six months | Deployers keep logs "of at least six months"[^euaiact] | Owner, compliance | **Auditability.** The [approval records](#templates) are the trail; collecting evidence across systems belongs to [ComplianceOps](../complianceops/). ### Cost, capacity and sustainability **Illustrative calculation** (assumed effort, not a benchmark): initial validation takes 120 hours for tier 1, 40 for tier 2 and 8 for tier 3; a periodic review a third of that. For 15 tier-1, 60 tier-2 and 150 tier-3 models reviewed every 12, 24 and 36 months: 15 × 40 + 60 × 13.3 ÷ 2 + 150 × 2.7 ÷ 3 ≈ 600 + 400 + 135 = **1,135 review hours a year**; ten new tier-2 models add 400. At about 1,500 productive hours per validator that is **one to 1.5 validators**; all 225 at tier-1 depth would take 9,000 hours, six people. Compute and token spend belong to [FinOps for AI](../finops-for-ai/), carbon to [GreenOps](../greenops/); ModelOps's own cost driver is hand-made evidence. ### Assurance of the operating model - **Internal audit** evaluates whether the practices "are rigorous and effective" without redoing validation.[^sr262a] - **Validate the tiering itself**: the PRA expects "periodic validation, or other objective and critical review" of it.[^prapdf] - **Reconcile** quarterly: trace twenty sampled production endpoints and AI service accounts to the inventory. - **Risk indicators** (this Atlas's thresholds): any unapproved or lapsed use; a tier-1 review 30 days overdue; an exception past expiry; coverage below 95%. ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause and diagnosis | Remediation | Prevention | |---|---|---|---| | Audit finds unlisted models | Self-reporting only; check cloud and procurement data | Add, tier and triage them | Automated discovery | | Everything is tier 1, or nothing is | Vague rubric; incentives to under-tier | Re-tier a sample with two reviewers | Measurable criteria; tier review | | Validation backlog delays every release | Same depth for all tiers and changes | Tier templates; change envelopes; provisional use | Capacity planning | | Vendor model drifts with no explanation | No outcome monitoring on your data | Outcomes analysis; overlay; vendor escalation | Contracted change notices | ### Worked runbook: a model breaches its limits An **illustrative scenario**: a marketing campaign changes the applicant mix, and the tier-1 credit model breaches two approval conditions: population stability (how far applicants have moved from the development sample, a drift measure from [MLOps](../mlops/)) and an approval-rate band. OSFI expects contingency plans "for model unavailability, deterioration in model performance, or outright failure".[^osfi] ::: figure Model breach handling: contain, decide, remediate, re-approve ```mermaid flowchart TB D["Monitoring breach on tier-1 model"] --> S{"Harm ongoing to people or money?"} S -- "Yes" --> C["Contain: fallback model, tighter cut-off or manual review"] S -- "No" --> W["Watch: owner reviews within 2 business days"] C --> N["Notify approver, MRM function, users"] W --> N N --> R{"Cause"} R -- "Data or pipeline defect" --> F["Fix with MLOps; backfill decisions"] R -- "Population changed" --> O["Overlay with expiry; redevelop"] R -- "Used outside approved scope" --> U["Stop the out-of-scope use"] F --> V["Validator reviews the fix"] O --> V U --> V V --> A{"Approver decision"} A -- "Restore" --> OK["Approved: conditions updated"] A -- "Not yet" --> X["Restricted or suspended; exception logged"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class W,N,F,O,U,V,OK core class S,R,A decision class D,C,X risk ``` Text equivalent: If harm is ongoing, contain it (fallback model, tighter cut-off or manual review); otherwise the owner reviews within two business days. The approver, MRM function and users are notified. A data defect is fixed with MLOps and decisions backfilled; a changed population gets an expiring overlay and redevelopment; out-of-scope use stops. After the validator's review, the approver restores approval with new conditions, or keeps the model restricted or suspended. ::: 1. **Record** the breach against the approval conditions; **contain** if harm is ongoing. 2. **Notify** the approver and users; in the EU, once high-risk duties apply, a deployer reports a serious incident to the provider first, then the authorities (Article 26).[^euaiact] 3. **Diagnose** with [MLOps](../mlops/): data defect, drift, or use outside the validated population? 4. **Remediate**: fix, expiring overlay, or redevelopment (a material change). 5. **Re-approve** on the validator's review; add campaigns to the review triggers. ### Anti-patterns and myths | Anti-pattern or myth | Why it fails | Warning signal | Do instead | |---|---|---|---| | "ModelOps is a tool we buy" | A platform stores decisions; it does not make them | Platform live, no rubric | Rubric and inventory first | | One process for every model and change | Tier-1 depth everywhere creates queues and workarounds | Teams avoid intake | Tiers and change classes | | "The vendor validated it" | Their population is not yours ([Case 1](#use-cases-and-scenarios)) | No outcomes analysis on your data | Validate your use | | "Not a model under the regulation, so not governed" | Rules and LLM apps cause harm too ([Case 4](#use-cases-and-scenarios)) | Inventory equals regulatory scope | Record regulatory scope as a field | | "A fairness metric passed, so it is fair" | Metrics conflict; proxies hide harm ([Case 2](#use-cases-and-scenarios)) | One metric in the report | A primary metric per use, with reasons | ## Deciding and adopting ### When to use and when not to use **Build an explicit practice when any of these hold:** models inform decisions about people's access to credit, jobs, insurance, care or public services; your supervisor publishes model risk expectations, or you deploy EU AI Act Annex III systems; models are bought or called through APIs faster than anyone reviews them, or agents can act; nobody can list production models with owners. **Keep it minimal, or wait, when** a few low-stakes models have clear owners (an inventory and quarterly review suffice); approvals would have no delivery pipeline to enforce them (fix [MLOps](../mlops/) first); or the "programme" would be a policy with no inventory. **Hidden responsibilities:** independent validators, a vendor-review budget, and someone who can say no to a launch. ### Decision framework The tree encodes the policy file's matrix and override, so hand tiering and the linter agree. ::: figure Decision tree: assigning a risk tier to a new model or a material change ```mermaid %%{init: {"flowchart": {"wrappingWidth": 130, "nodeSpacing": 24, "rankSpacing": 34}}}%% flowchart TB S["New model or material change"] --> Q1{"Output used in a decision?"} Q1 -- "No" --> T4A["Tier 4: inventory only"] Q1 -- "Yes" --> QH{"Materiality high?"} QH -- "Yes" --> Q3{"Complexity low?"} Q3 -- "No" --> T1["Tier 1: independent validation, committee approval"] Q3 -- "Yes" --> T2["Tier 2: independent validation, MRM approval"] QH -- "No" --> QP{"Affects individuals' access or rights?"} QP -- "Yes" --> T2 QP -- "No" --> QM{"Materiality medium?"} QM -- "Yes" --> QC1{"Complexity low?"} QC1 -- "No" --> T2 QC1 -- "Yes" --> T3["Tier 3: peer review, owner approval"] QM -- "No, low" --> QC2{"Complexity low?"} QC2 -- "No" --> T3 QC2 -- "Yes" --> T4B["Tier 4: inventory only"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class S neutral class Q1,QH,Q3,QP,QM,QC1,QC2 decision class T1 risk class T2,T3,T4A,T4B core ``` Text equivalent: A model whose output is not used in a decision is tier 4 (inventory only). Otherwise: high materiality gives tier 1 (committee approval), or tier 2 if complexity is low; a model affecting individuals' access or rights is tier 2 or stricter; medium materiality gives tier 2, or tier 3 if complexity is low; low materiality gives tier 3, or tier 4 if complexity is low. ::: **How much ModelOps?** If three or more conditions above hold, staff a part-time MRM function and an independent reviewer; if a supervisor oversees your models, plan a separate validation function. ### Maturity model These levels are **this Atlas's model**; no official ModelOps maturity model exists. | Level | Observable behaviour | Exit criteria | |---|---|---| | 0. Unknown portfolio | Models found in incidents; developers approve | An inventory of production models with owners | | 1. Inventoried | One list, owners named, all model types | Owned share 100%; discovery reconciles monthly | | 2. Tiered | Documented rubric; tiers and review dates set | Reviewers agree on ≥ 80% of tiers; no tier-1 model unvalidated | | 3. Enforced | Approval records gate releases; conditions expire; change classes applied | Zero unapproved or lapsed use for two quarters | | 4. Portfolio-managed | Aggregate risk and capacity steer investment; yearly self-assessment | Sustain; external review | ### Adoption roadmap: 30/60/90 days | Window | Goals | Activities | Metrics at the end | |---|---|---|---| | Days 1–30: see the portfolio | Inventory | Collect models from registries, cloud usage and procurement; name owners; run Lab 1 | Coverage; owned share | | Days 31–60: tier and decide | Rubric and roles | Agree rubric and change classes; tier everything; name validators and approvers; approve or suspend unapproved use | Unapproved use = 0 | | Days 61–90: enforce and report | Gates and reviews | Release gate for tiers 1–2 (Lab 3); exceptions register; first portfolio report | Review currency; time to decision | Dependencies: a registry to hook into ([MLOps](../mlops/)); a risk appetite ([RiskOps](../riskops/)). ## Toolkit ### Checklists #### Intake (every new model or material change) - [ ] Purpose, decision supported and approved uses in one paragraph; type and origin recorded. - [ ] Owner is a named person; data sources and regulatory scope set; provisional tier within five working days. - [ ] For a change: class recorded; material changes re-tiered. #### Before approval (tiers 1–2) - [ ] Validation covers conceptual soundness, outcomes analysis and a monitoring plan. - [ ] Performance by group, with the primary fairness metric named; explanations fit each audience. - [ ] Limits, conditions and expiry written into the approval record that the release gate reads. - [ ] Vendor or API models: documentation received; change and retirement notices routed to the owner. - [ ] Agents: tool permissions, action limits and human checkpoints listed. #### Periodic review - [ ] Used only for approved purposes; consumers current; breaches, overrides and overlays examined. - [ ] Tier still right; change envelope still valid; findings closed or re-dated with reasons. #### Retirement - [ ] Consumers notified with a date; traffic and access stopped; downstream effects watched. - [ ] Model, documentation and approval history retained for the agreed period. - [ ] Third-party contracts and API keys closed ([ProcurementOps](../procurementops/)). ### Templates #### One-page ModelOps charter ```text MODEL GOVERNANCE CHARTER — Owner: Review: yearly Mission: Every model we rely on is known, owned, used within approved limits, and retired on purpose. Scope: ML, rules, optimisation, LLM applications, agents, vendor and API models, spreadsheets used in decisions. Regulatory scope recorded per model. Principles: Inventory before policy · Rigour follows risk · Independence where it matters · Approve a use, not a model · Evidence lives with the model · Bought is not exempt · Retire on purpose. Tiers: 1 committee approval · 2 MRM approval · 3 owner approval · 4 inventory only. Changes: material = new approval · bounded = validator + approver · in envelope = automated record · record only = log. Goals: Owned share 100% · unapproved or lapsed use 0 · review currency >= 95%. Decision rights: Approvers per tier: . Exceptions: , maximum days. Non-goals: Approving every spreadsheet; replacing developers' testing; legal advice. ``` #### Model intake and inventory record ```text MODEL RECORD — Date: Name and description: Type: ml | rules | optimisation | llm-app | agent | other Origin: internal | vendor | API Owner (person): Developer: Users: Purpose and decision supported: Approved uses: Not approved for: Materiality (H/M/L) and why: Complexity (H/M/L) and why: Affects individuals' access or rights? Y/N Can act without a person? Y/N Regulatory scope: bank MRM Y/N · EU AI Act class <...> · other <...> Data sources and limitations: Dependencies (models, data, providers): Tier: proposed confirmed by Approval: by · conditions <...> · conditions until Change envelope: Last material change: Last review: Next review: Monitoring plan: Provider retirement date (if any): ``` #### Approval record with conditions ```text APPROVAL — v Tier: Date: Validation report: Validator: Independent of developer: Y/N Decision: approved | approved_with_conditions | provisional | rejected Approved uses: Prohibited uses: Conditions: Provisional only: exception ID · expiry <= 14 days · renewals 0 | 1 Open findings and due dates: Expires / next review: Triggers for early review: Approver: Recorded in the inventory by: ``` #### Exception request (provisional or out-of-scope use) ```text EXCEPTION — Requested: Type: use before validation | use outside approved scope | performance breach Business need and why it cannot wait: Limits while the exception lasts: Extra monitoring and reporting: Expiry date (maximum days): Renewal allowed: once | never Compensating controls owner: Approver: Logged in register: Y/N ``` ### Deep comparisons | Approach | Strength | Weakness | Choose it when | |---|---|---|---| | ModelOps (this page) | One inventory and risk language for everything that decides | Unsettled label; can sprawl | Many model types, high stakes | | Bank MRM (SR 26-2, SS1/23, E-23) | Mature practice, supervisory expectations | Definitions differ by country | You are supervised: extend it | | AI governance programme (NIST AI RMF, ISO/IEC 42001) | Organisation-wide management system | Can stay at policy level | You need a management-system frame | | [MLOps](../mlops/) and [LLMOps](../llmops/) alone | Automation, evaluations | No authority to say no | Few low-stakes models | ### FAQ **Is ModelOps the same as MLOps?** Some use the terms interchangeably;[^modelop2021] this Atlas separates building and running (MLOps) from deciding allowed uses (ModelOps). **Does a rules engine or a spreadsheet count as a model?** It depends on whose definition ([what ModelOps is not](#what-modelops-is-not)); inventory anything whose output drives a decision. **We only call a foundation model through an API. What is there to validate?** Your use: task, prompts and retrieval, outcomes on your cases, limits, and the provider's retirement date ([Scenario 5](#scenario-5-approving-the-support-assistant-illustrative)). [LLMOps](../llmops/) supplies the evaluations; ModelOps decides whether they suffice. **Does every retrained version need a committee?** No: retraining inside a pre-approved envelope is recorded automatically; a new method, use or provider model is a material change ([change classes](#playbook-model-changes-and-materiality)). **Can an agent be approved at all?** Yes, as a bounded use: named tools, data and actions, limits and human checkpoints.[^mgfagentic] An agent with open-ended permissions cannot be tiered meaningfully. ## Going further ### Advanced topics and emerging direction - **Aggregate model risk.** Models sharing a data source, vendor or assumption fail together; US guidance assesses risk "both individually and in aggregate".[^sr262a] Inventory dependency fields make this measurable. - **Generative and agentic AI in banks.** The US agencies plan "a request for information" on model risk, "including generative AI and agentic AI" (none in the Federal Register as of 18 October 2026).[^occ] ### Learning path - **Prerequisites:** [MLOps](../mlops/) basics; a confusion matrix; risk vocabulary ([RiskOps](../riskops/)). - **One day:** this page; SR 26-2 (12 pages) and OSFI E-23's inventory appendix; Lab 1.[^sr262a][^osfi] - **One week:** Labs 2 and 3; model cards and datasheets;[^modelcards][^datasheets] tier ten real models with a colleague. - **Thirty days:** the roadmap's first month for one business unit; one tier-2 validation report; PRA SS1/23.[^prapdf] - **Specialise:** fairness (Kleinberg et al.; Fairlearn's user guide[^kleinberg][^fairlearnst]), EU AI Act work with [ComplianceOps](../complianceops/), or agent evaluation with [LLMOps](../llmops/). **Project:** inventory, rubric and gate for an open-source model zoo. ## Wrap-up ### Executive recap and cheat sheet | Topic | What to remember | |---|---| | Definition | Governance and lifecycle management of every model you rely on; a new label for an old banking practice. | | Artefacts | Inventory, rubric, change classes, validation reports, expiring approvals, exceptions register. | | Metrics | Owned share, review currency, unapproved or lapsed use (target 0). | | 2026 changes | SR 26-2 replaced SR 11-7 (17 April); EU Annex III duties from 2 December 2027; OSFI E-23 from 1 May 2027. | | Watch-outs | One process for all; a gate that trusts developer-written evidence; "the vendor validated it". | | Next actions | List models with owners; agree a rubric and change classes; suspend unapproved agents; date every approval. | ### A judgement call: launch the refreshed fraud model before validation finishes? **Situation (illustrative).** A card issuer's tier-1 fraud model misses a new attack; losses rise by an estimated 200,000 (currency units) a week. A retrained version is ready; full validation takes six weeks, a focused review two. **Option A:** wait. **Option B:** deploy now under a provisional approval. **What the guidance allows.** US guidance accepts that "certain circumstances (e.g., an urgent business need) may necessitate using the model before validation is completed", with controls such as "placing limits on model use or more closely monitoring its performance".[^sr262a] The PRA treats it as a temporary exception supported by senior management.[^prapdf] **The attractive choice and why it can be wrong.** Option B fails if "provisional" has no teeth: the model may block legitimate customers in a segment the retraining over-weighted (a harm the loss figure ignores), and the exception is renewed while validation never finishes. **Recommendation.** Option B, with conditions: (1) 72 hours in shadow, comparing decline rates by segment with the old model; (2) a score band sent to manual review instead of auto-decline; (3) daily monitoring of declines and complaints by segment; (4) a `provisional` record with an exception ID, a senior approver, the monitoring plan and a 14-day expiry, renewable once after the focused review, as the [Lab 3](#hands-on-labs) gate checks; (5) full validation scheduled. **Uncertainty:** the loss estimate may be off by half; if one segment's declines jump in shadow, revert to A with a rule-based patch. ### Learning objectives and mastery check You should now be able to **explain** the meanings of ModelOps, **build** an inventory and tier models, **design** approval and change evidence, **evaluate** fairness reports, and **decide** exceptions. Self-check: 1. A team says its assistant needs no review because "it just calls the provider's model". What is wrong? ::: answer Show answer for question 1 The organisation answers for the assistant's output (*Moffatt v. Air Canada*).[^moffatt] The use needs a tier, an owner, approved uses and evaluations; a provider's replacement model is a material change.[^oaidepr][^anthdepr] ::: 2. Your US bank's MRM policy covers "all models per SR 11-7". What should change after April 2026? ::: answer Show answer for question 2 First check applicability: SR 26-2, which superseded SR 11-7 on 17 April 2026, is expected to be most relevant above $30 billion in total assets, though it may also be relevant to smaller banks with significant model risk. It narrows the definition (no simple spreadsheet arithmetic or theory-free deterministic rules), leaves out generative and agentic AI, and drops the "at least annually" review.[^sr262][^sr262a][^sr117] Keep excluded systems governed: the bank's practices "should guide the determination of appropriate governance and controls" for them.[^sr262a] ::: 3. In Lab 2, raising group B's approvals lifts its impact ratio from 0.75 to 0.85. Is the model now fair? ::: answer Show answer for question 3 Not necessarily. The true-positive-rate gap is still 0.078 and B's defaults among the approved rose to 8.9%. Fairness criteria can be incompatible,[^kleinberg] the four-fifths screen is a US employment heuristic,[^uggsp] and the target may be a biased proxy.[^obermeyer] Policy must name the primary metric, with reasons. ::: 4. The inventory shows 400 models, all tier 1, and a two-year validation backlog. What do you do first? ::: answer Show answer for question 4 Re-tier a sample with a measurable rubric and two reviewers: if everything is tier 1, nothing is tiered. Then validate true tier-1 models first, give tiers 2–3 their lighter scopes and routine retraining an envelope, and grant time-boxed provisional approvals where needed.[^prapdf][^osfi] ::: 5. An EU bank retrains a third-party credit-scoring system on its own data until its behaviour changes materially. What changes under the AI Act? ::: answer Show answer for question 5 Credit scoring is an Annex III high-risk use, and a deployer that substantially modifies a high-risk system becomes a provider under Article 25.[^euaiact] Whether this retraining qualifies is a legal question; ModelOps treats it as a material change and escalates it. Annex III duties apply from 2 December 2027.[^omnibus] ::: ### Conclusion ModelOps is a young name for an old duty. Banks learned that a sound model used in the wrong place is a risk, that someone independent must be able to say no, and that control starts with a list. What changed is the portfolio: rules, vendor scores, LLM applications and agents now shape decisions, and no single rulebook covers them. The useful version is modest: one inventory, an agreed rubric, approvals that name uses and expire, a gate only approvers can satisfy, and validation where the stakes justify it. Organisations that skip it still have model risk; they find it in an audit, a court or a loss. ## Sources and further reading Each source was opened on its "Checked" date; other dates are the source's own. Claim-level support is in `research/modelops-ledger.md`. [^gartner]: [ModelOps — Gartner IT Glossary (archived)](https://web.archive.org/web/20250517082631/https://www.gartner.com/en/information-technology/glossary/modelops) — Gartner, snapshot of 2025-05-17 (live page blocks automated access). Checked 2026-10-17. [^ibm2019]: [ModelOps: Cloud-based lifecycle management for reliable and trusted AI](https://research.ibm.com/publications/modelops-cloud-based-lifecycle-management-for-reliable-and-trusted-ai) — W. Hummer et al., IEEE IC2E 2019 (IBM Research), 2019-06-01. Checked 2026-10-17. [^modelop2021]: [MLOps and ModelOps: What's the Difference and Why it Matters](https://www.kdnuggets.com/2021/09/mlops-modelops-difference.html) — Stu Bailey (ModelOp co-founder), KDnuggets, 2021-09-28. Checked 2026-10-17. [^modelophome]: [ModelOp home page](https://www.modelop.com/) — ModelOp, as of October 2026 (vendor claims). Checked 2026-10-17. [^wxgov]: [IBM watsonx.governance](https://www.ibm.com/products/watsonx-governance) — IBM product page, as of October 2026 (vendor claims). Checked 2026-10-17. [^sr262]: [SR 26-2: Revised Guidance on Model Risk Management](https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm) — Board of Governors of the Federal Reserve System, 2026-04-17. Checked 2026-10-17. [^sr262a]: [Supervisory Guidance on Model Risk Management (SR 26-2 attachment)](https://www.federalreserve.gov/supervisionreg/srletters/SR2602a1.pdf) — Federal Reserve, FDIC, OCC, 2026-04-17. Checked 2026-10-18. [^sr117]: [Supervisory Guidance on Model Risk Management (SR 11-7 attachment, Internet Archive snapshot)](https://web.archive.org/web/20260316235822/https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf) — Federal Reserve and OCC, 2011-04-04; superseded 2026-04-17 (live URL now 404). Checked 2026-10-17. [^occ]: [OCC Bulletin 2026-13: Model Risk Management: Revised Guidance](https://www.occ.gov/news-issuances/bulletins/2026/bulletin-2026-13.html) — Office of the Comptroller of the Currency, 2026-04-17. Checked 2026-10-17. [^pra]: [SS1/23 – Model risk management principles for banks](https://www.bankofengland.co.uk/prudential-regulation/publication/2023/may/model-risk-management-principles-for-banks-ss) — Bank of England PRA; 2023-05-17, effective 2024-05-17, current version 2026-04-23. Checked 2026-10-17. [^prapdf]: [Supervisory Statement 1/23 – April 2026 (PDF)](https://www.bankofengland.co.uk/-/media/boe/files/prudential-regulation/supervisory-statement/2026/liaf0126app5.pdf) — PRA, April 2026. Checked 2026-10-18. [^osfi]: [Guideline E-23 – Model Risk Management (2027)](https://www.osfi-bsif.gc.ca/en/guidance/guidance-library/guideline-e-23-model-risk-management-2027) — OSFI Canada, 2025-09-11, effective 2027-05-01. Checked 2026-10-18. [^euaiact]: [Regulation (EU) 2024/1689 (Artificial Intelligence Act)](https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng) — Official Journal of the EU, 2024-07-12 (as adopted; amended by the Digital Omnibus). Checked 2026-10-17. [^omnibus]: [Regulation (EU) 2026/1744 (Digital Omnibus on AI)](https://eur-lex.europa.eu/eli/reg/2026/1744/oj/eng) — Official Journal of the EU, adopted 2026-07-08, published 2026-07-24, in force 2026-07-27. Checked 2026-10-17. [^ecaiact]: [AI Act: regulatory framework for AI](https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai) — European Commission, updated 2026-08-03. Checked 2026-10-17. [^nistrmf]: [AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) — NIST, living page (AI RMF 1.0 of 2023-01-26; under revision). Checked 2026-10-17. [^nist100]: [Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf) — NIST, 2023-01. Checked 2026-10-17. [^iso42001]: [ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system](https://webstore.iec.ch/en/publication/90574) — ISO/IEC (IEC Webstore listing), 2023-12-18. Checked 2026-10-17. [^mgfagentic]: [Singapore Launches New Model AI Governance Framework for Agentic AI](https://www.mddi.gov.sg/newsroom/singapore-launches-new-model-ai-governance-framework-for-agentic-ai--/) — Singapore MDDI, 2026-01-22. Checked 2026-10-17. [^m2521]: [M-25-21: Accelerating Federal Use of AI through Innovation, Governance, and Public Trust](https://www.whitehouse.gov/wp-content/uploads/2025/02/M-25-21-Accelerating-Federal-Use-of-AI-through-Innovation-Governance-and-Public-Trust.pdf) — US OMB, 2025-04-03 (scanned PDF); no rescission in OMB's [memoranda list](https://www.whitehouse.gov/omb/information-resources/guidance/memoranda/). Checked 2026-10-18. [^nyc144]: [Automated Employment Decision Tools (AEDT)](https://www.nyc.gov/site/dca/about/automated-employment-decision-tools.page) — NYC DCWP, living page (enforcement since 2023-07-05). Checked 2026-10-17. [^regb]: [12 CFR 1002.9 — Notifications (Regulation B)](https://www.ecfr.gov/current/title-12/chapter-X/part-1002/section-1002.9) — eCFR, up to date as of 2026-10-15. Checked 2026-10-17. [^cfpbwithdraw]: [Interpretive Rules, Policy Statements, and Advisory Opinions; Withdrawal](https://www.federalregister.gov/documents/2025/05/12/2025-08286/interpretive-rules-policy-statements-and-advisory-opinions-withdrawal) — CFPB, Federal Register, 2025-05-12. Checked 2026-10-17. [^uggsp]: [29 CFR 1607.4 — Information on impact (Uniform Guidelines on Employee Selection Procedures)](https://www.ecfr.gov/current/title-29/subtitle-B/chapter-XIV/part-1607/section-1607.4) — eCFR, up to date as of 2026-10-15. Checked 2026-10-17. [^kleinberg]: [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://arxiv.org/abs/1609.05807) — J. Kleinberg et al., arXiv:1609.05807, 2016. Checked 2026-10-17. [^chouldechova]: [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://arxiv.org/abs/1610.07524) — A. Chouldechova, arXiv:1610.07524, 2016. Checked 2026-10-17. [^rudin]: [Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead](https://arxiv.org/abs/1811.10154) — C. Rudin, Nature Machine Intelligence 1, 2019 (arXiv version). Checked 2026-10-17. [^shap]: [A Unified Approach to Interpreting Model Predictions](https://arxiv.org/abs/1705.07874) — S. Lundberg, S.-I. Lee, arXiv:1705.07874, 2017. Checked 2026-10-17. [^modelcards]: [Model Cards for Model Reporting](https://arxiv.org/abs/1810.03993) — M. Mitchell et al., FAT* 2019 (arXiv:1810.03993). Checked 2026-10-17. [^datasheets]: [Datasheets for Datasets](https://arxiv.org/abs/1803.09010) — T. Gebru et al., arXiv:1803.09010, 2018 (Communications of the ACM, December 2021). Checked 2026-10-17. [^fairlearn]: [Assessment — Fairlearn user guide](https://fairlearn.org/main/user_guide/assessment/index.html) — Fairlearn project, living documentation (development version). Checked 2026-10-17. [^fairlearnst]: [Fairness in Machine Learning — Fairlearn user guide](https://fairlearn.org/main/user_guide/fairness_in_machine_learning.html) — Fairlearn project, living documentation. Checked 2026-10-17. [^mlflowreg]: [MLflow Model Registry](https://mlflow.org/docs/latest/ml/model-registry/) — MLflow documentation, living page (MLflow 3 docs). Checked 2026-10-17. [^mlflowclient]: [mlflow.client API reference](https://mlflow.org/docs/latest/api_reference/python_api/mlflow.client.html) — MLflow 3.17.1 documentation, living page. Checked 2026-10-18. [^mlflowauth]: [Authentication with Username and Password](https://mlflow.org/docs/latest/self-hosting/security/basic-http-auth/) — MLflow 3.17.1 documentation, living page. Checked 2026-10-18. [^mlflowrbac]: [Role-Based Access Control (RBAC)](https://mlflow.org/docs/latest/self-hosting/security/role-based-access-control/) — MLflow 3.17.1 documentation, living page. Checked 2026-10-18. [^smapprove]: [Update the Approval Status of a Model](https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry-approve.html) — Amazon SageMaker AI Developer Guide, living page. Checked 2026-10-17. [^smcards]: [Amazon SageMaker Model Cards](https://docs.aws.amazon.com/sagemaker/latest/dg/model-cards.html) — Amazon SageMaker AI Developer Guide, living page. Checked 2026-10-17. [^oaidepr]: [Deprecations](https://developers.openai.com/api/docs/deprecations) — OpenAI API documentation, living page (latest entry 2026-10-14). Checked 2026-10-17. [^anthdepr]: [Model deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations) — Anthropic, Claude Platform documentation, living page (latest entry 2026-09-30). Checked 2026-10-17. [^epic]: [External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients](https://doi.org/10.1001/jamainternmed.2021.2626) — A. Wong et al., JAMA Internal Medicine, 2021 (abstract via Europe PMC). Checked 2026-10-17. [^obermeyer]: [Dissecting racial bias in an algorithm used to manage the health of populations](https://doi.org/10.1126/science.aax2342) — Z. Obermeyer et al., Science, 2019 (abstract via Europe PMC). Checked 2026-10-17. [^zillow8k]: [Zillow Group Reports Third-Quarter 2021 Financial Results (Form 8-K, Exhibit 99.1)](https://www.sec.gov/Archives/edgar/data/1617640/000161764021000085/q32021991.htm) — Zillow Group, 2021-11-02. Checked 2026-10-17. [^zillowletter]: [Q3 2021 Shareholder Letter (Form 8-K, Exhibit 99.3)](https://www.sec.gov/Archives/edgar/data/1617640/000161764021000085/exhibit993.htm) — Zillow Group, 2021-11-02. Checked 2026-10-17. [^moffatt]: [Moffatt v. Air Canada, 2024 BCCRT 149](https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do) — British Columbia Civil Resolution Tribunal, 2024-02-14. Checked 2026-10-17. **Further reading:** Google Cloud, [Practitioners guide to MLOps](https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf) (2021), "Model governance"; the AI Act Explorer's [Digital Omnibus view](https://artificialintelligenceact.eu/ai-act-explorer/digital-omnibus/). **Changelog.** 2026-10-17, r0: first draft. 2026-10-17, r1 (fact check `reviews/modelops-r1.md`): gate reads approvals from the inventory; one provisional path; change classes added; three figures redrawn; attributions corrected. 2026-10-18, post-score polish: SR 26-2 exclusion in full; validation depth by tier; MLflow pin and grant preconditions (`reviews/modelops-polish-check.md`); tier wording; Figures 1–2 relaid.