DecisionOps

DecisionOps is the work of running automated decisions the way good teams run software. A decision here is a repeated operational choice made by a system: approve, refer or decline a refund; which driver gets which delivery; which offer a customer sees; whether a payment is blocked. Its logic usually mixes business rules, model scores and sometimes optimisation. DecisionOps gives that logic an owner, a version, tests, a controlled release, a log of every decision made, and a loop that compares decisions with their outcomes so the logic improves.

The idea in one line: the model predicts; the decision acts; DecisionOps operates the act. A fraud model that is 2% better is worth nothing if the threshold that turns its score into “block” is edited by hand in production and nobody can say which version blocked a customer last Tuesday.

Where to start. Beginners: the glossary and Lab 1. Engineers: the reference architecture and the testing ladder. Leaders: the decision framework and the judgement call. Last verified 2026-10-23.

Organisations automate thousands of small decisions a day and rarely treat them as a product. The publisher’s description of Taylor and Raden’s 2007 book on enterprise decision management says many business decisions “tend to be hidden in systems that make poor decisions, or don’t make them at all”.^[11] Hidden decisions fail in recognisable ways:

• Logic nobody can see whole. A threshold lives in a config file, an exclusion in a stored procedure, a “temporary” override in a spreadsheet. Nobody can say what the policy is, only what the code does.
• Changes nobody can test. A rules analyst edits a cut-off; it goes live at once; the effect shows up weeks later in losses or complaints. Stripe’s documentation warns that a rule “can automatically allow a large number of elevated or high-risk payments or block a large number of legitimate payments, if it isn’t set up properly”.^[14]
• Decisions nobody can explain. A customer complains; nobody can reconstruct which version, inputs and rule produced the outcome. In US credit, reasons for adverse action must be specific; “failed to achieve a qualifying score” is not enough.^[15]
• Outcomes nobody joins back. Decisions sit in one system, outcomes arrive months later in another.
• Dead logic that wakes up. In 2012 Knight Capital’s new trading code reused a flag that once switched on retired “Power Peg” code; one of eight servers missed the deployment, and the loss ultimately exceeded $460 million.^[16] ReleaseOps tells the case in full.

• Not MLOps for rules. MLOps produces reliable scores; DecisionOps decides what to do with them, together with policy rules, costs and capacity.
• Not ModelOps. Gartner’s ModelOps definition covers “decision models” including rules and optimisation.^[17] This Atlas’s editorial distinction (not an industry rule): ModelOps governs models and rule sets (inventory, tier, approval); DecisionOps runs the decision logic built from them (authoring, testing, release, logging, measurement).
• Not “decision intelligence” by another name. Gartner’s decision intelligence covers how decisions are made and “how outcomes are evaluated, managed and improved via feedback”, human or automated;^[13] DecisionOps is the slice run by software. Nor is it a product: a decision list, a rules engine or plain code, a log table and discipline are enough.

The pieces are older than the word. A Fair Isaac patent filed in 2000 describes champion/challenger testing: every account gets a two-digit “random digit”, a challenger strategy runs on a slice such as 10% of a portfolio, and a winner is rolled out “as a new champion”.^[18] Taylor and Raden argued for enterprise decision management in 2007.^[11] OMG published DMN 1.0 in September 2015 and 1.6 in September 2026.^[12] Gartner’s Critical Capabilities report for Decision Intelligence Platforms is dated 27 January 2026, as FICO reports.^[19] The label DecisionOps came from optimisation: Nextmv’s 1.0 launch (2023) asked “So what if we treated decisions as code?”^[2]

Mental model: a decision is a function with a contract, a version and a memory. The contract fixes inputs and allowed outputs (including “refer to a person”); the version says which logic ran; the memory is the decision log, later joined to the outcome. DecisionOps keeps all three true while the logic changes weekly.

Also: DataOps supplies the input data; PrivacyOps the lawful basis for using it; ComplianceOps collects audit evidence; KnowledgeOps maintains the policy documents rules come from.

flowchart TB
  IN["Inputs: MLOps scores, DataOps data, AnalyticsOps metric definitions"]
  BZ["BizOps: policy intent from leaders"]
  DO["DecisionOps: logic, tests, release, decision log, outcomes"]
  ACT["Actions in business systems"]
  GOV["ModelOps and RiskOps: approvals and limits"]
  IN --> DO
  BZ -.->|"intent"| DO
  DO --> ACT
  DO <-.->|"approval, limits"| GOV
  ACT -->|"outcomes, overrides"| DO
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class DO core
  class IN,BZ,GOV neutral
  class ACT decision

No official manifesto exists; this charter is this Atlas’s proposal.

Mission. Every automated decision has an owner, a definition anyone can read, a tested version, a record of each time it ran, and evidence that it does what its owner intends.

1. One decision, one owner, one definition. The owner is a business role accountable for outcomes, not the engineer who wrote the code.
2. Logic is versioned with its tests. Tables, thresholds and strategies live in version control with test cases, whatever runs them.
3. No silent outcomes. A DMN table with no matching rule returns null unless a default is set.^[20] Every decision ends in an allowed output, and “refer to a person” counts as an output.
4. Every decision leaves a record of inputs, version, path, output, assignment probability, reviewer and, later, outcome.
5. Change climbs a ladder whose height matches the risk: a threshold inside an approved band needs less than a new data source.
6. Human review must be real. Reviewers need time, information and authority; otherwise it is, in the Article 29 Working Party’s words, “just a token gesture”.^[25]
7. Learn from randomised comparisons, not stories. Outcomes observed only for cases the old logic accepted cannot prove a new one.^[23]
8. Explainable and contestable by design. Reason codes are written with the rule, not reverse-engineered after a complaint.

Non-goals. Approving models (ModelOps) and setting risk appetite (RiskOps).

R = responsible, A = accountable, C = consulted, I = informed.

Small organisations combine roles, but an owner should not approve their own material changes.

Governance turns on change classes (this Atlas’s proposal; align them with your ModelOps change classes):

Cadence. Weekly: alerts, referrals, override reasons. Monthly: the scorecard. Quarterly: strategy and inventory review. Redo the impact assessment when scope changes, as Canada’s directive requires.^[26]

A decision service evaluates a pinned version of the decision logic. Assignment chooses the version, the log records everything, and outcomes return later.

flowchart TB
  CALL["Calling system: checkout, claims, dispatch"]
  ASG["Assignment: champion or challenger, with probability"]
  subgraph SVC["Decision service: one pinned version"]
    direction TB
    IN["Assemble and validate inputs"]
    POL["Policy rules: eligibility, hard limits"]
    SC["Model score call"]
    STR["Strategy table: bands, reason codes"]
    OPT["Optimiser (optional)"]
  end
  REV["Review queue: people with authority"]
  LOG[("Decision log: append-only")]
  ACT["Action and notice to the customer"]
  OUT["Outcome join, days to months later"]
  REG["Version registry and tests in Git"]
  CALL --> ASG -->|"routes to one version"| SVC
  IN -->|"valid"| POL
  POL -->|"eligible"| SC --> STR
  IN -->|"invalid: refer"| REV
  STR -->|"no optimiser"| LOG
  STR -->|"allocation needed"| OPT
  STR -->|"refer band"| REV
  POL -->|"rule refuses: refer"| REV
  OPT --> LOG
  REV -->|"reviewer's decision"| LOG
  LOG -->|"write, then act"| ACT
  ACT --> OUT
  OUT -.->|"join by decision id"| LOG
  REG -.->|"release"| SVC
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class IN,POL,SC,STR,OPT core
  class CALL,ACT,OUT,REG neutral
  class ASG,REV decision
  class LOG risk

Design choices and trade-offs.

• Rules before scores, so a score can never approve what policy forbids, and the reason is a rule, not a probability.
• Stateless service. DMN decision services are stateless with no side effects;^[20] counters such as refunds_90d are inputs computed elsewhere, so a replay reproduces the decision.
• Pin the version per request, and make every replica run it (Knight’s failure above).
• No record, no action. If logging fails, refer or fail closed. Open Policy Agent, by contrast, drops decision-log events beyond a configured rate limit;^[27] know what your stack does.
• Rules engine or code? DMN gives readable tables and a standard, but engines differ: on the DMN TCK (results page updated August 2026), engines with submitted results range from 3,391 of 3,391 tests passed to 2,741, and the suite reflects DMN 1.4, so it says nothing about 1.5 or 1.6 features.^[28] Test portability yourself.

Happy path (an illustrative refund request):

1. Checkout calls refund_auto_approval with order value, customer tenure and refunds in the last 90 days.
2. Assignment hashes the customer id with the experiment salt: bucket 45, champion 1.3.0, probability 0.9.
3. Inputs validate; policy rules pass; the strategy table matches rule R2: APPROVE, reason code LOW_VALUE_LONG_TENURE.
4. The service writes the decision record, then returns APPROVE with its reason code; the refund is issued.
5. Thirty days later the outcome job marks the refund not_disputed and joins it to the record.

Exception paths:

• Input missing or out of range (tenure null): the service returns REFER with reason INPUT_INCOMPLETE and alerts if the rate rises; it never guesses.
• No rule matched: the default rule returns REFER and increments a default_hit counter that should stay at zero.
• Score service timeout: the pre-approved fallback strategy (policy rules only, wider referral band) runs, and the record says so.
• Referral: a reviewer sees inputs, reason codes and the recommendation, decides, and records an override reason if they disagree.
• Contest: support retrieves the record by decision id, gives the reason codes, and a person with authority reconsiders; what the law requires differs by jurisdiction (regulation table).

Search for decisions, not systems. For each, record owner, daily volume, outputs, effect on people, outcome window, where the logic lives and whether people review. Rank by volume × cost of a wrong decision × effect on people; start with the top three. Canada’s definition is a useful net: “Any technology that either assists or replaces the judgment of human decision makers”, including “rules-based systems”.^[26]

• Declare the hit policy. DMN’s default, Unique, forbids overlapping rules; First-hit tables “are not considered good practice because they do not offer a clear overview of the decision logic”.^[20] If order matters, say so and test it.
• Close every gap. Either make the table complete or add an explicit default that returns REFER, never null.^[20]
• Attach reason codes to rules, so every output carries a specific reason, as US Regulation B requires for adverse credit action.^[15]
• Type and bound inputs. Ranges, units and null handling belong in the decision card; validation failures become REFER.
• Keep thresholds as data with an approved band, so class-P changes do not need code changes.
• Delete retired logic. A disabled rule or reused flag is a loaded path (Knight Capital).

Each rung answers a question the previous one cannot. Nextmv separates offline tests (“run on stored data, which does not involve the production system”) from online shadow and switchback tests;^[21] Stripe Radar offers a backtest, shadow at 0% traffic, then a rising share.^[14]

flowchart TB
  T1["1. Table checks and unit tests: is the logic well formed?"]
  T2["2. Replay on stored inputs: which decisions change, and how many?"]
  T3["3. Acceptance criteria: are change rates and known outcomes inside limits?"]
  T4["4. Shadow on live inputs: does it run correctly at production volume?"]
  T5["5. Randomised share or switchback: what happens to outcomes?"]
  T6["6. Full rollout with a named rollback version"]
  T7["7. Post-release review after the outcome window"]
  STOP["Stop or redesign"]
  T1 -->|"pass"| T2 -->|"pass"| T3 -->|"pass"| T4
  T4 -->|"pass"| T5 -->|"guardrails held"| T6 --> T7
  T1 -.->|"fail"| STOP
  T3 -.->|"fail"| STOP
  T5 -.->|"guardrail breached or SRM"| STOP
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class T1,T2,T3,T4,T6,T7 core
  class T5 decision
  class STOP risk

Rungs 1–4 cannot tell you how newly approved cases behave, because the past contains outcomes only for cases the old logic accepted (selective labels). Only rung 5 measures that. For a class-P change you may stop at rung 3 if the decision-change rate is tiny; for class M, climb every rung.

Choosing rung 5. Randomise per unit (customer, account) when units do not interact: classic champion/challenger with a stable hash.^[18] When they share capacity, randomise time windows and regions. DoorDash found A/B testing “is not optimal for testing our assignment algorithm”, because giving one delivery the only available Dasher affects the other; its switchback tests randomise “on geographic region and time ‘units’ rather than users”, and analysis respects those units.^[22][29]

Before reading any result, check the split. A sample ratio mismatch, “the situation when the observed sample ratio in the experiment is different from the expected”, is a symptom of data-quality problems; Fabijan et al. warn that ignoring it “may result in a bad product modification appearing to be good”.^[30] ProductOps teaches experiment hygiene in depth; for decisions the extra work is a stable assignment and a logged propensity per decision, which Lab 3 builds before a short split check.

• Refer a band, not everything: borderline scores, missing inputs, high-impact cases; audit a random sample of the rest.
• Give reviewers time, context and authority, and capture override reasons. WP29: oversight “should be carried out by someone who has the authority and competence to change the decision”.^[25]
• Expect automation bias. A review found it “occurs in both naive and expert participants, cannot be prevented by training or instructions”.^[31] Show the recommendation only after a first look in a sample of cases; track override rates per reviewer, where near zero is a warning.
• Canada’s rule for high-impact systems is blunt: at levels III and IV, “The final decision must be made by a human.”^[26]

1. Build an immutable artefact (tables, thresholds, model versions) with its tests and replay report.
2. Release by repointing an instance (Nextmv: “a representation of a version and optional configuration”)^[32] or route; rollback is the same pointer change, recorded.
3. Check every replica reports the new version before raising traffic.
4. In emergencies, switch to a pre-approved fallback strategy; never edit rules live.
5. Mind tools whose UI edits apply at once: Stripe keeps rule history “for the past 180 days” (October 2026);^[14] export it if you need longer.

• Define the outcome and its window in the card (default within 12 months, dispute within 30 days); confirm early proxies when the real outcome arrives. Join by decision id.
• Beware selective labels. In Lakkaraju et al.'s bail example, failure to appear is observed “only if the human judge decides to release the defendant”.^[23] Loans, refunds and fraud are the same: you see outcomes only where you said yes.
• Buy information deliberately. A small randomised share, with each action’s probability logged, lets you estimate what other logic would have done; Bottou et al. stress such data “must be collected with an experiment involving active randomization”.^[33] Vowpal Wabbit’s bandit format stores exactly this: action:cost:probability.^[24]

Regulation that shapes automated decisions (October 2026; jurisdiction-specific; not legal advice):

Categories first; product facts are dated October 2026 and change often.

Select on: readable logic, replay in a test harness, a record per decision, version pinning and rollback, and export of logic and logs.

• LLMs drafting rules. Stripe’s Radar Assistant drafts a rule from a natural-language request, then “Test rule” runs it against history (October 2026; Stripe may use the chat entries for training).^[14] Treat generated logic as a class-L change.
• LLMs or agents inside a decision. Same contract: allowed outputs, reason codes, a REFER exit, full logging, limits enforced by rules after the model. Evaluations belong to LLMOps.
• Automating the operation (mix alerts, fallback on timeout) is cheap and useful; promoting challengers with no person reading the result is not, for decisions with significant effects.

Illustrative scenario: a retailer automates refund approvals: about 4,000 requests a day, outputs APPROVE or REFER (people decide every refusal), outcome = dispute or abuse within 30 days. The minimum artefacts:

1. The decision card (the contract). Example status: YAML syntax-checked by the validator; not executed.

decision: refund_auto_approval
owner: head-of-customer-care          # a business role, not a person's name
purpose: Approve low-risk refunds instantly; refer the rest to agents
outputs: [APPROVE, REFER]             # no automated refusals in this design
inputs:
  order_value: {type: number, unit: EUR, min: 0}
  tenure_months: {type: integer, min: 0}
  refunds_90d: {type: integer, min: 0}
on_invalid_input: REFER               # reason INPUT_INCOMPLETE
on_no_rule_match: REFER               # reason DEFAULT_RULE; alert if > 0
reason_codes: [LOW_VALUE, LOW_VALUE_LONG_TENURE, NEW_CUSTOMER,
               RECENT_REFUNDS, HIGH_VALUE, INPUT_INCOMPLETE, DEFAULT_RULE]
approved_bands:
  approve_max_order_value: {min: 400, max: 600}  # live value 500; class P inside
service_levels: {p99_latency_ms: 150, log_completeness: 1.0}
outcome: {event: refund_disputed_or_abusive, window_days: 30}
experiments: {unit: customer_id, max_challenger_share: 0.10}
human_review: {queue: refunds-l2, target_handling_minutes: 6}
effects_on_people: financial, reversible; customer may contest via support

2. One decision record. Example status: JSON syntax-checked; not executed. Fields mirror the OPA decision-log idea (decision id, input, result, version)^[27] plus what experiments and audits need. recommendation is what the logic suggests (for a referral, what the reviewer is shown); final_output is what was done, set by the reviewer when there is one.

{
  "decision_id": "d-2026-10-23-000481",
  "decision": "refund_auto_approval",
  "version": "1.3.0",
  "arm": "champion",
  "propensity": 0.9,
  "timestamp": "2026-10-23T09:14:07Z",
  "inputs": {"order_value": 120, "tenure_months": 30, "refunds_90d": 1},
  "path": ["policy:eligible", "table:R2"],
  "recommendation": "APPROVE",
  "output": "APPROVE",
  "reason_codes": ["LOW_VALUE_LONG_TENURE"],
  "fallback_used": false,
  "reviewer": null,
  "final_output": "APPROVE",
  "outcome": null
}

3. A release manifest. Example status: YAML syntax-checked; not executed.

release: refund_auto_approval 1.4.0
change_class: L
summary: >-
  Raise long-tenure approve limit from 500 to 600 EUR (inside the approved band,
  class P on its own); new rule R6 makes the release class L
evidence:
  table_checks: pass            # no overlaps, no gaps, 14/14 tests
  replay: {inputs: 90 days, decision_change_rate: 0.031, report: replay-1.4.0.html}
  shadow: {days: 7, errors: 0, p99_latency_ms: 41}
rollout:
  challenger_share: 0.10
  min_days: 21
  guardrail: dispute_rate <= champion + 0.5pp
srm_gate: p >= 0.001 before reading outcomes
rollback_to: 1.3.0
approvals: {owner: head-of-customer-care, compliance: not required (no refusals)}

4. Outcome join and scorecard query. It assumes the log is loaded into a decisions table whose columns are the record’s top-level fields, and an outcomes table (decision_id, disputed). The override rate divides by reviewed cases only, as the KPI table defines it. Example status: executed on 2026-10-23 in SQLite 3.31 on 100 synthetic rows (20 reviewed, 2 overridden, 5 disputed): override_rate 0.1, referral_rate 0.2, dispute_rate 0.05 as computed by hand; not run on real data.

SELECT d.version, d.arm,
       COUNT(*) AS decisions,
       AVG(CASE WHEN d.output = 'REFER' THEN 1.0 ELSE 0 END) AS referral_rate,
       SUM(CASE WHEN d.reviewer IS NOT NULL
                 AND d.final_output <> d.recommendation THEN 1.0 ELSE 0 END)
         / NULLIF(SUM(CASE WHEN d.reviewer IS NOT NULL THEN 1 ELSE 0 END), 0)
         AS override_rate,
       AVG(CASE WHEN o.disputed = 1 THEN 1.0 ELSE 0 END) AS dispute_rate
FROM decisions d
LEFT JOIN outcomes o ON o.decision_id = d.decision_id
WHERE d.decision = 'refund_auto_approval'
  AND d.timestamp < DATE('now', '-30 days')    -- outcome window has closed
GROUP BY d.version, d.arm;

Three labs, standard library Python 3.8 or later, no network, no cost. All three were executed on 2026-10-23 (Python 3.8.10); the outputs shown are copied from those runs. Teardown for every lab: delete the working folder.

Objective. See why hit policies matter by checking a Unique table automatically. Prerequisites. Python 3.8+; an empty folder. Cost and risk. None. Save the table as refund_table.json. It contains two planted defects that mirror real policy drafting: R1 and R4 disagree about customers with exactly two recent refunds, and R5 was typed as “over 1,000” while the other rules stop at 500. Example status: executed.

{
  "decision": "refund_auto_approval",
  "version": "1.3.0-draft",
  "hit_policy": "UNIQUE",
  "inputs": ["order_value", "tenure_months", "refunds_90d"],
  "output": "action",
  "rules": [
    {"id": "R1", "order_value": [0, 50], "tenure_months": null,
     "refunds_90d": [0, 3], "action": "APPROVE"},
    {"id": "R2", "order_value": [50, 500], "tenure_months": [12, null],
     "refunds_90d": [0, 2], "action": "APPROVE"},
    {"id": "R3", "order_value": [50, 500], "tenure_months": [0, 12],
     "refunds_90d": [0, 2], "action": "REFER"},
    {"id": "R4", "order_value": [0, 500], "tenure_months": null,
     "refunds_90d": [2, null], "action": "REFER"},
    {"id": "R5", "order_value": [1000, null], "tenure_months": null,
     "refunds_90d": null, "action": "REFER"}
  ],
  "tests": [
    {"in": {"order_value": 20, "tenure_months": 1, "refunds_90d": 0},
     "expect": "APPROVE"},
    {"in": {"order_value": 120, "tenure_months": 30, "refunds_90d": 1},
     "expect": "APPROVE"},
    {"in": {"order_value": 120, "tenure_months": 3, "refunds_90d": 0},
     "expect": "REFER"},
    {"in": {"order_value": 900, "tenure_months": 60, "refunds_90d": 0},
     "expect": "REFER"}
  ]
}

Save the checker as check_table.py (intervals are [low, high); null means unbounded or “any”):

"""Check a DMN-style decision table (hit policy UNIQUE).

Reports overlapping rules, gaps (no rule matches) and failing test cases.

Usage: python3 check_table.py refund_table.json
Intervals are [low, high): low inclusive, high exclusive; null means unbounded or "any".
"""
import itertools
import json
import sys


def matches(cond, value):
    if cond is None:
        return True
    low, high = cond
    return (low is None or value >= low) and (high is None or value < high)


def probe_values(rules, name):
    """Each bound plus the values 0.5 below and above it."""
    bounds = {0}
    for rule in rules:
        cond = rule[name]
        if cond is not None:
            bounds.update(b for b in cond if b is not None)
    values = set()
    for b in bounds:
        values.update({b - 0.5, b, b + 0.5})
    return sorted(v for v in values if v >= 0)


def evaluate(table, case):
    return [r for r in table["rules"]
            if all(matches(r[name], case[name]) for name in table["inputs"])]


def main(path):
    table = json.load(open(path))
    names = table["inputs"]
    grids = [probe_values(table["rules"], n) for n in names]
    overlaps, gaps, seen = [], [], set()
    for combo in itertools.product(*grids):
        case = dict(zip(names, combo))
        hits = evaluate(table, case)
        if len(hits) > 1:
            key = tuple(r["id"] for r in hits)
            if key not in seen:
                seen.add(key)
                overlaps.append((key, case))
        elif not hits and len(gaps) < 3:
            gaps.append(case)
    print(f"{table['decision']} v{table['version']} hit policy {table['hit_policy']}")
    print(f"probe points: {len(list(itertools.product(*grids)))}")
    for key, case in overlaps:
        print(f"OVERLAP {'+'.join(key)} at {case}")
    for case in gaps:
        print(f"GAP (no rule, result null) at {case}")
    failed = 0
    for t in table["tests"]:
        hits = evaluate(table, t["in"])
        got = hits[0]["action"] if len(hits) == 1 else f"{len(hits)} hits"
        if got != t["expect"]:
            failed += 1
            print(f"TEST FAIL {t['in']}")
            print(f"  expected {t['expect']} got {got}")
    print(f"tests: {len(table['tests']) - failed}/{len(table['tests'])} passed")
    ok = not overlaps and not gaps and not failed
    print("RESULT", "PASS" if ok else "FAIL")
    return 0 if ok else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Run python3 check_table.py refund_table.json. Expected output (exit code 1):

refund_auto_approval v1.3.0-draft hit policy UNIQUE
probe points: 385
OVERLAP R1+R4 at {'order_value': 0, 'tenure_months': 0, 'refunds_90d': 2}
GAP (no rule, result null) at {'order_value': 500, 'tenure_months': 0, 'refunds_90d': 0}
GAP (no rule, result null) at {'order_value': 500, 'tenure_months': 0, 'refunds_90d': 0.5}
GAP (no rule, result null) at {'order_value': 500, 'tenure_months': 0, 'refunds_90d': 1.5}
TEST FAIL {'order_value': 900, 'tenure_months': 60, 'refunds_90d': 0}
  expected REFER got 0 hits
tests: 3/4 passed
RESULT FAIL

Validation. Change R1’s refunds_90d to [0, 2] and R5’s order_value to [500, null]: RESULT PASS, exit code 0 (verified). Troubleshooting. KeyError: a rule lacks a declared input; for non-integer inputs add finer probes than ± 0.5. Lesson. Three of four example tests passed, yet the table was wrong twice: systematic checks find what examples miss.

Objective. Compare a champion and a challenger credit-style strategy on stored inputs and read the decision-change matrix. Prerequisites. Lab 1 folder. Cost and risk. None; data are synthetic, generated with a fixed seed. Save as replay.py. Example status: executed.

"""Replay stored inputs through the champion and a challenger; compare decisions.

Usage: python3 replay.py
Synthetic data (fixed seed). Outcomes exist only where the champion approved
(selective labels).
"""
import collections
import random

ACTIONS = ("APPROVE", "REFER", "DECLINE")


def champion(app):  # strategy v4.2.0
    if app["pd"] > 0.20 or (app["amount"] > 5000 and not app["verified"]):
        return "DECLINE"
    if app["pd"] > 0.12:
        return "REFER"
    return "APPROVE"


def challenger(app):  # strategy v4.3.0-rc1: wider approve band, higher amount limit
    if app["pd"] > 0.22 or (app["amount"] > 8000 and not app["verified"]):
        return "DECLINE"
    if app["pd"] > 0.14:
        return "REFER"
    return "APPROVE"


def make_input_set(n=2000, seed=7):
    rng = random.Random(seed)
    apps = []
    for i in range(n):
        pd = min(0.6, rng.betavariate(2, 14))
        app = {"id": f"A{i:05d}", "pd": round(pd, 3),
               "amount": rng.choice([500, 1500, 3000, 6000, 9000]),
               "verified": rng.random() < 0.7}
        # Outcome is observed only if the champion approved the application in production.
        if champion(app) == "APPROVE":
            app["defaulted"] = rng.random() < app["pd"]
        apps.append(app)
    return apps


def main():
    apps = make_input_set()
    matrix = collections.Counter((champion(a), challenger(a)) for a in apps)
    print("champion -> challenger    count")
    for c in ACTIONS:
        for d in ACTIONS:
            if matrix[(c, d)]:
                flag = "" if c == d else "  <- changed"
                print(f"{c:>8} -> {d:<10} {matrix[(c, d)]:>6}{flag}")
    changed = sum(v for (c, d), v in matrix.items() if c != d)
    print(f"decision-change rate: {changed / len(apps):.1%} of {len(apps)}")
    swap_in = [a for a in apps if champion(a) != "APPROVE" and challenger(a) == "APPROVE"]
    swap_out = [a for a in apps
                if champion(a) == "APPROVE" and challenger(a) != "APPROVE"]
    labelled = [a for a in swap_in if "defaulted" in a]
    print(f"swap-ins (newly approved): {len(swap_in)}, "
          f"with observed outcome: {len(labelled)}")
    print(f"swap-outs (no longer approved): {len(swap_out)}")
    kept = [a for a in apps if champion(a) == "APPROVE" and challenger(a) == "APPROVE"]
    rate = sum(a["defaulted"] for a in kept) / len(kept)
    print(f"default rate where both approve (observed): {rate:.1%} of {len(kept)}")
    print("swap-in default rate: UNKNOWN from history -> needs a controlled test")


if __name__ == "__main__":
    main()

Expected output of python3 replay.py:

champion -> challenger    count
 APPROVE -> APPROVE      1006
   REFER -> APPROVE       133  <- changed
   REFER -> REFER         341
 DECLINE -> APPROVE        72  <- changed
 DECLINE -> REFER          89  <- changed
 DECLINE -> DECLINE       359
decision-change rate: 14.7% of 2000
swap-ins (newly approved): 205, with observed outcome: 0
swap-outs (no longer approved): 0
default rate where both approve (observed): 5.3% of 1006
swap-in default rate: UNKNOWN from history -> needs a controlled test

Validation. Explain the 72 DECLINE → APPROVE cases (unverified requests for 6,000, now under the 8,000 limit). Lesson. The 205 swap-ins are the challenger’s whole point, and history holds no outcome for any of them: replay sizes the change, only a randomised test prices it. Troubleshooting. Other numbers mean a different random implementation; the structure is what matters.

Objective. Assign units deterministically and log, for every decision, the version and the probability that chose it: the raw material for fair comparisons and counterfactual estimates. A short split check at the end stops a broken experiment from being read (sizing and SRM practice in depth: ProductOps). Prerequisites. Python 3.8+. Cost and risk. None; writes decisions.jsonl (about 8 MB) in the current folder. Save as experiment.py. Example status: executed.

"""Champion/challenger assignment, a decision log with propensities, an SRM check.

Usage: python3 experiment.py [--drop-rate 0.08]
Writes decisions.jsonl in the current directory. Standard library only.
"""
import argparse
import hashlib
import json
import math
import random

SALT = "refund-exp-2026-10"     # new salt per experiment, never reused
CHALLENGER_BUCKETS = range(0, 10)  # buckets 0-9 = 10 % challenger
VERSIONS = {"champion": "1.3.0", "challenger": "1.4.0-rc1"}


def bucket(unit_id):
    digest = hashlib.sha256(f"{SALT}:{unit_id}".encode()).hexdigest()
    return int(digest[:8], 16) % 100


def assign(unit_id):
    b = bucket(unit_id)
    arm = "challenger" if b in CHALLENGER_BUCKETS else "champion"
    propensity = len(CHALLENGER_BUCKETS) / 100 if arm == "challenger" else 0.9
    return arm, b, propensity


def srm_p_value(observed, expected_share):
    """Chi-square goodness of fit, 1 degree of freedom."""
    total = sum(observed)
    expected = [total * s for s in expected_share]
    chi2 = sum((o - e) ** 2 / e for o, e in zip(observed, expected))
    return chi2, math.erfc(math.sqrt(chi2 / 2))


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--drop-rate", type=float, default=0.0,
                        help="simulate a logging bug that loses challenger events")
    args = parser.parse_args()
    rng = random.Random(11)
    counts = {"champion": 0, "challenger": 0}
    with open("decisions.jsonl", "w") as log:
        for i in range(50000):
            unit = f"cust-{i:06d}"
            arm, b, p = assign(unit)
            if arm == "challenger" and rng.random() < args.drop_rate:
                continue  # the bug: event never written
            event = {"decision_id": f"d-{i:06d}", "decision": "refund_auto_approval",
                     "version": VERSIONS[arm], "arm": arm, "bucket": b,
                     "propensity": p, "unit": unit}
            log.write(json.dumps(event) + "\n")
            counts[arm] += 1
    chi2, p_value = srm_p_value([counts["champion"], counts["challenger"]], [0.9, 0.1])
    print(f"logged champion={counts['champion']} challenger={counts['challenger']}")
    print(f"expected share 90/10, chi2={chi2:.1f}, p={p_value:.2g}")
    if p_value < 0.001:
        print("SRM DETECTED: do not read outcome metrics; find the cause first")
    else:
        print("no SRM: outcome comparison may proceed")


if __name__ == "__main__":
    main()

Expected output. python3 experiment.py prints logged champion=45063 challenger=4937, chi2=0.9, p=0.35 and “no SRM”. python3 experiment.py --drop-rate 0.08 (a bug that loses 8% of challenger records) prints logged champion=45063 challenger=4524, chi2=42.3, p=7.7e-11 and “SRM DETECTED”. Validation. Run the first command twice: the same customers land in the same arms, because assignment depends only on the salt and the id. Troubleshooting. If the split drifts with no drop rate, check that the salt and bucket ranges match the experiment plan. Extension. Give 5% of champion-arm REFER cases an APPROVE at random, log propensity for that action, and you have the raw material for counterfactual estimates.^[33]

The Royal Commission’s recommendations 17.1 and 17.2 read like DecisionOps controls: a clear path to review, plain-language explanations, “business rules and algorithms should be made available, to enable independent expert scrutiny”, and a body to monitor and audit automated decision-making.^[48]

Measure each decision, per version and arm. Outcome metrics come from the business (AnalyticsOps owns their definitions); the rest are operational.

Mix, referrals, default hits and log completeness are daily leading signals; outcomes lag by their window. Set targets from the owner’s risk appetite, not another company’s numbers.

• Integrity of logic. Who can change rules is a security question: reviewed releases only, a protected registry, alerts on changes outside the pipeline. Canada’s directive asks departments to “secure data and model integrity to prevent tampering and unauthorized modifications”.^[26] Treat allow rules as privileged: in Stripe Radar they “override all your other rules”.^[14]
• Decision logs hold personal data. Omit or mask what is not needed to explain or reproduce (OPA can erase or modify fields before upload),^[27] restrict access and set retention with PrivacyOps.
• Fairness and versions. Canada requires testing for “unintended biases” before production, scheduled outcome monitoring, and “Obtaining and safeguarding all released versions of software components”.^[26]
• Regulatory mapping sits with ModelOps, evidence with ComplianceOps.

The main costs are people and learning. Review labour is referrals × handling time (960 hours above). Learning cost is the extra loss while a weaker challenger runs; cap it with share, guardrails and a stop rule. Solver compute: HiGHS stops a mixed-integer solve at a relative gap of 0.0001 by default, with no time limit unless set;^[44] set limits and log the gap reached. Energy matters only for large solves (GreenOps).

• Monthly: the owner reads the scorecard; a sample of logged decisions is replayed with their logged versions (re-performance; below 100% is a defect).
• Quarterly: reconcile the inventory; review override reasons (recurring reasons are requirements).
• Independent review for high-impact decisions; Canada scales peer review by impact level.^[26]

Illustrative incident. At 09:30 on a Tuesday the refund approval rate drops from 71% to 52%; referrals triple; nobody announced a change.

%%{init: {"flowchart": {"wrappingWidth": 140, "nodeSpacing": 25, "rankSpacing": 40}}}%%
flowchart TB
  A["Alert: approvals 52%, normal 71%"]
  B{"One version or host only?"}
  C["Pin all replicas to last-known-good"]
  D{"Any change in 48 hours?"}
  E["Redeploy or roll back the released version"]
  F{"Inputs shifted?"}
  G["Broken feed: policy-only fallback"]
  H["Genuine shift: owner decides"]
  I["Re-decide affected cases; record the incident"]
  A --> B
  B -->|"yes"| C
  B -->|"no"| D
  D -->|"yes"| E
  D -->|"no"| F
  F -->|"broken feed"| G
  F -->|"genuine"| H
  C --> I
  E --> I
  G --> I
  H --> I
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class A risk
  class B,D,F decision
  class C,E,G,H,I core

1. Confirm from the decision log: approval rate by version, host and hour.
2. Isolate: all hosts report 1.4.0, released last week, but the rule tool’s history shows a threshold edited at 09:12 by a user with direct access, so production no longer matches 1.4.0.
3. Contain: redeploy 1.4.0 from the registry, overwriting the edit; approvals return to 70%.
4. Recover: about 1,900 requests were referred instead of approved; nobody was refused (refusals are human here). Clear the queue oldest first.
5. Prevent: remove direct edit rights; alert on changes made outside the pipeline; record the incident with decision ids.

• “The model is the decision.” A better AUC changes nothing until thresholds change, and may break them.
• “Replay proved the challenger.” Replay cannot observe outcomes for newly approved cases.^[23]
• “A person clicks approve, so it is not automated.” Fabricated involvement does not escape GDPR Article 22.^[25]
• Logging only the output, optimising a proxy (approval rate, review cost) and disabling instead of deleting retired logic.

Use DecisionOps when a decision is repeated at volume, its logic changes more than a few times a year, a wrong decision costs money or harms people, outcomes can be observed, or regulators and customers ask “why?”.

Do not, or not yet:

• One-off or strategic decisions: BizOps decision records.
• Low volume: a written policy and human judgement are cheaper.
• No observable outcome: you can test correctness but not learn; use decision support.
• No lawful basis or no way to contest, or no specific reasons: automate the preparation, not the decision.
• Unstable policy: agree the policy first.

Hidden responsibilities: a review team, an outcome pipeline, log retention, change governance and on-call.

flowchart TB
  Q1{"Repeated, with stable policy and measurable outcomes?"}
  N1["Keep human judgement; use decision support"]
  Q2{"Significant effects on people?"}
  H["Design real human review and contest; check the law"]
  Q3{"Do units interact through shared capacity?"}
  SB["Switchback test over time and region"]
  Q4{"May you randomise who gets which version?"}
  AB["Champion/challenger per unit, propensity logged"]
  SH["Replay, shadow, staged rollout; state the blind spot"]
  Q1 -->|"no"| N1
  Q1 -->|"yes"| Q2
  Q2 -->|"yes"| H --> Q3
  Q2 -->|"no"| Q3
  Q3 -->|"yes"| SB
  Q3 -->|"no"| Q4
  Q4 -->|"yes"| AB
  Q4 -->|"no"| SH
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class Q1,Q2,Q3,Q4 decision
  class SB,AB,SH,H core
  class N1 risk

Weighted check for the level of automation (score each 1–5; weights are this Atlas’s suggestion):

Scores above about 60 of 75 suggest full automation with sampling; 40–60, automation with a referral band; below 40, decision support only. Treat the cut-offs as a starting point for discussion, not a rule.

This is this Atlas’s model, not an industry standard.

Before releasing a decision change

• Change class set; approver named
• Table checks pass: no overlaps under Unique, no gaps or an explicit REFER default
• Every rule has a reason code; test cases cover each rule and each boundary
• Replay report reviewed: decision-change matrix, swap-ins and swap-outs explained
• Shadow run clean at production volume; latency within service level
• Experiment plan: unit, share, guardrails, stop rule, SRM gate, outcome window
• Rollback version named and deployable; fallback strategy tested

Operating readiness

• Decision card current; owner and on-call named
• Every decision logged with version, inputs, path, output, propensity
• Mix, default-hit, latency and log-completeness alerts live
• Review queue staffed for the expected referral volume; override reasons captured

Quarterly governance audit

• Inventory reconciled with code and tools
• Re-performance sample at 100%
• Override and contest reasons reviewed; changes raised
• Retired rules deleted; no direct edit rights outside the pipeline
• Legal and impact assessments still match the decision’s scope

Decision change request (copy into your tracker):

Decision:             refund_auto_approval
Change class:         P | L | M | E
Requested by / owner: <role> / <role>
What changes:         <rules, thresholds, inputs, models referenced>
Why:                  <problem, evidence, expected effect on outcome metric>
Replay result:        <decision-change rate; swap-ins; swap-outs; report link>
Effect on people:     <who gets a different outcome; can they contest?>
Test plan:            <shadow days; share; guardrails; stop rule; outcome window>
Rollback:             <version>; fallback strategy <name>
Approvals:            owner <date>; compliance <date or n/a>; ModelOps <n/a>

Reviewer guidance card (one page per referral reason):

Referral reason:  RECENT_REFUNDS
What the system saw: refunds_90d >= 2, order value, tenure, reason codes
What you can check that it could not: order notes, carrier proof, contact history
You may: approve, request evidence, decline (give the specific reason)
Record:   final outcome + override reason from list + free text if "other"
Time:     target 6 minutes; escalate rather than guess

The decision card and the release manifest above complete the set.

Kinds of decision logic

Is DecisionOps a job title or a certification? We found neither (October 2026); the label comes from vendors.^[2][6]

Do we need DMN or a rules engine? No. DMN helps when business owners must read and change tables and you want a standard;^[20] plain code with the same tests, versions and logs is DecisionOps too.

Our decisions only recommend; a person decides. Does this apply? Yes: the recommendation still shapes outcomes, and automation bias means people often follow it.^[31] In the EU, a third party’s score can count as the decision when it plays a determining role.^[35]

How big should a challenger share be? Large enough to detect the effect you care about within the window, small enough that a bad challenger’s losses are affordable; the Fair Isaac patent’s 10% is an example, not a rule.^[18]

Can we skip randomised tests if replay looks good? Only if the change cannot approve anything the old logic declined. Otherwise replay is blind exactly where it matters (Lab 2).

How long should we keep decision logs? As long as decisions can be contested or audited, no longer than privacy law allows.

• Counterfactual evaluation and bandits: with logged propensities, estimate logic you did not run;^[33] common in advertising research, rarer elsewhere.
• Simulation (FICO markets “business outcome simulations”)^[6] screens changes; validate with live tests.
• Standards drift: DMN 1.7 is in beta;^[12] check the TCK before relying on new features.^[28]

Situation (illustrative). A lender automates small personal-loan decisions: 3,000 automated denials a day. Compliance proposes that a person reviews every denial (A). The operations lead proposes reviewing a borderline band plus a 5% random sample, with a fast reconsideration route for anyone who asks (B). Assumptions: 12 reviewers, 7 productive hours a day each; EU consumer credit; denials have significant effects; the lender must decide which Article 22 exception applies, and from 20 November 2026 the Consumer Credit Directive lets every applicant assessed with automated processing request human intervention and a review, and rejections must say so.^[36] Transposition and scope are counsel’s job (not legal advice).

Reasoning. (1) Capacity: A gives each denial 12 × 420 minutes ÷ 3,000 ≈ 1.7 minutes. Under that pressure, with the recommendation visible, automation bias predicts agreement;^[31] WP29 requires involvement that is “meaningful, rather than just a token gesture”.^[25] A buys the appearance of human decisions and may mislead the legal analysis. (2) B’s risk: clear denials are automated, so the lender needs a lawful exception and safeguards, including human intervention and contest.^[34] (3) Quality: assume the borderline band holds about 150 denials a day and about 30 applicants a day ask for reconsideration; with the 150 sampled cases that is 330 cases, so B’s 5,040 reviewer-minutes give each about 15 minutes, where judgement changes outcomes, and the random sample measures how often automated denials would be overturned. Under the directive, both options need the request-and-review route; A simply has no capacity left to run it. (4) What would change the answer: if counsel finds no exception, every denial needs a genuinely human decision: more reviewers or fewer automated denials, not faster clicking.

Decision: B, with the override and reconsideration rates published monthly to the owner and compliance. Uncertainty: the borderline band’s width is a guess until the first month’s sample shows where reviewers disagree, and reconsideration requests may rise once rejection notices mention the right to human assessment; size the team from the first months’ data.

After this page you can explain the term’s status and boundaries; write a checkable decision table; choose a testing rung; design human review; read a replay and an experiment; and run a decision incident.

DecisionOps is a young vendor word for an old, under-practised discipline. Its value is not a platform; it is a shift of attention from the model to the act: what the system decided, under which version, for which reason, with what result. Teams that treat each automated decision as a product, with a contract, tests, a ladder of evidence and a memory, change policy quickly and can answer the customer, the auditor and themselves. Teams that do not will keep discovering their policies in incident reviews, or, as Robodebt showed, in a royal commission.^[48] Facts are dated October 2026; re-check before relying on them.

Vendor and documentation pages carry no publication date unless stated. Ledger with claim-by-claim notes: research/decisionops-ledger.md.

Further reading

• DMN 1.6 specification, clause 11 (worked examples, including a lending decision).
• Directive on Automated Decision-Making, Appendix C, as a control checklist by impact level.

Changelog. 2026-10-23: first version (research, ledger, draft, diagrams, three executed labs).

Changelog. 2026-10-23 (r1 revision, after the independent fact check): fixed the scorecard SQL (override rate over reviewed cases; record gains recommendation and final_output; re-run on synthetic rows); added the EU Consumer Credit Directive 2023/2225 and the UK Arts 22A–22D commencement; made the worked example consistent; redrew the reference architecture (write, then act; referral exits; optional optimiser); dated term-status evidence (Nextmv and FICO posts, EURO Practitioners’ Forum); corrected the TCK range, Taylor–Raden attribution, Knight and Robodebt wording; Stripe plan limits; ProductOps link; derived the judgement call’s capacity numbers.

Changelog. 2026-10-23, post-score polish: FICO definition traced to Nextmv (2024); Gartner date narrowed; figure 2 re-routed; experimentation platforms and a Robodebt measure; trims.