---
title: "WorkflowOps"
slug: workflowops
category: delivery
term_status: ambiguous
evidence_status: verified
summary: "WorkflowOps is not a standard term. This Atlas uses it for running workflow engines well: keeping long-running, multi-step processes (DAGs, state machines, durable executions) correct through retries, crashes, partial failure and code changes."
audience: [Backend engineers, Platform engineers, Data engineers, Architects, SREs, Beginners]
last_verified: 2026-10-23
key_takeaways:
  - "\"WorkflowOps\" has no agreed definition: the name belongs to an Oracle WebLogic Portal interface and to a business-approvals product. This page uses it for workflow orchestration and durable execution, an editorial choice."
  - "A workflow engine records progress so a crashed process resumes instead of restarting. It does not make your side effects safe: every step can run more than once, so each external call needs an idempotency key."
  - "Retries are configured, not free. Temporal retries Activities without limit by default, a Step Functions Retry block three times; retries at several layers multiply load (AWS shows 243 times for five layers of three tries)."
  - "When a step cannot be undone, model the process as a saga: compensable steps, one pivot, retryable steps after it, and a human queue. A pivot with an unknown outcome is settled by a status query, not by compensation."
  - "Running instances outlive deployments. Replay-based engines (Temporal, Durable Functions, Lambda durable functions) need patching or version pinning; Step Functions keeps old executions on the old definition."
  - "Measure the workflow, not the worker: completion within deadline, stuck-instance age, compensation rate, retries per completed run and cost per completed run."
  - "A cron job plus an idempotent script is often enough. Adopt an engine when processes span minutes to months, cross several systems or wait for people."
related: [dataops, integrationops, automationops, serverlessops, observabilityops, mlops, sre, devops]
---

## What WorkflowOps is

### In 30 seconds

**WorkflowOps**, as this Atlas uses the word, is the practice of designing, running and changing **workflow engines** and the
long-running processes they execute, so that those processes **finish correctly even when steps fail, workers crash, people take days to
approve, and the code changes while thousands of instances are still running**.

A workflow is a multi-step process with state: *reserve stock, charge the card, book a courier, wait for dispatch*. A workflow
engine (Temporal, AWS Step Functions, Apache Airflow, Argo Workflows, Azure Durable Functions and others) stores each instance's
progress, retries steps, fires timers and resumes after a crash. WorkflowOps is everything around the engine that decides whether the
business outcome is right: retry and timeout policy, idempotency, compensation, versioning, backfills, observability and runbooks.

**For** backend and platform engineers who own an engine, data engineers whose schedulers behave like one, architects choosing
between a queue, a scheduler and a durable engine, and SREs who are paged when instances pile up.

::: note The label is ambiguous — read this first
Searches on 2026-10-23 found no standard, foundation or vendor that defines "WorkflowOps" as an engineering discipline. The name is
used for an Oracle WebLogic Portal Java interface that manages content workflows from DRAFT to PUBLISHED[^oracle] and for a commercial
approvals platform.[^wfopspl] The established names for this practice are **workflow orchestration** and **durable execution**; reading
"WorkflowOps" as the operation of workflow engines is this Atlas's **editorial emphasis**.
:::

### At a glance

| Item | Summary |
|---|---|
| Term status | **Ambiguous.** The name has unrelated uses (a Java interface, a product); the techniques (sagas, durable execution, retries) are well documented. |
| Goal | Every workflow instance reaches a correct end state (completed, compensated, or parked with a named owner) within its deadline. |
| In scope | DAGs and state machines; durable execution and replay; retries, backoff, timeouts; idempotency; sagas and compensation; timers and human waits; versioning running workflows; backfills; history limits; observing runs. |
| Out of scope | Pipeline content and data quality → [DataOps](../dataops/); brokers and message flows → [IntegrationOps](../integrationops/); runbook automation and its safety → [AutomationOps](../automationops/); single-function event semantics → [ServerlessOps](../serverlessops/); process modelling with BPMN. |
| Prerequisites | Delivery basics ([DevOps](../devops/)); what a transaction and a timeout are; some Python for the labs. |
| Key outputs | A design record per workflow (steps, keys, timeouts, retries, compensation); a versioning procedure; stuck-instance dashboards; runbooks. |

### The problem WorkflowOps solves

A process that spans several systems and minutes to months cannot be one database transaction. The 1987 paper that named the
**saga** described the problem precisely: long lived transactions "hold on to database resources for relatively long periods of time",
so the authors proposed writing such a transaction as "a sequence of transactions that can be interleaved with other transactions",
where the system guarantees "either all the transactions in a saga are successfully completed or compensating transactions are run to
amend a partial execution".[^sagas] Almost every problem on this page is a modern version of that sentence.

Without an engine and the practices around it, teams hand-build status columns, cron sweepers for "stuck orders" and fixed-sleep
retry loops. The failures are predictable:

- **Double effects.** A worker charges a card, crashes before recording success, and the retry charges again.
- **Half-finished processes.** Stock is reserved and payment taken, but shipping failed and nothing undoes the first two steps.
- **Retry storms.** Every layer retries, so a struggling dependency receives many times its normal load.
- **Deploys that break running work.** New code assumes a step that instances started last week never ran.
- **Silent stalls.** An instance waits forever for a callback, and no metric counts it.

An engine solves the bookkeeping (durable state, timers, retries, history), not correctness. That remainder is WorkflowOps.

### What WorkflowOps is not

- **Not BPMN modelling.** OMG's BPMN 2.0.2 is "the de-facto standard for business processes diagrams";[^bpmn] modelling a process
  with stakeholders is a different job. Engines that *execute* BPMN (Camunda 8, Flowable) are in scope as engines: this page starts
  when a process runs.
- **Not data pipeline quality** ([DataOps](../dataops/)): Airflow appears here only as an engine.
- **Not messaging** ([IntegrationOps](../integrationops/)): a workflow often *uses* a queue; the queue is not the workflow.
- **Not runbook automation** ([AutomationOps](../automationops/)): that page owns scripts and their guards; this one owns the engine
  that may run them as steps.

## Where it comes from and where it sits

### Origins and evolution

There is no history of "WorkflowOps" as a movement, only of the techniques:

| Period | Development | Why it matters here |
|---|---|---|
| 1987 | Garcia-Molina and Salem publish *Sagas*: long transactions as sequences of local transactions with compensations[^sagas] | The compensation model used here |
| 1999 onward | The Workflow Patterns initiative (Eindhoven University of Technology and Queensland University of Technology) catalogues control-flow, data, resource and exception-handling patterns[^wfpatterns] | Vendor-neutral vocabulary for comparing engines |
| 2000s–2010s | Temporal's founders worked on Amazon Simple Workflow Service and Azure's Durable Task Framework, then in 2015 co-created Cadence at Uber, Temporal's predecessor by its own account[^temporal-about] | Code-defined workflows with replay |
| 2020 | The Open Workflow Specification (hosted at serverlessworkflow.io) approved as a CNCF Sandbox project; Argo accepted to CNCF the same year and graduated in 2022[^openwf][^cncf-argo] | Open definitions and a graduated Kubernetes-native engine |
| 2025 | AWS launches Lambda durable functions at re:Invent 2025: multi-step workflows for up to one year using checkpoint and replay[^l-blog][^l-durable] | The technique moves into mainstream serverless runtimes |

### Glossary and mental model

| Term | Meaning on this page | Example |
|---|---|---|
| Workflow definition / instance | The code or document describing the steps / one run of it, with its own state and history | A Temporal workflow class / order `A1`'s fulfilment |
| Step / activity / task | A unit of work that touches the outside world | "charge card", "run container" |
| DAG / state machine | Tasks in a directed acyclic graph, usually per data interval / explicit states and error transitions | An Airflow Dag / Step Functions `Retry` and `Catch` |
| Durable execution | Progress is persisted; after a crash the code resumes from recorded results instead of starting over[^l-durable][^t-def] | Temporal, Durable Functions, Lambda durable functions |
| Replay | Re-running workflow code against its history to rebuild state; requires deterministic code[^t-def][^df-constraints] | A restarted worker replays 40 events |
| Idempotency key | A caller-supplied identifier that makes a repeated request have no further effect[^bl-idem] | `charge-A1` sent to the payment API |
| Saga, compensation | A long process split into local transactions, each with an undo (compensating) action[^sagas][^az-comp] | Refund, then release stock |
| Pivot step | The point of no return; after it, steps must be retried until they succeed rather than compensated[^az-saga] | Handing the parcel to the courier |

**Mental model: the engine is a ledger, the steps are promises.** The engine keeps an exact ledger of what each instance has done.
Each step is a promise to an outside system, and the outside system does not read your ledger. Correctness therefore lives at the
boundary: every step must be safe to repeat, and every promise that cannot be kept needs a defined undo or a human owner.

### How WorkflowOps relates to other Ops

::: figure Where WorkflowOps sits among its neighbours
```mermaid
flowchart TB
  WF["WorkflowOps: engine semantics, retries, sagas, versioning running work"]
  D["DataOps: what pipelines compute, data quality"]
  I["IntegrationOps: brokers, message replay, schemas"]
  A["AutomationOps: runbook scripts and their guards"]
  S["ServerlessOps: one function, its events and quotas"]
  O["ObservabilityOps: telemetry pipelines and tools"]
  M["MLOps: training and serving lifecycle"]
  D -->|"runs on an engine"| WF
  M -->|"training pipelines as workflows"| WF
  A -->|"long multi-step jobs"| WF
  WF -->|"uses queues and events"| I
  WF -->|"steps may be functions"| S
  WF -->|"emits run metrics and traces"| O
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class WF core
  class D,I,A,S,O,M neutral
```
Text equivalent: WorkflowOps sits in the centre. DataOps and MLOps pipelines run on a workflow engine, and AutomationOps hands it long
multi-step jobs. WorkflowOps uses IntegrationOps' queues and events, may run steps as ServerlessOps functions, and emits run metrics and
traces to ObservabilityOps.
:::

| Neighbour | Shared ground | This page owns | Go there for |
|---|---|---|---|
| [DataOps](../dataops/) | Airflow, Dagster, schedules | Retry, catchup and backfill mechanics; idempotent tasks | Data tests, freshness, pipeline CI |
| [IntegrationOps](../integrationops/) | At-least-once delivery, retries | Multi-step state, compensation, timers | Brokers, DLQs, schema evolution |
| [AutomationOps](../automationops/) | Idempotency, approvals | Durable engines and long waits | Blast-radius guards, kill switches |
| [ServerlessOps](../serverlessops/) | Retries, idempotency keys | Orchestration across many steps | Single-invocation semantics, concurrency |
| [MLOps](../mlops/) | Pipelines on Argo or Airflow | Engine behaviour | Features, training, model registry |
| [ObservabilityOps](../observabilityops/) | Metrics, traces | Which workflow signals matter | Instrumentation and telemetry pipelines |
| [SRE](../sre/) | SLOs, toil | Workflow-level objectives | SLO method, error budgets |

## Principles and operating model

### A proposed charter and principles

The charter below is **this Atlas's proposal**; we found no published WorkflowOps manifesto.

**Mission:** every workflow instance ends in a known, correct state — completed, compensated, or parked with an owner — within a
stated deadline, across failures and deployments.

1. **Every step may run more than once**: workers die between doing and recording. Temporal Activities "may be executed more than
   once";[^t-activity] asynchronous Step Functions Express workflows are at-least-once.[^sf-types]
2. **Every external effect carries an idempotency key derived from the instance**, and the receiving system records key and effect
   atomically.[^bl-idem]
3. **Retry at one layer, with backoff, jitter and a cap.**[^bl-jitter][^sre-cascade]
4. **Every wait has a deadline and an owner.** Defaults are unlimited or very long (Temporal Schedule-To-Close ∞; Step Functions
   `TimeoutSeconds` 99,999,999 seconds).[^t-timeouts][^sf-task]
5. **Know which steps can be undone.** Mark compensable steps, the pivot and retryable steps before writing code.[^az-saga]
6. **Running instances are a compatibility contract**: a deployment is finished when old instances have drained or been migrated.
7. **Measure outcomes per instance**, not only worker health.

**Non-goals:** modelling processes with stakeholders; exactly-once delivery between systems (not available end to end); deciding
refund policy (a business rule).

### Building blocks

| Block | What it does | Typical form |
|---|---|---|
| Definition | Describes steps, branches, waits | Code (Temporal, Durable Functions), JSON/YAML (Step Functions, Argo), BPMN, Python DAG (Airflow) |
| History store, scheduler, timers | Record every event; start runs; fire timers and timeouts | Engine database and service; size and retention limits |
| Task queues and workers | Execute steps; scale independently | Worker processes, pods, functions |
| Step contracts | Key, timeout, retry policy, compensation | Design record + code |
| Control plane | Start, signal, cancel, reset, redrive, backfill | CLI, API, console |

### Roles, ownership and RACI

| Activity | Workflow owner (product team) | Engine/platform team | SRE / on-call | Downstream system owner | Business owner |
|---|---|---|---|---|---|
| Workflow design record, step contracts | A/R | C | C | C | C |
| Idempotency at the receiving API | C | I | I | A/R | I |
| Compensation rules (refunds, cancellations) | R | I | I | C | A |
| Engine operation, upgrades, capacity | I | A/R | C | I | I |
| Versioning a running workflow | A/R | C | I | I | I |
| Stuck-instance triage and repair | R | C | A | C | I |
| Backfills and bulk resets | A/R | C | C | C | I |

R = responsible, A = accountable, C = consulted, I = informed. The key split: **the platform team owns the engine; the workflow
owner owns the correctness of each workflow.** When a platform team is asked to "fix stuck orders", ownership is already wrong.

### Lifecycle of a workflow definition

**Design (Day 0)**: the design record ([Templates](#templates)). **Build**: test steps, inject failures, replay recorded histories
(replay engines).[^t-test] **Release (Day 1)**: keep old versions until running instances drain. **Operate (Day 2)**: watch the
[metrics](#metrics-and-kpis), run the runbooks and backfills. **Change**: classify every change ([Versioning running
workflows](#versioning-running-workflows)). **Retire**: stop new starts, drain or migrate the rest, delete histories per retention.

### Operating model and governance

- **One design record per workflow**, reviewed like an API change, including retry and timeout changes (they move load onto other teams).
- **A versioning gate in the deployment pipeline** for replay-based engines: recorded histories from production replay cleanly against
  the new build, or the deploy stops.
- **A weekly stuck-and-parked review**: every instance older than its deadline, every compensation that failed, every manual repair.
- **Bulk operations** (reset, terminate, redrive, backfill) need a second person and a written scope query.
- **Escalation**: business-impacting failure rates page the workflow owner; engine health (task backlog, persistence errors) pages
  the platform team.

## Architecture and workflow

### Reference architecture

::: figure Reference architecture of a durable workflow platform
```mermaid
flowchart TB
  C["Callers: API, events, schedules, operators"]
  E["Workflow engine: history store, timers, task queues"]
  W["Workflow workers: deterministic code, no I/O"]
  S["Step workers: I/O with idempotency keys"]
  X["External systems: payments, stock, couriers"]
  P["Human queue: approvals, parked instances"]
  O["Telemetry: metrics, traces, run search"]
  C -->|"start, signal, cancel"| E
  E <-->|"workflow tasks, commands"| W
  E <-->|"step tasks, results"| S
  S -->|"calls with keys"| X
  E -->|"waits, callbacks"| P
  E -->|"run data"| O
  S -->|"spans, metrics"| O
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class E,W core
  class C,S,O neutral
  class X,P risk
```
Text equivalent: Callers (APIs, events, schedules, operators) start, signal or cancel instances in the engine, which holds history,
timers and task queues. Workflow workers run deterministic code without input or output. Step workers are the only components that call
external systems (payments, stock, couriers), always with idempotency keys. The engine routes waits to a human queue and run data to
telemetry, which also receives step workers' spans. External systems and the human queue are marked as risk points.
:::

**Component responsibilities and trust boundaries.**

- **Engine.** Owns ordering, timers and history; its database is critical state (back it up, size it, respect
  [history limits](#long-lived-instances-and-history-size)).
- **Workflow code.** In replay-based engines it must be deterministic: no clocks, random numbers or network calls except through the
  engine's APIs.[^t-def][^df-constraints] Where the definition is data (Step Functions, Argo, BPMN), the risks move to data size and
  expression errors.
- **Step code.** The only place with side effects; the trust boundary to other systems. Credentials live here, scoped per step.
- **Payload size.** Keep large data out of the history: Step Functions has a 256 KiB payload quota; Temporal Cloud limits a single
  request to 2 MB.[^sf-quotas][^t-cloud-limits] Pass references (object keys), not documents.

**Alternatives:** see [Deep comparisons](#deep-comparisons) and the [decision framework](#decision-framework).

### End-to-end workflow: happy path and exception path

The illustrative order-fulfilment workflow used throughout this page: reserve stock → charge card → book courier → wait for dispatch
confirmation (up to 48 hours) → complete.

::: figure Crash after a side effect: replay plus an idempotency key prevents a double charge
```mermaid
sequenceDiagram
  autonumber
  participant E as Engine
  participant W as Worker
  participant P as Payment API
  E->>W: task: charge order A1
  W->>P: charge 100, key charge-A1
  P-->>W: charged
  Note over W: worker crashes before reporting
  Note over E: start-to-close timeout fires
  E->>W: retry task: charge order A1
  W->>P: charge 100, key charge-A1
  P-->>W: same result, no second charge
  W-->>E: completed
```
Text equivalent: The engine gives a worker the task to charge order A1. The worker calls the payment API with amount 100 and the key
charge-A1, and the payment succeeds, but the worker crashes before it reports back. The engine's start-to-close timeout fires and it hands
the task out again. The worker repeats the call with the same key; the payment API recognises the key and returns the original result
without charging twice, and the worker reports completion to the engine.
:::

**Happy path.** Trigger (order placed) → engine starts instance `order-A1` → three steps succeed → a timer waits for the dispatch
signal → completed; metrics record duration and attempts. The order ID is the workflow ID, which blocks duplicates only under
conditions. Temporal allows one *open* execution per ID; under the default reuse policy, Allow Duplicate, a late duplicate (a webhook
retried after completion) starts a second order, so set **Reject Duplicate**, which checks closed executions within the namespace's
retention period.[^t-wfid] Step Functions Standard returns the original response to a start with the same name and input while the execution runs and `ExecutionAlreadyExists`
for 90 days after it closes.[^sf-start] Older duplicates need a business-level check.

**Exception paths.**

1. *Transient step failure* (timeout, HTTP 503): retried by policy with backoff; no human involvement.
2. *Crash after a side effect* (diagram above): the engine re-issues the task; the idempotency key prevents a second effect.
3. *Permanent failure before the pivot, or pivot rejected* (card declined, courier refuses): compensate completed steps, end as
   `compensated`.
4. *Pivot outcome unknown* (courier API down for hours, or the booking call timed out): keep retrying with the same key while the
   business deadline allows, then query the booking by its key. Booked → continue; not booked → compensate; still unknown → a person
   decides. Never compensate on a guess: the courier may already hold the booking.
5. *Failure after the pivot* (confirmation service down): retry forward within a bounded window, then park; never compensate.
6. *Compensation fails* (refund API rejects): park the instance with full context for a human; never drop it.
7. *Wait never ends* (no dispatch signal): the timer fires at 48 hours and routes to a person.

## Practices, frameworks and tools

### Core practices and playbooks

Six playbooks follow. The workflow owner runs all of them (with the downstream owner for keys and the business owner for
compensation), and each ends in a test: fault injection, a duplicate call, a failure at each saga step, a CI replay, a dry backfill
list, and a history-size metric.

#### Retries and timeouts

Engines differ sharply in their defaults, and the defaults decide how a failure behaves:

| Engine (docs as of October 2026) | Default retry | Default timeout | Note |
|---|---|---|---|
| Temporal Activity | Initial 1 s, coefficient 2.0, max interval 100 × initial, **unlimited attempts**[^t-retry] | Schedule-To-Close ∞, but an Activity with neither Start-To-Close nor Schedule-To-Close is rejected; Temporal "strongly" recommends Start-To-Close[^t-timeouts] | Workflows themselves do not retry by default[^t-retry] |
| Step Functions Task | Only if a `Retry` field exists; then IntervalSeconds 1, MaxAttempts 3, BackoffRate 2.0, jitter NONE[^sf-errors] | `TimeoutSeconds` 99,999,999[^sf-task] | Without `Catch`, an error fails the execution[^sf-errors] |
| Airflow task | **None**: `default_task_retries` is 0 (delay 300 s when set);[^af-config] retries use a fixed count and delay regardless of error type; `ExceptionRetryPolicy` can fail fast per exception[^af-tasks] | `execution_timeout` if set[^af-tasks] | A policy cannot extend past `retries`[^af-tasks] |
| Argo template | **None** unless `retryStrategy` is set; its policy is OnFailure by default (Always when an `expression` is set)[^ar-retry] | `activeDeadlineSeconds` if set[^ar-cost] | Backoff via `duration`, `factor`, `maxDuration`, `cap`[^ar-backoff] |

::: example Worked calculation: two defaults, two behaviours
A step fails for ten minutes because a dependency is down. With **Temporal's default policy**, retries start 1, 2, 4, 8, 16, 32 and 64 seconds
after the previous attempt, then every 100 seconds (the cap). Ignoring execution time, attempts happen at t = 0, 1, 3, 7, 15, 31, 63, 127,
227, 327, 427 and 527 seconds: **12 attempts in the first ten minutes, then 36 per hour, without end.** With a **Step Functions `Retry` using
the defaults**, retries come 1, 2 and 4 seconds apart: the step **gives up after about 7 seconds** and the execution fails unless a `Catch`
handles it. Choose either deliberately.
:::

Rules that hold across engines:

1. **Classify errors.** Validation errors, declined cards and "not found" are non-retryable; timeouts, throttling and 5xx are
   retryable. Temporal and Step Functions both let you name non-retryable errors; Airflow 3.3 can fail fast on an exception type.[^t-retry][^sf-errors][^af-tasks]
2. **Retry at one layer.** In AWS's example, five layers that each make three tries multiply database load 243 times (3^5); retry "at
   a single point in the stack"[^bl-jitter] (the arithmetic is in [IntegrationOps](../integrationops/#charter-and-principles)). Count
   *attempts*, not retries: Step Functions `MaxAttempts: 3` means four tries, and the SRE book's three retries per layer give 4^3 = 64
   attempts.[^sre-cascade] If the engine retries a step, keep SDK retries inside it small, and use a retry budget.
3. **Add jitter and a cap.** Step Functions jitter is off unless `JitterStrategy` is `FULL`.[^sf-errors]
4. **Bound the total.** Use an overall deadline (Schedule-To-Close, `MaxAttempts`, `maxDuration`) per step, plus a workflow deadline (a timer in replay engines).
5. **Heartbeat long steps.** A step that runs for minutes should report heartbeats so a dead worker is detected in seconds, not at the
   end of a long timeout (Temporal heartbeat timeout; Step Functions `HeartbeatSeconds`).[^t-timeouts][^sf-task]

#### Idempotency

An engine guarantees that it records each step's result once; it cannot guarantee that the step's effect happened once. Temporal says
it plainly: lack of idempotency "might affect the correctness of your application but does not affect the Temporal Platform".[^t-activity]

1. **Derive the key from the instance, not the attempt**: `charge-<order_id>`, or workflow ID + step name. A random key per attempt
   defeats the purpose.
2. **The receiver records key and effect atomically.** Amazon's guidance: recording the token and the mutation must be one ACID
   operation, and a retry should receive a "semantically equivalent" response rather than an error such as "already exists".[^bl-idem]
3. **Where the receiver has no key support**, check-then-act is not enough: write conditionally (compare-and-set on a version). In
   the October 2025 DynamoDB outage, an automation component's one-time check that its plan was newer "was stale by this time", so an
   older plan overwrote a newer one, and a clean-up left "an inconsistent state that prevented subsequent plan updates".[^aws-ddb] A
   check at the start of a slow, retried operation is not a guard at its end.
4. **Make data tasks overwrite, not append.** Airflow's best practices: treat tasks "equivalent to transactions in a database", replace
   INSERT with UPSERT, and read a specific partition rather than "the latest available data".[^af-bp]
5. **Keep retries inside the receiver's key retention.** Amazon's guidance returns an equivalent response "for some
   interval";[^bl-idem] Stripe's API v1 may prune keys at least 24 hours old, after which a reused key creates a new request, and it
   rejects a reused key with different parameters (October 2026).[^stripe-idem] Use one key per step (`refund-<order_id>`, never the
   charge's key), keep each step's retry and park horizon inside the window, and after a longer pause query by key before resending.

#### Sagas and compensation

::: figure A saga with compensable steps, a pivot, and a human queue
```mermaid
flowchart TB
  S1["Reserve stock (compensable)"] --> S2["Charge card (compensable)"]
  S2 --> PV{"Book courier: pivot"}
  PV -->|"booked"| R1["Send confirmation (retry forward)"]
  R1 --> DONE["Completed"]
  PV -->|"outage or timeout"| ST{"Retry, then query status by key"}
  ST -->|"booked"| R1
  ST -->|"not booked"| C1
  PV -->|"rejected"| C1["Refund card"]
  C1 --> C2["Release stock"]
  C2 --> COMP["Compensated"]
  ST -->|"still unknown"| PARK["Parked: human queue with context"]
  C1 -->|"refund keeps failing"| PARK
  C2 -->|"release keeps failing"| PARK
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class S1,S2,R1,DONE,COMP core
  class PV,ST decision
  class C1,C2,PARK risk
```
Text equivalent: The saga reserves stock and charges the card; both steps can be compensated. Booking the courier is the pivot. If the
booking succeeds, the confirmation is retried forward and the saga completes. If the courier rejects the booking, the saga refunds the
card and releases the stock, ending as compensated. If the booking fails through an outage or timeout, the saga retries and then
queries the booking status by its key: booked continues to the confirmation, not booked leads to the refund, and a still-unknown
outcome goes to a human queue. A refund or release that keeps failing is also parked in the human queue with its context.
:::

Microsoft's saga guidance names the three kinds of step this diagram uses: **compensable** transactions that can be undone, a
**pivot** that is "the point of no return", and **retryable** transactions after it, which are idempotent so the saga can always
finish.[^az-saga] Practical rules:

- **Put the pivot as late as possible**, after reversible checks (fraud score, address validation).
- **Retry before compensating**, as Microsoft's guidance advises ("retries first to preserve forward progress").[^az-comp] **A pivot
  with an unknown outcome is resolved by a status query, not by compensation**: a timed-out booking may have succeeded, so ask the
  receiver by key, and send a person when the receiver cannot answer.
- **Compensation is a business action, not a rollback.** A refund may be partial; a cancelled booking may carry a fee. The business
  owner signs off the compensation table.[^az-comp]
- **Compensations can fail.** Record progress so compensation resumes where it stopped, make each one idempotent, and park with full
  detail when only a person can finish.[^az-comp]
- **Order need not be strictly reverse**: undo first in the system most sensitive to inconsistency.[^az-comp]
- **Sagas lack isolation.** Other processes can see intermediate states: lost updates, dirty reads and fuzzy reads are possible;
  countermeasures include semantic locks (a "pending" flag) and reordering so updates happen in retryable steps.[^az-saga]
- **Orchestration or choreography?** An orchestrator (the workflow engine) holds the state and decides; in choreography each service
  reacts to the others' events and no component holds the whole process.[^az-saga] Choreography suits a few steps owned by separate
  teams with no central deadline; once you need compensation order, timers, a human queue or one place to ask "where is order A1?",
  an orchestrator is easier to operate. This page assumes orchestration; brokers and event delivery are in
  [IntegrationOps](../integrationops/).

#### Versioning running workflows

The hardest WorkflowOps problem is changing a workflow while instances started on the old code still run. Engines take two
approaches:

- **Definition-as-data engines keep each execution on its definition.** Updating a Step Functions state machine leaves running
  executions on "the previous definition and roleArn"; versions and aliases let you shift new executions gradually.[^sf-update][^sf-versions]
  Airflow 3 with a versioned Dag bundle (GitDagBundle) runs a Dag run on the same code throughout; local, S3 and GCS bundles always use
  the latest code.[^af-bundles]
- **Replay engines re-run your code against old histories.** Changing the order or kind of engine commands breaks determinism: Temporal's
  docs show a timer-then-activity workflow changed to activity-then-timer failing with a "nondeterminism error" on replay.[^t-def] Azure
  Durable Functions warns that deploying breaking changes with no strategy can leave orchestrations failing or "stuck indefinitely in a
  Running status".[^df-version]

::: figure Choosing how to ship a change to a workflow with running instances
```mermaid
flowchart TB
  Q1{"Changes the command sequence of running instances?"}
  Q1 -->|"no: step internals, logging"| A1["Deploy normally; replay test in CI"]
  Q1 -->|"yes"| Q2{"Do old instances finish soon (hours)?"}
  Q2 -->|"yes"| A2["Pin old instances to the old version; drain, then retire it"]
  Q2 -->|"no: days or months"| Q3{"Must old instances redo a part they already passed?"}
  Q3 -->|"no"| A3["Patch at the change point: runs already past it keep the old path"]
  Q3 -->|"yes"| A4["New workflow type or migration: signal, reset or restart with state"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class Q1,Q2,Q3 decision
  class A1,A2,A3 core
  class A4 risk
```
Text equivalent: If the change does not alter the sequence of engine commands (a change inside a step, logging), deploy normally
with a replay test in CI. If it does and old instances finish within hours, pin them to the old version and retire it after they drain.
If they run for days or months and need not redo a part they already passed, patch at the change point: runs already past it keep the
old path, later arrivals take the new one. If they must redo it, create a new workflow type or migrate them by signal, reset or
restart with state; this last path is the riskiest.
:::

Engine mechanisms, as documented in October 2026:

| Engine | Mechanism | How it behaves |
|---|---|---|
| Temporal | `patched("id")` → `deprecate_patch("id")` → remove | New runs record a marker; replays without it take the old path; remove the old path only when no open execution, retained history or CI replay history needs it, and prune the replay suite at the same time[^t-patch][^t-pyver] |
| Temporal | Worker Versioning | **Pinned** workflows finish on one worker deployment version; **Auto-Upgrade** workflows move to the current version and still need patching; new versions can ramp by percentage[^t-wver] |
| Azure Durable Functions | Orchestration versioning (recommended), side-by-side deployment, or stopping in-flight instances | Each instance keeps the version it was created with; older workers cannot run newer instances[^df-version] |
| Step Functions | Versions and aliases | Running executions keep their definition; aliases route new ones[^sf-update][^sf-versions] |
| Airflow 3 | Versioned Dag bundles | Runs keep their code version; reruns and backfills can choose latest or original via `rerun_with_latest_version`[^af-bundles] |
| Camunda 8 (BPMN) | New definition versions; process instance migration | Running instances stay on the version they started with;[^cam-version] a migration plan maps active elements to another definition, but an element cannot change type and jobs, expressions and input mappings are not recreated[^cam-migrate] |

#### Backfills and catchup

Airflow 3 does not catch up missed intervals by default (`catchup_by_default=False`), and catchup also fires when a paused Dag is re-enabled.[^af-dagrun]
A backfill is explicit and scoped:

_Example status: syntax-checked, not executed (requires an Airflow 3.3 deployment; command form from the Airflow docs)._

```bash
# Re-run only failed or missing runs for one week, three at a time, newest first.
airflow backfill create --dag-id orders_daily \
  --from-date 2026-06-01 \
  --to-date 2026-06-07 \
  --reprocess-behavior failed \
  --max-active-runs 3 \
  --run-backwards
```

`--reprocess-behavior` accepts `none`, `failed` or `completed`; runs that are still queued or running are never duplicated.[^af-backfill]
Agree concurrency with the downstream owners: a backfill is a load test on every system it touches.

#### Long-lived instances and history size

Every event of an instance is stored and replayed. Temporal's default limits apply on every deployment: 51,200 events or 50 MB of
history, with a warning at 10,240 events or 10 MB; they are configurable when self-hosted and fixed on Temporal Cloud. Step Functions
fails an execution at 25,000 events.[^t-events][^t-cloud-limits][^sf-quotas]
For loops, subscriptions and polling, **continue-as-new** closes the execution and starts a fresh one with the same workflow ID and a new
run ID, carrying only the state it needs;[^t-can] alternatively, split work into child workflows. Never poll in a tight loop inside
a workflow: each poll is history.

### Methods, standards and frameworks

| Item | Type | Use it for |
|---|---|---|
| *Sagas* (Garcia-Molina and Salem, 1987) | Foundational paper[^sagas] | The compensation model |
| Workflow Patterns | Academic pattern catalogue (control-flow, data, resource, exception handling)[^wfpatterns] | Comparing what engines can express |
| Saga and Compensating Transaction patterns | Vendor architecture guidance (Microsoft)[^az-saga][^az-comp] | Step classification, anomalies, countermeasures |
| Amazon Builders' Library on retries and idempotent APIs | Vendor engineering practice[^bl-jitter][^bl-idem] | Retry placement, jitter, request identifiers |
| Google SRE book, cascading failures | Industry practice[^sre-cascade] | Retry budgets, backoff |
| Open Workflow Specification | CNCF Sandbox DSL, 1.0.3 as of October 2026[^openwf][^openwf-site] | Vendor-neutral definitions; engine support varies |
| BPMN 2.0.2 | OMG standard[^bpmn] | The notation BPMN engines execute; modelling with stakeholders is out of scope |

None of these is mandatory. There is no WorkflowOps certification or standards body.

### Tooling landscape

Vendor-neutral categories first, examples second. Versions and limits are as of October 2026.

| Category | Examples (verified) | Strengths | Limits and risks | Choose when |
|---|---|---|---|---|
| Durable execution, self-hosted or managed | Temporal (server v1.32.1, MIT licence; Python SDK 1.34.0)[^releases]; Cadence (its predecessor)[^temporal-about] | Workflows as code; waits of months; versioning tools | Determinism discipline; a cluster to run, or Temporal Cloud billed by Actions[^t-pricing] | Long, branching application processes |
| Durable execution inside a function platform | AWS Lambda durable functions (up to one year, checkpoint and replay)[^l-durable]; Azure Durable Functions[^df-constraints] | No separate engine to run; pay-per-use waits | Same determinism rules; platform lock-in; new and changing | Teams already on that platform |
| Managed state machines | AWS Step Functions (Standard up to one year, Express up to five minutes)[^sf-types]; Google Cloud Workflows (YAML/JSON; can wait up to a year)[^gcw] | No servers; definitions as data; console history | JSON/YAML logic gets unwieldy; payload limits; per-transition pricing[^sf-pricing] | Orchestrating cloud services and functions |
| BPMN execution engines | Camunda 8 (8.10; source under the Camunda License v1, production self-managed use needs the Enterprise Edition);[^cam-migrate][^cam-licence] Flowable (Apache-2.0, 8.0.0)[^flowable] | One model that analysts read and the engine runs; user tasks; instance migration | Licence terms (Camunda); model-code drift | Human-heavy processes that business owners read |
| DAG schedulers | Apache Airflow 3.3.2[^releases]; Argo Workflows v4.1.6 (CNCF graduated)[^cncf-argo][^releases] | Time-based batch, backfills, container steps on Kubernetes | Not designed for per-request processes or long human waits | Data and ML batch work |

Other code-first durable execution engines exist; this page does not rank them, and the same criteria apply. Selection criteria are in
the [decision framework](#decision-framework); add one often missed: managed engines store your step inputs and outputs in their
history, so data residency applies to them.

### Automation and AI

- **AI agents as workflows.** AWS positions Lambda durable functions for "AI workflows" that chain model calls and human
  feedback,[^l-durable] and Temporal places LLM invocations in Activities because they are non-deterministic.[^t-retry] Model calls are
  steps with timeouts, retries and cost caps.
- **Assisted triage** (summarising a failed instance's history) should link to the raw history; bulk operations stay behind the
  two-person rule.

## WorkflowOps in practice

### Reference implementation

The same order saga in two engines, plus a Kubernetes batch step. Assumptions: the order ID is unique and becomes the workflow ID or
execution name; every downstream API accepts an idempotency key; refunds are full refunds (a business decision).

**Temporal (Python SDK).** Each compensation is registered *before* its step runs, because a step that timed out may still have
taken effect. At the pivot, `CourierRejected` leads to compensation, while an outage is retried for up to 12 hours with the same key
and then settled by a status query or a person, never by a refund. The confirmation after it is retried for up to a day, then parked.
Each park waits for a `resolve` signal for its own item and sends a daily reminder: a timer, not a workflow timeout, which Temporal
generally advises against.[^t-wffail]

_Example status: syntax-checked, not executed (requires a Temporal server and workers; API names checked against the Python SDK 1.34.0
source)._

```python
import asyncio
from datetime import timedelta
from typing import Optional

from temporalio import activity, workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError, ApplicationError

STEP_RETRY = RetryPolicy(
    initial_interval=timedelta(seconds=2),
    maximum_interval=timedelta(minutes=1),
    maximum_attempts=8,
    non_retryable_error_types=["CardDeclined", "OutOfStock"],
)
# Unlimited attempts, but only together with a schedule-to-close deadline.
PIVOT_RETRY = RetryPolicy(
    initial_interval=timedelta(seconds=30),
    maximum_interval=timedelta(minutes=15),
    non_retryable_error_types=["CourierRejected"],
)
FORWARD_RETRY = RetryPolicy(maximum_interval=timedelta(minutes=10))
UNDO_RETRY = RetryPolicy(maximum_interval=timedelta(minutes=5), maximum_attempts=20)


@activity.defn
async def reserve_stock(order_id: str) -> str:
    return "reserved"  # call stock API with key f"reserve-{order_id}"


@activity.defn
async def release_stock(order_id: str) -> str:
    return "released"  # key f"release-{order_id}"; releasing twice is a no-op


@activity.defn
async def charge_card(order_id: str) -> str:
    return "charged"  # call payment API with key f"charge-{order_id}"


@activity.defn
async def refund_card(order_id: str) -> str:
    return "refunded"  # key f"refund-{order_id}"; no-op if never charged


@activity.defn
async def fraud_check(order_id: str) -> str:
    return "ok"


@activity.defn
async def book_courier(order_id: str) -> str:
    return "booked"  # key f"courier-{order_id}"; raises CourierRejected if refused


@activity.defn
async def courier_status(order_id: str) -> str:
    return "booked"  # look up key f"courier-{order_id}": booked/not_booked/unknown


@activity.defn
async def send_confirmation(order_id: str) -> str:
    return "sent"


@activity.defn
async def open_ticket(subject: str) -> str:
    return "opened"  # human queue; a person later sends the resolve signal


def rejected(err: ActivityError) -> bool:
    cause = err.cause
    return isinstance(cause, ApplicationError) and cause.type == "CourierRejected"


@workflow.defn
class OrderWorkflow:
    def __init__(self) -> None:
        self.decisions = {}  # parked item -> a person's decision

    @workflow.signal
    def resolve(self, item: str, decision: str) -> None:
        self.decisions[item] = decision

    async def step(
        self, fn, arg: str, retry: RetryPolicy = STEP_RETRY,
        limit: Optional[timedelta] = None,
    ) -> str:
        return await workflow.execute_activity(
            fn,
            arg,
            start_to_close_timeout=timedelta(seconds=30),
            schedule_to_close_timeout=limit,
            retry_policy=retry,
        )

    async def park(self, order_id: str, item: str) -> str:
        await self.step(open_ticket, f"{order_id}: {item}")
        while item not in self.decisions:
            try:
                await workflow.wait_condition(
                    lambda: item in self.decisions, timeout=timedelta(days=1)
                )
            except asyncio.TimeoutError:
                await self.step(open_ticket, f"{order_id}: {item} (reminder)")
        return self.decisions[item]

    async def compensate(self, undo: list, order_id: str) -> str:
        for comp in reversed(undo):
            try:
                await self.step(comp, order_id, UNDO_RETRY)
            except ActivityError:
                await self.park(order_id, comp.__name__)  # one decision per item
        return "compensated"

    async def pivot_status(self, order_id: str) -> str:
        try:
            status = await self.step(courier_status, order_id)
        except ActivityError:
            status = "unknown"
        if status == "unknown":  # never guess: a person checks with the courier
            status = await self.park(order_id, "courier booking")
        return status

    @workflow.run
    async def run(self, order_id: str) -> str:
        undo = []
        try:
            undo.append(release_stock)
            await self.step(reserve_stock, order_id)
            if workflow.patched("fraud-check-2026-10"):
                await self.step(fraud_check, order_id)
            undo.append(refund_card)
            await self.step(charge_card, order_id)
        except ActivityError:
            return await self.compensate(undo, order_id)
        # The pivot: a rejection is definitive; a timeout or an outage is not.
        try:
            await self.step(book_courier, order_id, PIVOT_RETRY, timedelta(hours=12))
            outcome = "booked"
        except ActivityError as err:
            outcome = "not_booked" if rejected(err) else await self.pivot_status(order_id)
        if outcome != "booked":
            return await self.compensate(undo, order_id)
        # After the pivot: forward only. Bounded retries, then a person.
        try:
            await self.step(send_confirmation, order_id, FORWARD_RETRY, timedelta(days=1))
        except ActivityError:
            await self.park(order_id, "send_confirmation")
        return "completed"
```

Start it with the order ID as workflow ID and the reuse policy `REJECT_DUPLICATE` (the client's `id_reuse_policy` argument; see
the [happy path](#end-to-end-workflow-happy-path-and-exception-path)).

**AWS Step Functions (Standard).** The same saga as data. Each Task builds its own key from the execution name (the order ID) with
`States.Format`, so a refund never reuses the charge's key.[^sf-intrinsic] The pivot retries an outage for hours; `CourierRejected`
goes to the refund, any other final error to a status query and then, if still unknown, to a person through `.waitForTaskToken` (a
message with the task token goes to a queue; an operator's tool calls `SendTaskSuccess`).[^sf-callback] Every Task has a timeout, and a
compensation that keeps failing parks the execution.

_Example status: JSON syntax-checked, not deployed (placeholders `REGION` and `ACCOUNT_ID`; Lambda functions and the SQS queue not shown)._

```json
{
  "Comment": "Order saga (illustrative). Execution name = order ID.",
  "StartAt": "ReserveStock",
  "TimeoutSeconds": 2592000,
  "States": {
    "ReserveStock": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:reserve-stock",
        "Payload": {
          "order.$": "$.order",
          "key.$": "States.Format('reserve-{}', $$.Execution.Name)"
        }
      },
      "ResultPath": "$.reserve",
      "TimeoutSeconds": 30,
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException",
                          "States.Timeout"],
          "IntervalSeconds": 2, "MaxAttempts": 5, "BackoffRate": 2.0,
          "MaxDelaySeconds": 60, "JitterStrategy": "FULL"
        }
      ],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
                  "Next": "ReleaseStock" }],
      "Next": "ChargeCard"
    },
    "ChargeCard": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:charge-card",
        "Payload": {
          "order.$": "$.order",
          "key.$": "States.Format('charge-{}', $$.Execution.Name)"
        }
      },
      "ResultPath": "$.charge",
      "TimeoutSeconds": 30,
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "States.Timeout"],
          "IntervalSeconds": 2, "MaxAttempts": 5, "BackoffRate": 2.0,
          "MaxDelaySeconds": 60, "JitterStrategy": "FULL"
        }
      ],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
                  "Next": "RefundCard" }],
      "Next": "BookCourier"
    },
    "BookCourier": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:book-courier",
        "Payload": {
          "order.$": "$.order",
          "key.$": "States.Format('courier-{}', $$.Execution.Name)"
        }
      },
      "ResultPath": "$.booking",
      "TimeoutSeconds": 30,
      "Retry": [
        {
          "ErrorEquals": ["CourierUnavailable", "Lambda.ServiceException",
                          "States.Timeout"],
          "IntervalSeconds": 30, "MaxAttempts": 40, "BackoffRate": 2.0,
          "MaxDelaySeconds": 900, "JitterStrategy": "FULL"
        }
      ],
      "Catch": [
        { "ErrorEquals": ["CourierRejected"], "ResultPath": "$.error",
          "Next": "RefundCard" },
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
          "Next": "CheckCourier" }
      ],
      "Next": "Completed"
    },
    "CheckCourier": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:courier-status",
        "Payload": {
          "key.$": "States.Format('courier-{}', $$.Execution.Name)"
        }
      },
      "ResultSelector": { "status.$": "$.Payload.status" },
      "ResultPath": "$.courier",
      "TimeoutSeconds": 30,
      "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 10,
                  "MaxAttempts": 5, "BackoffRate": 2.0, "JitterStrategy": "FULL" }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
                  "Next": "AskCourierOutcome" }],
      "Next": "PivotOutcome"
    },
    "PivotOutcome": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.courier.status", "StringEquals": "booked",
          "Next": "Completed" },
        { "Variable": "$.courier.status", "StringEquals": "not_booked",
          "Next": "RefundCard" }
      ],
      "Default": "AskCourierOutcome"
    },
    "AskCourierOutcome": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
      "Parameters": {
        "QueueUrl": "https://sqs.REGION.amazonaws.com/ACCOUNT_ID/parked-orders",
        "MessageBody": { "question": "Was the courier booked?", "input.$": "$",
                         "taskToken.$": "$$.Task.Token" }
      },
      "ResultPath": "$.courier",
      "TimeoutSeconds": 604800,
      "Next": "PivotOutcome"
    },
    "RefundCard": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:refund-card",
        "Payload": {
          "order.$": "$.order",
          "key.$": "States.Format('refund-{}', $$.Execution.Name)"
        }
      },
      "ResultPath": "$.refund",
      "TimeoutSeconds": 30,
      "Retry": [
        {
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 5, "MaxAttempts": 10, "BackoffRate": 2.0,
          "MaxDelaySeconds": 300, "JitterStrategy": "FULL"
        }
      ],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
                  "Next": "ParkForHuman" }],
      "Next": "ReleaseStock"
    },
    "ParkForHuman": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
      "Parameters": {
        "QueueUrl": "https://sqs.REGION.amazonaws.com/ACCOUNT_ID/parked-orders",
        "MessageBody": { "input.$": "$", "taskToken.$": "$$.Task.Token" }
      },
      "ResultPath": "$.human",
      "TimeoutSeconds": 604800,
      "Next": "ReleaseStock"
    },
    "ReleaseStock": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:release-stock",
        "Payload": {
          "order.$": "$.order",
          "key.$": "States.Format('release-{}', $$.Execution.Name)"
        }
      },
      "ResultPath": "$.release",
      "TimeoutSeconds": 30,
      "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 5,
                  "MaxAttempts": 10, "BackoffRate": 2.0, "JitterStrategy": "FULL" }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
                  "Next": "ParkForHuman" }],
      "Next": "Compensated"
    },
    "Compensated": { "Type": "Pass", "Result": "compensated",
                     "ResultPath": "$.outcome", "End": true },
    "Completed": { "Type": "Succeed" }
  }
}
```

Cautions: `States.ALL` does not catch `States.Runtime` errors, so alert on failed executions.[^sf-errors] The 30-day execution
timeout is a backstop; the 48-hour objective is measured as a [KPI](#metrics-and-kpis). A park unanswered for seven days fails the
execution, which should page the owner; redrive can resume it within 14 days.[^sf-redrive] `ParkForHuman` returns to the idempotent
`ReleaseStock`, so one park state serves both compensations; after a failed refund the operator refunds by hand before returning the token.
`Compensated` is a `Pass` end state (`outcome: compensated`), so `ExecutionsFailed` counts only real failures.[^sf-metrics] A long park can outlive the receiver's key retention
([Idempotency](#idempotency), rule 5).

**Argo Workflows (a batch step on Kubernetes).** Retry only transient errors with capped backoff, bound the whole run, always run an exit
handler, and clean up finished workflows. `podGC: OnPodSuccess` deletes pods that succeeded but keeps failed ones, whose logs you need
to troubleshoot a failed reconcile; `OnPodCompletion` would delete those too unless logs are archived. Argo's "transient" means its own
list (mainly Kubernetes API, quota and network errors) or `TRANSIENT_ERROR_PATTERN` matches; a container exiting non-zero after a
downstream 503 needs `OnFailure` (or `Always`).[^ar-retry][^ar-backoff][^ar-exit][^ar-cost][^ar-fields]

_Example status: YAML syntax-checked, not executed (requires Argo Workflows on a Kubernetes cluster; image tag illustrative)._

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: nightly-reconcile-
spec:
  entrypoint: reconcile
  onExit: report
  arguments:
    parameters:
      - name: business-date
        value: "2026-10-22"
  activeDeadlineSeconds: 7200
  ttlStrategy:
    secondsAfterCompletion: 86400
  podGC:
    strategy: OnPodSuccess
  templates:
    - name: reconcile
      retryStrategy:
        limit: "5"
        retryPolicy: OnTransientError
        backoff:
          duration: "10s"
          factor: "2"
          cap: "5m"
          maxDuration: "30m"
      container:
        image: registry.example.com/reconcile:1.4.2
        args: ["--date", "{{workflow.parameters.business-date}}"]
    - name: report
      container:
        image: registry.example.com/notify:1.0.0
        args: ["--status", "{{workflow.status}}"]
```

### Hands-on labs

The labs use `wfdemo.py`, a small teaching model of durable execution (standard-library Python, SQLite), **not** a production engine:
a journal of step results, replay after a crash, recorded failures, compensation and a Temporal-style patch marker.

- **Prerequisites:** Python 3.8 or later; a terminal; no network or accounts. **Cost and risk:** none; files stay in a temporary directory.
- **Execution status:** all three labs and the Lab 2 extension were **executed on 2026-10-23** with Python 3.8.10 on Ubuntu; the
  outputs below are copied from those runs.

_Example status: executed (Python 3.8.10), see above._

```python
"""wfdemo.py - a teaching model of durable execution (not a production engine)."""
import argparse
import os
import sqlite3
import sys

DB = os.environ.get("WFDEMO_DB", "wfdemo.sqlite")


class NonDeterminismError(Exception):
    """The code asked for a different step than the journal recorded."""


class StepFailed(Exception):
    """A step failed permanently (retries exhausted or non-retryable)."""


def db():
    con = sqlite3.connect(DB)
    con.executescript(
        "CREATE TABLE IF NOT EXISTS journal(wf TEXT, seq INT, step TEXT,"
        " result TEXT, PRIMARY KEY(wf, seq));"
        "CREATE TABLE IF NOT EXISTS charges(idem_key TEXT UNIQUE, order_id TEXT,"
        " amount INT);"
        "CREATE TABLE IF NOT EXISTS stock(order_id TEXT PRIMARY KEY);"
        "CREATE TABLE IF NOT EXISTS parked(wf TEXT, reason TEXT);"
    )
    return con


class Workflow:
    def __init__(self, con, wf_id):
        self.con, self.wf, self.seq = con, wf_id, 0
        rows = con.execute(
            "SELECT seq, step, result FROM journal WHERE wf=? ORDER BY seq",
            (wf_id,),
        ).fetchall()
        self.history = [(step, result) for _, step, result in rows]

    def replaying(self):
        return self.seq < len(self.history)

    def step(self, name, fn, retries=3):
        if self.replaying():
            recorded, result = self.history[self.seq]
            if recorded != name:
                raise NonDeterminismError(
                    f"seq {self.seq}: journal has {recorded!r}, code asked {name!r}"
                )
            self.seq += 1
            print(f"  replay  {name} -> {result}")
            if result.startswith("FAILED: "):  # failures are history too
                raise StepFailed(result[8:])
            return result
        failure = None
        for attempt in range(1, retries + 1):
            try:
                result = fn()
                break
            except StepFailed as exc:  # permanent: do not retry
                failure = str(exc)
                break
            except Exception as exc:  # transient: retry
                print(f"  attempt {attempt} of {name} failed: {exc}")
        else:
            failure = f"{name}: retries exhausted"
        if failure:
            result = "FAILED: " + failure
        if os.environ.get("CRASH_AFTER") == name:
            print(f"  CRASH after {name} ran, before the journal write")
            sys.stdout.flush()
            os._exit(3)  # simulate a killed worker: no cleanup, no journal
        self.con.execute(
            "INSERT INTO journal VALUES(?,?,?,?)", (self.wf, self.seq, name, result)
        )
        self.con.commit()
        self.seq += 1
        print(f"  run     {name} -> {result}")
        if failure:
            raise StepFailed(failure)
        return result

    def patched(self, patch_id):
        """Temporal-style patch marker: old runs keep the old path."""
        marker = "patch:" + patch_id
        if self.replaying():
            if self.history[self.seq][0] == marker:
                self.seq += 1
                return True
            return False
        self.step(marker, lambda: "marker")
        return True


# ---- "external systems" -------------------------------------------------
def charge(con, order_id, amount, use_key):
    key = f"charge-{order_id}" if use_key else None
    # One atomic statement records key and effect (no check-then-insert race).
    # A NULL key never collides, so the no-key run can charge twice.
    cur = con.execute(
        "INSERT OR IGNORE INTO charges VALUES(?,?,?)", (key, order_id, amount)
    )
    con.commit()
    return "charged" if cur.rowcount == 1 else "charged (duplicate suppressed)"


def refund(con, order_id):
    con.execute("DELETE FROM charges WHERE order_id=?", (order_id,))
    con.commit()
    return "refunded"


def reserve(con, order_id):
    con.execute("INSERT OR IGNORE INTO stock VALUES(?)", (order_id,))
    con.commit()
    return "reserved"


def release(con, order_id):
    con.execute("DELETE FROM stock WHERE order_id=?", (order_id,))
    con.commit()
    return "released"


def book_courier(order_id):
    if os.environ.get("COURIER") == "down":
        raise StepFailed("courier rejected the booking (non-retryable)")
    return "booked"


# ---- the workflow ---------------------------------------------------------
def order_workflow(wf, con, order_id, use_key, version):
    wf.step("reserve", lambda: reserve(con, order_id))
    if version == 2:  # unsafe change: a new step inserted for everyone
        wf.step("fraud_check", lambda: "ok")
    if version == 3 and wf.patched("fraud-check"):  # safe change
        wf.step("fraud_check", lambda: "ok")
    wf.step("charge", lambda: charge(con, order_id, 100, use_key))
    try:
        wf.step("courier", lambda: book_courier(order_id))
    except StepFailed as exc:
        print(f"  compensating: {exc}")
        try:
            wf.step("refund", lambda: refund(con, order_id))
            wf.step("release", lambda: release(con, order_id))
        except StepFailed as comp_exc:
            con.execute("INSERT INTO parked VALUES(?,?)", (wf.wf, str(comp_exc)))
            con.commit()
            return "parked for a human"
        return "compensated"
    return "completed"


def main():
    p = argparse.ArgumentParser()
    p.add_argument("cmd", choices=["run", "report", "reset"])
    p.add_argument("--order", default="A1")
    p.add_argument("--no-key", action="store_true", help="omit idempotency key")
    p.add_argument("--version", type=int, default=1)
    a = p.parse_args()
    if a.cmd == "reset":
        if os.path.exists(DB):
            os.remove(DB)
        print("removed", DB)
        return 0
    con = db()
    if a.cmd == "report":
        n = con.execute("SELECT COUNT(*) FROM charges WHERE order_id=?", (a.order,))
        s = con.execute("SELECT COUNT(*) FROM stock WHERE order_id=?", (a.order,))
        print(f"order {a.order}: charges={n.fetchone()[0]} stock={s.fetchone()[0]}")
        for row in con.execute("SELECT wf, seq, step FROM journal ORDER BY wf, seq"):
            print("  journal", row)
        return 0
    wf = Workflow(con, "order-" + a.order)
    try:
        outcome = order_workflow(wf, con, a.order, not a.no_key, a.version)
    except NonDeterminismError as exc:
        print("NON-DETERMINISM:", exc)
        return 2
    print("outcome:", outcome)
    return 0


if __name__ == "__main__":
    sys.exit(main())
```

#### Lab 1 (beginner): a crash between doing and recording

**Objective:** see why replay alone does not prevent a double charge, and how an idempotency key does.

1. Create a working directory and save the file above as `wfdemo.py` in it.
2. Run without a key, crashing after the charge, then resume:

```bash
mkdir -p ~/wfdemo-lab && cd ~/wfdemo-lab   # save wfdemo.py here first
python3 wfdemo.py reset
CRASH_AFTER=charge python3 wfdemo.py run --order A1 --no-key; echo "exit=$?"
python3 wfdemo.py run --order A1 --no-key
python3 wfdemo.py report --order A1
```

3. Repeat with the key (drop `--no-key`).

**Expected output** (abridged from the executed run):

```text
  run     reserve -> reserved
  CRASH after charge ran, before the journal write
exit=3
  replay  reserve -> reserved
  run     charge -> charged
  run     courier -> booked
outcome: completed
order A1: charges=2 stock=1        <- without the key: charged twice
...
  run     charge -> charged (duplicate suppressed)
order A1: charges=1 stock=1        <- with the key
```

**Validation:** `charges=2` without the key, `charges=1` with it. Replay skipped `reserve`; the journal had no record of `charge`, so
it ran again. **Troubleshooting:** if the first command prints nothing, output was buffered; exit code 3 still confirms the crash. **Teardown:** `python3 wfdemo.py reset`.

#### Lab 2 (intermediate): compensation, and failures as history

**Objective:** run a saga to the `compensated` state and see that a rerun replays the failure instead of booking again.

```bash
python3 wfdemo.py reset
COURIER=down python3 wfdemo.py run --order B7
python3 wfdemo.py report --order B7
python3 wfdemo.py run --order B7      # same instance, courier now "up"
```

**Expected output** (journal lines of the report omitted):

```text
  run     reserve -> reserved
  run     charge -> charged
  run     courier -> FAILED: courier rejected the booking (non-retryable)
  compensating: courier rejected the booking (non-retryable)
  run     refund -> refunded
  run     release -> released
outcome: compensated
order B7: charges=0 stock=0
  replay  reserve -> reserved
  replay  charge -> charged
  replay  courier -> FAILED: courier rejected the booking (non-retryable)
  compensating: courier rejected the booking (non-retryable)
  replay  refund -> refunded
  replay  release -> released
outcome: compensated
```

**Validation:** no charge or reservation remains, and the rerun changes nothing although the courier is now up: the history, not the
current world, decides; real engines record failures as events for this reason. **Extension:** make `refund` raise `StepFailed` and confirm the outcome `parked for a human` and a row in the `parked` table.
**Teardown:** `python3 wfdemo.py reset`.

#### Lab 3 (advanced): deploy a change while an instance is running

**Objective:** break a running instance with an unsafe change, then ship the same change safely with a patch marker.

```bash
python3 wfdemo.py reset
CRASH_AFTER=courier python3 wfdemo.py run --order C3 --version 1   # in flight
python3 wfdemo.py run --order C3 --version 2; echo "exit=$?"       # unsafe
python3 wfdemo.py run --order C3 --version 3                       # patched
python3 wfdemo.py run --order D4 --version 3                       # new order
python3 wfdemo.py report --order D4
```

**Expected output** (abridged: the first run's lines and most of the report are left out):

```text
  replay  reserve -> reserved
NON-DETERMINISM: seq 1: journal has 'charge', code asked 'fraud_check'
exit=2
  replay  reserve -> reserved
  replay  charge -> charged
  run     courier -> booked
outcome: completed
  run     reserve -> reserved
  run     patch:fraud-check -> marker
  run     fraud_check -> ok
  ...
  journal ('order-D4', 1, 'patch:fraud-check')
```

**Validation:** version 2 fails on the in-flight instance; version 3 lets `C3` finish on the old path while `D4` records the marker and
runs the fraud check. This mirrors Temporal's `patched()` behaviour.[^t-patch] **Troubleshooting:** if `C3` completes under version 2,
you skipped the crash step. The crash came after `courier` ran, so a real courier API would have been called twice: that call needs
its own key too. **Teardown:** `python3 wfdemo.py reset`, then
`rm -r ~/wfdemo-lab`.

### Use cases and scenarios

All three are **illustrative scenarios**, not documented case studies.

| Scenario | Context and method | Trade-offs | Success criteria |
|---|---|---|---|
| Order fulfilment (this page's example) | Durable engine; order ID as workflow ID; saga with courier pivot; 48-hour dispatch timer | Engine to operate; determinism to learn | ≥ 99.5 % completed or compensated within 48 h; no double charges |
| Nightly ledger reconciliation | Airflow or Argo DAG per business date; overwrite partitions; backfills scoped by date and capped concurrency | Not suited to per-request work | Run finished by 06:00 on ≥ 29 of 30 days; reruns produce identical output |
| Customer onboarding with human review | Managed state machine; callback task for KYC review; reminder and escalation timers | Human waits dominate; small payloads | No instance waits past 7 days without escalation |

## Measure, secure and pay

### Metrics and KPIs

Measure what the business sees (instances reaching a correct end state on time), then the signals that predict it.

| KPI | Definition / formula | Data source | Pitfall | Review |
|---|---|---|---|---|
| Correct end state within deadline | (completed + compensated within deadline) ÷ instances whose deadline fell in the period | Engine run search; status and close time | Counting "not failed" as success hides open instances | Weekly; SLO |
| Stuck instances | Open instances older than their deadline; also the age of the oldest | Engine list query | Averages hide one very old instance | Daily |
| Compensation rate | compensated ÷ closed (parked instances are still open) | Engine | A falling rate can mean failures are being parked instead | Weekly |
| Parked backlog | Instances waiting for a human, and their age | Human queue | Nobody owns the queue | Daily |
| Retries per completed run | retry attempts ÷ completed runs, per step | Engine/SDK metrics | Retries hidden inside step code are invisible | Weekly; alert on spikes |
| Task start latency | Time from task scheduled to picked up | Temporal `temporal_activity_schedule_to_start_latency`, `temporal_workflow_task_schedule_to_start_latency`[^t-metrics] | High values mean too few workers | Real time |
| Workflow task failures | Rate of failed workflow tasks (non-determinism, code errors) | Temporal `temporal_workflow_task_execution_failed`;[^t-metrics] Step Functions `ExecutionsFailed`, `ExecutionsTimedOut`[^sf-metrics] | Spikes right after deploys are versioning bugs | Real time |
| Reconciliation gaps | Effects downstream without a matching instance, or the reverse | Join engine records with payment/stock records | The only metric that detects double effects | Daily |

::: example Worked calculation (illustrative week)
10,000 order instances had their 48-hour deadline in the week. 9,870 completed and 85 were compensated on time; 12 are parked; 33 are
still open past deadline. **Correct end state within deadline** = (9,870 + 85) ÷ 10,000 = **99.55 %**, just above a 99.5 % objective.
**Compensation rate** = 85 ÷ 9,955 closed (the 12 parked instances are still open) = 0.85 %. **Retries per completed run**: 4,200 retry attempts ÷ 9,870 = 0.43, of which 3,900
were on the courier step: one dependency to discuss. The 33 stuck and 12 parked instances go to the weekly review by name.
:::

Leading: task start latency, retries per run, history size. Lagging: correct end state, reconciliation gaps. SLO method:
[SRE](../sre/).

### Security, privacy and compliance

- **Histories contain data.** Step Functions keeps closed execution history for 90 days (reducible to 30 by quota request); Lambda
  durable functions retain data 1–90 days after completion (default 14).[^sf-quotas][^l-pricing] Pass identifiers, not personal data;
  where personal data must flow, encrypt payloads and set retention deliberately.
- **Least privilege per step**: each step (function, worker, pod) gets its own identity scoped to the systems it calls.
- **Control-plane rights** (reset, terminate, redrive, signal, backfill) change outcomes in bulk: restrict, log, two-person rule.
- **Signals and task tokens are inputs**: one that completes a human review must come from an authenticated tool.
- **Audit trails**: a history is evidence only if retention matches the rule that applies; export it before the engine deletes it.

### Cost, capacity and sustainability

Cost drivers are steps, retries, waiting and stored history. Step Functions Standard bills per state transition "including
retries", $0.000025 each in US East (N. Virginia) after 4,000 free a month (October 2026).[^sf-pricing] Temporal Cloud bills Actions
such as starting workflows and recording heartbeats;[^t-pricing] Lambda durable functions bill compute (including replays), durable
operations, data written and retention.[^l-pricing]

::: example Worked calculation: what retries cost (illustrative, Step Functions Standard, US East prices as of October 2026)
Counting graph nodes as AWS's own pricing example does (Start and End included),[^sf-pricing] the happy path of the state machine
above is Start, three Tasks, `Completed` and End: 6 transitions. 1,000,000 orders a month × 6 = 6,000,000 transitions. Minus 4,000
free = 5,996,000 × $0.000025 = **$149.90**. Assume a courier outage makes 5 % of orders retry the booking 20 times on average: 1,000,000
× 0.05 × 20 = 1,000,000 extra transitions = **$25.00**. The bill moves a little; the load on the struggling courier API (1,000,000 extra
calls) is the real cost, which is why retry policy is a capacity decision agreed with the downstream owner.
:::

Capacity: size workers by task start latency, not CPU; cap concurrency for backfills; keep history small (continue-as-new) so replay stays
fast. Sustainability follows compute: Google Cloud Workflows charges nothing while idle,[^gcw] and waits in on-demand Lambda durable
functions incur no duration charges, although data retention is still billed;[^l-pricing] polling loops cost compute.

### Assurance of the operating model

The [operating model](#operating-model-and-governance) is real only if it leaves evidence: CI logs of replay tests on exported
production histories (`temporal workflow show --output json` plus the SDK replayer),[^t-cli][^t-test] fault-injection results, a daily
reconciliation report, notes of the weekly review, and a ticket with scope query and second approver per bulk operation. Audit a sample
each quarter.

## When things go wrong

### Common failures and troubleshooting

| Symptom | Likely cause | Diagnosis | Remediation | Prevention |
|---|---|---|---|---|
| Duplicate effects (two charges) | Step retried after effect, no key | Reconciliation gap; provider logs | Refund duplicates; add keys | Per-step keys; duplicate-call test |
| Instances stuck "Running" | Missing timeout or callback; no workers | Task start latency; last event | Add timeout; send callback; scale workers | Deadline on every wait; age alert |
| Spike in workflow task failures after deploy | Non-deterministic change (replay engines) | Error message names the mismatch; replay the history locally | Roll back workers; then patch | Replay tests in CI |
| Execution terminated or failed near N events | History limit (51,200 Temporal; 25,000 Step Functions) | Event count per instance | Restart from state with a new instance | Continue-as-new; child workflows[^t-events][^sf-quotas] |
| Dependency overwhelmed during an incident | Retries at several layers, no jitter | Retry metrics per layer | Lower retries; add jitter; pause backfills | One retry layer; budgets[^bl-jitter] |
| Backfill rewrote good data or duplicated rows | Non-idempotent task (INSERT, `now()`) | Compare outputs across reruns | Restore; fix task | UPSERT, partition by interval[^af-bp] |
| Payload or state too large | Documents passed through history | Engine error on size | Store objects externally, pass keys | Payload budget in the design record |

### Worked runbook: non-determinism errors after a deploy

_Context: a Temporal deployment; command forms from the Temporal CLI docs; not executed._

1. **Detect.** `temporal_workflow_task_execution_failed` rises within minutes of a worker release; instances stop progressing.[^t-metrics]
2. **Confirm.** Pick one affected workflow and export its history; the worker log names the mismatch (expected event vs new command).

   _Example status: syntax-checked, not executed (requires the Temporal CLI and a namespace)._

   ```bash
   temporal workflow list --limit 20 \
     --query 'ExecutionStatus="Running" AND WorkflowType="OrderWorkflow"'
   temporal workflow show --workflow-id order-A1 --output json > order-A1.json
   ```

3. **Contain first: roll the workers back.** Workflow tasks are not governed by retry policies; they keep retrying with backoff up to a
   10-minute interval until the workflow's execution timeout,[^t-retry] so affected instances resume on the old build without data loss. Do
   **not** terminate or reset them.
4. **Reproduce.** Replay `order-A1.json` against the new build with the SDK replayer; it should fail the same way.[^t-test]
5. **Fix.** Wrap the change in `workflow.patched("…")` (or pin the workflow type to its worker deployment version) and add the exported
   history to the CI replay suite.[^t-patch][^t-wver]
6. **Re-deploy and verify.** Failure metric back to baseline; old instances complete on the old path; new instances record the patch marker.
7. **Only if instances are unrecoverable**, use `temporal workflow reset --workflow-id … --event-id …` from a known-good event, with a
   second approver and a written list of affected IDs.[^t-cli]
8. **Review** why CI did not replay a production history.

### Anti-patterns and myths

- **"The engine makes it exactly once."** Step Functions' exactly-once claim for
  Standard workflows explicitly excludes steps with `Retry`.[^sf-types]
- **Retry everything forever.** Non-retryable errors then loop for days. Classify errors.
- **The workflow as a database.** One eternal instance with growing state hits history limits; keep business data in its system of record.
- **Side effects in scheduler callbacks.** Hooks are not retried, versioned or visible like steps.
- **Compensation as an afterthought.** If no one has written the refund rule, the engine cannot run it.
- **Bulk reset as a fix.** Resetting thousands of instances replays effects: treat it as a data migration.

## Deciding and adopting

### When to use and when not to use

**Use a workflow engine when** a process has several steps across systems and needs any of: surviving crashes mid-process,
compensation, waits of minutes to months (timers, people, callbacks), or a step-by-step audit trail. Duration alone is not the test: a
seconds-long transfer across three systems that needs compensation belongs on an engine too.

**Do not adopt one when:**

- One idempotent job on a schedule does the work: cron (or a Kubernetes CronJob) plus a lock and an idempotent script is cheaper.
- The process is a single request-response under a few seconds.
- Steps are a stream of independent messages: use a queue with idempotent consumers ([IntegrationOps](../integrationops/)).
- The team cannot yet guarantee idempotent downstream calls: an engine will retry the problem faster.

Hidden responsibilities: engine operation or vendor cost, determinism training, versioning discipline, history retention, and a
staffed human queue.

### Decision framework

::: figure Choosing the simplest mechanism that keeps a multi-step process correct
```mermaid
flowchart TB
  Q1{"Several steps with side effects?"}
  Q1 -->|"no"| R1["Plain job or request"]
  Q1 -->|"yes"| Q2{"Compensation, long or human waits, or step audit?"}
  Q2 -->|"no"| Q3{"Batch per time interval?"}
  Q3 -->|"yes"| R2["DAG scheduler"]
  Q3 -->|"no"| R3["Queue and idempotent consumers"]
  Q2 -->|"yes"| Q4{"Mostly calls cloud services, little branching?"}
  Q4 -->|"yes"| R4["Managed state machine"]
  Q4 -->|"no"| R5["Durable execution or BPMN engine"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q1,Q2,Q3,Q4 decision
  class R2,R4,R5 core
  class R1,R3 neutral
```
Text equivalent: If there is only one step with side effects, use a plain job or request with a timeout. If there are several, ask
whether the process needs compensation across systems, waits for minutes to months or for people, or must be audited step by step;
any one of these leads to an engine, however short the process. If none applies, a batch processed per time interval suits a DAG
scheduler such as Airflow or Argo Workflows, and independent messages suit a queue with idempotent consumers. On the engine branch, a
process that mostly calls cloud services with little branching suits a managed state machine such as Step Functions or Google Cloud
Workflows; complex logic suits a durable execution engine such as Temporal, Azure Durable Functions or Lambda durable functions, or a
BPMN engine when business owners must read the model.
:::

For a shortlist of engines, score each 1–5 on weighted criteria agreed in advance (example weights, adjust to your context):

| Criterion | Weight | Question |
|---|---|---|
| Fit to longest wait and duration | 25 % | Can it hold an instance long enough? |
| Versioning of running instances | 20 % | What happens to in-flight instances on deploy? |
| Team skills and language | 15 % | Code, JSON/YAML or BPMN? Which SDKs? |
| Operational burden and cost | 15 % | Who runs it; how does cost scale with retries? |
| Limits | 15 % | History, payload, throughput |
| Exit cost | 10 % | How much is in a proprietary DSL or SDK? |

### Maturity model

**This Atlas's model**, not an industry standard.

| Level | Observable behaviour | Exit criteria |
|---|---|---|
| 1 Ad hoc | Status columns, cron sweepers, manual fixes | Inventory of multi-step processes |
| 2 Engine adopted | Processes on an engine with default policies | Every step has a timeout and explicit retry policy |
| 3 Correct by design | Idempotency keys, saga tables, deadlines on waits | Fault-injection tests pass per step; daily reconciliation |
| 4 Safe to change | Replay tests in CI; versioning per change; bulk ops governed | No deploy-caused failures in a quarter |
| 5 Measured and improved | SLO on correct end state; weekly review drives fixes | Objective met two quarters running; parked backlog shrinking |

### Adoption roadmap: 30/60/90 days

| Window | Actions | Milestone |
|---|---|---|
| Days 1–30 | Inventory multi-step processes; pick one with real pain; write its design record; agree keys with downstream owners | Design record approved; engine chosen |
| Days 31–60 | Build it with timeouts, retries, compensation, a human queue and fault-injection tests; KPI dashboards | Live for a share of traffic; stuck-instance alert on |
| Days 61–90 | Replay tests in CI or version/alias routing; daily reconciliation; first weekly review | First change shipped to running instances without failures |

Quick wins: a deadline on every wait, an alert on the oldest open instance, no nested SDK retries inside engine-retried steps.

## Toolkit

### Checklists

**Before production (per workflow)**

- [ ] Design record approved: steps, keys, timeouts, retries, compensation, pivot, deadline, owner
- [ ] Every step has a timeout and an explicit retry policy; non-retryable errors listed
- [ ] Every external call carries a per-step, instance-derived key; each receiver's key retention exceeds the step's retry and park horizon
- [ ] Fault-injection tests: timeout, crash after effect, permanent error at each step
- [ ] Human queue exists, is staffed and has an age alert
- [ ] Payloads are references; history size estimated against engine limits

**Before each deployment**

- [ ] Change classified (compatible / patch / new version / new type)
- [ ] Replay tests pass on recent production histories, or versions/aliases route new executions
- [ ] Rollback path for workers or definitions tested

### Templates

_Template status: plain-text forms; adapt the fields to your engine._

```text
WORKFLOW DESIGN RECORD
Workflow name / type:            Owner (team, on-call):
Workflow ID and reuse policy:    e.g. "order-<id>", Reject Duplicate
Deadline (whole workflow):       Expected volume (starts/day, steps/run):
Max duration and history size:   Engine limits checked (events, payload):

STEP TABLE
| Step | System | Idempotency key | Key retention | Timeout | Retry policy |
| Non-retryable errors | Kind: compensable / pivot / retryable | Compensation | Owner |

Human waits: what, who, deadline, escalation
Data in history: fields, personal data (y/n), retention
Versioning plan: patch / pin / new type; replay test source
Metrics and alerts: correct-end-state SLO, oldest open, parked age
Approvals: workflow owner, downstream owners, business owner (compensation)
```

```text
CHANGE TO A RUNNING WORKFLOW
Workflow:                 Change summary:
Running instances now:    Oldest started:      Expected drain date:
Alters command sequence?  yes / no  (evidence: replay test run id)
Strategy:                 deploy / patch id ____ / pin version ____ / new type
Bulk operation needed?    none / reset / signal / migrate  (scope query attached)
Second approver:          Rollback plan:
After drain: remove old path on (date) ________
```

### Deep comparisons

| Mechanism | Best at | Weak at | State lives in |
|---|---|---|---|
| Queue + idempotent consumers | Independent messages at volume | Multi-step state, timers, compensation | Messages and consumers' databases |
| DAG scheduler (Airflow, Argo) | Batch per interval, backfills, containers | Per-request processes, long human waits | Scheduler metadata DB; outputs |
| Managed state machine (Step Functions, Cloud Workflows) | Calling cloud services; no servers | Complex logic in JSON/YAML; payload size | Provider's execution history |
| Durable execution (Temporal, Durable Functions, Lambda durable) | Long, branching processes written as code | Determinism discipline; versioning effort | Engine history, replayed into code |
| BPMN engine (Camunda 8, Flowable) | Human-heavy processes that business owners read | Model and code drift; licence terms | Engine state per process instance |

### FAQ

**Is "WorkflowOps" an established discipline?** No. The name has unrelated uses (an Oracle API, a product);[^oracle][^wfopspl] the practice
here is known as workflow orchestration or durable execution.

**Is a BPMN engine a workflow engine?** Yes. Camunda 8 or Flowable runs process instances with the same concerns as on this page
(retries, idempotent steps, versions of running instances); only the modelling work with stakeholders is out of scope.

**Airflow or Temporal?** Different jobs: batch work per data interval with backfills, versus application processes as code with long
waits. Teams often run both; pipeline practice is in [DataOps](../dataops/).

**When should I compensate instead of retrying?** Only when forward progress is impossible: a non-retryable error or exhausted retries,
and only before the pivot. If the pivot's outcome is unknown, query it by key first.[^az-saga][^az-comp]

**What should page the on-call engineer?** The oldest open instance past its deadline, a jump in workflow task failures, and task start
latency; not single step retries, which are expected.

## Going further

### Advanced topics and emerging direction

- **Durable execution inside platforms** is recent (Lambda durable functions launched at re:Invent 2025);[^l-blog] re-check quotas and
  pricing.[^l-pricing]
- **AI agents on durable engines** treat each model call as a step; per-instance cost caps and compensation for an agent's actions are
  open design questions.
- **Portable definitions** (Open Workflow Specification 1.0.3) exist; portability is a goal, not a given.[^openwf-site]

### Learning path

| Stage | Do this |
|---|---|
| Prerequisites | Transactions, timeouts, HTTP status codes; [DevOps](../devops/) basics |
| 1-day primer | The Sagas abstract, Microsoft's saga pattern, Amazon's retry article; Labs 1–2 |
| 1 week | Lab 3; the order saga on one engine's local tier, with fault injection |
| 30 days | One real workflow with a design record, KPIs and a replay or versioning test in CI |
| Specialise | Engine operation at scale; [DataOps](../dataops/); [IntegrationOps](../integrationops/) |
| Portfolio idea | A public repository with a saga, its failure tests and a documented versioning change |

## Wrap-up

### Executive recap and cheat sheet

| Topic | Remember |
|---|---|
| Core risk | Steps run more than once; effects need instance-derived keys |
| Retries | One layer, backoff, jitter, cap, classified errors |
| Failure model | Compensable steps, one pivot (query it when unsure), retryable steps, a human queue |
| Change | Running instances are a contract: patch, pin or new type |
| Measure | Correct end state within deadline; oldest open instance; reconciliation |

### A judgement call: where to put a patch for a rule that covers running instances

::: decision Scenario (illustrative)
A lender's loan-approval workflow runs for up to 90 days; 40,000 instances are open, about 24,000 of them not yet disbursed. A new rule
requires a second affordability check before disbursement for **every** loan not yet disbursed. The engine replays workflow code
(Temporal). Option A: add the check at intake, where it logically belongs, behind `patched("affordability-2")`. Option B: add it
immediately before the disbursement step, behind the same kind of patch. Option C: create a new workflow type and migrate open instances.
:::

- **Option A is the attractive mistake.** Open instances passed intake long ago. When they replay, their history has no marker at that
  point, so `patched()` returns false and they keep the old path for the rest of the run;[^t-patch] the 24,000 open loans skip the check.
- **Option B meets the rule with no migration.** Temporal documents that an execution which met no `patched()` call during replay uses the
  new code once past the replayed history,[^t-patch] so open instances that later reach disbursement run the check (new loans too,
  if a late check is acceptable to the business).
- **Option C** also works, but migrating 24,000 instances is a bulk operation for no gain here.

Decision: Option B, released after replay tests on histories of instances at several stages, plus a report of loans disbursed
without the check (expected: none). Uncertainty: an execution that already met a `patched()` call with the same ID would behave
differently, so use a new, unique patch ID.

### Learning objectives and mastery check

After this page you can explain why an engine does not prevent double effects, set retry and timeout policies, design a saga with a
pivot, version running instances, calculate workflow KPIs and run the runbook.

::: answer A worker charged a card and crashed before reporting. What prevents a second charge, and what does not?
The engine's replay does not: it has no record of the charge, so it re-issues the step. An idempotency key derived from the instance
(for example `charge-<order_id>`), recorded atomically by the payment service, prevents the second charge (Lab 1).
:::

::: answer Which steps of a saga are compensated, and which are retried?
Steps before the pivot are compensable and are undone on permanent failure; steps after the pivot are retryable and must be retried until
they succeed, because the process can no longer go back.
:::

::: answer You must add a step to a workflow whose instances run for months. Old instances need not run it. What do you do?
Patch: branch on a version marker (Temporal `patched()`, Durable Functions orchestration versioning) so instances already past that
point keep the old path (those not yet there run the new step), add a replay test, and remove the old path only when no open or
retained history needs it.
:::

::: answer Five layers each make three tries. What happens during a database overload?
Attempts multiply: 3^5 = 243 times the load at the database in Amazon's example, which can keep it from recovering. (With three
*retries* per layer, it is four tries each.) Retry at one layer, with jitter and a budget.
:::

::: answer The courier booking timed out after hours of retries. Refund the customer?
Not yet. The booking may have succeeded, so query it by its key: booked → continue forward; not booked → compensate; no answer → a
person decides.
:::

### Conclusion

"WorkflowOps" is a label without a settled meaning, but the work it names here is real: processes that outlive the workers,
deployments and sometimes the people who started them. Engines turn the bookkeeping into infrastructure; they do not make a
charge safe to repeat, decide what a refund means, or protect a running instance from tomorrow's code. Teams that do this well write down
every step's key, timeout, retry and undo before writing code, treat running instances as a contract, and measure whether each
instance ended correctly. That discipline matters more than the choice of engine.

## Sources and further reading

All sources were opened and checked on 2026-10-23. Product limits, prices and versions change; re-check them before relying on them.

[^oracle]: [WorkflowOps (Oracle Fusion Middleware Java API for Oracle WebLogic Portal 10.3.5)](https://docs.oracle.com/cd/E35968_01/wlp.1035/e14255/com/bea/content/spi/WorkflowOps.html) — Oracle, undated. Checked 2026-10-23.
[^wfopspl]: [WorkflowOps: Operations Workflow & Approvals Platform](https://www.workflowops.pl/en/) — workflowops.pl, undated. Checked 2026-10-23.
[^sagas]: [SAGAS, technical report TR-070-87](https://www.cs.princeton.edu/research/techreps/598) — Hector Garcia-Molina and Kenneth Salem, Princeton University, January 1987 (also SIGMOD 1987). Abstract quoted. Checked 2026-10-23.
[^wfpatterns]: [Workflow Patterns home page](http://www.workflowpatterns.com/) — Workflow Patterns initiative, undated. Checked 2026-10-23.
[^temporal-about]: [About Temporal](https://temporal.io/about) — Temporal Technologies, undated (company-authored history). Checked 2026-10-23.
[^bpmn]: [Business Process Model and Notation, Version 2.0.2](https://www.omg.org/spec/BPMN/) — Object Management Group. Checked 2026-10-23.
[^openwf]: [Open Workflow Specification (serverlessworkflow/specification)](https://github.com/serverlessworkflow/specification) — GitHub, README states CNCF Sandbox since 2020-07-14. Checked 2026-10-23.
[^openwf-site]: [Open Workflow Specification](https://serverlessworkflow.io/) — project site, current DSL version 1.0.3. Checked 2026-10-23.
[^cncf-argo]: [Argo](https://www.cncf.io/projects/argo/) — Cloud Native Computing Foundation (accepted 2020-03-26, graduated 2022-12-06). Checked 2026-10-23.
[^releases]: GitHub releases: [Temporal Server v1.32.1](https://github.com/temporalio/temporal/releases/tag/v1.32.1) (2026-10-08), [Temporal Python SDK 1.34.0](https://github.com/temporalio/sdk-python/releases/tag/1.34.0) (2026-09-30), [Apache Airflow 3.3.2](https://github.com/apache/airflow/releases/tag/3.3.2) (2026-09-17), [Argo Workflows v4.1.6](https://github.com/argoproj/argo-workflows/releases/tag/v4.1.6) (2026-10-22). Checked 2026-10-23.
[^t-def]: [Workflow Definition](https://docs.temporal.io/workflow-definition) — Temporal documentation. Checked 2026-10-23.
[^t-retry]: [Retry Policies](https://docs.temporal.io/encyclopedia/retry-policies) — Temporal documentation. Checked 2026-10-23.
[^t-timeouts]: [Detecting Activity failures](https://docs.temporal.io/encyclopedia/detecting-activity-failures) — Temporal documentation. Checked 2026-10-23.
[^t-events]: [Events and Event History](https://docs.temporal.io/workflow-execution/event) — Temporal documentation. Checked 2026-10-23.
[^t-cloud-limits]: [Temporal Cloud limits](https://docs.temporal.io/cloud/limits) — Temporal documentation. Checked 2026-10-23.
[^t-can]: [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) — Temporal documentation. Checked 2026-10-23.
[^t-patch]: [Patching](https://docs.temporal.io/patching) — Temporal documentation. Checked 2026-10-23.
[^t-pyver]: [Versioning (Python SDK)](https://docs.temporal.io/develop/python/versioning) — Temporal documentation. Checked 2026-10-23.
[^t-wver]: [Worker Versioning](https://docs.temporal.io/worker-versioning) — Temporal documentation. Checked 2026-10-23.
[^t-activity]: [Activity Definition](https://docs.temporal.io/activity-definition) — Temporal documentation. Checked 2026-10-23.
[^t-wfid]: [Workflow Id and Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid) — Temporal documentation. Checked 2026-10-23.
[^t-cli]: [Temporal CLI workflow command reference](https://docs.temporal.io/cli/workflow) — Temporal documentation. Checked 2026-10-23.
[^t-test]: [Testing (Python SDK)](https://docs.temporal.io/develop/python/testing-suite) — Temporal documentation. Checked 2026-10-23.
[^t-metrics]: [SDK metrics](https://docs.temporal.io/references/sdk-metrics) — Temporal documentation. Checked 2026-10-23.
[^t-pricing]: [Temporal Cloud pricing](https://docs.temporal.io/cloud/pricing) — Temporal documentation. Checked 2026-10-23.
[^sf-types]: [Choosing workflow type in Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/choosing-workflow-type.html) — AWS documentation. Checked 2026-10-23.
[^sf-errors]: [Handling errors in Step Functions workflows](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html) — AWS documentation. Checked 2026-10-23.
[^sf-task]: [Task workflow state](https://docs.aws.amazon.com/step-functions/latest/dg/state-task.html) — AWS documentation. Checked 2026-10-23.
[^sf-quotas]: [Step Functions service quotas](https://docs.aws.amazon.com/step-functions/latest/dg/service-quotas.html) — AWS documentation. Checked 2026-10-23.
[^sf-versions]: [State machine versions in Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-state-machine-version.html) — AWS documentation. Checked 2026-10-23.
[^sf-update]: [UpdateStateMachine](https://docs.aws.amazon.com/step-functions/latest/apireference/API_UpdateStateMachine.html) — AWS Step Functions API reference. Checked 2026-10-23.
[^sf-callback]: [Discover service integration patterns in Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html) — AWS documentation. Checked 2026-10-23.
[^sf-metrics]: [Monitoring Step Functions metrics using Amazon CloudWatch](https://docs.aws.amazon.com/step-functions/latest/dg/procedure-cw-metrics.html) — AWS documentation. Checked 2026-10-23.
[^sf-pricing]: [AWS Step Functions Pricing](https://aws.amazon.com/step-functions/pricing/) — AWS, as of October 2026. Checked 2026-10-23.
[^l-durable]: [Lambda durable functions](https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html) — AWS documentation. Checked 2026-10-23.
[^l-blog]: [Building fault-tolerant long-running applications with AWS Lambda durable functions](https://aws.amazon.com/blogs/compute/building-fault-tolerant-long-running-application-with-aws-lambda-durable-functions/) — Rahul Pisal and Ben Freiberg, AWS Compute Blog, 2026-02-06. Checked 2026-10-23.
[^l-pricing]: [AWS Lambda Pricing (durable functions section)](https://aws.amazon.com/lambda/pricing/) — AWS, as of October 2026. Checked 2026-10-23.
[^gcw]: [Workflows overview](https://cloud.google.com/workflows/docs/overview) — Google Cloud documentation. Checked 2026-10-23.
[^df-constraints]: [Orchestrator function code constraints](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-code-constraints) — Microsoft Learn. Checked 2026-10-23.
[^df-version]: [Versioning in Durable Functions](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-versioning) — Microsoft Learn. Checked 2026-10-23.
[^af-bp]: [Best Practices](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html) — Apache Airflow 3.3.2 documentation. Checked 2026-10-23.
[^af-dagrun]: [Dag Runs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html) — Apache Airflow 3.3.2 documentation. Checked 2026-10-23.
[^af-backfill]: [Backfill](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/backfill.html) — Apache Airflow 3.3.2 documentation. Checked 2026-10-23.
[^af-tasks]: [Tasks](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html) — Apache Airflow 3.3.2 documentation. Checked 2026-10-23.
[^af-bundles]: [Dag Bundles](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/dag-bundles.html) — Apache Airflow 3.3.2 documentation. Checked 2026-10-23.
[^ar-retry]: [Retries](https://argo-workflows.readthedocs.io/en/latest/retries/) — Argo Workflows documentation. Checked 2026-10-23.
[^ar-backoff]: [examples/retry-backoff.yaml](https://github.com/argoproj/argo-workflows/blob/main/examples/retry-backoff.yaml) — argoproj/argo-workflows, GitHub. Checked 2026-10-23.
[^ar-exit]: [Exit handlers](https://argo-workflows.readthedocs.io/en/latest/walk-through/exit-handlers/) — Argo Workflows documentation. Checked 2026-10-23.
[^ar-cost]: [Cost optimisation](https://argo-workflows.readthedocs.io/en/latest/cost-optimisation/) — Argo Workflows documentation. Checked 2026-10-23.
[^az-saga]: [Saga distributed transactions pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/saga) — Microsoft Azure Architecture Center. Checked 2026-10-23.
[^az-comp]: [Compensating Transaction pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction) — Microsoft Azure Architecture Center. Checked 2026-10-23.
[^bl-jitter]: [Timeouts, retries, and backoff with jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) — Marc Brooker, Amazon Builders' Library. Checked 2026-10-23.
[^bl-idem]: [Making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) — Malcolm Featonby, Amazon Builders' Library. Checked 2026-10-23.
[^sre-cascade]: [Addressing Cascading Failures](https://sre.google/sre-book/addressing-cascading-failures/) — *Site Reliability Engineering*, Google, 2016. Checked 2026-10-23.
[^aws-ddb]: [Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/101925/) — AWS, October 2025. Checked 2026-10-23.
[^t-wffail]: [Detecting Workflow failures](https://docs.temporal.io/encyclopedia/detecting-workflow-failures) — Temporal documentation. Checked 2026-10-23.
[^sf-start]: [StartExecution](https://docs.aws.amazon.com/step-functions/latest/apireference/API_StartExecution.html) — AWS Step Functions API reference. Checked 2026-10-23.
[^sf-intrinsic]: [Intrinsic functions for JSONPath states in Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/intrinsic-functions.html) — AWS documentation. Checked 2026-10-23.
[^sf-redrive]: [Restarting state machine executions with redrive](https://docs.aws.amazon.com/step-functions/latest/dg/redrive-executions.html) — AWS documentation. Checked 2026-10-23.
[^stripe-idem]: [Idempotent requests](https://docs.stripe.com/api/idempotent_requests) — Stripe API reference (API v1), as of October 2026. Checked 2026-10-23.
[^af-config]: [Configuration Reference](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html) — Apache Airflow 3.3.2 documentation (`default_task_retries`, `default_task_retry_delay`). Checked 2026-10-23.
[^ar-fields]: [Field Reference (PodGC strategy)](https://argo-workflows.readthedocs.io/en/latest/fields/) — Argo Workflows documentation. Checked 2026-10-23.
[^cam-migrate]: [Process instance migration](https://docs.camunda.io/docs/components/concepts/process-instance-migration/) — Camunda 8 documentation, version 8.10. Checked 2026-10-23.
[^cam-version]: [Versioning process definitions](https://docs.camunda.io/docs/components/best-practices/operations/versioning-process-definitions/) — Camunda 8 documentation (best practices), version 8.10. Checked 2026-10-23.
[^cam-licence]: [Licensing](https://docs.camunda.io/docs/reference/licenses/) — Camunda 8 documentation. Checked 2026-10-23.
[^flowable]: [flowable/flowable-engine release flowable-8.0.0](https://github.com/flowable/flowable-engine/releases/tag/flowable-8.0.0) — Flowable on GitHub, 2026-02-27 (repository licence Apache-2.0). Checked 2026-10-23.

**Further reading**

- Temporal's [Temporal overview](https://docs.temporal.io/temporal) for the durable-execution vocabulary used on this page.
- Neighbouring Atlas pages: [DataOps](../dataops/), [IntegrationOps](../integrationops/), [AutomationOps](../automationops/),
  [ServerlessOps](../serverlessops/), [ObservabilityOps](../observabilityops/).

**Changelog**

- 2026-10-23 — First draft: term status confirmed as ambiguous; research ledger `research/workflowops-ledger.md`; Labs 1–3 executed
  with Python 3.8.10; other examples syntax-checked only.
- 2026-10-23 — Round 1 revision after the independent review (`reviews/workflowops-r1.md`, response `reviews/workflowops-r1-response.md`):
  pivot outcomes (rejected / unknown / failure after it) fixed in text and both implementations; per-step keys and key retention;
  decision tree routes compensation needs to an engine; BPMN engines and instance migration added; 16 minor fixes; length trimmed;
  labs re-run.
- 2026-10-23, post-score polish: Argo's "transient" errors defined (a container's non-zero exit needs `OnFailure`); ASL compensations
  end in a `Pass` state so `ExecutionsFailed` counts only real failures; manual refund before answering a refund park stated; small
  trims to stay in the length band (`reviews/workflowops-polish.md`).