WorkflowOps

WorkflowOps, as this Atlas uses the word, is the practice of designing, running and changing workflow engines and the long-running processes they execute, so that those processes finish correctly even when steps fail, workers crash, people take days to approve, and the code changes while thousands of instances are still running.

A workflow is a multi-step process with state: reserve stock, charge the card, book a courier, wait for dispatch. A workflow engine (Temporal, AWS Step Functions, Apache Airflow, Argo Workflows, Azure Durable Functions and others) stores each instance’s progress, retries steps, fires timers and resumes after a crash. WorkflowOps is everything around the engine that decides whether the business outcome is right: retry and timeout policy, idempotency, compensation, versioning, backfills, observability and runbooks.

For backend and platform engineers who own an engine, data engineers whose schedulers behave like one, architects choosing between a queue, a scheduler and a durable engine, and SREs who are paged when instances pile up.

A process that spans several systems and minutes to months cannot be one database transaction. The 1987 paper that named the saga described the problem precisely: long lived transactions “hold on to database resources for relatively long periods of time”, so the authors proposed writing such a transaction as “a sequence of transactions that can be interleaved with other transactions”, where the system guarantees “either all the transactions in a saga are successfully completed or compensating transactions are run to amend a partial execution”.^[3] Almost every problem on this page is a modern version of that sentence.

Without an engine and the practices around it, teams hand-build status columns, cron sweepers for “stuck orders” and fixed-sleep retry loops. The failures are predictable:

• Double effects. A worker charges a card, crashes before recording success, and the retry charges again.
• Half-finished processes. Stock is reserved and payment taken, but shipping failed and nothing undoes the first two steps.
• Retry storms. Every layer retries, so a struggling dependency receives many times its normal load.
• Deploys that break running work. New code assumes a step that instances started last week never ran.
• Silent stalls. An instance waits forever for a callback, and no metric counts it.

An engine solves the bookkeeping (durable state, timers, retries, history), not correctness. That remainder is WorkflowOps.

• Not BPMN modelling. OMG’s BPMN 2.0.2 is “the de-facto standard for business processes diagrams”;^[4] modelling a process with stakeholders is a different job. Engines that execute BPMN (Camunda 8, Flowable) are in scope as engines: this page starts when a process runs.
• Not data pipeline quality (DataOps): Airflow appears here only as an engine.
• Not messaging (IntegrationOps): a workflow often uses a queue; the queue is not the workflow.
• Not runbook automation (AutomationOps): that page owns scripts and their guards; this one owns the engine that may run them as steps.

There is no history of “WorkflowOps” as a movement, only of the techniques:

Mental model: the engine is a ledger, the steps are promises. The engine keeps an exact ledger of what each instance has done. Each step is a promise to an outside system, and the outside system does not read your ledger. Correctness therefore lives at the boundary: every step must be safe to repeat, and every promise that cannot be kept needs a defined undo or a human owner.

flowchart TB
  WF["WorkflowOps: engine semantics, retries, sagas, versioning running work"]
  D["DataOps: what pipelines compute, data quality"]
  I["IntegrationOps: brokers, message replay, schemas"]
  A["AutomationOps: runbook scripts and their guards"]
  S["ServerlessOps: one function, its events and quotas"]
  O["ObservabilityOps: telemetry pipelines and tools"]
  M["MLOps: training and serving lifecycle"]
  D -->|"runs on an engine"| WF
  M -->|"training pipelines as workflows"| WF
  A -->|"long multi-step jobs"| WF
  WF -->|"uses queues and events"| I
  WF -->|"steps may be functions"| S
  WF -->|"emits run metrics and traces"| O
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class WF core
  class D,I,A,S,O,M neutral

The charter below is this Atlas’s proposal; we found no published WorkflowOps manifesto.

Mission: every workflow instance ends in a known, correct state — completed, compensated, or parked with an owner — within a stated deadline, across failures and deployments.

1. Every step may run more than once: workers die between doing and recording. Temporal Activities “may be executed more than once”;^[16] asynchronous Step Functions Express workflows are at-least-once.^[17]
2. Every external effect carries an idempotency key derived from the instance, and the receiving system records key and effect atomically.^[13]
3. Retry at one layer, with backoff, jitter and a cap.^[18][19]
4. Every wait has a deadline and an owner. Defaults are unlimited or very long (Temporal Schedule-To-Close ∞; Step Functions TimeoutSeconds 99,999,999 seconds).^[20][21]
5. Know which steps can be undone. Mark compensable steps, the pivot and retryable steps before writing code.^[15]
6. Running instances are a compatibility contract: a deployment is finished when old instances have drained or been migrated.
7. Measure outcomes per instance, not only worker health.

Non-goals: modelling processes with stakeholders; exactly-once delivery between systems (not available end to end); deciding refund policy (a business rule).

R = responsible, A = accountable, C = consulted, I = informed. The key split: the platform team owns the engine; the workflow owner owns the correctness of each workflow. When a platform team is asked to “fix stuck orders”, ownership is already wrong.

Design (Day 0): the design record (Templates). Build: test steps, inject failures, replay recorded histories (replay engines).^[22] Release (Day 1): keep old versions until running instances drain. Operate (Day 2): watch the metrics, run the runbooks and backfills. Change: classify every change (Versioning running workflows). Retire: stop new starts, drain or migrate the rest, delete histories per retention.

• One design record per workflow, reviewed like an API change, including retry and timeout changes (they move load onto other teams).
• A versioning gate in the deployment pipeline for replay-based engines: recorded histories from production replay cleanly against the new build, or the deploy stops.
• A weekly stuck-and-parked review: every instance older than its deadline, every compensation that failed, every manual repair.
• Bulk operations (reset, terminate, redrive, backfill) need a second person and a written scope query.
• Escalation: business-impacting failure rates page the workflow owner; engine health (task backlog, persistence errors) pages the platform team.

flowchart TB
  C["Callers: API, events, schedules, operators"]
  E["Workflow engine: history store, timers, task queues"]
  W["Workflow workers: deterministic code, no I/O"]
  S["Step workers: I/O with idempotency keys"]
  X["External systems: payments, stock, couriers"]
  P["Human queue: approvals, parked instances"]
  O["Telemetry: metrics, traces, run search"]
  C -->|"start, signal, cancel"| E
  E <-->|"workflow tasks, commands"| W
  E <-->|"step tasks, results"| S
  S -->|"calls with keys"| X
  E -->|"waits, callbacks"| P
  E -->|"run data"| O
  S -->|"spans, metrics"| O
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class E,W core
  class C,S,O neutral
  class X,P risk

Component responsibilities and trust boundaries.

• Engine. Owns ordering, timers and history; its database is critical state (back it up, size it, respect history limits).
• Workflow code. In replay-based engines it must be deterministic: no clocks, random numbers or network calls except through the engine’s APIs.^[11][12] Where the definition is data (Step Functions, Argo, BPMN), the risks move to data size and expression errors.
• Step code. The only place with side effects; the trust boundary to other systems. Credentials live here, scoped per step.
• Payload size. Keep large data out of the history: Step Functions has a 256 KiB payload quota; Temporal Cloud limits a single request to 2 MB.^[23][24] Pass references (object keys), not documents.

Alternatives: see Deep comparisons and the decision framework.

The illustrative order-fulfilment workflow used throughout this page: reserve stock → charge card → book courier → wait for dispatch confirmation (up to 48 hours) → complete.

sequenceDiagram
  autonumber
  participant E as Engine
  participant W as Worker
  participant P as Payment API
  E->>W: task: charge order A1
  W->>P: charge 100, key charge-A1
  P-->>W: charged
  Note over W: worker crashes before reporting
  Note over E: start-to-close timeout fires
  E->>W: retry task: charge order A1
  W->>P: charge 100, key charge-A1
  P-->>W: same result, no second charge
  W-->>E: completed

Happy path. Trigger (order placed) → engine starts instance order-A1 → three steps succeed → a timer waits for the dispatch signal → completed; metrics record duration and attempts. The order ID is the workflow ID, which blocks duplicates only under conditions. Temporal allows one open execution per ID; under the default reuse policy, Allow Duplicate, a late duplicate (a webhook retried after completion) starts a second order, so set Reject Duplicate, which checks closed executions within the namespace’s retention period.^[25] Step Functions Standard returns the original response to a start with the same name and input while the execution runs and ExecutionAlreadyExists for 90 days after it closes.^[26] Older duplicates need a business-level check.

Exception paths.

1. Transient step failure (timeout, HTTP 503): retried by policy with backoff; no human involvement.
2. Crash after a side effect (diagram above): the engine re-issues the task; the idempotency key prevents a second effect.
3. Permanent failure before the pivot, or pivot rejected (card declined, courier refuses): compensate completed steps, end as compensated.
4. Pivot outcome unknown (courier API down for hours, or the booking call timed out): keep retrying with the same key while the business deadline allows, then query the booking by its key. Booked → continue; not booked → compensate; still unknown → a person decides. Never compensate on a guess: the courier may already hold the booking.
5. Failure after the pivot (confirmation service down): retry forward within a bounded window, then park; never compensate.
6. Compensation fails (refund API rejects): park the instance with full context for a human; never drop it.
7. Wait never ends (no dispatch signal): the timer fires at 48 hours and routes to a person.

Six playbooks follow. The workflow owner runs all of them (with the downstream owner for keys and the business owner for compensation), and each ends in a test: fault injection, a duplicate call, a failure at each saga step, a CI replay, a dry backfill list, and a history-size metric.

Engines differ sharply in their defaults, and the defaults decide how a failure behaves:

Rules that hold across engines:

1. Classify errors. Validation errors, declined cards and “not found” are non-retryable; timeouts, throttling and 5xx are retryable. Temporal and Step Functions both let you name non-retryable errors; Airflow 3.3 can fail fast on an exception type.^[27][28][30]
2. Retry at one layer. In AWS’s example, five layers that each make three tries multiply database load 243 times (3^5); retry “at a single point in the stack”^[18] (the arithmetic is in IntegrationOps). Count attempts, not retries: Step Functions MaxAttempts: 3 means four tries, and the SRE book’s three retries per layer give 4^3 = 64 attempts.^[19] If the engine retries a step, keep SDK retries inside it small, and use a retry budget.
3. Add jitter and a cap. Step Functions jitter is off unless JitterStrategy is FULL.^[28]
4. Bound the total. Use an overall deadline (Schedule-To-Close, MaxAttempts, maxDuration) per step, plus a workflow deadline (a timer in replay engines).
5. Heartbeat long steps. A step that runs for minutes should report heartbeats so a dead worker is detected in seconds, not at the end of a long timeout (Temporal heartbeat timeout; Step Functions HeartbeatSeconds).^[20][21]

An engine guarantees that it records each step’s result once; it cannot guarantee that the step’s effect happened once. Temporal says it plainly: lack of idempotency “might affect the correctness of your application but does not affect the Temporal Platform”.^[16]

1. Derive the key from the instance, not the attempt: charge-<order_id>, or workflow ID + step name. A random key per attempt defeats the purpose.
2. The receiver records key and effect atomically. Amazon’s guidance: recording the token and the mutation must be one ACID operation, and a retry should receive a “semantically equivalent” response rather than an error such as “already exists”.^[13]
3. Where the receiver has no key support, check-then-act is not enough: write conditionally (compare-and-set on a version). In the October 2025 DynamoDB outage, an automation component’s one-time check that its plan was newer “was stale by this time”, so an older plan overwrote a newer one, and a clean-up left “an inconsistent state that prevented subsequent plan updates”.^[34] A check at the start of a slow, retried operation is not a guard at its end.
4. Make data tasks overwrite, not append. Airflow’s best practices: treat tasks “equivalent to transactions in a database”, replace INSERT with UPSERT, and read a specific partition rather than “the latest available data”.^[35]
5. Keep retries inside the receiver’s key retention. Amazon’s guidance returns an equivalent response “for some interval”;^[13] Stripe’s API v1 may prune keys at least 24 hours old, after which a reused key creates a new request, and it rejects a reused key with different parameters (October 2026).^[36] Use one key per step (refund-<order_id>, never the charge’s key), keep each step’s retry and park horizon inside the window, and after a longer pause query by key before resending.

flowchart TB
  S1["Reserve stock (compensable)"] --> S2["Charge card (compensable)"]
  S2 --> PV{"Book courier: pivot"}
  PV -->|"booked"| R1["Send confirmation (retry forward)"]
  R1 --> DONE["Completed"]
  PV -->|"outage or timeout"| ST{"Retry, then query status by key"}
  ST -->|"booked"| R1
  ST -->|"not booked"| C1
  PV -->|"rejected"| C1["Refund card"]
  C1 --> C2["Release stock"]
  C2 --> COMP["Compensated"]
  ST -->|"still unknown"| PARK["Parked: human queue with context"]
  C1 -->|"refund keeps failing"| PARK
  C2 -->|"release keeps failing"| PARK
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class S1,S2,R1,DONE,COMP core
  class PV,ST decision
  class C1,C2,PARK risk

Microsoft’s saga guidance names the three kinds of step this diagram uses: compensable transactions that can be undone, a pivot that is “the point of no return”, and retryable transactions after it, which are idempotent so the saga can always finish.^[15] Practical rules:

• Put the pivot as late as possible, after reversible checks (fraud score, address validation).
• Retry before compensating, as Microsoft’s guidance advises (“retries first to preserve forward progress”).^[14] A pivot with an unknown outcome is resolved by a status query, not by compensation: a timed-out booking may have succeeded, so ask the receiver by key, and send a person when the receiver cannot answer.
• Compensation is a business action, not a rollback. A refund may be partial; a cancelled booking may carry a fee. The business owner signs off the compensation table.^[14]
• Compensations can fail. Record progress so compensation resumes where it stopped, make each one idempotent, and park with full detail when only a person can finish.^[14]
• Order need not be strictly reverse: undo first in the system most sensitive to inconsistency.^[14]
• Sagas lack isolation. Other processes can see intermediate states: lost updates, dirty reads and fuzzy reads are possible; countermeasures include semantic locks (a “pending” flag) and reordering so updates happen in retryable steps.^[15]
• Orchestration or choreography? An orchestrator (the workflow engine) holds the state and decides; in choreography each service reacts to the others’ events and no component holds the whole process.^[15] Choreography suits a few steps owned by separate teams with no central deadline; once you need compensation order, timers, a human queue or one place to ask “where is order A1?”, an orchestrator is easier to operate. This page assumes orchestration; brokers and event delivery are in IntegrationOps.

The hardest WorkflowOps problem is changing a workflow while instances started on the old code still run. Engines take two approaches:

• Definition-as-data engines keep each execution on its definition. Updating a Step Functions state machine leaves running executions on “the previous definition and roleArn”; versions and aliases let you shift new executions gradually.^[37][38] Airflow 3 with a versioned Dag bundle (GitDagBundle) runs a Dag run on the same code throughout; local, S3 and GCS bundles always use the latest code.^[39]
• Replay engines re-run your code against old histories. Changing the order or kind of engine commands breaks determinism: Temporal’s docs show a timer-then-activity workflow changed to activity-then-timer failing with a “nondeterminism error” on replay.^[11] Azure Durable Functions warns that deploying breaking changes with no strategy can leave orchestrations failing or “stuck indefinitely in a Running status”.^[40]

flowchart TB
  Q1{"Changes the command sequence of running instances?"}
  Q1 -->|"no: step internals, logging"| A1["Deploy normally; replay test in CI"]
  Q1 -->|"yes"| Q2{"Do old instances finish soon (hours)?"}
  Q2 -->|"yes"| A2["Pin old instances to the old version; drain, then retire it"]
  Q2 -->|"no: days or months"| Q3{"Must old instances redo a part they already passed?"}
  Q3 -->|"no"| A3["Patch at the change point: runs already past it keep the old path"]
  Q3 -->|"yes"| A4["New workflow type or migration: signal, reset or restart with state"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class Q1,Q2,Q3 decision
  class A1,A2,A3 core
  class A4 risk

Engine mechanisms, as documented in October 2026:

Airflow 3 does not catch up missed intervals by default (catchup_by_default=False), and catchup also fires when a paused Dag is re-enabled.^[46] A backfill is explicit and scoped:

Example status: syntax-checked, not executed (requires an Airflow 3.3 deployment; command form from the Airflow docs).

# Re-run only failed or missing runs for one week, three at a time, newest first.
airflow backfill create --dag-id orders_daily \
  --from-date 2026-06-01 \
  --to-date 2026-06-07 \
  --reprocess-behavior failed \
  --max-active-runs 3 \
  --run-backwards

--reprocess-behavior accepts none, failed or completed; runs that are still queued or running are never duplicated.^[47] Agree concurrency with the downstream owners: a backfill is a load test on every system it touches.

Every event of an instance is stored and replayed. Temporal’s default limits apply on every deployment: 51,200 events or 50 MB of history, with a warning at 10,240 events or 10 MB; they are configurable when self-hosted and fixed on Temporal Cloud. Step Functions fails an execution at 25,000 events.^[48][24][23] For loops, subscriptions and polling, continue-as-new closes the execution and starts a fresh one with the same workflow ID and a new run ID, carrying only the state it needs;^[49] alternatively, split work into child workflows. Never poll in a tight loop inside a workflow: each poll is history.

None of these is mandatory. There is no WorkflowOps certification or standards body.

Vendor-neutral categories first, examples second. Versions and limits are as of October 2026.

Other code-first durable execution engines exist; this page does not rank them, and the same criteria apply. Selection criteria are in the decision framework; add one often missed: managed engines store your step inputs and outputs in their history, so data residency applies to them.

• AI agents as workflows. AWS positions Lambda durable functions for “AI workflows” that chain model calls and human feedback,^[10] and Temporal places LLM invocations in Activities because they are non-deterministic.^[27] Model calls are steps with timeouts, retries and cost caps.
• Assisted triage (summarising a failed instance’s history) should link to the raw history; bulk operations stay behind the two-person rule.

The same order saga in two engines, plus a Kubernetes batch step. Assumptions: the order ID is unique and becomes the workflow ID or execution name; every downstream API accepts an idempotency key; refunds are full refunds (a business decision).

Temporal (Python SDK). Each compensation is registered before its step runs, because a step that timed out may still have taken effect. At the pivot, CourierRejected leads to compensation, while an outage is retried for up to 12 hours with the same key and then settled by a status query or a person, never by a refund. The confirmation after it is retried for up to a day, then parked. Each park waits for a resolve signal for its own item and sends a daily reminder: a timer, not a workflow timeout, which Temporal generally advises against.^[57]

Example status: syntax-checked, not executed (requires a Temporal server and workers; API names checked against the Python SDK 1.34.0 source).

import asyncio
from datetime import timedelta
from typing import Optional

from temporalio import activity, workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError, ApplicationError

STEP_RETRY = RetryPolicy(
    initial_interval=timedelta(seconds=2),
    maximum_interval=timedelta(minutes=1),
    maximum_attempts=8,
    non_retryable_error_types=["CardDeclined", "OutOfStock"],
)
# Unlimited attempts, but only together with a schedule-to-close deadline.
PIVOT_RETRY = RetryPolicy(
    initial_interval=timedelta(seconds=30),
    maximum_interval=timedelta(minutes=15),
    non_retryable_error_types=["CourierRejected"],
)
FORWARD_RETRY = RetryPolicy(maximum_interval=timedelta(minutes=10))
UNDO_RETRY = RetryPolicy(maximum_interval=timedelta(minutes=5), maximum_attempts=20)


@activity.defn
async def reserve_stock(order_id: str) -> str:
    return "reserved"  # call stock API with key f"reserve-{order_id}"


@activity.defn
async def release_stock(order_id: str) -> str:
    return "released"  # key f"release-{order_id}"; releasing twice is a no-op


@activity.defn
async def charge_card(order_id: str) -> str:
    return "charged"  # call payment API with key f"charge-{order_id}"


@activity.defn
async def refund_card(order_id: str) -> str:
    return "refunded"  # key f"refund-{order_id}"; no-op if never charged


@activity.defn
async def fraud_check(order_id: str) -> str:
    return "ok"


@activity.defn
async def book_courier(order_id: str) -> str:
    return "booked"  # key f"courier-{order_id}"; raises CourierRejected if refused


@activity.defn
async def courier_status(order_id: str) -> str:
    return "booked"  # look up key f"courier-{order_id}": booked/not_booked/unknown


@activity.defn
async def send_confirmation(order_id: str) -> str:
    return "sent"


@activity.defn
async def open_ticket(subject: str) -> str:
    return "opened"  # human queue; a person later sends the resolve signal


def rejected(err: ActivityError) -> bool:
    cause = err.cause
    return isinstance(cause, ApplicationError) and cause.type == "CourierRejected"


@workflow.defn
class OrderWorkflow:
    def __init__(self) -> None:
        self.decisions = {}  # parked item -> a person's decision

    @workflow.signal
    def resolve(self, item: str, decision: str) -> None:
        self.decisions[item] = decision

    async def step(
        self, fn, arg: str, retry: RetryPolicy = STEP_RETRY,
        limit: Optional[timedelta] = None,
    ) -> str:
        return await workflow.execute_activity(
            fn,
            arg,
            start_to_close_timeout=timedelta(seconds=30),
            schedule_to_close_timeout=limit,
            retry_policy=retry,
        )

    async def park(self, order_id: str, item: str) -> str:
        await self.step(open_ticket, f"{order_id}: {item}")
        while item not in self.decisions:
            try:
                await workflow.wait_condition(
                    lambda: item in self.decisions, timeout=timedelta(days=1)
                )
            except asyncio.TimeoutError:
                await self.step(open_ticket, f"{order_id}: {item} (reminder)")
        return self.decisions[item]

    async def compensate(self, undo: list, order_id: str) -> str:
        for comp in reversed(undo):
            try:
                await self.step(comp, order_id, UNDO_RETRY)
            except ActivityError:
                await self.park(order_id, comp.__name__)  # one decision per item
        return "compensated"

    async def pivot_status(self, order_id: str) -> str:
        try:
            status = await self.step(courier_status, order_id)
        except ActivityError:
            status = "unknown"
        if status == "unknown":  # never guess: a person checks with the courier
            status = await self.park(order_id, "courier booking")
        return status

    @workflow.run
    async def run(self, order_id: str) -> str:
        undo = []
        try:
            undo.append(release_stock)
            await self.step(reserve_stock, order_id)
            if workflow.patched("fraud-check-2026-10"):
                await self.step(fraud_check, order_id)
            undo.append(refund_card)
            await self.step(charge_card, order_id)
        except ActivityError:
            return await self.compensate(undo, order_id)
        # The pivot: a rejection is definitive; a timeout or an outage is not.
        try:
            await self.step(book_courier, order_id, PIVOT_RETRY, timedelta(hours=12))
            outcome = "booked"
        except ActivityError as err:
            outcome = "not_booked" if rejected(err) else await self.pivot_status(order_id)
        if outcome != "booked":
            return await self.compensate(undo, order_id)
        # After the pivot: forward only. Bounded retries, then a person.
        try:
            await self.step(send_confirmation, order_id, FORWARD_RETRY, timedelta(days=1))
        except ActivityError:
            await self.park(order_id, "send_confirmation")
        return "completed"

Start it with the order ID as workflow ID and the reuse policy REJECT_DUPLICATE (the client’s id_reuse_policy argument; see the happy path).

AWS Step Functions (Standard). The same saga as data. Each Task builds its own key from the execution name (the order ID) with States.Format, so a refund never reuses the charge’s key.^[58] The pivot retries an outage for hours; CourierRejected goes to the refund, any other final error to a status query and then, if still unknown, to a person through .waitForTaskToken (a message with the task token goes to a queue; an operator’s tool calls SendTaskSuccess).^[59] Every Task has a timeout, and a compensation that keeps failing parks the execution.

Example status: JSON syntax-checked, not deployed (placeholders REGION and ACCOUNT_ID; Lambda functions and the SQS queue not shown).

{
  "Comment": "Order saga (illustrative). Execution name = order ID.",
  "StartAt": "ReserveStock",
  "TimeoutSeconds": 2592000,
  "States": {
    "ReserveStock": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:reserve-stock",
        "Payload": {
          "order.$": "$.order",
          "key.$": "States.Format('reserve-{}', $$.Execution.Name)"
        }
      },
      "ResultPath": "$.reserve",
      "TimeoutSeconds": 30,
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException",
                          "States.Timeout"],
          "IntervalSeconds": 2, "MaxAttempts": 5, "BackoffRate": 2.0,
          "MaxDelaySeconds": 60, "JitterStrategy": "FULL"
        }
      ],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
                  "Next": "ReleaseStock" }],
      "Next": "ChargeCard"
    },
    "ChargeCard": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:charge-card",
        "Payload": {
          "order.$": "$.order",
          "key.$": "States.Format('charge-{}', $$.Execution.Name)"
        }
      },
      "ResultPath": "$.charge",
      "TimeoutSeconds": 30,
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "States.Timeout"],
          "IntervalSeconds": 2, "MaxAttempts": 5, "BackoffRate": 2.0,
          "MaxDelaySeconds": 60, "JitterStrategy": "FULL"
        }
      ],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
                  "Next": "RefundCard" }],
      "Next": "BookCourier"
    },
    "BookCourier": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:book-courier",
        "Payload": {
          "order.$": "$.order",
          "key.$": "States.Format('courier-{}', $$.Execution.Name)"
        }
      },
      "ResultPath": "$.booking",
      "TimeoutSeconds": 30,
      "Retry": [
        {
          "ErrorEquals": ["CourierUnavailable", "Lambda.ServiceException",
                          "States.Timeout"],
          "IntervalSeconds": 30, "MaxAttempts": 40, "BackoffRate": 2.0,
          "MaxDelaySeconds": 900, "JitterStrategy": "FULL"
        }
      ],
      "Catch": [
        { "ErrorEquals": ["CourierRejected"], "ResultPath": "$.error",
          "Next": "RefundCard" },
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
          "Next": "CheckCourier" }
      ],
      "Next": "Completed"
    },
    "CheckCourier": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:courier-status",
        "Payload": {
          "key.$": "States.Format('courier-{}', $$.Execution.Name)"
        }
      },
      "ResultSelector": { "status.$": "$.Payload.status" },
      "ResultPath": "$.courier",
      "TimeoutSeconds": 30,
      "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 10,
                  "MaxAttempts": 5, "BackoffRate": 2.0, "JitterStrategy": "FULL" }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
                  "Next": "AskCourierOutcome" }],
      "Next": "PivotOutcome"
    },
    "PivotOutcome": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.courier.status", "StringEquals": "booked",
          "Next": "Completed" },
        { "Variable": "$.courier.status", "StringEquals": "not_booked",
          "Next": "RefundCard" }
      ],
      "Default": "AskCourierOutcome"
    },
    "AskCourierOutcome": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
      "Parameters": {
        "QueueUrl": "https://sqs.REGION.amazonaws.com/ACCOUNT_ID/parked-orders",
        "MessageBody": { "question": "Was the courier booked?", "input.$": "$",
                         "taskToken.$": "$$.Task.Token" }
      },
      "ResultPath": "$.courier",
      "TimeoutSeconds": 604800,
      "Next": "PivotOutcome"
    },
    "RefundCard": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:refund-card",
        "Payload": {
          "order.$": "$.order",
          "key.$": "States.Format('refund-{}', $$.Execution.Name)"
        }
      },
      "ResultPath": "$.refund",
      "TimeoutSeconds": 30,
      "Retry": [
        {
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 5, "MaxAttempts": 10, "BackoffRate": 2.0,
          "MaxDelaySeconds": 300, "JitterStrategy": "FULL"
        }
      ],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
                  "Next": "ParkForHuman" }],
      "Next": "ReleaseStock"
    },
    "ParkForHuman": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
      "Parameters": {
        "QueueUrl": "https://sqs.REGION.amazonaws.com/ACCOUNT_ID/parked-orders",
        "MessageBody": { "input.$": "$", "taskToken.$": "$$.Task.Token" }
      },
      "ResultPath": "$.human",
      "TimeoutSeconds": 604800,
      "Next": "ReleaseStock"
    },
    "ReleaseStock": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:REGION:ACCOUNT_ID:function:release-stock",
        "Payload": {
          "order.$": "$.order",
          "key.$": "States.Format('release-{}', $$.Execution.Name)"
        }
      },
      "ResultPath": "$.release",
      "TimeoutSeconds": 30,
      "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 5,
                  "MaxAttempts": 10, "BackoffRate": 2.0, "JitterStrategy": "FULL" }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error",
                  "Next": "ParkForHuman" }],
      "Next": "Compensated"
    },
    "Compensated": { "Type": "Pass", "Result": "compensated",
                     "ResultPath": "$.outcome", "End": true },
    "Completed": { "Type": "Succeed" }
  }
}

Cautions: States.ALL does not catch States.Runtime errors, so alert on failed executions.^[28] The 30-day execution timeout is a backstop; the 48-hour objective is measured as a KPI. A park unanswered for seven days fails the execution, which should page the owner; redrive can resume it within 14 days.^[60] ParkForHuman returns to the idempotent ReleaseStock, so one park state serves both compensations; after a failed refund the operator refunds by hand before returning the token. Compensated is a Pass end state (outcome: compensated), so ExecutionsFailed counts only real failures.^[61] A long park can outlive the receiver’s key retention (Idempotency, rule 5).

Argo Workflows (a batch step on Kubernetes). Retry only transient errors with capped backoff, bound the whole run, always run an exit handler, and clean up finished workflows. podGC: OnPodSuccess deletes pods that succeeded but keeps failed ones, whose logs you need to troubleshoot a failed reconcile; OnPodCompletion would delete those too unless logs are archived. Argo’s “transient” means its own list (mainly Kubernetes API, quota and network errors) or TRANSIENT_ERROR_PATTERN matches; a container exiting non-zero after a downstream 503 needs OnFailure (or Always).^{[31][33][62][32][63]}

Example status: YAML syntax-checked, not executed (requires Argo Workflows on a Kubernetes cluster; image tag illustrative).

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: nightly-reconcile-
spec:
  entrypoint: reconcile
  onExit: report
  arguments:
    parameters:
      - name: business-date
        value: "2026-10-22"
  activeDeadlineSeconds: 7200
  ttlStrategy:
    secondsAfterCompletion: 86400
  podGC:
    strategy: OnPodSuccess
  templates:
    - name: reconcile
      retryStrategy:
        limit: "5"
        retryPolicy: OnTransientError
        backoff:
          duration: "10s"
          factor: "2"
          cap: "5m"
          maxDuration: "30m"
      container:
        image: registry.example.com/reconcile:1.4.2
        args: ["--date", "{{workflow.parameters.business-date}}"]
    - name: report
      container:
        image: registry.example.com/notify:1.0.0
        args: ["--status", "{{workflow.status}}"]

The labs use wfdemo.py, a small teaching model of durable execution (standard-library Python, SQLite), not a production engine: a journal of step results, replay after a crash, recorded failures, compensation and a Temporal-style patch marker.

• Prerequisites: Python 3.8 or later; a terminal; no network or accounts. Cost and risk: none; files stay in a temporary directory.
• Execution status: all three labs and the Lab 2 extension were executed on 2026-10-23 with Python 3.8.10 on Ubuntu; the outputs below are copied from those runs.

Example status: executed (Python 3.8.10), see above.

"""wfdemo.py - a teaching model of durable execution (not a production engine)."""
import argparse
import os
import sqlite3
import sys

DB = os.environ.get("WFDEMO_DB", "wfdemo.sqlite")


class NonDeterminismError(Exception):
    """The code asked for a different step than the journal recorded."""


class StepFailed(Exception):
    """A step failed permanently (retries exhausted or non-retryable)."""


def db():
    con = sqlite3.connect(DB)
    con.executescript(
        "CREATE TABLE IF NOT EXISTS journal(wf TEXT, seq INT, step TEXT,"
        " result TEXT, PRIMARY KEY(wf, seq));"
        "CREATE TABLE IF NOT EXISTS charges(idem_key TEXT UNIQUE, order_id TEXT,"
        " amount INT);"
        "CREATE TABLE IF NOT EXISTS stock(order_id TEXT PRIMARY KEY);"
        "CREATE TABLE IF NOT EXISTS parked(wf TEXT, reason TEXT);"
    )
    return con


class Workflow:
    def __init__(self, con, wf_id):
        self.con, self.wf, self.seq = con, wf_id, 0
        rows = con.execute(
            "SELECT seq, step, result FROM journal WHERE wf=? ORDER BY seq",
            (wf_id,),
        ).fetchall()
        self.history = [(step, result) for _, step, result in rows]

    def replaying(self):
        return self.seq < len(self.history)

    def step(self, name, fn, retries=3):
        if self.replaying():
            recorded, result = self.history[self.seq]
            if recorded != name:
                raise NonDeterminismError(
                    f"seq {self.seq}: journal has {recorded!r}, code asked {name!r}"
                )
            self.seq += 1
            print(f"  replay  {name} -> {result}")
            if result.startswith("FAILED: "):  # failures are history too
                raise StepFailed(result[8:])
            return result
        failure = None
        for attempt in range(1, retries + 1):
            try:
                result = fn()
                break
            except StepFailed as exc:  # permanent: do not retry
                failure = str(exc)
                break
            except Exception as exc:  # transient: retry
                print(f"  attempt {attempt} of {name} failed: {exc}")
        else:
            failure = f"{name}: retries exhausted"
        if failure:
            result = "FAILED: " + failure
        if os.environ.get("CRASH_AFTER") == name:
            print(f"  CRASH after {name} ran, before the journal write")
            sys.stdout.flush()
            os._exit(3)  # simulate a killed worker: no cleanup, no journal
        self.con.execute(
            "INSERT INTO journal VALUES(?,?,?,?)", (self.wf, self.seq, name, result)
        )
        self.con.commit()
        self.seq += 1
        print(f"  run     {name} -> {result}")
        if failure:
            raise StepFailed(failure)
        return result

    def patched(self, patch_id):
        """Temporal-style patch marker: old runs keep the old path."""
        marker = "patch:" + patch_id
        if self.replaying():
            if self.history[self.seq][0] == marker:
                self.seq += 1
                return True
            return False
        self.step(marker, lambda: "marker")
        return True


# ---- "external systems" -------------------------------------------------
def charge(con, order_id, amount, use_key):
    key = f"charge-{order_id}" if use_key else None
    # One atomic statement records key and effect (no check-then-insert race).
    # A NULL key never collides, so the no-key run can charge twice.
    cur = con.execute(
        "INSERT OR IGNORE INTO charges VALUES(?,?,?)", (key, order_id, amount)
    )
    con.commit()
    return "charged" if cur.rowcount == 1 else "charged (duplicate suppressed)"


def refund(con, order_id):
    con.execute("DELETE FROM charges WHERE order_id=?", (order_id,))
    con.commit()
    return "refunded"


def reserve(con, order_id):
    con.execute("INSERT OR IGNORE INTO stock VALUES(?)", (order_id,))
    con.commit()
    return "reserved"


def release(con, order_id):
    con.execute("DELETE FROM stock WHERE order_id=?", (order_id,))
    con.commit()
    return "released"


def book_courier(order_id):
    if os.environ.get("COURIER") == "down":
        raise StepFailed("courier rejected the booking (non-retryable)")
    return "booked"


# ---- the workflow ---------------------------------------------------------
def order_workflow(wf, con, order_id, use_key, version):
    wf.step("reserve", lambda: reserve(con, order_id))
    if version == 2:  # unsafe change: a new step inserted for everyone
        wf.step("fraud_check", lambda: "ok")
    if version == 3 and wf.patched("fraud-check"):  # safe change
        wf.step("fraud_check", lambda: "ok")
    wf.step("charge", lambda: charge(con, order_id, 100, use_key))
    try:
        wf.step("courier", lambda: book_courier(order_id))
    except StepFailed as exc:
        print(f"  compensating: {exc}")
        try:
            wf.step("refund", lambda: refund(con, order_id))
            wf.step("release", lambda: release(con, order_id))
        except StepFailed as comp_exc:
            con.execute("INSERT INTO parked VALUES(?,?)", (wf.wf, str(comp_exc)))
            con.commit()
            return "parked for a human"
        return "compensated"
    return "completed"


def main():
    p = argparse.ArgumentParser()
    p.add_argument("cmd", choices=["run", "report", "reset"])
    p.add_argument("--order", default="A1")
    p.add_argument("--no-key", action="store_true", help="omit idempotency key")
    p.add_argument("--version", type=int, default=1)
    a = p.parse_args()
    if a.cmd == "reset":
        if os.path.exists(DB):
            os.remove(DB)
        print("removed", DB)
        return 0
    con = db()
    if a.cmd == "report":
        n = con.execute("SELECT COUNT(*) FROM charges WHERE order_id=?", (a.order,))
        s = con.execute("SELECT COUNT(*) FROM stock WHERE order_id=?", (a.order,))
        print(f"order {a.order}: charges={n.fetchone()[0]} stock={s.fetchone()[0]}")
        for row in con.execute("SELECT wf, seq, step FROM journal ORDER BY wf, seq"):
            print("  journal", row)
        return 0
    wf = Workflow(con, "order-" + a.order)
    try:
        outcome = order_workflow(wf, con, a.order, not a.no_key, a.version)
    except NonDeterminismError as exc:
        print("NON-DETERMINISM:", exc)
        return 2
    print("outcome:", outcome)
    return 0


if __name__ == "__main__":
    sys.exit(main())

Objective: see why replay alone does not prevent a double charge, and how an idempotency key does.

1. Create a working directory and save the file above as wfdemo.py in it.
2. Run without a key, crashing after the charge, then resume:

mkdir -p ~/wfdemo-lab && cd ~/wfdemo-lab   # save wfdemo.py here first
python3 wfdemo.py reset
CRASH_AFTER=charge python3 wfdemo.py run --order A1 --no-key; echo "exit=$?"
python3 wfdemo.py run --order A1 --no-key
python3 wfdemo.py report --order A1

3. Repeat with the key (drop --no-key).

Expected output (abridged from the executed run):

  run     reserve -> reserved
  CRASH after charge ran, before the journal write
exit=3
  replay  reserve -> reserved
  run     charge -> charged
  run     courier -> booked
outcome: completed
order A1: charges=2 stock=1        <- without the key: charged twice
...
  run     charge -> charged (duplicate suppressed)
order A1: charges=1 stock=1        <- with the key

Validation: charges=2 without the key, charges=1 with it. Replay skipped reserve; the journal had no record of charge, so it ran again. Troubleshooting: if the first command prints nothing, output was buffered; exit code 3 still confirms the crash. Teardown: python3 wfdemo.py reset.

Objective: run a saga to the compensated state and see that a rerun replays the failure instead of booking again.

python3 wfdemo.py reset
COURIER=down python3 wfdemo.py run --order B7
python3 wfdemo.py report --order B7
python3 wfdemo.py run --order B7      # same instance, courier now "up"

Expected output (journal lines of the report omitted):

  run     reserve -> reserved
  run     charge -> charged
  run     courier -> FAILED: courier rejected the booking (non-retryable)
  compensating: courier rejected the booking (non-retryable)
  run     refund -> refunded
  run     release -> released
outcome: compensated
order B7: charges=0 stock=0
  replay  reserve -> reserved
  replay  charge -> charged
  replay  courier -> FAILED: courier rejected the booking (non-retryable)
  compensating: courier rejected the booking (non-retryable)
  replay  refund -> refunded
  replay  release -> released
outcome: compensated

Validation: no charge or reservation remains, and the rerun changes nothing although the courier is now up: the history, not the current world, decides; real engines record failures as events for this reason. Extension: make refund raise StepFailed and confirm the outcome parked for a human and a row in the parked table. Teardown: python3 wfdemo.py reset.

Objective: break a running instance with an unsafe change, then ship the same change safely with a patch marker.

python3 wfdemo.py reset
CRASH_AFTER=courier python3 wfdemo.py run --order C3 --version 1   # in flight
python3 wfdemo.py run --order C3 --version 2; echo "exit=$?"       # unsafe
python3 wfdemo.py run --order C3 --version 3                       # patched
python3 wfdemo.py run --order D4 --version 3                       # new order
python3 wfdemo.py report --order D4

Expected output (abridged: the first run’s lines and most of the report are left out):

  replay  reserve -> reserved
NON-DETERMINISM: seq 1: journal has 'charge', code asked 'fraud_check'
exit=2
  replay  reserve -> reserved
  replay  charge -> charged
  run     courier -> booked
outcome: completed
  run     reserve -> reserved
  run     patch:fraud-check -> marker
  run     fraud_check -> ok
  ...
  journal ('order-D4', 1, 'patch:fraud-check')

Validation: version 2 fails on the in-flight instance; version 3 lets C3 finish on the old path while D4 records the marker and runs the fraud check. This mirrors Temporal’s patched() behaviour.^[41] Troubleshooting: if C3 completes under version 2, you skipped the crash step. The crash came after courier ran, so a real courier API would have been called twice: that call needs its own key too. Teardown: python3 wfdemo.py reset, then rm -r ~/wfdemo-lab.

All three are illustrative scenarios, not documented case studies.

Measure what the business sees (instances reaching a correct end state on time), then the signals that predict it.

Leading: task start latency, retries per run, history size. Lagging: correct end state, reconciliation gaps. SLO method: SRE.

• Histories contain data. Step Functions keeps closed execution history for 90 days (reducible to 30 by quota request); Lambda durable functions retain data 1–90 days after completion (default 14).^[23][65] Pass identifiers, not personal data; where personal data must flow, encrypt payloads and set retention deliberately.
• Least privilege per step: each step (function, worker, pod) gets its own identity scoped to the systems it calls.
• Control-plane rights (reset, terminate, redrive, signal, backfill) change outcomes in bulk: restrict, log, two-person rule.
• Signals and task tokens are inputs: one that completes a human review must come from an authenticated tool.
• Audit trails: a history is evidence only if retention matches the rule that applies; export it before the engine deletes it.

Cost drivers are steps, retries, waiting and stored history. Step Functions Standard bills per state transition “including retries”, $0.000025 each in US East (N. Virginia) after 4,000 free a month (October 2026).^[54] Temporal Cloud bills Actions such as starting workflows and recording heartbeats;^[52] Lambda durable functions bill compute (including replays), durable operations, data written and retention.^[65]

Capacity: size workers by task start latency, not CPU; cap concurrency for backfills; keep history small (continue-as-new) so replay stays fast. Sustainability follows compute: Google Cloud Workflows charges nothing while idle,^[53] and waits in on-demand Lambda durable functions incur no duration charges, although data retention is still billed;^[65] polling loops cost compute.

The operating model is real only if it leaves evidence: CI logs of replay tests on exported production histories (temporal workflow show --output json plus the SDK replayer),^[66][22] fault-injection results, a daily reconciliation report, notes of the weekly review, and a ticket with scope query and second approver per bulk operation. Audit a sample each quarter.

Context: a Temporal deployment; command forms from the Temporal CLI docs; not executed.

1. Detect. temporal_workflow_task_execution_failed rises within minutes of a worker release; instances stop progressing.^[64]

2. Confirm. Pick one affected workflow and export its history; the worker log names the mismatch (expected event vs new command).

   Example status: syntax-checked, not executed (requires the Temporal CLI and a namespace).

   temporal workflow list --limit 20 \
     --query 'ExecutionStatus="Running" AND WorkflowType="OrderWorkflow"'
   temporal workflow show --workflow-id order-A1 --output json > order-A1.json
   

3. Contain first: roll the workers back. Workflow tasks are not governed by retry policies; they keep retrying with backoff up to a 10-minute interval until the workflow’s execution timeout,^[27] so affected instances resume on the old build without data loss. Do not terminate or reset them.

4. Reproduce. Replay order-A1.json against the new build with the SDK replayer; it should fail the same way.^[22]

5. Fix. Wrap the change in workflow.patched("…") (or pin the workflow type to its worker deployment version) and add the exported history to the CI replay suite.^[41][43]

6. Re-deploy and verify. Failure metric back to baseline; old instances complete on the old path; new instances record the patch marker.

7. Only if instances are unrecoverable, use temporal workflow reset --workflow-id … --event-id … from a known-good event, with a second approver and a written list of affected IDs.^[66]

8. Review why CI did not replay a production history.

• “The engine makes it exactly once.” Step Functions’ exactly-once claim for Standard workflows explicitly excludes steps with Retry.^[17]
• Retry everything forever. Non-retryable errors then loop for days. Classify errors.
• The workflow as a database. One eternal instance with growing state hits history limits; keep business data in its system of record.
• Side effects in scheduler callbacks. Hooks are not retried, versioned or visible like steps.
• Compensation as an afterthought. If no one has written the refund rule, the engine cannot run it.
• Bulk reset as a fix. Resetting thousands of instances replays effects: treat it as a data migration.

Use a workflow engine when a process has several steps across systems and needs any of: surviving crashes mid-process, compensation, waits of minutes to months (timers, people, callbacks), or a step-by-step audit trail. Duration alone is not the test: a seconds-long transfer across three systems that needs compensation belongs on an engine too.

Do not adopt one when:

• One idempotent job on a schedule does the work: cron (or a Kubernetes CronJob) plus a lock and an idempotent script is cheaper.
• The process is a single request-response under a few seconds.
• Steps are a stream of independent messages: use a queue with idempotent consumers (IntegrationOps).
• The team cannot yet guarantee idempotent downstream calls: an engine will retry the problem faster.

Hidden responsibilities: engine operation or vendor cost, determinism training, versioning discipline, history retention, and a staffed human queue.

flowchart TB
  Q1{"Several steps with side effects?"}
  Q1 -->|"no"| R1["Plain job or request"]
  Q1 -->|"yes"| Q2{"Compensation, long or human waits, or step audit?"}
  Q2 -->|"no"| Q3{"Batch per time interval?"}
  Q3 -->|"yes"| R2["DAG scheduler"]
  Q3 -->|"no"| R3["Queue and idempotent consumers"]
  Q2 -->|"yes"| Q4{"Mostly calls cloud services, little branching?"}
  Q4 -->|"yes"| R4["Managed state machine"]
  Q4 -->|"no"| R5["Durable execution or BPMN engine"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q1,Q2,Q3,Q4 decision
  class R2,R4,R5 core
  class R1,R3 neutral

For a shortlist of engines, score each 1–5 on weighted criteria agreed in advance (example weights, adjust to your context):

This Atlas’s model, not an industry standard.

Quick wins: a deadline on every wait, an alert on the oldest open instance, no nested SDK retries inside engine-retried steps.

Before production (per workflow)

• Design record approved: steps, keys, timeouts, retries, compensation, pivot, deadline, owner
• Every step has a timeout and an explicit retry policy; non-retryable errors listed
• Every external call carries a per-step, instance-derived key; each receiver’s key retention exceeds the step’s retry and park horizon
• Fault-injection tests: timeout, crash after effect, permanent error at each step
• Human queue exists, is staffed and has an age alert
• Payloads are references; history size estimated against engine limits

Before each deployment

• Change classified (compatible / patch / new version / new type)
• Replay tests pass on recent production histories, or versions/aliases route new executions
• Rollback path for workers or definitions tested

Template status: plain-text forms; adapt the fields to your engine.

WORKFLOW DESIGN RECORD
Workflow name / type:            Owner (team, on-call):
Workflow ID and reuse policy:    e.g. "order-<id>", Reject Duplicate
Deadline (whole workflow):       Expected volume (starts/day, steps/run):
Max duration and history size:   Engine limits checked (events, payload):

STEP TABLE
| Step | System | Idempotency key | Key retention | Timeout | Retry policy |
| Non-retryable errors | Kind: compensable / pivot / retryable | Compensation | Owner |

Human waits: what, who, deadline, escalation
Data in history: fields, personal data (y/n), retention
Versioning plan: patch / pin / new type; replay test source
Metrics and alerts: correct-end-state SLO, oldest open, parked age
Approvals: workflow owner, downstream owners, business owner (compensation)

CHANGE TO A RUNNING WORKFLOW
Workflow:                 Change summary:
Running instances now:    Oldest started:      Expected drain date:
Alters command sequence?  yes / no  (evidence: replay test run id)
Strategy:                 deploy / patch id ____ / pin version ____ / new type
Bulk operation needed?    none / reset / signal / migrate  (scope query attached)
Second approver:          Rollback plan:
After drain: remove old path on (date) ________

Is “WorkflowOps” an established discipline? No. The name has unrelated uses (an Oracle API, a product);^[1][2] the practice here is known as workflow orchestration or durable execution.

Is a BPMN engine a workflow engine? Yes. Camunda 8 or Flowable runs process instances with the same concerns as on this page (retries, idempotent steps, versions of running instances); only the modelling work with stakeholders is out of scope.

Airflow or Temporal? Different jobs: batch work per data interval with backfills, versus application processes as code with long waits. Teams often run both; pipeline practice is in DataOps.

When should I compensate instead of retrying? Only when forward progress is impossible: a non-retryable error or exhausted retries, and only before the pivot. If the pivot’s outcome is unknown, query it by key first.^[15][14]

What should page the on-call engineer? The oldest open instance past its deadline, a jump in workflow task failures, and task start latency; not single step retries, which are expected.

• Durable execution inside platforms is recent (Lambda durable functions launched at re:Invent 2025);^[9] re-check quotas and pricing.^[65]
• AI agents on durable engines treat each model call as a step; per-instance cost caps and compensation for an agent’s actions are open design questions.
• Portable definitions (Open Workflow Specification 1.0.3) exist; portability is a goal, not a given.^[50]

• Option A is the attractive mistake. Open instances passed intake long ago. When they replay, their history has no marker at that point, so patched() returns false and they keep the old path for the rest of the run;^[41] the 24,000 open loans skip the check.
• Option B meets the rule with no migration. Temporal documents that an execution which met no patched() call during replay uses the new code once past the replayed history,^[41] so open instances that later reach disbursement run the check (new loans too, if a late check is acceptable to the business).
• Option C also works, but migrating 24,000 instances is a bulk operation for no gain here.

Decision: Option B, released after replay tests on histories of instances at several stages, plus a report of loans disbursed without the check (expected: none). Uncertainty: an execution that already met a patched() call with the same ID would behave differently, so use a new, unique patch ID.

After this page you can explain why an engine does not prevent double effects, set retry and timeout policies, design a saga with a pivot, version running instances, calculate workflow KPIs and run the runbook.

“WorkflowOps” is a label without a settled meaning, but the work it names here is real: processes that outlive the workers, deployments and sometimes the people who started them. Engines turn the bookkeeping into infrastructure; they do not make a charge safe to repeat, decide what a refund means, or protect a running instance from tomorrow’s code. Teams that do this well write down every step’s key, timeout, retry and undo before writing code, treat running instances as a contract, and measure whether each instance ended correctly. That discipline matters more than the choice of engine.

All sources were opened and checked on 2026-10-23. Product limits, prices and versions change; re-check them before relying on them.

Further reading

• Temporal’s Temporal overview for the durable-execution vocabulary used on this page.
• Neighbouring Atlas pages: DataOps, IntegrationOps, AutomationOps, ServerlessOps, ObservabilityOps.

Changelog

• 2026-10-23 — First draft: term status confirmed as ambiguous; research ledger research/workflowops-ledger.md; Labs 1–3 executed with Python 3.8.10; other examples syntax-checked only.
• 2026-10-23 — Round 1 revision after the independent review (reviews/workflowops-r1.md, response reviews/workflowops-r1-response.md): pivot outcomes (rejected / unknown / failure after it) fixed in text and both implementations; per-step keys and key retention; decision tree routes compensation needs to an engine; BPMN engines and instance migration added; 16 minor fixes; length trimmed; labs re-run.
• 2026-10-23, post-score polish: Argo’s “transient” errors defined (a container’s non-zero exit needs OnFailure); ASL compensations end in a Pass state so ExecutionsFailed counts only real failures; manual refund before answering a refund park stated; small trims to stay in the length band (reviews/workflowops-polish.md).