IntegrationOps

IntegrationOps is operating the flows that move business events and records between systems: an order becoming an invoice in an ERP, a ticket mirrored between two service desks, a database change feeding a search index. The code is usually small; the operational work is large, because flows cross team and company boundaries, brokers may deliver a message twice, and failures stay invisible until a customer or an auditor notices.

This page centres on reliability at the message level: what happens to one message when a consumer crashes, a payload is malformed, a downstream system is down, or someone replays last Tuesday. The goal: integrations that fail visibly and recover without losing or doubling data. It is for integration engineers, administrators of integration platforms as a service (iPaaS), architects, and leaders deciding who owns a flow that belongs to no single team.

Integrations fail quietly. A call that times out after the remote side committed, a consumer that crashes after writing but before acknowledging, a mapping that drops a renamed field: each creates a duplicate, a gap or a corrupt record, and none turns a dashboard red. AWS states the root fact for SQS standard queues: “you might get that message copy again”, so applications should be idempotent^[1].

Without an operating discipline the same four failures recur: silent loss (RabbitMQ, for example, drops messages that need dead-lettering if the dead-letter exchange is missing^[2]); silent duplication (two invoices, two refunds); retry storms (every layer retries, so a short outage becomes sustained overload); and orphaned flows (the project ended, the flow still runs, nobody owns its dead-letter queue). IntegrationOps replaces these with owned flows, explicit guarantees, rehearsed recovery and record-level proof that nothing went missing.

Term status: vendor-specific. The label is promoted mainly by ONEiO, a Helsinki-based managed integration service provider. Its April 2026 explainer defines Integration Operations as “the structured, continuous management of integrations as an operational capability” and says “ONEiO pioneers Integration Ops as a managed service”^[3]. ONEiO also writes “IntOps” and offers an Integration Ops book^[4], not reviewed here. A 2025 funding report calls the company “a Managed Integration Service Provider delivering IntegrationOps”^[5]. We found no standards body, independent definition, certification or vendor-neutral community using the label, so this page treats it as a vendor’s name for a practice documented since Hohpe and Woolf’s Enterprise Integration Patterns (2003)^[6]. ONEiO’s offer, which its co-founder calls “a domain-specific, managed operating model for IT services”^[7], is one way to staff that practice (Deep comparisons).

Beyond the neighbours under Out of scope above, it is not “integrated operations” in oil and gas, Norway’s term for real-time onshore–offshore work processes; the Research Council of Norway funded a Center for Integrated Operations in the Petroleum Industry from 2007 to 2015^[8]. That is a name collision.

Hohpe and Woolf named four integration styles (file transfer, shared database, remote procedure invocation, messaging)^[9] and described the central message broker, “sometimes referred to as hub-and-spoke architectural style”^[10]. Enterprise service bus (ESB) products then put routing, transformation and business rules into middleware, until Lewis and Fowler’s 2014 microservices article argued for “smart endpoints and dumb pipes”, naming the ESB as the example of putting “significant smarts into the communication mechanism itself”^[11]. Log-based brokers such as Apache Kafka keep events after consumption^[12], which made replay routine, and iPaaS moved connectors and mappings into SaaS. The IntegrationOps label dates from 2025–2026 marketing^[5][3]; the problems it names are two decades old.

Mental model: four parts and a ledger. A contract (what a message means), a transport (broker, queue, file or API call), a handler (message to side effect) and a recovery path (retry, park, redrive, replay), plus the ledger: business records on both sides that prove the flow worked. Most incidents are one part assuming another’s guarantee.

There is no official IntegrationOps manifesto; this charter is this Atlas’s proposal. Mission: every business record that should cross a boundary arrives, in effect, exactly once, or is visibly parked with a named owner and a deadline.

1. Assume duplicates. Exactly-once features stop at the broker’s edge (FAQ).
2. Deduplicate on business identity, never on a broker’s delivery identifier, which changes on redrive^[18].
3. Retry in one place. In a five-deep stack retrying three times per layer, load on the bottom tier can rise 243-fold^[19].
4. Park, don’t loop. Messages that cannot succeed leave the live path quickly, with their failure context.
5. Every parked message has an owner and a clock.
6. Replay is a change: approved, rate-limited and verified.
7. Prove completeness in business terms. Broker metrics cannot show what never entered the broker.

Non-goals: one broker for everything; all mapping logic in one central team (the ESB lesson^[11]).

Integrations fail between teams. The flow owner, a person rather than a mailbox, is accountable for the flow’s guarantees, DLQ and reconciliation.

R = responsible, A = accountable, C = consulted, I = informed. One person may hold several columns; each row needs exactly one A.

• Day 0 (design): name the business event and its identifier; choose the delivery guarantee and ordering scope; register the flow; write the failure policy.
• Day 1 (go-live): broker policies as code; a duplicate-injection test; a first reconciliation on real data; one redrive rehearsed in staging.
• Day 2 (operate): scheduled DLQ triage, daily reconciliation, schema reviews, quarterly replay rehearsal.
• Retirement: drain or archive parked messages, delete queues and credentials. Orphaned queues keep costs and personal data alive.

Keep it light: an integration register (one flow card per flow); a fifteen-minute weekly DLQ review where every parked message is fixed, redriven, written off with a reason, or escalated; change control by risk (schema changes through the API Ops contract gate, broker policies through pull requests, redrive and replay with the flow owner’s recorded approval); and escalation of completeness or latency SLO breaches as incidents, coordinated through IncidentOps.

A common, vendor-neutral shape for a flow such as order-to-invoice; simpler flows may drop the outbox or the parking queue.

flowchart TB
  subgraph SRC["Producer side"]
    APP["Order service"]:::neutral --> DB[("Orders + outbox table")]:::neutral
    DB --> RELAY["Outbox relay"]:::core
  end
  RELAY --> BRK["Broker: orders topic or queue"]:::core
  BRK --> CON["Consumer with inbox table"]:::core
  CON --> TGT[("ERP: invoices")]:::neutral
  CON -->|"transient error"| RTY["Delayed retry, bounded"]:::decision
  RTY --> BRK
  CON -->|"invalid or poison"| DLQ["Parking queue with failure context"]:::risk
  DLQ --> TRI["Triage, then approved redrive"]:::decision
  TRI --> BRK
  REC["Reconciliation job"]:::core -.-> DB
  REC -.-> TGT
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

• Outbox closes the dual-write gap, but the relay “might publish a message more than once”, so consumers must be idempotent^[17]. Debezium’s outbox router keys messages by aggregate ID, keeping one aggregate’s events in one Kafka partition^[22].
• Parking queue separates “not now” from “never” and keeps failure context (RabbitMQ adds x-death headers^[2]).
• Reconciliation sits outside the message path, so it catches losses before the broker and after a premature ack.
• Trust boundaries separate producer, broker and consumer teams; the redrive path needs the strictest credentials.

Which to choose is in the Decision framework.

sequenceDiagram
  participant P as Producer and outbox
  participant B as Broker
  participant C as Consumer
  participant D as Consumer database
  P->>B: publish order-123 (relay)
  B-->>P: publisher confirm
  B->>C: deliver order-123
  C->>D: insert inbox key, write local invoice row
  D-->>C: commit
  C--xB: ack lost (consumer restarts)
  B->>C: redeliver order-123
  C->>D: insert inbox key
  D-->>C: key exists, skip
  C->>B: ack
  B->>C: deliver order-124 (malformed)
  C->>B: reject, no requeue
  Note over B: dead-letter to parking queue with reason

This is atomic only because the invoice is a local row. With an external ERP no transaction spans inbox and ERP; the ERP’s idempotency key protects that call (the handler).

The consumer’s code, not the broker, decides the guarantee. RabbitMQ’s publisher confirms and consumer acknowledgements are “entirely orthogonal” (you need both), and automatic acknowledgement “should be considered unsafe”^[23]. Kafka’s design notes say a consumer that processes and then saves its position gets at-least-once; one that saves its position first gets at-most-once^[13].

Seven playbooks, each with a trigger, owner and output.

Trigger: any new consumer. Owner: consumer team. Output: one effect per business event.

Hohpe and Woolf give two routes: detect and discard duplicates, or define messages so that a second copy has no further effect^[14]. “Set stock to 40” is naturally idempotent; “subtract 2” is not. When you cannot change the semantics, use an inbox: a unique-key table written in the same transaction as the side effect.

Choose the key carefully. SQS gives redriven messages “a new messageID and enqueueTime”^[18], and Pub/Sub publish retries can produce copies with different message IDs^[24]. Use an identifier the producer assigns once; CloudEvents requires source + id to be unique per distinct event and lets consumers treat identical pairs as duplicates^[25].

For an external API, pass an idempotency key (AWS uses the same caller-provided token idea^[26]) and learn two rules of the target. Keys expire: Stripe’s v1 API can drop a key once it is at least 24 hours old, and a repeat then counts as a new request. Keys also remember answers: Stripe v1 returns the saved first response to every retry with that key, “including 500 errors”^[27], so a key can block recovery as well as duplicates.

Duplicates also arrive with no crash at all, when the work outlasts the broker’s lease and another consumer receives the message (Common failures). Verify: inject every message twice in a test environment; business totals must not change (Lab 1).

Trigger: any call that can fail transiently. Owner: consumer team. Output: a written retry budget.

Retries hide in libraries (as of October 2026 AWS documents three attempts, including the first, for its updated, still opt-in SDK retry behaviour^[28]); a consumer loop adds a second layer and broker redelivery a third. Marc Brooker recommends retrying “at a single point in the stack” and warns that “APIs with side effects aren’t safe to retry unless they provide idempotency”^[19]. Record per flow which layer retries, how often, and what happens when the budget is spent (park). Broker-side delay is often the cleanest layer: RabbitMQ 4.3 added delayed retry for quorum queues, delay = min(min_delay * delivery_count, max_delay)^[29]. The formula has no random part, so messages that fail together return together; add jitter in the client if that hurts the target.

Trigger: any handler error. Owner: consumer team. Output: a failure policy in code.

flowchart TB
  E["Handler error"]:::risk --> Q1{"Can the payload ever succeed?"}:::decision
  Q1 -->|"no: malformed or invalid"| PARK["Park now with reason, no retries"]:::risk
  Q1 -->|"yes"| Q2{"Is the downstream system down or throttling?"}:::decision
  Q2 -->|"yes"| PAUSE["Pause or slow consumption; keep messages in the queue"]:::core
  Q2 -->|"no"| Q3{"Transient error with budget left?"}:::decision
  Q3 -->|"yes"| RETRY["Delayed retry with backoff"]:::core
  Q3 -->|"no: budget spent or unknown"| DL["Dead-letter with attempt history"]:::risk
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029

An invalid message^[16] needs a producer fix; a dead letter^[15] often needs only time. Pause during downstream outages: most queues count redeliveries towards a dead-letter threshold, so an outage turns healthy messages into dead letters.

Trigger: fixable parked messages, or state to rebuild. Owner: flow owner approves; consumer team executes. Output: a replay record (Templates).

Both re-execute side effects, so first confirm the root cause is fixed, the dedup key survives, external keys are valid and hold no stored failure, and the rate suits the downstream system. SQS redrive caps custom velocity at 500 messages per second and “doesn’t support filtering and modifying messages while redriving”^[18]. Kafka offset resets need inactive consumer instances and only plan the reset until you add --execute^[30]; replay is bounded by topic retention, seven days by default^[31].

Trigger: any schema change. Owner: producer team; the contract gate is API Ops. Output: a compatibility mode per subject and an upgrade order.

Confluent Schema Registry defaults to BACKWARD so that you can “rewind consumers to the beginning of the topic”, but the default is non-transitive: new schemas are checked “only against the latest schema”^[21]. Version 3 may fail on version 1 data although each step passed. For subjects replayed from the start, use BACKWARD_TRANSITIVE or FULL_TRANSITIVE. Upgrade consumers first under backward compatibility, producers first under forward, either under full^[21].

Trigger: a consumer that breaks on disorder. Owner: flow owner. Output: an explicit ordering scope.

Ordering is per partition, queue or group: Kafka puts events with the same key in one partition and consumers read a partition in write order^[12]. The price is head-of-line blocking; in an SQS FIFO queue a failed message “blocks its message group until it’s deleted or expires”^[32], and AWS advises against giving a FIFO queue a dead-letter queue at all where exact order must hold^[33]. Kafka share groups (production-ready since 4.2^[34]) make the opposite trade: several consumers share a partition and acknowledge records one by one^[13], which buys broker-counted retries, but records “can be delivered out of order to a consumer, in particular when redeliveries occur”^[35]. Prefer consumers that compare versions and ignore stale updates; keep strict ordering for operations that do not commute.

Trigger: every flow carrying business value. Owner: consumer team; flow owner signs off. Output: a completeness report per closed window.

Compare business records at source and target for a closed window and explain each difference: in flight, parked, written off, or unexplained, the flow’s key metric (Metrics and KPIs). Hohpe’s “Starbucks Does Not Use Two-Phase Commit” names three responses: write-off, retry, and a compensating action “to put the system back into a consistent state”^[36].

Pact’s own guidance fits integration hubs: for a “pass through” API that forwards to a queue without validating, the contract that matters is “between your consumer and the downstream system”^[40].

Defaults decide failure behaviour more than architecture diagrams do. As of October 2026:

A RabbitMQ detail catches teams out: since 4.3 the delivery limit counts only genuine failures (basic.reject, client crashes, or AMQP 1.0 modified with delivery-failed=true), so “unlimited explicit message returns” by basic.nack are allowed^[29]. A consumer that nacks a poison message with requeue loops for ever. Delayed retry of type all or returned only spaces those returns out, and type failed does not delay them at all^[29]. Only basic.reject, or an attempt counter that then rejects or parks, ends the loop.

Automate the mechanical parts: policies as code, DLQ alarms, reconciliation queries, rate-limited redrive scripts, duplicate-injection tests. Language models can draft field mappings or cluster DLQ errors, but mappings change money, so a human approves them and contract tests check them. No agent redrives or replays without the approval in Playbook 4.

One flow, order-to-invoice: an order service on PostgreSQL publishes OrderPlaced events; a billing consumer creates invoices through an ERP API. Assumes RabbitMQ 4.3+ quorum queues (or Amazon SQS) and PostgreSQL; all names are placeholders.

Keep the register entry in Git beside the broker policies.

Example status: YAML syntax-checked by the Atlas validator; not executed.

flow: order-to-invoice
owner: jane.doe@example.com          # a person, not a mailbox
producer: {team: shop, system: order-service}
consumer: {team: billing, system: invoice-consumer}
contract: https://git.example.com/contracts/order-placed/asyncapi.yaml
business_key: order_id               # survives redrive and replay
delivery: at-least-once
idempotence: inbox table billing.inbox(msg_key)
ordering: none                       # consumer ignores stale updates
transport: {broker: rabbitmq, queue: orders.in, parking: orders.parked}
retry: {layer: broker-delayed-retry, limit: 10, min_delay_s: 1, max_delay_s: 10}
retention: {parking_ttl_days: 14}    # message-ttl policy on orders.parked
slo:
  completeness_unexplained: 0        # per closed day
  latency_p95_minutes: 5
reconciliation: sql/recon_order_to_invoice.sql
runbook: https://wiki.example.com/runbooks/order-to-invoice
replay_approval: flow owner

Example status: illustrative PostgreSQL DDL; not executed.

-- Producer: the event is written in the same transaction as the order.
CREATE TABLE outbox (
  id            uuid PRIMARY KEY,          -- event id, unique per event
  aggregatetype text NOT NULL,             -- e.g. 'order'
  aggregateid   text NOT NULL,             -- order_id: the business key
  type          text NOT NULL,             -- e.g. 'OrderPlaced'
  payload       jsonb NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now()
);

-- Consumer: one row per processed business event.
CREATE TABLE inbox (
  msg_key      text PRIMARY KEY,           -- 'order:' || order_id [|| ':r2' on a re-send]
  processed_at timestamptz NOT NULL DEFAULT now()
);

The column names follow the defaults of Debezium’s outbox event router, which routes by aggregatetype and uses aggregateid as the message key^[22].

Example status: Python syntax-checked by the Atlas validator; not executed (written for psycopg2; needs a broker client and an ERP).

import json


class PermanentError(Exception):
    """The message can never succeed: park it without retries."""


def handle(conn, erp, body: bytes) -> None:
    """Process one OrderPlaced message at least once, with one effect."""
    try:
        event = json.loads(body)
        order_id = str(event["order_id"])
        amount = int(event["amount_cents"])
        # Set (e.g. "r2") only on re-sends confirmed absent in the ERP.
        generation = event.get("key_generation")
    except (ValueError, KeyError, TypeError, AttributeError) as exc:
        raise PermanentError(f"invalid payload: {exc}") from exc

    key = f"order:{order_id}" + (f":{generation}" if generation else "")
    with conn:  # psycopg2: commits or rolls back (psycopg 3 also closes)
        cur = conn.cursor()
        cur.execute(
            "INSERT INTO inbox (msg_key) VALUES (%s) "
            "ON CONFLICT (msg_key) DO NOTHING",
            (key,),
        )
        if cur.rowcount == 0:
            return  # duplicate delivery: already processed
        # External side effect: the ERP deduplicates on the same key.
        # If this call raises, the inbox row rolls back and the message
        # is retried with the same key. If the ERP stored a failure for
        # that key, it may replay it: see the runbook's recovery step.
        erp.create_invoice(order_id=order_id, amount_cents=amount,
                           idempotency_key=key, timeout_s=10)

If the ERP call fails, the inbox row rolls back and the retry cannot create a second invoice. It may not create the first either: a target that stores first responses returns the saved error until the key expires. The transaction and its row lock stay open during the ERP call, so keep the timeout short. If the ERP cannot deduplicate, use a “requested”/“confirmed” status table plus reconciliation, not a cleverer retry.

One RabbitMQ policy carries every setting, because at most one policy applies to a queue^[47]. At-least-once dead-lettering needs overflow set to reject-publish and the stream_queue feature flag; max-length stops unbounded growth^[29]. Side effects: at the limit the broker refuses publishes (a basic.nack when confirms are on)^[48], so the outbox relay must keep those rows and retry; and dead-lettered messages stay in the source queue until the target accepts them^[29], so a missing dead-letter target makes orders.in grow. Alarm on its length.

Example status: bash -n syntax-checked; not executed.

#!/usr/bin/env bash
set -euo pipefail
# Live queue: dead-letter to orders.dlx at least once, cap growth.
# Type "failed" delays rejects and crashes (1 s, 2 s ... 10 s: about
# 55 s in all before dead-lettering) but not basic.nack returns, so
# the consumer must reject, never nack, a message that fails.
rabbitmqctl set_policy orders-in '^orders\.in$' \
  '{"dead-letter-exchange": "orders.dlx",
    "dead-letter-strategy": "at-least-once",
    "overflow": "reject-publish",
    "max-length": 1000000,
    "delivery-limit": 10,
    "delayed-retry-type": "failed",
    "delayed-retry-min": 1000,
    "delayed-retry-max": 10000}' \
  --apply-to quorum_queues --priority 10
# Parking queue: RabbitMQ keeps messages until consumed, so set a
# 14-day TTL; expired parked messages are dropped.
rabbitmqctl set_policy orders-parked '^orders\.parked$' \
  '{"message-ttl": 1209600000}' --apply-to quorum_queues --priority 10

The same intent on Amazon SQS: a 14-day dead-letter queue (the maximum retention^[41]) that only the source queue may use, and a source queue that dead-letters after ten receives.

Example status: bash -n syntax-checked; not executed (needs an AWS account; creates billable resources).

#!/usr/bin/env bash
set -euo pipefail
: "${REGION:?}" "${ACCOUNT_ID:?}"
SRC_ARN="arn:aws:sqs:${REGION}:${ACCOUNT_ID}:orders-in"
DLQ_URL=$(aws sqs create-queue --queue-name orders-parked \
  --attributes MessageRetentionPeriod=1209600 \
  --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "${DLQ_URL}" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)
ALLOW=$(jq -cn --arg src "${SRC_ARN}" \
  '{redrivePermission: "byQueue", sourceQueueArns: [$src]}')
aws sqs set-queue-attributes --queue-url "${DLQ_URL}" \
  --attributes "$(jq -cn --arg p "${ALLOW}" '{RedriveAllowPolicy: $p}')"
REDRIVE=$(jq -cn --arg dlq "${DLQ_ARN}" \
  '{deadLetterTargetArn: $dlq, maxReceiveCount: "10"}')
aws sqs create-queue --queue-name orders-in --attributes \
  "$(jq -cn --arg p "${REDRIVE}" '{RedrivePolicy: $p}')"

Run it per closed window so in-flight messages do not count as gaps.

Example status: illustrative SQL for a database that can see both tables (a replica or a reporting copy); not executed.

-- Orders placed yesterday with no invoice, no parked message and no
-- write-off. ops.parked_messages: business keys recorded by the DLQ
-- triage tool, including written-off keys (status 'written_off').
SELECT o.order_id, o.created_at
FROM   shop.orders o
LEFT   JOIN billing.invoices i ON i.order_id = o.order_id
LEFT   JOIN ops.parked_messages p ON p.business_key = o.order_id
WHERE  o.created_at >= date_trunc('day', now()) - interval '1 day'
AND    o.created_at <  date_trunc('day', now())
AND    i.order_id IS NULL
AND    p.business_key IS NULL;   -- these are the unexplained gaps

AWS recommends ApproximateNumberOfMessagesVisible for DLQs, since automatic moves are not counted by NumberOfMessagesSent; and on the source queue, “Poison-pill messages … are excluded from this metric until successfully processed” (ApproximateAgeOfOldestMessage)^[32].

Example status: bash -n syntax-checked; not executed.

#!/usr/bin/env bash
set -euo pipefail
: "${SNS_TOPIC_ARN:?}"
aws cloudwatch put-metric-alarm \
  --alarm-name orders-parked-not-empty \
  --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=orders-parked \
  --statistic Maximum --period 300 --evaluation-periods 1 \
  --threshold 0 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions "${SNS_TOPIC_ARN}"

Lab 1 was executed for this page (Python 3.8.10). Labs 2 and 3 were not executed (the author’s environment was a shared server); their commands are syntax-checked against the cited documentation and source. Use a laptop or a disposable VM with Docker.

Objective. See at-least-once delivery inflate a ledger, and see why the deduplication key must survive redrive. Prerequisites. Python 3.8 or later; no network. Cost and risk. None (in-memory SQLite).

1. Save the script as lab1.py. It simulates a broker that redelivers 20 % of messages because the consumer “crashes” after the side effect but before the acknowledgement, then redrives 50 messages with new broker identifiers.
2. Run python3 -I lab1.py.

Example status: executed on 2026-10-23 with Python 3.8.10; output below is the real output.

"""At-least-once delivery, duplicates and an idempotent consumer."""
import random
import sqlite3
import uuid

CRASH_RATE = 0.2  # consumer dies after the side effect, before the ack


def make_events(n):
    # The producer assigns a business event id once.
    return [{"event_id": f"order-{i}", "amount": 10} for i in range(n)]


def deliver(events, handler):
    """At-least-once: a message is redelivered until it is acknowledged."""
    queue = [dict(e, broker_id=str(uuid.uuid4())) for e in events]
    deliveries = 0
    while queue:
        msg = queue.pop(0)
        deliveries += 1
        handler(msg)                      # side effect happens here
        if random.random() < CRASH_RATE:  # crash before ack: redelivery
            queue.append(msg)
    return deliveries


def redrive(events):
    """A redrive re-publishes copies: new broker ids, same business ids."""
    return [dict(e, broker_id=str(uuid.uuid4())) for e in events]


def run(name, key):
    db = sqlite3.connect(":memory:")
    db.execute("CREATE TABLE ledger (amount INTEGER)")
    db.execute("CREATE TABLE inbox (msg_key TEXT PRIMARY KEY)")

    def handler(msg):
        with db:  # one transaction: dedup record + side effect
            if key is not None:
                cur = db.execute("INSERT OR IGNORE INTO inbox VALUES (?)",
                                 (msg[key],))
                if cur.rowcount == 0:
                    return  # duplicate: skip the side effect
            db.execute("INSERT INTO ledger VALUES (?)", (msg["amount"],))

    events = make_events(1000)
    n = deliver(events, handler)
    for msg in redrive(events[:50]):  # an operator redrives 50 messages
        handler(msg)
    total = db.execute("SELECT SUM(amount) FROM ledger").fetchone()[0]
    print(f"{name:<24} deliveries={n} ledger_total={total} expected=10000")


random.seed(7); run("no dedup", None)
random.seed(7); run("dedup on broker id", "broker_id")
random.seed(7); run("dedup on business id", "event_id")

Expected output (seed 7):

no dedup                 deliveries=1277 ledger_total=13270 expected=10000
dedup on broker id       deliveries=1277 ledger_total=10500 expected=10000
dedup on business id     deliveries=1277 ledger_total=10000 expected=10000

Validation. 1,277 deliveries for 1,000 events is close to the 1,250 a 20 % redelivery rate predicts. Deduplicating on the broker identifier misses the 50 redriven copies (+500), as with SQS redrive^[18]. Only the business key gives 10,000. Troubleshooting. Another Python version may produce different numbers; the ordering of the three totals is what matters. Teardown. Delete the file.

Objective. Watch a poison message reach the delivery limit and be dead-lettered, then see why basic.nack with requeue does not count towards the limit on RabbitMQ 4.3 and later^[29]. Prerequisites. Docker; Python 3.8+ with pika (1.4.4 was current on 2026-10-23^[49]). Cost and risk. Local only. The image’s default login is guest / guest^[50], so the ports are bound to 127.0.0.1; plain -p 5672:5672 would expose the broker to everyone on your network.

1. Start a broker (image tag 4.3-management^[50]) and apply a policy to the live queue only; the anchored pattern keeps the parking queue from dead-lettering into itself.

Example status: bash -n syntax-checked; not executed.

#!/usr/bin/env bash
set -euo pipefail
docker run -d --name rmq-lab \
  -p 127.0.0.1:5672:5672 -p 127.0.0.1:15672:15672 \
  rabbitmq:4.3-management
sleep 30   # wait for the node to boot
docker exec rmq-lab rabbitmqctl set_policy lab '^orders\.in$' \
  '{"dead-letter-exchange": "orders.dlx", "delivery-limit": 3}' \
  --apply-to quorum_queues --priority 10

2. Run python3 lab2.py reject, then python3 lab2.py nack. The script publishes one malformed order and refuses it on every delivery; a safety stop acknowledges it after ten deliveries.

Example status: Python syntax-checked by the Atlas validator; not executed.

import sys

import pika

MODE = sys.argv[1] if len(sys.argv) > 1 else "reject"
conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.exchange_declare("orders.dlx", exchange_type="fanout", durable=True)
ch.queue_declare("orders.parked", durable=True,
                 arguments={"x-queue-type": "quorum"})
ch.queue_bind("orders.parked", "orders.dlx")
ch.queue_declare("orders.in", durable=True,
                 arguments={"x-queue-type": "quorum"})
ch.basic_publish("", "orders.in", b'{"order_id": "A-1", "amount": "ten"}',
                 pika.BasicProperties(delivery_mode=2, message_id="A-1"))

seen = 0
for method, props, body in ch.consume("orders.in", inactivity_timeout=5):
    if method is None:
        break  # nothing delivered for 5 s: dead-lettered or done
    seen += 1
    count = (props.headers or {}).get("x-delivery-count", 0)
    print(f"delivery {seen}: x-delivery-count={count}")
    if seen >= 10:
        ch.basic_ack(method.delivery_tag)  # safety stop for the lab
        print("stopped after 10 deliveries")
        break
    if MODE == "reject":
        ch.basic_reject(method.delivery_tag, requeue=True)
    else:
        ch.basic_nack(method.delivery_tag, requeue=True)
ch.cancel()
conn.close()

3. After each run, list the queues: docker exec rmq-lab rabbitmqctl list_queues name type messages.

Expected output. reject: x-delivery-count rises on each redelivery; once it exceeds the limit the message leaves orders.in and orders.parked shows 1 message. nack: the count stays at 0, the loop hits the safety stop, and orders.parked does not grow. Validation. In the management UI (http://localhost:15672) the parked message carries x-death and x-first-death-reason headers^[2]. Troubleshooting. Nothing parked after reject: check rabbitmqctl list_policies and that orders.in is a quorum queue. On 4.0–4.2 the counting rules differ, so the nack run may also be dead-lettered. Teardown. docker rm -f rmq-lab.

Objective. Practise the replay procedure on a log, and see that replay delivers duplicates by design. Prerequisites. Docker; the apache/kafka:4.3.1 image^[51] (scripts live in /opt/kafka/bin^[52]). Cost and risk. Local only; the broker has no authentication, so its port is bound to 127.0.0.1.

Example status: bash -n syntax-checked; option names checked against the Kafka 4.3.1 source^[53]; not executed.

#!/usr/bin/env bash
set -euo pipefail
K=/opt/kafka/bin
BS=localhost:9092
docker run -d --name kafka-lab -p 127.0.0.1:9092:9092 apache/kafka:4.3.1
sleep 20
docker exec kafka-lab $K/kafka-topics.sh --create --topic orders \
  --partitions 3 --bootstrap-server $BS
START=$(date -u +%Y-%m-%dT%H:%M:%S.000)
# Ten keyed events; each key always maps to the same partition.
for i in $(seq 1 10); do echo "order-$i:{\"order_id\":$i}"; done |
  docker exec -i kafka-lab $K/kafka-console-producer.sh --topic orders \
    --bootstrap-server $BS \
    --reader-property parse.key=true --reader-property key.separator=:
# First pass: the "billing" group reads everything and commits offsets.
docker exec kafka-lab $K/kafka-console-consumer.sh --topic orders \
  --group billing --from-beginning --timeout-ms 10000 \
  --bootstrap-server $BS
docker exec kafka-lab $K/kafka-consumer-groups.sh --describe \
  --group billing --bootstrap-server $BS
# Plan the replay with an explicit dry run, then execute it.
docker exec kafka-lab $K/kafka-consumer-groups.sh --reset-offsets \
  --group billing --topic orders --to-datetime "$START" --dry-run \
  --bootstrap-server $BS
docker exec kafka-lab $K/kafka-consumer-groups.sh --reset-offsets \
  --group billing --topic orders --to-datetime "$START" --execute \
  --bootstrap-server $BS
docker exec kafka-lab $K/kafka-console-consumer.sh --topic orders \
  --group billing --timeout-ms 10000 --bootstrap-server $BS

Expected output. The first consume prints ten values; --describe shows LAG 0 on every partition^[30]. The dry run prints the planned offsets without changing them; after --execute, the second consume prints the same ten values: a replay is a deliberate duplicate. Validation. Run --describe after the reset and confirm the LAG values add up to ten. Troubleshooting. The reset fails while a group member is active^[30]; --to-datetime expects YYYY-MM-DDThh:mm:ss.sss; --property is deprecated since 4.2 in favour of --reader-property, and Kafka 5.0 will require --dry-run or --execute^[53]. Teardown. docker rm -f kafka-lab.

All four are illustrative scenarios, not documented case studies.

Echo loops haunt two-way syncs: each update needs an origin marker or correlation identifier^[54] so a received change is not sent back.

Measure flows, not brokers.

Review gaps and completeness daily, the rest weekly. Worked calculation (illustrative). Yesterday the shop recorded 48,210 orders; the ERP holds 48,198 invoices; 9 orders are parked with reasons. Completeness = 48,198 ÷ 48,210 = 99.975 %; unexplained gaps = 48,210 − 48,198 − 9 = 3. The percentage looks excellent; the three untraced records are the incident (runbook). Set the SLO on the count: zero per closed day.

Backlog drain time = backlog ÷ (sustainable processing rate − arrival rate). With 180,000 queued messages, 20 new per second and an ERP that accepts 45 per second: 180,000 ÷ (45 − 20) = 7,200 s, two hours. More consumers cannot beat the ERP’s limit; they turn backlog into throttling and, through delivery counts, dead letters.

• Least privilege on recovery paths. An SQS redrive needs sqs:StartMessageMoveTask, sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on the DLQ, sqs:SendMessage on the destination, and KMS rights for encrypted queues^[18]. Give them to an operator role used through the approval step, not to application roles. A redrive allow policy can limit a DLQ to ten named source queues^[41].
• Parked messages are production data, often kept longer. Service Bus DLQs have no automatic cleanup and ignore time-to-live^[44]. The GDPR’s storage-limitation principle allows identifiable personal data only as long as the purpose needs^[55], so set parking retention deliberately, purge after triage, and coordinate deletion requests that reach logs and replays with PrivacyOps.
• Failure context can leak secrets. Microsoft recommends the exception type in DeadLetterReason and the stack trace in DeadLetterDescription^[44]; scrub tokens and personal data first.
• Auditability. Record every redrive and replay (who, why, range, rate, result). A message store, which captures “information about each message in a central location”^[37], is the pattern for a searchable trail; it needs a retention rule too.

Cost drivers are retention, headroom and people. Retention buys replay windows and multiplies storage (Kafka keeps seven days by default^[31]; SQS one minute to fourteen days^[41]). Headroom sets drain time. iPaaS and managed messaging are usually priced by usage, so compare on your own volumes and include triage time. Orphaned queues are idle servers by another name. Estate-wide spend and carbon belong to FinOps and GreenOps.

Check quarterly: every production queue, topic and iPaaS flow maps to a register entry with an owner; each tier-1 flow has rehearsed a redrive or replay in staging, followed by a clean reconciliation; no DLQ lacks an alarm and no parked message exceeds its triage SLA; and ten sampled records can be traced from source to target, for example with OpenTelemetry messaging spans^[39].

flowchart TB
  A["Unexplained gaps above 0"]:::risk --> B{"In the outbox?"}:::decision
  B -->|"no"| C["Producer defect"]:::risk
  B -->|"yes"| D{"Published?"}:::decision
  D -->|"no"| E["Relay stuck"]:::risk
  D -->|"yes"| F{"Inbox key?"}:::decision
  F -->|"no"| G["Lost in transit or expired"]:::risk
  F -->|"yes"| H["Acked without effect"]:::risk
  C --> R["Fix cause; re-send confirmed gaps"]:::core
  E --> R
  G --> R
  H --> R
  R --> V["Reconcile again: 0 gaps"]:::core
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029

1. Declare and scope. Open an incident; list the three order IDs. Leave the 9 parked messages alone: they are explained.
2. Trace each key: outbox row? published? inbox key in billing? ERP invoice with that idempotency key?
3. Classify. Here all three have inbox keys and no invoice. Logs show the ERP answered HTTP 202 (accepted) and later rejected the documents asynchronously; the handler treated 202 as success.
4. Contain. Record “requested”, not “done”, on 202 and add a confirmation check; deploy normally.
5. Recover. Search the ERP by order ID to confirm that the three invoices are absent. Under change control, delete the three false inbox keys and re-publish the events with key_generation set to r2 (key order:<id>:r2), recorded in the replay request: the original key may only replay the stored first response (Playbook 1). If the ERP documents a resubmission path, use that instead.
6. Verify. Reconciliation for the window shows zero gaps; the ERP holds exactly one invoice per order ID.
7. Close. Postmortem via IncidentOps: no broker metric could see this loss. Add a test for “accepted, later rejected” responses.

• Retry everywhere. Library, consumer and broker retries multiply into a storm^[19].
• The DLQ as a bin. Unowned parked messages expire silently.
• Redrive everything after an outage. Without a fixed cause it refills the DLQ; at full speed it recreates the overload.
• Event notification without a map. Fowler warns that a logical flow over several event notifications can be hard to see “as it’s not explicit in any program text”^[60]; the register and traces are that map.
• Monitoring the pipe, not the outcome. An empty queue can mean “all processed” or “nothing published”.

Invest when valuable records cross team or company boundaries; when finance, customers or auditors would notice a missing or doubled record; or when flows outlive the projects that built them.

Do not build the full model when one team owns both ends and a synchronous call fails loudly; when data is advisory and a lost update is harmless; or when a durable workflow engine owns the process (WorkflowOps). For two services owned by one team, a shared database, one of the four classic styles^[9], can be the honest answer.

Hidden costs: on-call for cross-team flows, longer retention, weekly triage time, and the political work of naming flow owners.

Two decisions recur: which style should carry a new flow, and how much operating model it deserves.

flowchart TB
  S["New flow between two systems"]:::neutral --> Q1{"Does the caller need the answer to continue?"}:::decision
  Q1 -->|"yes"| SYNC["Synchronous API call; contract via API Ops"]:::core
  Q1 -->|"no"| Q2{"Several consumers, or state rebuilt by replay?"}:::decision
  Q2 -->|"yes"| LOG["Event log with retention, such as Kafka"]:::core
  Q2 -->|"no"| Q3{"Mostly SaaS-to-SaaS with standard connectors?"}:::decision
  Q3 -->|"yes"| IPAAS["iPaaS flow; insist on DLQ and logs access"]:::core
  Q3 -->|"no"| Q4{"Partner can only exchange files?"}:::decision
  Q4 -->|"yes"| FILE["Batch file transfer with manifest and checksums"]:::core
  Q4 -->|"no"| QUEUE["Queue with DLQ: managed or RabbitMQ"]:::core
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

To size the operating model, score each flow 0–2 on five questions:

Tier 1 (7–10): outbox, inbox, alarmed DLQ, daily reconciliation, rehearsed replay. Tier 2 (4–6): idempotent consumer, alarmed DLQ, weekly reconciliation. Tier 3 (0–3): an owner, a DLQ and an alarm. The weights are this Atlas’s proposal.

This is the Atlas’s maturity model, not an industry standard.

Aim for level 2 everywhere and level 3 for tier-1 flows; level 4 pays off when replay is routine.

• Days 1–30: see the flows. Inventory queues, topics and iPaaS flows; register them; name owners; alarm every DLQ; pick the three highest-impact flows.
• Days 31–60: make tier-1 flows recoverable. Business-key inboxes; broker policies in Git; written retry budgets; one redrive rehearsed in staging; a weekly DLQ review.
• Days 61–90: prove completeness. Daily reconciliation for tier-1 flows; the zero-unexplained-gaps SLO; duplicate-injection tests in CI; a first replay rehearsal. Report unexplained gaps per day and time to triage.

Dependencies: read access to both ends’ business data.

New flow (preflight)

• Register entry with a named owner and a business key
• Delivery guarantee and ordering scope written down
• Duplicate-injection test passes
• One retry layer, bounded and jittered; failures classified
• DLQ exists, is alarmed, and outlives the source’s retention
• Transitive schema compatibility if consumers replay history
• Reconciliation query run once on real data

Weekly

• Every parked message has a decision; the oldest is within SLA
• Unexplained gaps reviewed; lag and oldest-message age within SLO

Before any redrive or replay

• Root cause fixed and deployed; dedup key survives the operation
• External idempotency windows open, or reconciliation run first
• Rate below the downstream limit; approval recorded; verification query ready
• Records with a stored failure in the target go out under a new key generation

Replay or redrive request (keep with the incident or change record):

REPLAY / REDRIVE REQUEST
Flow:             order-to-invoice           Requested by:  <name>
Source:           orders-parked (DLQ) | topic orders, group billing
Range:            <message count> | <from timestamp> to <to timestamp>
Root cause fixed: <change id>, deployed <date>
Dedup key:        order_id (business key, survives redrive)
External keys:    ERP keys valid until <date>; reconciled first? yes/no
Key generation:   original | r2 (only for records confirmed absent in target)
Rate:             <n> msg/s (downstream limit <m> msg/s)
Verification:     reconciliation query <link>; duplicate check <link>
Rollback:         stop task / cancel move task; compensate per <SOP>
Approved by:      <flow owner>                Date: <YYYY-MM-DD>
Result:           <messages moved>, gaps after = <n>, duplicates = <n>

DLQ triage record:

DLQ TRIAGE                        Date: <YYYY-MM-DD>   Triage by: <name>
Queue:          <name>            Depth: <n>   Oldest: <age>
Top reasons:    <reason>: <n>, <reason>: <n>
Class:          invalid | poison | downstream outage | unknown
Decision:       fix producer | fix consumer | redrive | write off
Write-offs:     <ids>, reason <text>, approved by <business owner>
Follow-up:      <ticket>                 Due: <date>

The flow card in the Reference implementation serves as the register template.

• Established: transactional outbox and log-based CDC^[17][22]; schema registries; broker-native delayed retry (RabbitMQ 4.3^[29]).
• Maturing: trace context across brokers; the OpenTelemetry messaging conventions are still in Development and do not yet fix how context is attached to messages^[39].
• New in brokers: Kafka share groups, production-ready since 4.2^[34], bring per-record retries.
• Marketing to read carefully: “IntegrationOps as a Service”^[5]; judge triage times, log access and exit terms, not the label.

• One day: the dead-letter channel, idempotent receiver and guaranteed delivery patterns^[15][14]; Lab 1.
• One week: Labs 2 and 3; your broker’s delivery and DLQ documentation; a flow card for one real flow.
• Thirty days: an inbox and a reconciliation query for a tier-1 flow; a staged redrive.
• Specialise and show: an order-to-invoice demo with outbox, broker, DLQ, idempotent consumer, reconciliation and a replay procedure.

• Decide: an owner per flow; which flows are tier 1; one retry layer per flow.
• Practise: business-key idempotence; classify failures; park fast; redrive and replay only with approval.
• Measure: unexplained gaps per closed window (target zero), p95 latency, oldest message age, triage time.
• Watch out for: broker IDs as dedup keys; basic.nack loops on RabbitMQ 4.3+; at-most-once dead-lettering by default on quorum queues; leases shorter than the handler; non-transitive schemas before a full replay; DLQs that expire before their sources.
• Next action: alarm every DLQ, then write the first reconciliation query.

Option A: redrive all 12,000 now at full speed. Attractive, since the consumer is idempotent, and wrong here. The inbox covers only what the consumer committed; if an earlier attempt reached the ERP but the consumer’s transaction rolled back, the ERP key was the only protection, and for the 1,500 older messages it has expired. Full speed also hits a recovering ERP with backlog plus live traffic.

Option B: split by risk. Redrive the 10,500 messages whose keys are valid at a rate that leaves room for live orders: with an ERP limit of 45 per second and 20 used by live traffic, 10,500 ÷ 25 = 420 s, about seven minutes. Reconcile the 1,500 older ones against the ERP by order ID first and re-publish only those with no invoice. Valid keys can also block: an ERP that stores first responses returns a saved error again, so check a sample and send orders confirmed absent under a new key generation.

The decision turns on “which protection covers each message now?”, not “is the consumer idempotent?”. Check the ERP’s real key lifetime and response storage, whether it can be searched by order ID, and its sustainable rate. If it cannot be searched, the 1,500 need manual review and may miss the close; that is finance’s decision, not engineering’s.

You should now be able to explain delivery semantics and leases, design an idempotent consumer with a key that survives recovery, classify failures, set a retry budget, run a governed redrive or replay, measure completeness, and size the operating model to a flow.

IntegrationOps is a new name, mostly one vendor’s, for an old job: making sure that what one system says happened actually happens, once, in the next. The tools moved from ESBs to logs, managed queues and iPaaS; the hard parts did not. Brokers redeliver, leases expire, recovery repeats side effects, and dashboards watch pipes rather than records. What works is small and strict: named owners, business-key idempotence, one retry layer, fast parking, governed replay, and a daily count of records nobody can explain. If you adopt one practice, adopt reconciliation; it sees the failures your broker cannot.

Defaults were checked on 2026-10-23 (Kafka 4.3.1, RabbitMQ 4.3/4.4, Confluent Platform 8.3, Debezium 3.7, AWS CLI 2.38.5); re-check before relying on them.

Term and context

Patterns and architecture

Brokers and managed messaging

Contracts, schemas and telemetry

Retries and idempotency

Privacy

Changelog. 2026-10-23: first draft (r0) for Kafka 4.3, RabbitMQ 4.3/4.4, current AWS, Azure and Google Cloud documentation; term status confirmed as vendor-specific. 2026-10-23 (r1, after review): Kafka share groups; stored idempotency responses and new key generations in recovery; nack loops and delayed-retry types corrected; lease-expiry duplicates; RabbitMQ policy, retention and lab ports fixed; neighbour figure removed. 2026-10-23, post-score polish: share-group ordering quoted from KIP-932; handler takes a key generation; clearer runbook labels.