---
title: "IntegrationOps"
slug: integrationops
category: delivery
term_status: vendor-specific
evidence_status: partially_verified
summary: "IntegrationOps is the work of running system-to-system integrations as owned production flows: messages that may arrive twice, late or broken are deduplicated, parked, replayed and reconciled so that no business record is silently lost."
audience: [Beginners, Integration engineers, Middleware and iPaaS teams, Architects, Engineering leaders]
last_verified: 2026-10-23
key_takeaways:
  - "IntegrationOps is a vendor-promoted label (ONEiO's managed 'Integration Ops' service); this page teaches the vendor-neutral practice: operating integration flows."
  - "Assume at-least-once delivery everywhere: every consumer must tolerate duplicates, keyed on a business identifier that survives redrive and replay."
  - "Classify failures before retrying: transient faults get bounded, jittered retries in one layer; invalid and poison messages are parked at once."
  - "A dead-letter queue is not a safety net until someone owns it: alarm on depth and age, triage on a schedule, and keep it longer than the source."
  - "Replay is a production change that repeats side effects: approve it, rate-limit it and verify it."
  - "Green broker dashboards do not prove completeness: reconcile source and target business records and drive the unexplained gap to zero."
related: [api-ops, workflowops, serverlessops, dataops, observabilityops, incidentops]
---

## What IntegrationOps is

### IntegrationOps in 30 seconds

**IntegrationOps** is operating the flows that move business events and records between systems: an order becoming an invoice in an
ERP, a ticket mirrored between two service desks, a database change feeding a search index. The code is usually small; the operational
work is large, because flows cross team and company boundaries, brokers may deliver a message twice, and failures stay invisible until
a customer or an auditor notices.

This page centres on **reliability at the message level**: what happens to one message when a consumer crashes, a payload is malformed,
a downstream system is down, or someone replays last Tuesday. The goal: integrations that **fail visibly and recover without losing
or doubling data**. It is for integration engineers, administrators of integration platforms as a service (iPaaS), architects, and
leaders deciding who owns a flow that belongs to no single team.

### At a glance

| Item | Summary |
|---|---|
| Goal | Every business record that should cross a boundary arrives once in effect, or is visibly parked with an owner |
| Users | Integration and middleware engineers, broker platform teams, iPaaS administrators, producing and consuming application teams |
| Scope | Integration styles, brokers, idempotent consumers, retries, dead-letter queues, redrive and replay, schema evolution, ordering, monitoring, reconciliation, ownership |
| Out of scope | API contracts ([API Ops](../api-ops/)); workflow engines ([WorkflowOps](../workflowops/)); analytics pipelines ([DataOps](../dataops/)); function-side queue settings ([ServerlessOps](../serverlessops/)) |
| Prerequisites | HTTP, JSON, some SQL, what a queue is |
| Key outputs | Integration register, idempotent consumers, broker policies as code, replay procedures, reconciliation reports |
| Term status | **Vendor-specific label** for a long-established practice |

### The problem it solves

Integrations fail quietly. A call that times out after the remote side committed, a consumer that crashes after writing but before
acknowledging, a mapping that drops a renamed field: each creates a duplicate, a gap or a corrupt record, and none turns a dashboard red.
AWS states the root fact for SQS standard queues: "you might get that message copy again", so applications should be idempotent[^sqs-alo].

Without an operating discipline the same four failures recur: **silent loss** (RabbitMQ, for example, drops messages that need
dead-lettering if the dead-letter exchange is missing[^rmq-dlx]); **silent duplication** (two invoices, two refunds); **retry storms**
(every layer retries, so a short outage becomes sustained overload); and **orphaned flows** (the project ended, the flow still runs,
nobody owns its dead-letter queue). IntegrationOps replaces these with owned flows, explicit guarantees, rehearsed recovery and
record-level proof that nothing went missing.

### What IntegrationOps is not: boundaries and naming

**Term status: vendor-specific.** The label is promoted mainly by ONEiO, a Helsinki-based managed integration service provider. Its
April 2026 explainer defines Integration Operations as "the structured, continuous management of integrations as an operational
capability" and says "ONEiO pioneers Integration Ops as a managed service"[^oneio-what]. ONEiO also writes "IntOps" and offers an
Integration Ops book[^oneio-intops], not reviewed here. A 2025 funding report calls the company "a Managed Integration Service
Provider delivering IntegrationOps"[^techeu]. We found no standards body, independent definition, certification or vendor-neutral
community using the label, so this page treats it as a vendor's name for a practice documented since Hohpe and Woolf's *Enterprise
Integration Patterns* (2003)[^eip-book]. ONEiO's offer, which its co-founder calls "a domain-specific, managed operating model for IT
services"[^oneio-founder], is one way to staff that practice ([Deep comparisons](#deep-comparisons)).

::: note Editorial distinction
These boundaries are this Atlas's choices; no standard defines them. **API Ops owns the contract; IntegrationOps owns the flow.** Whether
a field may be removed from an event schema is a question for [API Ops](../api-ops/#the-contract-gate-what-blocks-a-merge); what happens
to messages already written with the old schema, and to the one that fails on the new consumer, belongs here.
:::

Beyond the neighbours under *Out of scope* above, it is **not "integrated operations"** in oil and gas, Norway's term for real-time onshore–offshore work processes; the Research Council of
Norway funded a Center for Integrated Operations in the Petroleum Industry from 2007 to 2015[^io-norway]. That is a name collision.

## Where it comes from and where it sits

### Origins and evolution

Hohpe and Woolf named four integration styles (file transfer, shared database, remote procedure
invocation, messaging)[^eip-home] and described the central message broker, "sometimes referred to as hub-and-spoke architectural
style"[^eip-broker]. Enterprise service bus (ESB) products then put routing, transformation and business rules into middleware, until
Lewis and Fowler's 2014 microservices article argued for "smart endpoints and dumb pipes", naming the ESB as the example of putting
"significant smarts into the communication mechanism itself"[^fowler-micro]. Log-based brokers such as Apache Kafka keep events after
consumption[^kafka-intro], which made replay routine, and iPaaS moved connectors and mappings into SaaS. The *IntegrationOps* label dates from 2025–2026 marketing[^techeu][^oneio-what]; the problems it names are two decades old.

### Glossary and mental model

| Term | Meaning | Why it matters |
|---|---|---|
| At-most-once / at-least-once / exactly-once | Messages "may be lost but are never redelivered" / "are never lost but may be redelivered" / are "processed once and only once" (Kafka's definitions)[^kafka-semantics] | At-least-once plus idempotence is the usual target |
| Idempotent receiver | A receiver "that can safely receive the same message multiple times"[^eip-idem] | The key property of a consumer |
| Dead-letter channel (DLQ) | Where a messaging system puts a message "it cannot deliver"[^eip-dlc], or one that failed too often | Parks failures; the flow keeps moving |
| Invalid message channel | Where a receiver puts a message "that makes no sense"[^eip-invalid] | Parked at once, not retried |
| Poison message | A message that fails every delivery | Blocks ordered flows |
| Redrive / replay | Moving parked messages back to a live queue / re-reading events from a retained log | Both repeat side effects |
| Transactional outbox | Store the message in the sender's database in the same transaction as the change, then relay it[^outbox] | No "saved the order, lost the event" |
| Consumer lag | How far a consumer group is behind the end of a log | Backlog signal for streaming |
| Lease | The time a broker lends a message to one consumer (SQS visibility timeout, Service Bus lock) | Outlast it and the message is delivered twice |
| CDC | Change data capture: publishing a database's committed changes from its log | Feeds outboxes and search indexes |
| SLO | Service level objective: a measurable target | Makes "reliable" a number |
| Reconciliation | Comparing business records at source and target and explaining every difference | The only proof of completeness |

**Mental model: four parts and a ledger.** A *contract* (what a message means), a *transport* (broker, queue, file or API call), a
*handler* (message to side effect) and a *recovery path* (retry, park, redrive, replay), plus the *ledger*: business records on both
sides that prove the flow worked. Most incidents are one part assuming another's guarantee.

### How IntegrationOps relates to other Ops

| Neighbour | Shared scope | This page's focus | Go there when |
|---|---|---|---|
| [API Ops](../api-ops/) | Schemas, AsyncAPI | Delivery, retries, DLQs, replay, reconciliation | You are deciding whether a change is breaking |
| [WorkflowOps](../workflowops/) | Retries, idempotency, sagas | Messages between independently owned systems | One engine owns the state of a multi-step process |
| [ServerlessOps](../serverlessops/) | Queues, DLQs, redrive | Broker-agnostic governance across brokers and teams | You tune a Lambda event source mapping |
| [DataOps](../dataops/) | Kafka, CDC, schemas | Operational flows whose consumers have side effects | The consumer is a warehouse or a model |
| [ObservabilityOps](../observabilityops/) | Traces, metrics | Which flow signals matter | You run the telemetry pipeline itself |
| [IncidentOps](../incidentops/) | Incidents | The technical recovery of a flow | You need roles and communications |
| [MarTechOps](../martechops/) | SaaS-to-SaaS syncs | Company-wide integration engineering | The question is the marketing tool stack |

## Principles and operating model

### Charter and principles

There is no official IntegrationOps manifesto; this charter is **this Atlas's proposal**. **Mission:** every business record that should
cross a boundary arrives, in effect, exactly once, or is visibly parked with a named owner and a deadline.

1. **Assume duplicates.** Exactly-once features stop at the broker's edge ([FAQ](#faq)).
2. **Deduplicate on business identity,** never on a broker's delivery identifier, which changes on redrive[^sqs-redrive].
3. **Retry in one place.** In a five-deep stack retrying three times per layer, load on the bottom tier can rise 243-fold[^builders-retries].
4. **Park, don't loop.** Messages that cannot succeed leave the live path quickly, with their failure context.
5. **Every parked message has an owner and a clock.**
6. **Replay is a change:** approved, rate-limited and verified.
7. **Prove completeness in business terms.** Broker metrics cannot show what never entered the broker.

**Non-goals:** one broker for everything; all mapping logic in one central team (the ESB lesson[^fowler-micro]).

### Fundamentals and building blocks

| Block | Job (products: [Tooling landscape](#tooling-landscape)) |
|---|---|
| Transport | Holds and delivers messages: broker, managed queue, SFTP drop |
| Integration runtime | Connectors, mapping, routing |
| Contract and schema store | Defines and versions messages (AsyncAPI[^asyncapi], a schema registry[^sr-evolution]) |
| Outbox relay | Publishes database changes reliably |
| Inbox | Remembers processed business keys in the consumer's database |
| Parking and replay | DLQs, SQS message move tasks, Kafka offset resets |
| Telemetry and reconciliation | Lag, age, traces; end-to-end record counts |
| Integration register | Owner and guarantees per flow ([the flow card](#1-the-flow-card)) |

### Roles, ownership and RACI

Integrations fail between teams. The **flow owner**, a person rather than a mailbox, is accountable for the flow's guarantees, DLQ and
reconciliation.

| Activity | Flow owner | Producer team | Consumer team | Platform (broker) team | Business data owner |
|---|---|---|---|---|---|
| Define delivery guarantee and SLO | A | C | C | C | C |
| Schema change | A | R | C | I | I |
| Idempotent handler | A | I | R | I | I |
| Broker policies (DLQ, limits, retention) | A | I | C | R | I |
| DLQ triage within SLA | A | C | R | C | I |
| Approve redrive or replay | A | C | R | C | C |
| Reconciliation and gap sign-off | A | C | R | I | C |
| Retire the flow | A | R | R | R | I |

R = responsible, A = accountable, C = consulted, I = informed. One person may hold several columns; each row needs exactly one A.

### Lifecycle: Day 0, Day 1, Day 2

- **Day 0 (design):** name the business event and its identifier; choose the delivery guarantee and ordering scope; register the flow;
  write the failure policy.
- **Day 1 (go-live):** broker policies as code; a duplicate-injection test; a first reconciliation on real data; one redrive rehearsed
  in staging.
- **Day 2 (operate):** scheduled DLQ triage, daily reconciliation, schema reviews, quarterly replay rehearsal.
- **Retirement:** drain or archive parked messages, delete queues and credentials. Orphaned queues keep costs and personal data
  alive.

### Operating model and governance

Keep it light: an **integration register** (one [flow card](#1-the-flow-card) per flow); a fifteen-minute **weekly DLQ review** where every parked message is fixed, redriven, written off with a reason,
or escalated; **change control by risk** (schema changes through the [API Ops](../api-ops/) contract gate, broker policies through pull
requests, redrive and replay with the flow owner's recorded approval); and **escalation** of completeness or latency SLO breaches as
incidents, coordinated through [IncidentOps](../incidentops/).

## Architecture and workflow

### Reference architecture

A common, vendor-neutral shape for a flow such as order-to-invoice; simpler flows may drop the outbox or the parking queue.

::: figure Reference architecture of a recoverable integration flow
```mermaid
flowchart TB
  subgraph SRC["Producer side"]
    APP["Order service"]:::neutral --> DB[("Orders + outbox table")]:::neutral
    DB --> RELAY["Outbox relay"]:::core
  end
  RELAY --> BRK["Broker: orders topic or queue"]:::core
  BRK --> CON["Consumer with inbox table"]:::core
  CON --> TGT[("ERP: invoices")]:::neutral
  CON -->|"transient error"| RTY["Delayed retry, bounded"]:::decision
  RTY --> BRK
  CON -->|"invalid or poison"| DLQ["Parking queue with failure context"]:::risk
  DLQ --> TRI["Triage, then approved redrive"]:::decision
  TRI --> BRK
  REC["Reconciliation job"]:::core -.-> DB
  REC -.-> TGT
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: the order service writes the order and an outbox row in one transaction; a relay publishes the row to the broker.
A consumer with an inbox table writes the invoice to the ERP. Transient errors go to a bounded, delayed retry back to the broker;
invalid or poison messages go to a parking queue, whose triage leads to an approved redrive. A reconciliation job compares orders in
the source database with invoices in the ERP.
:::

- **Outbox** closes the dual-write gap, but the relay "might publish a message more than once", so consumers must be idempotent[^outbox].
  Debezium's outbox router keys messages by aggregate ID, keeping one aggregate's events in one Kafka partition[^debezium-outbox].
- **Parking queue** separates "not now" from "never" and keeps failure context (RabbitMQ adds `x-death` headers[^rmq-dlx]).
- **Reconciliation** sits outside the message path, so it catches losses before the broker and after a premature ack.
- **Trust boundaries** separate producer, broker and consumer teams; the redrive path needs the strictest credentials.

### Integration styles compared

| Style | Strengths | Operational weaknesses |
|---|---|---|
| Point-to-point API calls | Simple; immediate errors | Connections grow with the square of the system count; no buffer when the target is down |
| Hub-and-spoke broker or ESB | Central routing and control[^eip-broker] | Central team becomes a bottleneck[^fowler-micro] |
| Event log (streaming) | Replay from retention; per-key ordering[^kafka-intro] | Lag and offset management; replay duplicates by design |
| Managed queues | No brokers to run; built-in DLQs | Provider-specific limits and semantics |
| iPaaS | SaaS connectors and mappings | Limited control of retries and idempotence |
| File transfer | Simple, auditable | Latency; partial files; whole-batch reprocessing |

Which to choose is in the [Decision framework](#decision-framework).

### End-to-end workflow: happy path and exception path

::: figure One message through the flow: duplicate and poison paths
```mermaid
sequenceDiagram
  participant P as Producer and outbox
  participant B as Broker
  participant C as Consumer
  participant D as Consumer database
  P->>B: publish order-123 (relay)
  B-->>P: publisher confirm
  B->>C: deliver order-123
  C->>D: insert inbox key, write local invoice row
  D-->>C: commit
  C--xB: ack lost (consumer restarts)
  B->>C: redeliver order-123
  C->>D: insert inbox key
  D-->>C: key exists, skip
  C->>B: ack
  B->>C: deliver order-124 (malformed)
  C->>B: reject, no requeue
  Note over B: dead-letter to parking queue with reason
```
Text equivalent: the relay publishes order-123 and gets a publisher confirm. The consumer inserts the inbox key and a local invoice
row in one commit, but its acknowledgement is lost when it restarts. On redelivery the inbox insert finds the key, so the consumer
skips the effect and acknowledges. Malformed order-124 is rejected without requeue and dead-lettered to the parking queue.
:::

This is atomic only because the invoice is a local row. With an external ERP no transaction spans inbox and ERP; the ERP's
idempotency key protects that call ([the handler](#3-an-idempotent-handler)).

The consumer's code, not the broker, decides the guarantee. RabbitMQ's publisher confirms and consumer acknowledgements are "entirely
orthogonal" (you need both), and automatic acknowledgement "should be considered unsafe"[^rmq-confirms]. Kafka's design notes say a
consumer that processes and then saves its position gets at-least-once; one that saves its position first gets at-most-once[^kafka-semantics].

## Practices, frameworks and tools

### Core practices and playbooks

Seven playbooks, each with a trigger, owner and output.

#### Playbook 1: make every consumer idempotent

**Trigger:** any new consumer. **Owner:** consumer team. **Output:** one effect per business event.

Hohpe and Woolf give two routes: detect and discard duplicates, or define messages so that a second copy has no further effect[^eip-idem].
"Set stock to 40" is naturally idempotent; "subtract 2" is not. When you cannot change the semantics, use an **inbox**: a unique-key
table written in the same transaction as the side effect.

Choose the key carefully. SQS gives redriven messages "a new `messageID` and `enqueueTime`"[^sqs-redrive], and Pub/Sub publish retries
can produce copies with different message IDs[^pubsub-eod]. Use an identifier the producer assigns once; CloudEvents requires `source` +
`id` to be unique per distinct event and lets consumers treat identical pairs as duplicates[^cloudevents].

For an external API, pass an idempotency key (AWS uses the same caller-provided token idea[^builders-idem]) and learn two rules of
the target. Keys expire: Stripe's v1 API can drop a key once it is at least 24 hours old, and a repeat then counts as a new request.
Keys also remember answers: Stripe v1 returns the saved first response to every retry with that key, "including 500
errors"[^stripe-idem], so a key can block recovery as well as duplicates.

Duplicates also arrive with no crash at all, when the work outlasts the broker's *lease* and another consumer receives the message
([Common failures](#common-failures-and-troubleshooting)). **Verify:** inject every message twice in a test environment; business
totals must not change ([Lab 1](#lab-1-beginner-executed-duplicates-and-an-idempotent-consumer)).

#### Playbook 2: retry in one layer, bounded and jittered

**Trigger:** any call that can fail transiently. **Owner:** consumer team. **Output:** a written retry budget.

Retries hide in libraries (as of October 2026 AWS documents three attempts, including the first, for its updated, still opt-in SDK
retry behaviour[^aws-sdk-retry]); a consumer loop adds a second layer and broker redelivery a third. Marc Brooker recommends retrying "at
a single point in the stack" and warns that "APIs with side effects aren't safe to retry unless they provide idempotency"[^builders-retries].
Record per flow which layer retries, how often, and what happens when the budget is spent (park). Broker-side delay is often the
cleanest layer: RabbitMQ 4.3 added delayed retry for quorum queues, `delay = min(min_delay * delivery_count, max_delay)`[^rmq-qq].
The formula has no random part, so messages that fail together return together; add jitter in the client if that hurts the target.

#### Playbook 3: classify failures before acting

**Trigger:** any handler error. **Owner:** consumer team. **Output:** a failure policy in code.

::: figure Classifying a failed message before deciding what to do
```mermaid
flowchart TB
  E["Handler error"]:::risk --> Q1{"Can the payload ever succeed?"}:::decision
  Q1 -->|"no: malformed or invalid"| PARK["Park now with reason, no retries"]:::risk
  Q1 -->|"yes"| Q2{"Is the downstream system down or throttling?"}:::decision
  Q2 -->|"yes"| PAUSE["Pause or slow consumption; keep messages in the queue"]:::core
  Q2 -->|"no"| Q3{"Transient error with budget left?"}:::decision
  Q3 -->|"yes"| RETRY["Delayed retry with backoff"]:::core
  Q3 -->|"no: budget spent or unknown"| DL["Dead-letter with attempt history"]:::risk
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
```
Text equivalent: if the payload can never succeed (malformed or invalid), park it at once without retries. If the downstream system is
down or throttling, pause or slow consumption and leave messages queued. Otherwise retry transient errors with backoff while budget
remains; when the budget is spent or the error is unknown, dead-letter the message with its attempt history.
:::

An *invalid* message[^eip-invalid] needs a producer fix; a *dead letter*[^eip-dlc] often needs only time. Pause during downstream
outages: most queues count redeliveries towards a dead-letter threshold, so an outage turns healthy messages into dead letters.

#### Playbook 4: govern redrive and replay

**Trigger:** fixable parked messages, or state to rebuild. **Owner:** flow owner approves; consumer team executes. **Output:** a replay
record ([Templates](#templates)).

Both re-execute side effects, so first confirm the root cause is fixed, the dedup key survives, external keys are valid and hold no
stored failure, and the rate suits the downstream system. SQS redrive caps custom velocity at 500 messages per second and "doesn't support filtering and
modifying messages while redriving"[^sqs-redrive]. Kafka offset resets need inactive consumer instances and only plan the reset until
you add `--execute`[^kafka-ops]; replay is bounded by topic retention, seven days by default[^kafka-topic].

#### Playbook 5: evolve schemas without breaking replay

**Trigger:** any schema change. **Owner:** producer team; the contract gate is [API Ops](../api-ops/#the-contract-gate-what-blocks-a-merge).
**Output:** a compatibility mode per subject and an upgrade order.

Confluent Schema Registry defaults to `BACKWARD` so that you can "rewind consumers to the beginning of the topic", but the default is
non-transitive: new schemas are checked "only against the latest schema"[^sr-evolution]. Version 3 may fail on version 1 data
although each step passed. For subjects replayed from the start, use `BACKWARD_TRANSITIVE` or `FULL_TRANSITIVE`. Upgrade
consumers first under backward compatibility, producers first under forward, either under full[^sr-evolution].

#### Playbook 6: buy ordering only where it is needed

**Trigger:** a consumer that breaks on disorder. **Owner:** flow owner. **Output:** an explicit ordering scope.

Ordering is per partition, queue or group: Kafka puts events with the same key in one partition and consumers read a partition in write
order[^kafka-intro]. The price is head-of-line blocking; in an SQS FIFO queue a failed message "blocks its message group until it's
deleted or expires"[^sqs-metrics], and AWS advises against giving a FIFO queue a dead-letter queue at all where exact order must
hold[^sqs-dlq]. Kafka share groups (production-ready since 4.2[^kafka-42]) make the opposite trade: several consumers share a
partition and acknowledge records one by one[^kafka-semantics], which buys broker-counted retries, but records "can be delivered out of
order to a consumer, in particular when redeliveries occur"[^kip-932]. Prefer consumers that compare versions and ignore stale updates; keep strict ordering for
operations that do not commute.

#### Playbook 7: reconcile at the business level

**Trigger:** every flow carrying business value. **Owner:** consumer team; flow owner signs off. **Output:** a completeness report per
closed window.

Compare business records at source and target for a closed window and explain each difference: in flight, parked, written off, or
*unexplained*, the flow's key metric ([Metrics and KPIs](#metrics-and-kpis)). Hohpe's "Starbucks Does Not Use Two-Phase Commit"
names three responses: write-off, retry, and a compensating action "to put the system back into a consistent state"[^starbucks].

### Methods, standards and frameworks

| Source | Type | Use it for |
|---|---|---|
| *Enterprise Integration Patterns*[^eip-book] | Pattern language | Shared vocabulary: dead-letter channel, idempotent receiver, message store[^eip-store] |
| CloudEvents 1.0, CNCF graduated[^cloudevents][^cloudevents-cncf] | Specification | A common event envelope with required `id`, `source`, `specversion`, `type` |
| AsyncAPI 3.1.0[^asyncapi] | Specification | Contracts for message-driven APIs (owned by [API Ops](../api-ops/)) |
| OpenTelemetry messaging conventions[^otel-msg] | Conventions, status Development | Producer and consumer spans linked across brokers |
| Schema-registry compatibility rules[^sr-evolution] | Vendor documentation | Schema evolution enforced by the registry |
| Pact message testing[^pact] | Testing method | Consumer-driven contracts for messages, broker abstracted away |

Pact's own guidance fits integration hubs: for a "pass through" API that forwards to a queue without validating, the contract that
matters is "between your consumer and the downstream system"[^pact].

### Tooling landscape

| Category | Representative options | Selection criteria |
|---|---|---|
| Self-run broker | RabbitMQ, Apache Kafka | Team skill, replay needs, ordering, operations budget |
| Managed messaging | Amazon SQS/SNS/EventBridge, Azure Service Bus, Google Pub/Sub | Cloud alignment, limits, DLQ and redrive features |
| Integration runtime | Apache Camel, Kafka Connect, iPaaS products | Connector coverage, control over retries and idempotence, exportable configuration |
| CDC and outbox | Debezium[^debezium-outbox], polling publishers | Database support, ordering by key |
| Schema registry | Confluent Schema Registry and alternatives | Formats, transitive modes |
| Observability | Broker metrics, OpenTelemetry, APM tools | Per-flow views, business identifiers in traces |
| Managed integration service | Vendors such as ONEiO | See [Deep comparisons](#deep-comparisons) |

### Delivery and dead-letter behaviour by broker

Defaults decide failure behaviour more than architecture diagrams do. As of October 2026:

| Broker | Default delivery | Retry limit before dead-letter | Dead-letter caveat |
|---|---|---|---|
| Amazon SQS standard | At-least-once[^sqs-alo] | `maxReceiveCount` in the redrive policy, 1–1,000[^sqs-attrs] | DLQ expiry uses the original enqueue time, so make DLQ retention longer than the source's[^sqs-dlq] |
| RabbitMQ quorum queue (4.x) | Depends on acks and confirms[^rmq-confirms] | `delivery-limit` defaults to 20 since 4.0[^rmq-qq] | Dead-lettering is at-most-once unless `dead-letter-strategy` is `at-least-once` with `overflow` `reject-publish`[^rmq-qq] |
| Apache Kafka 4.3, consumer groups | At-least-once[^kafka-semantics] | None in the broker; the consumer decides | Kafka Connect sink connectors offer `errors.deadletterqueue.topic.name`, empty by default[^kafka-connect] |
| Apache Kafka 4.3, share groups (production-ready since 4.2[^kafka-42]) | Records acknowledged one by one; one partition can feed several consumers[^kafka-semantics] | Broker counts deliveries: `group.share.delivery.count.limit`, default 5, range 2–10[^kafka-broker]; a consumer can *Reject* a record as unprocessable[^kafka-semantics] | No dead-letter topic in the 4.3 documentation: log or publish the record yourself before rejecting it |
| Azure Service Bus | Peek-lock redelivery | Max delivery count, default 10[^asb-dlq] | No automatic cleanup; time-to-live is not observed in the DLQ[^asb-dlq] |
| Google Pub/Sub | At-least-once, unordered, by default[^pubsub-subs]; exactly-once optional for pull[^pubsub-eod] | Max delivery attempts 5–100, default 5, approximate[^pubsub-dlq] | Service agent needs the publisher role on the dead-letter topic and the subscriber role on the source subscription; attempts are counted only when this is set up[^pubsub-dlq] |

A RabbitMQ detail catches teams out: since 4.3 the delivery limit counts only genuine failures (`basic.reject`, client crashes, or
AMQP 1.0 `modified` with `delivery-failed=true`), so "unlimited explicit message returns" by `basic.nack` are allowed[^rmq-qq]. A
consumer that nacks a poison message with requeue loops for ever. Delayed retry of type `all` or `returned` only spaces those returns
out, and type `failed` does not delay them at all[^rmq-qq]. Only `basic.reject`, or an attempt counter that then rejects or parks,
ends the loop.

### Automation and AI

Automate the mechanical parts: policies as code, DLQ alarms, reconciliation queries, rate-limited redrive scripts, duplicate-injection
tests. Language models can draft field mappings or cluster DLQ errors, but mappings change money, so a human approves them and contract
tests check them. No agent redrives or replays without the approval in [Playbook 4](#playbook-4-govern-redrive-and-replay).

## IntegrationOps in practice

### Reference implementation

One flow, **order-to-invoice**: an order service on PostgreSQL publishes `OrderPlaced` events; a billing consumer creates invoices
through an ERP API. Assumes RabbitMQ 4.3+ quorum queues (or Amazon SQS) and PostgreSQL; all names are placeholders.

#### 1. The flow card

Keep the register entry in Git beside the broker policies.

_Example status: YAML syntax-checked by the Atlas validator; not executed._

```yaml
flow: order-to-invoice
owner: jane.doe@example.com          # a person, not a mailbox
producer: {team: shop, system: order-service}
consumer: {team: billing, system: invoice-consumer}
contract: https://git.example.com/contracts/order-placed/asyncapi.yaml
business_key: order_id               # survives redrive and replay
delivery: at-least-once
idempotence: inbox table billing.inbox(msg_key)
ordering: none                       # consumer ignores stale updates
transport: {broker: rabbitmq, queue: orders.in, parking: orders.parked}
retry: {layer: broker-delayed-retry, limit: 10, min_delay_s: 1, max_delay_s: 10}
retention: {parking_ttl_days: 14}    # message-ttl policy on orders.parked
slo:
  completeness_unexplained: 0        # per closed day
  latency_p95_minutes: 5
reconciliation: sql/recon_order_to_invoice.sql
runbook: https://wiki.example.com/runbooks/order-to-invoice
replay_approval: flow owner
```

#### 2. Outbox and inbox tables

_Example status: illustrative PostgreSQL DDL; not executed._

```sql
-- Producer: the event is written in the same transaction as the order.
CREATE TABLE outbox (
  id            uuid PRIMARY KEY,          -- event id, unique per event
  aggregatetype text NOT NULL,             -- e.g. 'order'
  aggregateid   text NOT NULL,             -- order_id: the business key
  type          text NOT NULL,             -- e.g. 'OrderPlaced'
  payload       jsonb NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now()
);

-- Consumer: one row per processed business event.
CREATE TABLE inbox (
  msg_key      text PRIMARY KEY,           -- 'order:' || order_id [|| ':r2' on a re-send]
  processed_at timestamptz NOT NULL DEFAULT now()
);
```

The column names follow the defaults of Debezium's outbox event router, which routes by `aggregatetype` and uses `aggregateid` as the
message key[^debezium-outbox].

#### 3. An idempotent handler

_Example status: Python syntax-checked by the Atlas validator; not executed (written for psycopg2; needs a broker client and an
ERP)._

```python
import json


class PermanentError(Exception):
    """The message can never succeed: park it without retries."""


def handle(conn, erp, body: bytes) -> None:
    """Process one OrderPlaced message at least once, with one effect."""
    try:
        event = json.loads(body)
        order_id = str(event["order_id"])
        amount = int(event["amount_cents"])
        # Set (e.g. "r2") only on re-sends confirmed absent in the ERP.
        generation = event.get("key_generation")
    except (ValueError, KeyError, TypeError, AttributeError) as exc:
        raise PermanentError(f"invalid payload: {exc}") from exc

    key = f"order:{order_id}" + (f":{generation}" if generation else "")
    with conn:  # psycopg2: commits or rolls back (psycopg 3 also closes)
        cur = conn.cursor()
        cur.execute(
            "INSERT INTO inbox (msg_key) VALUES (%s) "
            "ON CONFLICT (msg_key) DO NOTHING",
            (key,),
        )
        if cur.rowcount == 0:
            return  # duplicate delivery: already processed
        # External side effect: the ERP deduplicates on the same key.
        # If this call raises, the inbox row rolls back and the message
        # is retried with the same key. If the ERP stored a failure for
        # that key, it may replay it: see the runbook's recovery step.
        erp.create_invoice(order_id=order_id, amount_cents=amount,
                           idempotency_key=key, timeout_s=10)
```

If the ERP call fails, the inbox row rolls back and the retry cannot create a second invoice. It may not create the first either: a
target that stores first responses returns the saved error until the key expires. The transaction and its row lock stay open during
the ERP call, so keep the timeout short. If the ERP cannot deduplicate, use a "requested"/"confirmed" status table plus
reconciliation, not a cleverer retry.

#### 4. Broker policies as code

One RabbitMQ policy carries every setting, because at most one policy applies to a queue[^rmq-policies]. At-least-once dead-lettering
needs `overflow` set to `reject-publish` and the `stream_queue` feature flag; `max-length` stops unbounded growth[^rmq-qq]. Side
effects: at the limit the broker refuses publishes (a `basic.nack` when confirms are on)[^rmq-maxlength], so the outbox relay must keep those rows and
retry; and dead-lettered messages stay in the source queue until the target accepts them[^rmq-qq], so a missing dead-letter target
makes `orders.in` grow. Alarm on its length.

_Example status: `bash -n` syntax-checked; not executed._

```bash
#!/usr/bin/env bash
set -euo pipefail
# Live queue: dead-letter to orders.dlx at least once, cap growth.
# Type "failed" delays rejects and crashes (1 s, 2 s ... 10 s: about
# 55 s in all before dead-lettering) but not basic.nack returns, so
# the consumer must reject, never nack, a message that fails.
rabbitmqctl set_policy orders-in '^orders\.in$' \
  '{"dead-letter-exchange": "orders.dlx",
    "dead-letter-strategy": "at-least-once",
    "overflow": "reject-publish",
    "max-length": 1000000,
    "delivery-limit": 10,
    "delayed-retry-type": "failed",
    "delayed-retry-min": 1000,
    "delayed-retry-max": 10000}' \
  --apply-to quorum_queues --priority 10
# Parking queue: RabbitMQ keeps messages until consumed, so set a
# 14-day TTL; expired parked messages are dropped.
rabbitmqctl set_policy orders-parked '^orders\.parked$' \
  '{"message-ttl": 1209600000}' --apply-to quorum_queues --priority 10
```

The same intent on Amazon SQS: a 14-day dead-letter queue (the maximum retention[^sqs-attrs]) that only the source queue may use, and a
source queue that dead-letters after ten receives.

_Example status: `bash -n` syntax-checked; not executed (needs an AWS account; creates billable resources)._

```bash
#!/usr/bin/env bash
set -euo pipefail
: "${REGION:?}" "${ACCOUNT_ID:?}"
SRC_ARN="arn:aws:sqs:${REGION}:${ACCOUNT_ID}:orders-in"
DLQ_URL=$(aws sqs create-queue --queue-name orders-parked \
  --attributes MessageRetentionPeriod=1209600 \
  --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "${DLQ_URL}" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)
ALLOW=$(jq -cn --arg src "${SRC_ARN}" \
  '{redrivePermission: "byQueue", sourceQueueArns: [$src]}')
aws sqs set-queue-attributes --queue-url "${DLQ_URL}" \
  --attributes "$(jq -cn --arg p "${ALLOW}" '{RedriveAllowPolicy: $p}')"
REDRIVE=$(jq -cn --arg dlq "${DLQ_ARN}" \
  '{deadLetterTargetArn: $dlq, maxReceiveCount: "10"}')
aws sqs create-queue --queue-name orders-in --attributes \
  "$(jq -cn --arg p "${REDRIVE}" '{RedrivePolicy: $p}')"
```

#### 5. The reconciliation query

Run it per closed window so in-flight messages do not count as gaps.

_Example status: illustrative SQL for a database that can see both tables (a replica or a reporting copy); not executed._

```sql
-- Orders placed yesterday with no invoice, no parked message and no
-- write-off. ops.parked_messages: business keys recorded by the DLQ
-- triage tool, including written-off keys (status 'written_off').
SELECT o.order_id, o.created_at
FROM   shop.orders o
LEFT   JOIN billing.invoices i ON i.order_id = o.order_id
LEFT   JOIN ops.parked_messages p ON p.business_key = o.order_id
WHERE  o.created_at >= date_trunc('day', now()) - interval '1 day'
AND    o.created_at <  date_trunc('day', now())
AND    i.order_id IS NULL
AND    p.business_key IS NULL;   -- these are the unexplained gaps
```

#### 6. Alarms that see parked messages

AWS recommends `ApproximateNumberOfMessagesVisible` for DLQs, since automatic moves are not counted by `NumberOfMessagesSent`; and on
the source queue, "Poison-pill messages ... are excluded from this metric until successfully processed" (`ApproximateAgeOfOldestMessage`)[^sqs-metrics].

_Example status: `bash -n` syntax-checked; not executed._

```bash
#!/usr/bin/env bash
set -euo pipefail
: "${SNS_TOPIC_ARN:?}"
aws cloudwatch put-metric-alarm \
  --alarm-name orders-parked-not-empty \
  --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=orders-parked \
  --statistic Maximum --period 300 --evaluation-periods 1 \
  --threshold 0 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions "${SNS_TOPIC_ARN}"
```

### Hands-on labs

Lab 1 was **executed** for this page (Python 3.8.10). Labs 2 and 3 were **not executed** (the author's environment was a shared
server); their commands are syntax-checked against the cited documentation and source. Use a laptop or a disposable VM with Docker.

#### Lab 1 (beginner, executed): duplicates and an idempotent consumer

**Objective.** See at-least-once delivery inflate a ledger, and see why the deduplication key must survive redrive.
**Prerequisites.** Python 3.8 or later; no network. **Cost and risk.** None (in-memory SQLite).

1. Save the script as `lab1.py`. It simulates a broker that redelivers 20 % of messages because the consumer "crashes" after the side
   effect but before the acknowledgement, then redrives 50 messages with new broker identifiers.
2. Run `python3 -I lab1.py`.

_Example status: executed on 2026-10-23 with Python 3.8.10; output below is the real output._

```python
"""At-least-once delivery, duplicates and an idempotent consumer."""
import random
import sqlite3
import uuid

CRASH_RATE = 0.2  # consumer dies after the side effect, before the ack


def make_events(n):
    # The producer assigns a business event id once.
    return [{"event_id": f"order-{i}", "amount": 10} for i in range(n)]


def deliver(events, handler):
    """At-least-once: a message is redelivered until it is acknowledged."""
    queue = [dict(e, broker_id=str(uuid.uuid4())) for e in events]
    deliveries = 0
    while queue:
        msg = queue.pop(0)
        deliveries += 1
        handler(msg)                      # side effect happens here
        if random.random() < CRASH_RATE:  # crash before ack: redelivery
            queue.append(msg)
    return deliveries


def redrive(events):
    """A redrive re-publishes copies: new broker ids, same business ids."""
    return [dict(e, broker_id=str(uuid.uuid4())) for e in events]


def run(name, key):
    db = sqlite3.connect(":memory:")
    db.execute("CREATE TABLE ledger (amount INTEGER)")
    db.execute("CREATE TABLE inbox (msg_key TEXT PRIMARY KEY)")

    def handler(msg):
        with db:  # one transaction: dedup record + side effect
            if key is not None:
                cur = db.execute("INSERT OR IGNORE INTO inbox VALUES (?)",
                                 (msg[key],))
                if cur.rowcount == 0:
                    return  # duplicate: skip the side effect
            db.execute("INSERT INTO ledger VALUES (?)", (msg["amount"],))

    events = make_events(1000)
    n = deliver(events, handler)
    for msg in redrive(events[:50]):  # an operator redrives 50 messages
        handler(msg)
    total = db.execute("SELECT SUM(amount) FROM ledger").fetchone()[0]
    print(f"{name:<24} deliveries={n} ledger_total={total} expected=10000")


random.seed(7); run("no dedup", None)
random.seed(7); run("dedup on broker id", "broker_id")
random.seed(7); run("dedup on business id", "event_id")
```

**Expected output** (seed 7):

```text
no dedup                 deliveries=1277 ledger_total=13270 expected=10000
dedup on broker id       deliveries=1277 ledger_total=10500 expected=10000
dedup on business id     deliveries=1277 ledger_total=10000 expected=10000
```

**Validation.** 1,277 deliveries for 1,000 events is close to the 1,250 a 20 % redelivery rate predicts. Deduplicating on the broker
identifier misses the 50 redriven copies (+500), as with SQS redrive[^sqs-redrive]. Only the business key gives 10,000. **Troubleshooting.** Another Python version may produce different
numbers; the ordering of the three totals is what matters. **Teardown.** Delete the file.

#### Lab 2 (intermediate): poison messages on a RabbitMQ quorum queue

**Objective.** Watch a poison message reach the delivery limit and be dead-lettered, then see why `basic.nack` with requeue does not
count towards the limit on RabbitMQ 4.3 and later[^rmq-qq]. **Prerequisites.** Docker; Python 3.8+ with `pika` (1.4.4 was current on
2026-10-23[^pika]). **Cost and risk.** Local only. The image's default login is `guest` / `guest`[^rmq-docker], so the ports are
bound to 127.0.0.1; plain `-p 5672:5672` would expose the broker to everyone on your network.

1. Start a broker (image tag `4.3-management`[^rmq-docker]) and apply a policy to the live queue only; the anchored pattern keeps the
   parking queue from dead-lettering into itself.

_Example status: `bash -n` syntax-checked; not executed._

```bash
#!/usr/bin/env bash
set -euo pipefail
docker run -d --name rmq-lab \
  -p 127.0.0.1:5672:5672 -p 127.0.0.1:15672:15672 \
  rabbitmq:4.3-management
sleep 30   # wait for the node to boot
docker exec rmq-lab rabbitmqctl set_policy lab '^orders\.in$' \
  '{"dead-letter-exchange": "orders.dlx", "delivery-limit": 3}' \
  --apply-to quorum_queues --priority 10
```

2. Run `python3 lab2.py reject`, then `python3 lab2.py nack`. The script publishes one malformed order and refuses it on every
   delivery; a safety stop acknowledges it after ten deliveries.

_Example status: Python syntax-checked by the Atlas validator; not executed._

```python
import sys

import pika

MODE = sys.argv[1] if len(sys.argv) > 1 else "reject"
conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.exchange_declare("orders.dlx", exchange_type="fanout", durable=True)
ch.queue_declare("orders.parked", durable=True,
                 arguments={"x-queue-type": "quorum"})
ch.queue_bind("orders.parked", "orders.dlx")
ch.queue_declare("orders.in", durable=True,
                 arguments={"x-queue-type": "quorum"})
ch.basic_publish("", "orders.in", b'{"order_id": "A-1", "amount": "ten"}',
                 pika.BasicProperties(delivery_mode=2, message_id="A-1"))

seen = 0
for method, props, body in ch.consume("orders.in", inactivity_timeout=5):
    if method is None:
        break  # nothing delivered for 5 s: dead-lettered or done
    seen += 1
    count = (props.headers or {}).get("x-delivery-count", 0)
    print(f"delivery {seen}: x-delivery-count={count}")
    if seen >= 10:
        ch.basic_ack(method.delivery_tag)  # safety stop for the lab
        print("stopped after 10 deliveries")
        break
    if MODE == "reject":
        ch.basic_reject(method.delivery_tag, requeue=True)
    else:
        ch.basic_nack(method.delivery_tag, requeue=True)
ch.cancel()
conn.close()
```

3. After each run, list the queues: `docker exec rmq-lab rabbitmqctl list_queues name type messages`.

**Expected output.** `reject`: `x-delivery-count` rises on each redelivery; once it exceeds the limit the message leaves `orders.in`
and `orders.parked` shows 1 message. `nack`: the count stays at 0, the loop hits the safety stop, and `orders.parked` does not grow.
**Validation.** In the management UI (`http://localhost:15672`) the parked message carries `x-death` and `x-first-death-reason`
headers[^rmq-dlx]. **Troubleshooting.** Nothing parked after `reject`: check `rabbitmqctl list_policies` and that `orders.in` is a
quorum queue. On 4.0–4.2 the counting rules differ, so the `nack` run may also be dead-lettered.
**Teardown.** `docker rm -f rmq-lab`.

#### Lab 3 (advanced): replay a Kafka consumer group from a timestamp

**Objective.** Practise the replay procedure on a log, and see that replay delivers duplicates by design. **Prerequisites.** Docker;
the `apache/kafka:4.3.1` image[^kafka-quickstart] (scripts live in `/opt/kafka/bin`[^kafka-docker]). **Cost and risk.** Local only;
the broker has no authentication, so its port is bound to 127.0.0.1.

_Example status: `bash -n` syntax-checked; option names checked against the Kafka 4.3.1 source[^kafka-src]; not executed._

```bash
#!/usr/bin/env bash
set -euo pipefail
K=/opt/kafka/bin
BS=localhost:9092
docker run -d --name kafka-lab -p 127.0.0.1:9092:9092 apache/kafka:4.3.1
sleep 20
docker exec kafka-lab $K/kafka-topics.sh --create --topic orders \
  --partitions 3 --bootstrap-server $BS
START=$(date -u +%Y-%m-%dT%H:%M:%S.000)
# Ten keyed events; each key always maps to the same partition.
for i in $(seq 1 10); do echo "order-$i:{\"order_id\":$i}"; done |
  docker exec -i kafka-lab $K/kafka-console-producer.sh --topic orders \
    --bootstrap-server $BS \
    --reader-property parse.key=true --reader-property key.separator=:
# First pass: the "billing" group reads everything and commits offsets.
docker exec kafka-lab $K/kafka-console-consumer.sh --topic orders \
  --group billing --from-beginning --timeout-ms 10000 \
  --bootstrap-server $BS
docker exec kafka-lab $K/kafka-consumer-groups.sh --describe \
  --group billing --bootstrap-server $BS
# Plan the replay with an explicit dry run, then execute it.
docker exec kafka-lab $K/kafka-consumer-groups.sh --reset-offsets \
  --group billing --topic orders --to-datetime "$START" --dry-run \
  --bootstrap-server $BS
docker exec kafka-lab $K/kafka-consumer-groups.sh --reset-offsets \
  --group billing --topic orders --to-datetime "$START" --execute \
  --bootstrap-server $BS
docker exec kafka-lab $K/kafka-console-consumer.sh --topic orders \
  --group billing --timeout-ms 10000 --bootstrap-server $BS
```

**Expected output.** The first consume prints ten values; `--describe` shows `LAG` 0 on every partition[^kafka-ops]. The dry run
prints the planned offsets without changing them; after `--execute`, the second consume prints the same ten values: a replay is a
deliberate duplicate. **Validation.** Run `--describe`
after the reset and confirm the `LAG` values add up to ten. **Troubleshooting.** The reset fails while a group member is active[^kafka-ops];
`--to-datetime` expects `YYYY-MM-DDThh:mm:ss.sss`; `--property` is deprecated since 4.2 in favour of `--reader-property`, and
Kafka 5.0 will require `--dry-run` or `--execute`[^kafka-src]. **Teardown.** `docker rm -f kafka-lab`.

### Use cases and scenarios

All four are **illustrative scenarios**, not documented case studies.

| Scenario | Method | Trade-off | Success measure | Pitfall |
|---|---|---|---|---|
| Order-to-invoice between two teams | The reference implementation | A relay to operate | Zero unexplained gaps per day; p95 under five minutes | Deduplicating on broker message IDs |
| Service-desk exchange with an IT supplier | Tickets mirrored both ways between two ITSM tools, by a managed service or in-house iPaaS | A managed service saves staff but holds logs and mappings | Every update mirrored in the agreed time | Echo loops |
| CDC to a search index | Debezium reads the database log; the index is rebuilt by replay | Rebuild time bounded by retention | Index lag under a minute; rebuild rehearsed quarterly | Non-transitive schemas breaking a replay |
| CRM-to-billing sync through an iPaaS | Connectors with API rate limits | Fast to build; little retry control | Daily record counts match | Retries spending the target's rate limit during its outage |

**Echo loops** haunt two-way syncs: each update needs an origin marker or correlation identifier[^eip-corr] so a received change is
not sent back.

## Measure, govern, pay

### Metrics and KPIs

Measure flows, not brokers.

| KPI | Definition and formula | Source | Pitfall |
|---|---|---|---|
| Unexplained gaps (primary, lagging) | Source records in a closed window with no target record, no parked message and no recorded write-off | Reconciliation | Reconciling open windows counts in-flight records as gaps |
| Completeness | Matched records ÷ source records | Reconciliation | 99.9 % can hide 50 lost invoices a day |
| End-to-end latency p95 | Target commit time − business event time | Timestamps or traces | Broker publish time misses outbox delay |
| Oldest unprocessed age (leading) | Age of the oldest waiting message, or consumer lag in time | Broker metrics | SQS excludes poison messages from this metric[^sqs-metrics] |
| Dead-letter rate (leading) | Messages parked ÷ messages consumed | Broker metrics | A low rate at high volume is many records |
| Time to triage | Park to decision (fix, redrive, write off) | Triage log | SQS DLQ expiry counts from the original enqueue time[^sqs-dlq] |
| Duplicate suppression rate | Duplicates skipped ÷ deliveries | Consumer logs | Zero may mean the check is broken |

Review gaps and completeness daily, the rest weekly. **Worked calculation (illustrative).** Yesterday the shop recorded 48,210 orders;
the ERP holds 48,198 invoices; 9 orders are parked with reasons. Completeness = 48,198 ÷ 48,210 = **99.975 %**; unexplained gaps =
48,210 − 48,198 − 9 = **3**. The percentage looks excellent; the three untraced records are the incident
([runbook](#runbook-the-dashboard-is-green-but-invoices-are-missing)). Set the SLO on the count: zero per closed day.

**Backlog drain time** = backlog ÷ (sustainable processing rate − arrival rate). With 180,000 queued messages, 20 new per second and an
ERP that accepts 45 per second: 180,000 ÷ (45 − 20) = 7,200 s, **two hours**. More consumers cannot beat the ERP's limit; they turn
backlog into throttling and, through delivery counts, dead letters.

### Quality, security, privacy and compliance

- **Least privilege on recovery paths.** An SQS redrive needs `sqs:StartMessageMoveTask`, `sqs:ReceiveMessage`, `sqs:DeleteMessage`
  and `sqs:GetQueueAttributes` on the DLQ, `sqs:SendMessage` on the destination, and KMS rights for encrypted queues[^sqs-redrive]. Give
  them to an operator role used through the approval step, not to application roles. A redrive allow policy can limit a DLQ to ten
  named source queues[^sqs-attrs].
- **Parked messages are production data, often kept longer.** Service Bus DLQs have no automatic cleanup and ignore time-to-live[^asb-dlq].
  The GDPR's storage-limitation principle allows identifiable personal data only as long as the purpose needs[^gdpr-5], so set parking
  retention deliberately, purge after triage, and coordinate deletion requests that reach logs and replays with [PrivacyOps](../privacyops/).
- **Failure context can leak secrets.** Microsoft recommends the exception type in `DeadLetterReason` and the stack trace in
  `DeadLetterDescription`[^asb-dlq]; scrub tokens and personal data first.
- **Auditability.** Record every redrive and replay (who, why, range, rate, result). A message store, which captures "information about
  each message in a central location"[^eip-store], is the pattern for a searchable trail; it needs a retention rule too.

### Cost, capacity and sustainability

Cost drivers are retention, headroom and people. Retention buys replay windows and multiplies storage (Kafka keeps seven days by
default[^kafka-topic]; SQS one minute to fourteen days[^sqs-attrs]). Headroom sets drain time. iPaaS and managed messaging are usually
priced by usage, so compare on your own volumes and include triage time. Orphaned queues are idle servers by another name. Estate-wide
spend and carbon belong to [FinOps](../finops/) and [GreenOps](../greenops/).

### Assurance of the operating model

Check quarterly: every production queue, topic and iPaaS flow maps to a register entry with an owner; each tier-1 flow has rehearsed a
redrive or replay in staging, followed by a clean reconciliation; no DLQ lacks an alarm and no parked message exceeds its triage SLA; and
ten sampled records can be traced from source to target, for example with OpenTelemetry messaging spans[^otel-msg].

## When things go wrong

### Common failures and troubleshooting

| Symptom | Likely cause | Diagnosis | Fix and prevention |
|---|---|---|---|
| Duplicate invoices after an incident | Redelivery or redrive with dedup on broker IDs | Group target records by business key | Compensate with the business owner; business-key inbox ([Lab 1](#lab-1-beginner-executed-duplicates-and-an-idempotent-consumer)) |
| DLQ fills during a downstream outage | Healthy messages used up their delivery count | DLQ reasons all say timeout or 503 | Fix downstream, then slow redrive; pause consumption next time ([Playbook 3](#playbook-3-classify-failures-before-acting)) |
| A message loops for hours on RabbitMQ 4.3+ | `basic.nack` with requeue does not count towards `delivery-limit`[^rmq-qq] | `x-acquired-count` rises, `x-delivery-count` stays flat | Use `basic.reject`, or count attempts and park; delayed retry only slows the loop |
| Duplicate effects with no crash or error | Work outlived the lease: SQS visibility timeout (default 30 s[^sqs-vis]), Service Bus lock (default 1 minute, maximum 5[^asb-lock]), RabbitMQ consumer timeout (default 30 minutes[^rmq-consumers]; quorum-queue support since 4.3[^rmq-qq]), Kafka share-group lock (30 s[^kafka-semantics]) | Compare handler p99 with the lease | Extend or renew the lease, or shorten the work; Lambda in [ServerlessOps](../serverlessops/#charter-and-principles) |
| Messages vanish instead of being parked | Missing DLX (dropped silently[^rmq-dlx]); at-most-once dead-lettering; DLQ retention shorter than the source's | Policies, exchanges, retention | Policies as code; replay from source if still retained |
| Kafka group keeps rebalancing, lag grows | A batch takes longer than `max.poll.interval.ms` (default five minutes), so the member is declared failed[^kafka-consumer] | Consumer logs; `kafka-consumer-groups.sh --describe` | Smaller batches or a faster handler |
| New consumer group misses existing events | `auto.offset.reset` defaults to `latest`[^kafka-consumer] | Committed offsets start at the log end | Reset offsets and replay; set the policy explicitly |
| Old events fail after a schema change | Non-transitive compatibility met version 1 data on replay | Registry history | Fix the reader; `BACKWARD_TRANSITIVE` for replayed subjects[^sr-evolution] |
| Dashboards green, records missing | Loss before publish or after a premature ack | Reconciliation | The runbook below; outbox; ack after commit |

### Runbook: the dashboard is green but invoices are missing

::: example Illustrative scenario
The daily reconciliation for order-to-invoice reports 3 unexplained gaps (the worked calculation above). Broker metrics are normal, the
parking queue holds 9 messages with recorded reasons, and no alert fired.
:::

::: figure Runbook for unexplained gaps found by reconciliation
```mermaid
flowchart TB
  A["Unexplained gaps above 0"]:::risk --> B{"In the outbox?"}:::decision
  B -->|"no"| C["Producer defect"]:::risk
  B -->|"yes"| D{"Published?"}:::decision
  D -->|"no"| E["Relay stuck"]:::risk
  D -->|"yes"| F{"Inbox key?"}:::decision
  F -->|"no"| G["Lost in transit or expired"]:::risk
  F -->|"yes"| H["Acked without effect"]:::risk
  C --> R["Fix cause; re-send confirmed gaps"]:::core
  E --> R
  G --> R
  H --> R
  R --> V["Reconcile again: 0 gaps"]:::core
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
```
Text equivalent: trace each missing order along the flow. No outbox row: producer defect. Outbox row never published: relay stuck or
skipping rows. Published but no inbox key: lost in transit or expired, so check retention and the dead-letter exchange. Inbox key but no
invoice: acknowledged without effect. Every branch ends with fixing the cause, re-sending the records confirmed absent in the ERP, and
a reconciliation re-run showing zero gaps.
:::

1. **Declare and scope.** Open an incident; list the three order IDs. Leave the 9 parked messages alone: they are explained.
2. **Trace each key:** outbox row? published? inbox key in billing? ERP invoice with that idempotency key?
3. **Classify.** Here all three have inbox keys and no invoice. Logs show the ERP answered HTTP 202 (accepted) and later rejected the
   documents asynchronously; the handler treated 202 as success.
4. **Contain.** Record "requested", not "done", on 202 and add a confirmation check; deploy normally.
5. **Recover.** Search the ERP by order ID to confirm that the three invoices are absent. Under change control, delete the three false
   inbox keys and re-publish the events with `key_generation` set to `r2` (key `order:<id>:r2`), recorded in the replay request: the original key
   may only replay the stored first response ([Playbook 1](#playbook-1-make-every-consumer-idempotent)). If the ERP documents a
   resubmission path, use that instead.
6. **Verify.** Reconciliation for the window shows zero gaps; the ERP holds exactly one invoice per order ID.
7. **Close.** Postmortem via [IncidentOps](../incidentops/): no broker metric could see this loss. Add a test for "accepted, later
   rejected" responses.

### Anti-patterns and myths

- **Retry everywhere.** Library, consumer and broker retries multiply into a storm[^builders-retries].
- **The DLQ as a bin.** Unowned parked messages expire silently.
- **Redrive everything after an outage.** Without a fixed cause it refills the DLQ; at full speed it recreates the overload.
- **Event notification without a map.** Fowler warns that a logical flow over several event notifications can be hard to see "as it's
  not explicit in any program text"[^fowler-eda]; the register and traces are that map.
- **Monitoring the pipe, not the outcome.** An empty queue can mean "all processed" or "nothing published".

## Deciding and adopting

### When to use and when not to use

**Invest when** valuable records cross team or company boundaries; when finance, customers or auditors would notice a missing or
doubled record; or when flows outlive the projects that built them.

**Do not build the full model when** one team owns both ends and a synchronous call fails loudly; when data is advisory and a lost
update is harmless; or when a durable workflow engine owns the process ([WorkflowOps](../workflowops/)). For two services owned by one
team, a shared database, one of the four classic styles[^eip-home], can be the honest answer.

**Hidden costs:** on-call for cross-team flows, longer retention, weekly triage time, and the political work of naming flow owners.

### Decision framework

Two decisions recur: which style should carry a new flow, and how much operating model it deserves.

::: figure Choosing how a new flow should move data
```mermaid
flowchart TB
  S["New flow between two systems"]:::neutral --> Q1{"Does the caller need the answer to continue?"}:::decision
  Q1 -->|"yes"| SYNC["Synchronous API call; contract via API Ops"]:::core
  Q1 -->|"no"| Q2{"Several consumers, or state rebuilt by replay?"}:::decision
  Q2 -->|"yes"| LOG["Event log with retention, such as Kafka"]:::core
  Q2 -->|"no"| Q3{"Mostly SaaS-to-SaaS with standard connectors?"}:::decision
  Q3 -->|"yes"| IPAAS["iPaaS flow; insist on DLQ and logs access"]:::core
  Q3 -->|"no"| Q4{"Partner can only exchange files?"}:::decision
  Q4 -->|"yes"| FILE["Batch file transfer with manifest and checksums"]:::core
  Q4 -->|"no"| QUEUE["Queue with DLQ: managed or RabbitMQ"]:::core
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: if the caller needs the answer to continue, use a synchronous API call (contract via API Ops). Otherwise: several
consumers or replay-rebuilt state, an event log such as Kafka; SaaS-to-SaaS with standard connectors, an iPaaS flow with DLQ and log
access; file-only partners, batch transfer with manifest and checksums; anything else, a queue with a DLQ.
:::

To size the operating model, score each flow 0–2 on five questions:

| Question | 0 | 1 | 2 |
|---|---|---|---|
| Impact of one lost record | None | Rework | Money, compliance or customer harm |
| Volume per day | Under 100 | 100–100,000 | Over 100,000 |
| Boundaries crossed | One team | Two teams | Two companies |
| Replay needed | Never | Rarely | Routinely |
| External side effects | None | Idempotent API | Non-idempotent API |

**Tier 1 (7–10):** outbox, inbox, alarmed DLQ, daily reconciliation, rehearsed replay. **Tier 2 (4–6):** idempotent consumer, alarmed
DLQ, weekly reconciliation. **Tier 3 (0–3):** an owner, a DLQ and an alarm. The weights are this Atlas's proposal.

### Maturity model

This is **the Atlas's maturity model**, not an industry standard.

| Level | Observable behaviour | Exit criteria |
|---|---|---|
| 0. Ad hoc | Integrations are projects; users find failures | — |
| 1. Visible | Register with owners; every DLQ alarmed | All production flows registered and alarmed |
| 2. Recoverable | Business-key idempotence; retry budgets; governed redrive | Duplicate-injection tests pass for tier 1; every redrive recorded |
| 3. Reconciled | Completeness measured per closed window | Zero unexplained gaps held for a quarter |
| 4. Engineered | Transitive schema gates; quarterly replay rehearsals; per-flow SLOs | Reconciliation finds problems before customers do |

Aim for level 2 everywhere and level 3 for tier-1 flows; level 4 pays off when replay is routine.

### Adoption roadmap: 30/60/90 days

- **Days 1–30: see the flows.** Inventory queues, topics and iPaaS flows; register them; name owners; alarm every DLQ; pick the three
  highest-impact flows.
- **Days 31–60: make tier-1 flows recoverable.** Business-key inboxes; broker policies in Git; written retry budgets; one redrive
  rehearsed in staging; a weekly DLQ review.
- **Days 61–90: prove completeness.** Daily reconciliation for tier-1 flows; the zero-unexplained-gaps SLO; duplicate-injection tests
  in CI; a first replay rehearsal. Report unexplained gaps per day and time to triage.

Dependencies: read access to both ends' business data.

## Toolkit

### Checklists

**New flow (preflight)**

- [ ] Register entry with a named owner and a business key
- [ ] Delivery guarantee and ordering scope written down
- [ ] Duplicate-injection test passes
- [ ] One retry layer, bounded and jittered; failures classified
- [ ] DLQ exists, is alarmed, and outlives the source's retention
- [ ] Transitive schema compatibility if consumers replay history
- [ ] Reconciliation query run once on real data

**Weekly**

- [ ] Every parked message has a decision; the oldest is within SLA
- [ ] Unexplained gaps reviewed; lag and oldest-message age within SLO

**Before any redrive or replay**

- [ ] Root cause fixed and deployed; dedup key survives the operation
- [ ] External idempotency windows open, or reconciliation run first
- [ ] Rate below the downstream limit; approval recorded; verification query ready
- [ ] Records with a stored failure in the target go out under a new key generation

### Templates

**Replay or redrive request** (keep with the incident or change record):

```text
REPLAY / REDRIVE REQUEST
Flow:             order-to-invoice           Requested by:  <name>
Source:           orders-parked (DLQ) | topic orders, group billing
Range:            <message count> | <from timestamp> to <to timestamp>
Root cause fixed: <change id>, deployed <date>
Dedup key:        order_id (business key, survives redrive)
External keys:    ERP keys valid until <date>; reconciled first? yes/no
Key generation:   original | r2 (only for records confirmed absent in target)
Rate:             <n> msg/s (downstream limit <m> msg/s)
Verification:     reconciliation query <link>; duplicate check <link>
Rollback:         stop task / cancel move task; compensate per <SOP>
Approved by:      <flow owner>                Date: <YYYY-MM-DD>
Result:           <messages moved>, gaps after = <n>, duplicates = <n>
```

**DLQ triage record**:

```text
DLQ TRIAGE                        Date: <YYYY-MM-DD>   Triage by: <name>
Queue:          <name>            Depth: <n>   Oldest: <age>
Top reasons:    <reason>: <n>, <reason>: <n>
Class:          invalid | poison | downstream outage | unknown
Decision:       fix producer | fix consumer | redrive | write off
Write-offs:     <ids>, reason <text>, approved by <business owner>
Follow-up:      <ticket>                 Due: <date>
```

The flow card in the [Reference implementation](#1-the-flow-card) serves as the register template.

### Deep comparisons

| Question | Run in-house | Managed integration service (for example ONEiO) |
|---|---|---|
| Who triages parked messages? | Your teams | The provider, per contract |
| Access to raw messages and logs | Full | As contracted |
| Reconciliation against your ledgers | You | Usually still you |
| Exit cost | Your code and policies | Mappings and history sit with the provider |
| Fit | Many in-house systems, strong engineering teams | IT-service and SaaS tool integration, where ONEiO positions itself[^oneio-founder] |


### FAQ

::: answer Is IntegrationOps a recognised discipline?
The label is vendor-promoted and has no standard or certification; the practice it names is long established
([boundaries and naming](#what-integrationops-is-not-boundaries-and-naming)).
:::

::: answer If our broker offers exactly-once delivery, can we skip idempotent consumers?
No. Exactly-once features are scoped to the broker (Kafka transactions between Kafka topics[^kafka-semantics]; Pub/Sub pull
subscriptions within a region, with publish-side duplicates still possible[^pubsub-eod]). External side effects still need a key.
:::

::: answer How many retries before dead-lettering?
Enough to ride out the transient faults you have measured; defaults are in the
[broker table](#delivery-and-dead-letter-behaviour-by-broker). Invalid messages skip retries; downstream outages pause consumption.
:::

::: answer Who should own a flow between two teams?
One named person, usually on the consumer side, which sees failures and reconciles. The producer owns the contract and outbox; the
platform team owns the broker ([Roles, ownership and RACI](#roles-ownership-and-raci)).
:::

::: answer Is an iPaaS enough on its own?
It solves connectors and mappings. Check that it exposes failed runs with payloads, allows selective retries, passes idempotency keys
to targets and supports reconciliation; otherwise those gaps remain yours.
:::

## Going further

### Advanced topics and emerging direction

- **Established:** transactional outbox and log-based CDC[^outbox][^debezium-outbox]; schema registries; broker-native delayed retry
  (RabbitMQ 4.3[^rmq-qq]).
- **Maturing:** trace context across brokers; the OpenTelemetry messaging conventions are still in Development and do not yet fix how
  context is attached to messages[^otel-msg].
- **New in brokers:** Kafka share groups, production-ready since 4.2[^kafka-42], bring per-record retries.
- **Marketing to read carefully:** "IntegrationOps as a Service"[^techeu]; judge triage times, log access and exit terms, not the label.

### Learning path

- **One day:** the dead-letter channel, idempotent receiver and guaranteed delivery patterns[^eip-dlc][^eip-idem];
  [Lab 1](#lab-1-beginner-executed-duplicates-and-an-idempotent-consumer).
- **One week:** Labs 2 and 3; your broker's delivery and DLQ documentation; a flow card for one real flow.
- **Thirty days:** an inbox and a reconciliation query for a tier-1 flow; a staged redrive.
- **Specialise and show:** an order-to-invoice demo with outbox, broker, DLQ, idempotent consumer, reconciliation and a replay procedure.

## Wrap-up

### Executive recap and cheat sheet

- **Decide:** an owner per flow; which flows are tier 1; one retry layer per flow.
- **Practise:** business-key idempotence; classify failures; park fast; redrive and replay only with approval.
- **Measure:** unexplained gaps per closed window (target zero), p95 latency, oldest message age, triage time.
- **Watch out for:** broker IDs as dedup keys; `basic.nack` loops on RabbitMQ 4.3+; at-most-once dead-lettering by default on quorum
  queues; leases shorter than the handler; non-transitive schemas before a full replay; DLQs that expire before their sources.
- **Next action:** alarm every DLQ, then write the first reconciliation query.

### A judgement call: redrive everything now, or reconcile first?

::: example Illustrative scenario
A six-hour ERP outage ends at 16:00 on the last working day of the month; 12,000 order messages are parked. Finance wants invoices
before the 18:00 close. The consumer deduplicates on the business key; its ERP calls carry an idempotency key that the ERP keeps for
24 hours, and about 1,500 parked messages were first attempted more than 24 hours ago, during an earlier partial failure.
:::

**Option A: redrive all 12,000 now at full speed.** Attractive, since the consumer is idempotent, and wrong here. The inbox covers only
what the consumer committed; if an earlier attempt reached the ERP but the consumer's transaction rolled back, the ERP key was the only
protection, and for the 1,500 older messages it has expired. Full speed also hits a recovering ERP with backlog plus live traffic.

**Option B: split by risk.** Redrive the 10,500 messages whose keys are valid at a rate that leaves room for live orders: with an ERP
limit of 45 per second and 20 used by live traffic, 10,500 ÷ 25 = 420 s, about seven minutes. Reconcile the 1,500 older ones against the
ERP by order ID first and re-publish only those with no invoice. Valid keys can also block: an ERP that stores first responses
returns a saved error again, so check a sample and send orders confirmed absent under a new key generation.

**The decision turns on** "which protection covers each message *now*?", not "is the consumer idempotent?". Check the ERP's real key
lifetime and response storage, whether it can be searched by order ID, and its sustainable rate. If it cannot be searched, the 1,500
need manual review and may miss the close; that is finance's decision, not engineering's.

### Learning objectives and mastery check

You should now be able to explain delivery semantics and leases, design an idempotent consumer with a key that survives recovery,
classify failures, set a retry budget, run a governed redrive or replay, measure completeness, and size the operating model to a flow.

::: answer 1. A team deduplicates on the SQS message ID. Where does that fail?
On redrive, which assigns a new `messageID`[^sqs-redrive], and on publish-side duplicates. Use a business key assigned by the
producer.
:::

::: answer 2. A RabbitMQ 4.3 quorum queue has delivery-limit 5, yet one message has been redelivered 300 times. Why?
The consumer probably uses `basic.nack` with requeue, which since 4.3 does not increment `x-delivery-count`; only genuine failures such
as `basic.reject` or a crash count[^rmq-qq]. Use `basic.reject`, or an attempt counter that rejects or parks. Delayed retry only slows
the loop.
:::

::: answer 3. Every subject uses the registry default. You will replay a topic from the start after three schema changes. What could break?
`BACKWARD` is non-transitive: each version was checked only against the previous one[^sr-evolution], so the newest reader may fail on
the oldest data. Switch to `BACKWARD_TRANSITIVE` first.
:::

::: answer 4. An SQS consumer's handler takes up to 50 s; the queue keeps its default settings. No errors are logged, yet some orders are processed twice. Why?
The default visibility timeout is 30 s[^sqs-vis], so a slow message becomes visible again and a second consumer takes it while the first
is still working. Raise the timeout above the handler's p99 or extend it while working; the business-key inbox makes the overlap
harmless.
:::

::: answer 5. Reconciliation shows 99.98 % completeness and 4 unexplained gaps. Healthy?
No. The SLO is on the unexplained count; trace each key as in the [runbook](#runbook-the-dashboard-is-green-but-invoices-are-missing).
:::

### Conclusion

IntegrationOps is a new name, mostly one vendor's, for an old job: making sure that what one system says happened actually happens,
once, in the next. The tools moved from ESBs to logs, managed queues and iPaaS; the hard parts did not. Brokers redeliver, leases
expire, recovery repeats side effects, and dashboards watch pipes rather than records. What works is small and strict: named owners,
business-key idempotence, one retry layer, fast parking, governed replay, and a daily count of records nobody can explain. If you adopt one practice, adopt reconciliation; it sees the failures your broker cannot.

## Sources and further reading

Defaults were checked on 2026-10-23 (Kafka 4.3.1, RabbitMQ 4.3/4.4, Confluent Platform 8.3, Debezium 3.7, AWS CLI 2.38.5); re-check before relying on them.

**Term and context**

[^oneio-what]: [What Is Integration Ops?](https://www.oneio.cloud/blog/what-is-integration-ops) — Janne Kärkkäinen, ONEiO, 2026-04-09. Checked 2026-10-23.
[^techeu]: [ONEiO lands €8M to replace manual IT integrations with IntegrationOps](https://tech.eu/2025/09/11/oneio-lands-eur8m-to-replace-manual-it-integrations-with-integrationops/) — Tech.eu, 2025-09-11. Checked 2026-10-23.
[^oneio-founder]: [A founder's take: why we raised €8M and what it means for the IT industry](https://www.oneio.cloud/blog/a-founders-take-why-we-raised-eu8m-and-what-it-means-for-the-it-industry) — Juha Berghäll, ONEiO (page dated 2026-06-10). Checked 2026-10-23.
[^oneio-intops]: [ONEiO - Transforming IT Services with Integration Ops](https://oneio.cloud/blog/oneio-transforming-it-services-with-integration-ops) — Juha Berghäll, ONEiO, 2026-03-24. Checked 2026-10-23.
[^io-norway]: [Center for Integrated Operations in the Petroleum Industry](https://prosjektbanken.forskningsradet.no/en/project/FORISS/174963) — Research Council of Norway project bank, project period 2007–2015. Checked 2026-10-23.

**Patterns and architecture**

[^eip-book]: [Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions](https://martinfowler.com/books/eip.html) — Gregor Hohpe and Bobby Woolf, 2003 (book page on martinfowler.com). Checked 2026-10-23.
[^eip-home]: [Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/) — companion site (65 patterns; four integration styles). Checked 2026-10-23.
[^eip-broker]: [Message Broker](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageBroker.html) — Enterprise Integration Patterns. Checked 2026-10-23.
[^eip-dlc]: [Dead Letter Channel](https://www.enterpriseintegrationpatterns.com/patterns/messaging/DeadLetterChannel.html) — Enterprise Integration Patterns. Checked 2026-10-23.
[^eip-invalid]: [Invalid Message Channel](https://www.enterpriseintegrationpatterns.com/patterns/messaging/InvalidMessageChannel.html) — Enterprise Integration Patterns. Checked 2026-10-23.
[^eip-idem]: [Idempotent Receiver](https://www.enterpriseintegrationpatterns.com/patterns/messaging/IdempotentReceiver.html) — Enterprise Integration Patterns. Checked 2026-10-23.
[^eip-store]: [Message Store](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageStore.html) — Enterprise Integration Patterns. Checked 2026-10-23.
[^eip-corr]: [Correlation Identifier](https://www.enterpriseintegrationpatterns.com/patterns/messaging/CorrelationIdentifier.html) — Enterprise Integration Patterns. Checked 2026-10-23.
[^starbucks]: [Starbucks Does Not Use Two-Phase Commit](https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html) — Gregor Hohpe, 2004-11-19. Checked 2026-10-23.
[^fowler-micro]: [Microservices](https://martinfowler.com/articles/microservices.html) — James Lewis and Martin Fowler, 2014-03-25. Checked 2026-10-23.
[^fowler-eda]: [What do you mean by "Event-Driven"?](https://martinfowler.com/articles/201701-event-driven.html) — Martin Fowler, 2017-02-07. Checked 2026-10-23.
[^outbox]: [Pattern: Transactional outbox](https://microservices.io/patterns/data/transactional-outbox.html) — Chris Richardson, microservices.io (undated). Checked 2026-10-23.
[^debezium-outbox]: [Outbox Event Router](https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html) — Debezium 3.7 documentation. Checked 2026-10-23.

**Brokers and managed messaging**

[^kafka-semantics]: [Design: message delivery semantics](https://kafka.apache.org/43/design/design/) — Apache Kafka 4.3 documentation. Checked 2026-10-23.
[^kafka-intro]: [Introduction](https://kafka.apache.org/43/getting-started/introduction/) — Apache Kafka 4.3 documentation. Checked 2026-10-23.
[^kafka-consumer]: [Consumer configs](https://kafka.apache.org/43/configuration/consumer-configs/) — Apache Kafka 4.3 documentation. Checked 2026-10-23.
[^kafka-topic]: [Topic configs](https://kafka.apache.org/43/configuration/topic-configs/) — Apache Kafka 4.3 documentation. Checked 2026-10-23.
[^kafka-ops]: [Basic Kafka operations: managing consumer groups](https://kafka.apache.org/43/operations/basic-kafka-operations/) — Apache Kafka 4.3 documentation. Checked 2026-10-23.
[^kafka-connect]: [Kafka Connect configs](https://kafka.apache.org/43/configuration/kafka-connect-configs/) — Apache Kafka 4.3 documentation. Checked 2026-10-23.
[^kafka-broker]: [Broker configs](https://kafka.apache.org/43/configuration/broker-configs/) — Apache Kafka 4.3 documentation (`group.share.delivery.count.limit`). Checked 2026-10-23.
[^kip-932]: [KIP-932: Queues for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-932:+Queues+for+Kafka) — Apache Kafka (accepted KIP), "Ordering" section. Checked 2026-10-23.
[^kafka-42]: [Apache Kafka 4.2.0 release announcement](https://kafka.apache.org/blog/2026/02/17/apache-kafka-4.2.0-release-announcement/) — Apache Kafka, 2026-02-17. Checked 2026-10-23.
[^kafka-quickstart]: [Quickstart](https://kafka.apache.org/43/getting-started/quickstart/) — Apache Kafka 4.3 documentation (image `apache/kafka:4.3.1`). Checked 2026-10-23.
[^kafka-docker]: [apache/kafka](https://hub.docker.com/r/apache/kafka) — Docker Hub. Checked 2026-10-23.
[^kafka-src]: [ConsoleProducer.java](https://github.com/apache/kafka/blob/4.3.1/tools/src/main/java/org/apache/kafka/tools/ConsoleProducer.java) and [ConsumerGroupCommandOptions.java](https://github.com/apache/kafka/blob/4.3.1/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommandOptions.java) at tag 4.3.1 — Apache Kafka source. Checked 2026-10-23.
[^rmq-qq]: [Quorum Queues](https://www.rabbitmq.com/docs/quorum-queues) — RabbitMQ 4.4 documentation (poison-message handling, delayed retry, at-least-once dead-lettering). Checked 2026-10-23.
[^rmq-dlx]: [Dead Letter Exchanges](https://www.rabbitmq.com/docs/dlx) — RabbitMQ 4.4 documentation. Checked 2026-10-23.
[^rmq-consumers]: [Consumers: delivery acknowledgement timeout](https://www.rabbitmq.com/docs/consumers) — RabbitMQ documentation (default 30 minutes). Checked 2026-10-23.
[^rmq-maxlength]: [Queue Length Limit](https://www.rabbitmq.com/docs/maxlength) — RabbitMQ documentation (`reject-publish` and `basic.nack`). Checked 2026-10-23.
[^rmq-confirms]: [Consumer Acknowledgements and Publisher Confirms](https://www.rabbitmq.com/docs/confirms) — RabbitMQ documentation. Checked 2026-10-23.
[^rmq-policies]: [Policies](https://www.rabbitmq.com/docs/policies) — RabbitMQ documentation. Checked 2026-10-23.
[^rmq-docker]: [rabbitmq](https://hub.docker.com/_/rabbitmq) and its [tags](https://hub.docker.com/_/rabbitmq/tags?name=4.3) — Docker Hub official image. Checked 2026-10-23.
[^pika]: [pika](https://pypi.org/project/pika/) — PyPI, version 1.4.4 released 2026-08-06. Checked 2026-10-23.
[^sqs-alo]: [Amazon SQS at-least-once delivery](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html) — AWS documentation. Checked 2026-10-23.
[^sqs-vis]: [Amazon SQS visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) — AWS documentation. Checked 2026-10-23.
[^sqs-dlq]: [Using dead-letter queues in Amazon SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html) — AWS documentation. Checked 2026-10-23.
[^sqs-redrive]: [Configure a dead-letter queue redrive in Amazon SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue-redrive.html) — AWS documentation. Checked 2026-10-23.
[^sqs-metrics]: [Available CloudWatch metrics for Amazon SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html) — AWS documentation. Checked 2026-10-23.
[^sqs-attrs]: [SetQueueAttributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SetQueueAttributes.html) — Amazon SQS API Reference; console range for maximum receives from [Configure a dead-letter queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue.html). Checked 2026-10-23.
[^asb-dlq]: [Service Bus dead-letter queues](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues) — Microsoft Learn, 2026-07-16. Checked 2026-10-23.
[^asb-lock]: [Message transfers, locks, and settlement](https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement) — Microsoft Learn. Checked 2026-10-23.
[^pubsub-subs]: [Subscription overview](https://docs.cloud.google.com/pubsub/docs/subscription-overview) — Google Cloud documentation. Checked 2026-10-23.
[^pubsub-eod]: [Exactly-once delivery](https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery) — Google Cloud documentation, updated 2026-10-22. Checked 2026-10-23.
[^pubsub-dlq]: [Dead-letter topics (handling message failures)](https://docs.cloud.google.com/pubsub/docs/handling-failures) — Google Cloud documentation, updated 2026-10-22. Checked 2026-10-23.

**Contracts, schemas and telemetry**

[^sr-evolution]: [Schema Evolution and Compatibility for Schema Registry](https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html) — Confluent Platform 8.3 documentation, published 2026-06-29. Checked 2026-10-23.
[^cloudevents]: [CloudEvents specification](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) — CNCF CloudEvents (`specversion` 1.0). Checked 2026-10-23.
[^cloudevents-cncf]: [CloudEvents project page](https://www.cncf.io/projects/cloudevents/) — CNCF (graduated 2024-01-25). Checked 2026-10-23.
[^asyncapi]: [AsyncAPI Specification 3.1.0](https://www.asyncapi.com/docs/reference/specification/latest) — AsyncAPI Initiative. Checked 2026-10-23.
[^otel-msg]: [Semantic conventions for messaging spans](https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/) — OpenTelemetry (status: Development). Checked 2026-10-23.
[^pact]: [What is Pact good for?](https://docs.pact.io/getting_started/what_is_pact_good_for) and [How Pact works: non-HTTP testing](https://docs.pact.io/getting_started/how_pact_works#non-http-testing-message-pact) — Pact documentation. Checked 2026-10-23.

**Retries and idempotency**

[^builders-retries]: [Timeouts, retries, and backoff with jitter](https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter) — Marc Brooker, Amazon Builders' Library, 2019-12-05. Checked 2026-10-23.
[^builders-idem]: [Making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) — Malcolm Featonby, Amazon Builders' Library (undated). Checked 2026-10-23.
[^aws-sdk-retry]: [Retry behavior](https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html) — AWS SDKs and Tools Reference Guide (updated behaviour opt-in via `AWS_NEW_RETRIES_2026=true`). Checked 2026-10-23.
[^stripe-idem]: [Idempotent requests](https://docs.stripe.com/api/idempotent_requests) — Stripe API reference. Checked 2026-10-23.

**Privacy**

[^gdpr-5]: [Art. 5 GDPR: Principles relating to processing of personal data](https://gdpr-info.eu/art-5-gdpr/) — Regulation (EU) 2016/679, unofficial consolidated text on gdpr-info.eu. Checked 2026-10-23.

**Changelog.** 2026-10-23: first draft (r0) for Kafka 4.3, RabbitMQ 4.3/4.4, current AWS, Azure and Google Cloud documentation;
term status confirmed as vendor-specific. 2026-10-23 (r1, after review): Kafka share groups; stored idempotency responses and new key
generations in recovery; nack loops and delayed-retry types corrected; lease-expiry duplicates; RabbitMQ policy, retention and lab
ports fixed; neighbour figure removed. 2026-10-23, post-score polish: share-group ordering quoted from KIP-932;
handler takes a key generation; clearer runbook labels.