ServerlessOps

ServerlessOps is the operations side of serverless computing: running functions-as-a-service and request-billed containers (AWS Lambda, Azure Functions, Google Cloud Run) in production so that events are processed correctly despite retries and duplicates, scaling stays within platform and downstream limits, deployments are safe, and the bill follows business value. The provider runs servers, patching and capacity; you own failure semantics, configuration, concurrency, telemetry and cost. As the CNCF Serverless Working Group put it, serverless does not mean “that operations engineers are no longer required”.^[1]

In one line: the platform decides when your code runs and how often it retries; ServerlessOps makes that safe.

For developers owning functions in production, platform engineers setting guardrails, SREs on call and architects deciding whether serverless fits. Beginners can stop after the glossary; engineers should do the labs.

Serverless removes host work and adds failure modes that stay invisible until production:

The usual cost is not a visible outage but wrong data found later: double payments, missing orders, a backlog nobody owns.

• Not NoOps. Whether a team can run with little operations staff is evaluated on NoOps; this page teaches the mechanisms that remain.
• Not a product, method or certification. ServerlessOps was also Tom McLaughlin’s consultancy, which stopped consulting in July 2019.^[14]
• Not only Lambda. The questions apply to Azure Functions, Cloud Run and Knative too.
• Not workflow orchestration or messaging design. Those belong to WorkflowOps and IntegrationOps; this page covers the function’s side of those contracts.
• Editorial distinction: the Atlas uses “ServerlessOps” for operating request-billed compute; no standard defines it.

Mental model: three questions for every event path. (1) Who retries, how often and for how long? (2) Where does the event go after the last attempt, and who looks there? (3) What if the handler runs twice, or stops half-way? A trigger without three answers is not ready for production.

flowchart LR
  N["`NoOps
  is less operations realistic?`"]:::neutral
  C["`CloudOps
  accounts, quotas, Regions`"]:::neutral
  S["`ServerlessOps
  failure semantics, concurrency,
  safe deploys, cost per invocation`"]:::core
  W["`WorkflowOps
  state machines`"]:::neutral
  I["`IntegrationOps
  brokers and replay`"]:::neutral
  O["`ObservabilityOps
  telemetry pipelines`"]:::neutral
  F["`FinOps
  spend governance`"]:::neutral
  N --> S
  C --> S
  S --> W
  S --> I
  S --> O
  S --> F
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

This charter is this Atlas’s proposal; no official manifesto exists.

Mission: every event is processed exactly once in effect, within a known time, at a known cost, by a function someone owns.

1. Every handler is idempotent on a business key. AWS, Google and Microsoft document at-least-once delivery.^[5][34][35]
2. Every asynchronous path ends somewhere visible: a DLQ or destination with an owner, an alarm and a maximum age.
3. Retry at one layer. Marc Brooker calls retrying “at a single point in the stack” Amazon’s best practice for “low-cost control-plane and data-plane operations”; five layers of three retries multiply load 243 times.^[36]
4. Concurrency is a budget. Cap what reaches fragile dependencies; reserve what critical functions need.^[37][11]
5. Timeouts are nested. Downstream call < function timeout; SQS visibility timeout ≥ 6 × function timeout.^[38]
6. Cost is measured per business event, including retries and duplicates.
7. Releases go gradually to a version behind an alias, with alarms that see partial failures.^[25]

Non-goals: zero cold starts without a latency target; trusting platform exactly-once claims.

Who retries depends on how the event arrives. Lambda does not retry a synchronous invocation, whose code “might have run completely, partially, or not at all”.^[9] Asynchronous events get two more attempts on function errors and up to six hours of retries for throttles, then are discarded unless a DLQ or failure destination exists.^[6][39] Event source mappings retry the whole batch until it succeeds or expires; for queues, visibility timeout and redrive policy decide the rest.^[5][9]

flowchart LR
  subgraph SY["Synchronous"]
    direction TB
    A1["`Client or API
    calls`"] --> A2["Function"] --> A3["`Error returned:
    caller decides`"]
  end
  subgraph AS["Asynchronous"]
    direction TB
    B1["`S3, SNS,
    EventBridge`"] --> B2["`Internal queue,
    up to 6 h`"] --> B3["`Function, up to
    2 retries`"] --> B4["`DLQ or failure
    destination`"]
  end
  subgraph PO["Poll-based"]
    direction TB
    C1["`Queue or
    stream`"] --> C2["`Poller reads
    a batch`"] --> C3["Function"] --> C4["`Queue: DLQ.
    Stream: shard waits`"]
  end
  SY ~~~ AS ~~~ PO
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class A3,B4,C4 risk

The platforms differ in limits but share the semantics (as of October 2026):

The team that writes a function owns its event contract. A common split (R = responsible, A = accountable, C = consulted, I = informed):

With a few functions, developers on call often works; with hundreds, Hausenblas suggested “you probably need some on-call strategy”.^[48] Platform teams own shared defaults (templates, lint rules, alarm baselines), not functions.

• Day 0 (design): fill in the event-path design record and a cost estimate (Lab 2).
• Day 1 (release): deploy through an alias with a canary; load-test to peak; inject a duplicate and a poison message before real traffic does.
• Day 2 (operate): triage the DLQ, redrive after fixes, review cost per event, keep runtimes current. AWS gives at least 180 days’ notice of a runtime deprecation, but not for container images, and afterwards “may no longer apply security patches”.^[49]

Governance is mostly defaults as code plus reviews (this Atlas’s proposal):

• Policy checks in CI: every asynchronous function has a failure destination; every SQS mapping sets ReportBatchItemFailures and a maximum concurrency (or, in provisioned mode, a maximum poller count^[37]); visibility timeout ≥ 6 × function timeout; every DLQ has an alarm and an owner tag.
• Weekly DLQ review by the owning team; monthly cost and quota reviews with FinOps and CloudOps; a quarterly game day (duplicate storm, poison message, throttled dependency).
• Escalation: provider incidents follow IncidentOps.

The reference design is an order-payment path, where a duplicate is expensive and visible.

flowchart TB
  U["Client"] --> G["HTTPS API: authenticated, throttled"]
  G --> F1["Intake function: validate, enqueue, return 202"]
  F1 --> Q["Orders queue: visibility 180 s, maxReceiveCount 5"]
  Q --> E["Event source mapping: batch 10, max concurrency 20, partial batch responses"]
  E --> F2["Consumer function, alias live, timeout 30 s"]
  F2 <--> T["Idempotency table: key order_id, expiry 15 days"]
  F2 -->|"Idempotency-Key: order_id"| P["Payment provider API"]
  Q -->|"after 5 receives"| D["Dead-letter queue, 14-day retention"]
  D -->|"after a fix: rate-limited redrive"| Q
  F2 -.-> M["Alarms: failed records, errors, DLQ depth"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class F2,T,E core
  class D,P risk
  class U,G,F1,Q,M neutral

Why each element is there:

• Queue between API and payment: a fast answer for the caller, a controlled rate for the provider; the order is “accepted”, not “paid”, so the client needs a status endpoint. The API authenticates and throttles callers: every request is billed.
• Maximum concurrency 20, reserved 25: reserved concurrency should be at least the sum of maximum concurrency settings, or Lambda “might throttle your messages”;^[37] it comes from the shared pool, at most unreserved concurrency minus 100.^[50]
• Visibility timeout 180 s = 6 × 30 s, maxReceiveCount 5: the minimums AWS recommends, giving Lambda “a few chances to retry”.^[38]
• DLQ retention 14 days, longer than the source’s 4: standard-queue expiry counts from the original enqueue time, so a shorter DLQ deletes evidence.^[27][51] A redrive restarts retention and assigns new message IDs,^[52] so redriving into the source queue is safe.
• Idempotency expiry 15 days, so a message redriven on day 10 is still recognised locally (this page’s rule; Powertools defaults to one hour^[53]). The provider may forget sooner: see keys expire.

Alternatives. A FIFO queue prevents duplicate sends only within a five-minute deduplication interval;^[54] Pub/Sub exactly-once delivery works only for pull subscriptions within one Region;^[55] neither makes side effects idempotent. A multi-step payment with compensation belongs in a workflow engine (WorkflowOps).

Happy path. The consumer claims each order_id with a conditional write (IN_PROGRESS, lease = the invocation’s remaining time), calls the provider with Idempotency-Key: order_id, records COMPLETE, and returns an empty batchItemFailures list, so Lambda deletes the batch.^[31][56]

Exception path. The dangerous case is a failure after the side effect, before the acknowledgement.

sequenceDiagram
  participant Q as Orders queue
  participant F as Consumer and its table
  participant P as Payment API
  Q->>F: order-42, attempt 1
  Note over F: claim order-42: IN_PROGRESS, lease
  F->>P: charge, key order-42
  Note over F: times out before success is recorded
  Q->>F: order-42 again after 180 s
  Note over F: lease expired: claim again
  F->>P: charge, same key
  P-->>F: original result, no new charge
  Note over F: mark COMPLETE
  F-->>Q: returns success, mapping deletes message
  Q->>F: broker duplicate of order-42
  Note over F: record already COMPLETE
  F-->>Q: returns success, no side effect

Without the downstream key the second charge would be real: the local record cannot know whether the first call reached the provider. Hence Amazon’s preferred “unique caller-provided client request identifier”^[33] and Google’s advice to use the event ID as the idempotency key for external APIs.^[34] Platform IDs catch only redeliveries; prefer a business key like order_id. The Idempotency-Key header is a convention whose IETF draft expired (last revised October 2025).^[57]

Keys expire. Stripe’s API v1 may prune a key once it is 24 hours old, then treats a reuse as a new request; it also rejects a reused key whose parameters differ.^[58] Replay is safe only within the shorter of the two retentions: before a late redrive, or one after a fix that changes what is sent, look each order up at the provider and skip those already charged.

Timeouts in practice. Lambda’s default is 3 seconds; AWS warns against timeouts close to the average duration and advises testing “at the upper bounds” of expected data.^[59]

Streams are different from queues. A Kinesis or DynamoDB stream mapping keeps order per shard, so by default a failing batch is retried until it succeeds or expires (MaximumRetryAttempts and MaximumRecordAgeInSeconds default to −1, infinite).^[8] Set a finite retry count and record age, turn on BisectBatchOnFunctionError, report item failures, and add an on-failure destination; Lambda then splits the batch at the failing sequence number and retries only the rest.^[32] An SQS or SNS destination does not receive the failed records, only their shard and sequence numbers; they must be read back from the stream “before they expire and are lost”, while an S3 destination gets the full record.^[29] DynamoDB Streams keeps records 24 hours,^[60] so alarm on destination inflow and repair within that window.

None is mandatory; CloudEvents, the only normative one, defines the event format, not delivery guarantees.

Categories first, brands second (as of October 2026):

Select on: event-path answers visible in review; aliases, canaries and alarms expressible; licence and exit path.

Automate safe actions (pausing a mapping, lowering maximum concurrency, ticketing DLQ samples); keep human approval for money-moving redrives. Language models can cluster DLQ errors, but DLQ messages carry full payloads: treat them as production data (PrivacyOps). The core loop needs no AI.

Assumptions: AWS, one Region, Python 3.13 (deprecation scheduled for 30 June 2029 as of October 2026^[49]), Powertools for AWS Lambda, and a payment provider that honours an idempotency key.

Example status: YAML syntax-checked by the Atlas validator; not linted with cfn-lint; not deployed. Long-form intrinsics (Ref:, Fn::GetAtt:) keep it parseable as plain YAML.

AWSTemplateFormatVersion: "2010-09-09"
Transform:
  - AWS::LanguageExtensions          # a changed PaymentApiUrl then publishes a version
  - AWS::Serverless-2016-10-31
Description: Idempotent order consumer, partial batch responses, DLQ (illustrative)

Parameters:
  PaymentApiUrl:
    Type: String
    Default: https://payments.example.com/v1/charges   # a test endpoint you control

Resources:
  OrdersDLQ:
    Type: AWS::SQS::Queue
    Properties:
      MessageRetentionPeriod: 1209600       # 14 days, longer than the source queue
  OrdersQueue:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 180                # 6 x the 30 s function timeout
      MessageRetentionPeriod: 345600        # 4 days
      RedrivePolicy:
        deadLetterTargetArn:
          Fn::GetAtt: [OrdersDLQ, Arn]
        maxReceiveCount: 5
  IdempotencyTable:
    Type: AWS::DynamoDB::Table
    Properties:
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      TimeToLiveSpecification:
        AttributeName: expiration
        Enabled: true
  OrderConsumer:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.13
      Handler: app.handler
      CodeUri: src/
      MemorySize: 512
      Timeout: 30
      ReservedConcurrentExecutions: 25      # >= MaximumConcurrency; needs 125 unreserved
      AutoPublishAlias: live
      DeploymentPreference:
        Type: Canary10Percent10Minutes
        Alarms:                             # function-wide, all versions: see the FAQ
          - Ref: FailedRecordsAlarm
          - Ref: ErrorsAlarm
      Environment:
        Variables:
          IDEMPOTENCY_TABLE:
            Ref: IdempotencyTable
          PAYMENT_API_URL:
            Ref: PaymentApiUrl
          POWERTOOLS_SERVICE_NAME: order-consumer
          POWERTOOLS_METRICS_NAMESPACE: Orders
      Policies:                             # the four table actions Powertools documents
        - Statement:
            - Effect: Allow
              Action:
                - dynamodb:GetItem
                - dynamodb:PutItem
                - dynamodb:UpdateItem
                - dynamodb:DeleteItem
              Resource:
                Fn::GetAtt: [IdempotencyTable, Arn]
      Events:
        Orders:
          Type: SQS
          Properties:
            Queue:
              Fn::GetAtt: [OrdersQueue, Arn]
            BatchSize: 10
            FunctionResponseTypes: [ReportBatchItemFailures]
            ScalingConfig:
              MaximumConcurrency: 20        # protects the payment provider
  OrderConsumerLogs:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName:
        Fn::Sub: "/aws/lambda/${OrderConsumer}"
      RetentionInDays: 14                   # owned by the stack, so sam delete removes it
  FailedRecordsAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      Namespace: Orders
      MetricName: FailedRecords
      Dimensions:
        - Name: service
          Value: order-consumer
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      TreatMissingData: notBreaching          # add AlarmActions to page the owners
  ErrorsAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      Namespace: AWS/Lambda
      MetricName: Errors
      Dimensions:
        - Name: FunctionName
          Value:
            Ref: OrderConsumer
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      TreatMissingData: notBreaching
  DlqDepthAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      Namespace: AWS/SQS
      # Counts automatic redrive; NumberOfMessagesSent does not.
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value:
            Fn::GetAtt: [OrdersDLQ, QueueName]
      Statistic: Maximum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      TreatMissingData: notBreaching

Details that are easy to miss. The canary watches a failed-records metric: a batch that returns item failures is a successful invocation, so Errors stays at zero (this page’s inference from the metric definition;^[30] Powertools raises only when every record fails^[63]). SAM decides whether to publish a version from the unresolved template, so without AWS::LanguageExtensions a new PaymentApiUrl alone would reach only $LATEST, and the live alias would keep calling the old endpoint.^[66] Invocations “should use the alias qualifier”,^[25] so check which one the mapping invokes (Lab 3, step 3).

Example status: Python syntax-checked; not executed (needs AWS, DynamoDB and the aws-lambda-powertools package). APIs follow the Powertools documentation for idempotency, batch processing and metrics as of October 2026.^[53][63]

"""Order consumer: one payment per order even when SQS delivers a message twice (illustrative)."""
import json
import os
import urllib.request

from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit
from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, process_partial_response
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    IdempotencyConfig,
    idempotent_function,
)

metrics = Metrics()
processor = BatchProcessor(event_type=EventType.SQS)
persistence = DynamoDBPersistenceLayer(table_name=os.environ["IDEMPOTENCY_TABLE"])
config = IdempotencyConfig(
    event_key_jmespath="powertools_json(body).order_id",  # business key, not the SQS messageId
    raise_on_no_idempotency_key=True,                     # a message without a key fails loudly
    expires_after_seconds=15 * 24 * 3600,                 # longer than the 14-day DLQ retention
)


@idempotent_function(data_keyword_argument="record", config=config, persistence_store=persistence)
def record_handler(record: SQSRecord) -> dict:
    order = json.loads(record.body)
    # Illustrative: a real provider call adds an API key fetched from Secrets Manager, and
    # classifies permanent 4xx errors so they do not cycle through five receives.
    request = urllib.request.Request(
        os.environ["PAYMENT_API_URL"],
        data=json.dumps({"order_id": order["order_id"], "amount": order["amount"]}).encode(),
        headers={"Content-Type": "application/json", "Idempotency-Key": order["order_id"]},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=5) as response:  # far below the 30 s function timeout
        return {"order_id": order["order_id"], "status": response.status}


@metrics.log_metrics
def handler(event, context):
    config.register_lambda_context(context)  # the in-progress lease ends with this invocation's time limit
    response = process_partial_response(
        event=event, record_handler=record_handler, processor=processor, context=context
    )
    metrics.add_metric(name="FailedRecords", unit=MetricUnit.Count,
                       value=len(response.get("batchItemFailures", [])))
    return response

Why order_id, not messageId: Powertools’ batch example uses messageId but notes that a field “your producer intentionally set” may be more accurate.^[53] An order sent twice becomes two messages with different IDs, and a DLQ redrive assigns new IDs,^[52] so a messageId key recognises no redriven message.

Objective: measure double charges for three designs: no idempotency, a local record only, and a local record plus a downstream key. Prerequisites: Python 3.8 or later; no cloud account, cost or risk; 20 minutes.

Example status: executed on 2026-10-16 with Python 3.8.10; output below is from that run (seeded, deterministic).

"""Simulate at-least-once delivery and count double charges for three consumer designs."""
import random
import sys

VISIBILITY_TIMEOUT = 180  # seconds before an unacknowledged message is delivered again
LEASE = 30                # in-progress claim expires with the function timeout


class Payments:
    """Downstream API. With keys=True it ignores a repeated idempotency key."""

    def __init__(self, keys):
        self.keys, self.seen, self.charges = keys, set(), {}

    def charge(self, order_id):
        if self.keys and order_id in self.seen:
            return
        self.seen.add(order_id)
        self.charges[order_id] = self.charges.get(order_id, 0) + 1


class Store:
    """Idempotency records, as a table with conditional writes would hold them."""

    def __init__(self):
        self.items = {}

    def claim(self, key, now):
        item = self.items.get(key)
        if item is None or (item["status"] == "IN_PROGRESS" and item["expires"] <= now):
            self.items[key] = {"status": "IN_PROGRESS", "expires": now + LEASE}
            return "CLAIMED"
        return item["status"]

    def complete(self, key):
        self.items[key] = {"status": "COMPLETE", "expires": float("inf")}


def run(design, orders, seed, timeout_rate):
    rng = random.Random(seed)
    payments, store = Payments(keys=(design == "record+key")), Store()
    queue = [(0.0, o) for o in orders]
    queue += [(rng.uniform(0, 60), o) for o in orders if rng.random() < 0.02]  # broker duplicates
    deliveries = 0
    while queue:
        queue.sort()
        now, order = queue.pop(0)
        deliveries += 1
        if design != "naive":
            state = store.claim(order, now)
            if state == "COMPLETE":
                continue                                  # duplicate: acknowledge, do nothing
            if state == "IN_PROGRESS":
                queue.append((now + VISIBILITY_TIMEOUT, order))  # report failure; retry later
                continue
        payments.charge(order)
        if rng.random() < timeout_rate:                    # timeout after the charge, before the ack
            queue.append((now + VISIBILITY_TIMEOUT, order))
            continue
        if design != "naive":
            store.complete(order)
    doubles = sum(1 for n in payments.charges.values() if n > 1)
    return deliveries, sum(payments.charges.values()), doubles


def main():
    n = int(sys.argv[1]) if len(sys.argv) > 1 else 10000
    timeout_rate = float(sys.argv[2]) if len(sys.argv) > 2 else 0.03
    orders = ["order-%05d" % i for i in range(n)]
    print("%-14s %10s %8s %14s" % ("design", "deliveries", "charges", "double-charged"))
    for design in ("naive", "record-only", "record+key"):
        deliveries, charges, doubles = run(design, orders, seed=7, timeout_rate=timeout_rate)
        print("%-14s %10d %8d %14d" % (design, deliveries, charges, doubles))


if __name__ == "__main__":
    main()

1. Save as at_least_once.py; run python3 at_least_once.py (10,000 orders, 2% broker duplicates, 3% timeouts after the charge).
2. Run python3 at_least_once.py 10000 0 (no timeouts) and python3 at_least_once.py 10000 0.10 (10% timeouts).

Expected output (step 1):

design         deliveries  charges double-charged
naive               10529    10529            504
record-only         10525    10303            295
record+key          10525    10000              0

Step 2: with no timeouts record-only has 0 double charges (naive 217); with 10% timeouts, 986 (naive 1,179); record+key stays at 0. Validation: the local record stops duplicates arriving after completion; only the downstream key stops those caused by a timeout after the side effect. Limits: the IN_PROGRESS branch fires only 5 times per design (a duplicate inside a timed-out attempt’s lease), and the simulated provider never forgets a key; real ones do (exception path). Troubleshooting: ValueError means a non-numeric argument. Teardown: delete the file.

Objective: compute the monthly cost of the consumer path, cost per million events and peak concurrency, and see what a slow dependency does to both. Prerequisites: Python 3.8 or later; no cost or risk; 30 minutes.

Example status: executed on 2026-10-17 with Python 3.8.10. Defaults: US East (N. Virginia) list prices checked 2026-10-16 for Lambda x86,^[12] DynamoDB on-demand writes^[67] and SQS requests;^[68] two writes per attempt (Powertools claims, then completes or deletes^[53]); 1.2 queue requests per event is assumed.

"""Estimate monthly cost and peak concurrency of a queue-driven function and its idempotency table.

Prices are inputs. Defaults: US East (N. Virginia) list prices checked 2026-10-16 for AWS Lambda
(x86, on-demand), DynamoDB on-demand writes and SQS standard requests.
"""
import argparse

SECONDS_PER_MONTH = 30 * 24 * 3600


def estimate(a):
    attempts = a.events * a.attempts_per_event
    gb_seconds = attempts * (a.duration_ms / 1000) * (a.memory_mb / 1024)
    free_gbs, free_req = (400000, 1000000) if a.free_tier else (0, 0)
    compute = max(gb_seconds - free_gbs, 0) * a.price_gb_s
    requests = max(attempts - free_req, 0) / 1e6 * a.price_per_million
    table = attempts * a.writes_per_attempt / 1e6 * a.price_per_million_writes
    queue = a.events * a.queue_requests_per_event / 1e6 * a.price_per_million_queue
    total = compute + requests + table + queue + a.other_usd
    return {
        "invocations": attempts,
        "gb_seconds": gb_seconds,
        "lambda_usd": compute + requests,
        "table_usd": table,
        "queue_usd": queue,
        "total_usd": total,
        "usd_per_million_events": total / (a.events / 1e6),
        "lambda_share_pct": (compute + requests) / total * 100,
        "avg_busy_environments": attempts * (a.duration_ms / 1000) / SECONDS_PER_MONTH,
        "peak_concurrency": a.peak_rps * a.attempts_per_event * (a.duration_ms / 1000),
    }


def main():
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--events", type=float, default=30e6, help="business events per month")
    p.add_argument("--duration-ms", type=float, default=200)
    p.add_argument("--memory-mb", type=float, default=512)
    p.add_argument("--attempts-per-event", type=float, default=1.05, help="1 + retries + duplicates")
    p.add_argument("--peak-rps", type=float, default=400, help="peak business events per second")
    p.add_argument("--quota", type=float, default=1000, help="concurrency available to this function")
    p.add_argument("--price-gb-s", type=float, default=0.0000166667)
    p.add_argument("--price-per-million", type=float, default=0.20, help="Lambda requests")
    p.add_argument("--writes-per-attempt", type=float, default=2, help="idempotency table writes")
    p.add_argument("--price-per-million-writes", type=float, default=0.625)
    p.add_argument("--queue-requests-per-event", type=float, default=1.2, help="assumption")
    p.add_argument("--price-per-million-queue", type=float, default=0.40)
    p.add_argument("--other-usd", type=float, default=0, help="logs, API layer, data transfer")
    p.add_argument("--free-tier", action="store_true", help="Lambda free tier only")
    a = p.parse_args()
    r = estimate(a)
    for k, v in r.items():
        print("%-24s %16s" % (k, "{:,.2f}".format(v)))
    headroom = 1 - r["peak_concurrency"] / a.quota
    print("%-24s %15.1f%%" % ("peak_headroom", headroom * 100))
    if headroom < 0.3:
        print("WARNING: less than 30% concurrency headroom at peak")
        raise SystemExit(1)


if __name__ == "__main__":
    main()

1. Save as fn_cost.py; run python3 fn_cost.py (30 million events a month, 200 ms, 512 MB, 1.05 attempts, peak 400 per second).
2. Simulate a slow dependency: python3 fn_cost.py --duration-ms 2000.
3. Try --memory-mb 1024 --duration-ms 100 (more memory brings more CPU) and --free-tier.

Expected output: step 1: lambda_usd 58.80, table_usd 39.38, queue_usd 14.40, total_usd 112.58, usd_per_million_events 3.75, peak_concurrency 84.00, peak_headroom 91.6%, exit code 0. Step 2: lambda_usd 531.30, peak_concurrency 840.00, headroom 16.0%, the warning, exit code 1. Step 3: same cost, half the busy environments (1.22, not 2.43); free tier: Lambda $51.93. Validation: 3.15 million GB-s × $0.0000166667 = $52.50, plus 31.5 × $0.20 = $6.30; 63 × $0.625 = $39.38; 36 × $0.40 = $14.40. Troubleshooting: unrecognized arguments means a mistyped flag. Teardown: none. Lessons: Lambda is half of this path’s bill; in step 2 the dependency, not your code, set the bill and the capacity risk.

Objective: watch duplicates being absorbed and a poison message reach the DLQ, then redrive it. Prerequisites: a sandbox AWS account (never production), AWS CLI v2, AWS SAM CLI, the reference implementation as template.yaml and src/app.py, src/requirements.txt with aws-lambda-powertools, and a test endpoint you control that logs the Idempotency-Key header. New accounts have reduced concurrency quotas;^[10] the reservation needs 125 unreserved (25 + 100^[50]), so if step 0 shows less, remove ReservedConcurrentExecutions. Cost and risk: small charges; never use a real payment provider. Status: not executed (no AWS account was used).

Example status: bash syntax-checked; not executed.

# Placeholders: replace with your own values.
STACK=order-consumer-lab
QUEUE_URL="https://sqs.eu-west-1.amazonaws.com/ACCOUNT_ID/orders-queue"
QUEUE_ARN="arn:aws:sqs:eu-west-1:ACCOUNT_ID:orders-queue"
DLQ_URL="https://sqs.eu-west-1.amazonaws.com/ACCOUNT_ID/orders-dlq"
DLQ_ARN="arn:aws:sqs:eu-west-1:ACCOUNT_ID:orders-dlq"

# 0. Reserved concurrency 25 needs unreserved account concurrency of at least 125.
aws lambda get-account-settings --query 'AccountLimit.UnreservedConcurrentExecutions'

sam build && sam deploy --guided --stack-name "$STACK" \
  --parameter-overrides PaymentApiUrl=https://test-endpoint.example.com/charges

# 1. A good order, sent twice by the producer: expect ONE request at the test endpoint.
for i in 1 2; do
  aws sqs send-message --queue-url "$QUEUE_URL" \
    --message-body '{"order_id":"order-1","amount":100}'
done

# 2. A poison message (no order_id): expect it in the DLQ after about 5 x 180 s.
aws sqs send-message --queue-url "$QUEUE_URL" --message-body '{"amount":100}'
aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names ApproximateNumberOfMessages

# 3. Which function version or alias does the mapping invoke?
aws lambda list-event-source-mappings --event-source-arn "$QUEUE_ARN" \
  --query 'EventSourceMappings[].{uuid:UUID,target:FunctionArn,state:State}'

# 4. Redrive from the DLQ at a fixed, low rate (after fixing the cause).
aws sqs start-message-move-task --source-arn "$DLQ_ARN" \
  --max-number-of-messages-per-second 5

Expected output: step 0, 125 or more; step 1, one request at the endpoint; step 2, ApproximateNumberOfMessages of "1" after roughly 15 minutes; step 3, a target ARN ending in :live; step 4, a TaskHandle. Validation: alone in its batch, the poison message makes Powertools raise, so Errors counts it;^[63] mixed with good messages, only FailedRecords shows it. Troubleshooting: a deploy failing on concurrency means step 0 was below 125; on the mapping, visibility timeout < function timeout;^[38] no DLQ arrival points to the redrive policy. The move task accepts only DLQs whose sources are SQS queues, at most 500 messages per second.^[69] Teardown: sam delete --stack-name "$STACK" (deletes queues, messages, table and log group).

Scenario 4: a provider outage (sourced). In AWS’s account of the 19–20 October 2025 us-east-1 disruption, a subsystem polling SQS queues “failed and did not recover automatically” until 4:40 AM PDT; later AWS “throttled Lambda Event Source Mappings and asynchronous invocations to prioritize latency-sensitive synchronous invocations”, and backlogs were processed by 2:15 PM PDT.^[70] This page’s lessons: asynchronous work can wait hours, so retention, event age and idempotency expiry must cover a long outage; the backlog returns as a surge for maximum concurrency to absorb; and replays invite duplicates.

Scenario 5: per-step charges (sourced, secondary). InfoQ reports that Prime Video’s stream monitor ran several Step Functions state transitions per second of stream and kept intermediate frames in S3; those charges and an account limit on transitions ended with one process on ECS.^[71] For tiny, continuous units of work, per-step pricing and limits set the cost; NoOps discusses the case.

Worked calculation (illustrative). 30 million orders; 31.5 million consumer runs; 63,000 throttles; 120 repeated keys that the provider de-duplicated.

• Throttle ratio = 63,000 ÷ (31,500,000 + 63,000) = 0.20%.
• Attempts per order = 31.5 ÷ 30 = 1.05: 5% of compute pays for retries and duplicates.
• Duplicates prevented by keys = 120 ÷ 30,000,000 = 4 per million, each a double charge without keys.
• Cost per million orders = $112.58 ÷ 30 = $3.75 for the consumer path (Lab 2); Lambda alone suggests $1.96.

Targets come from business outcomes (no duplicate charges; DLQ empty by end of day); review DLQ and backlog daily, cost monthly. Vanity metrics: number of functions, “100% serverless”, Errors = 0 on a partial-batch consumer.

• Least privilege. One execution role per function, scoped to exact actions and resources.
• Who may invoke. An S3 bucket ARN carries no account ID: once you delete a bucket, another account could create one with the same name and invoke the function, so add aws:SourceAccount.^[73]
• Public endpoints. With auth type NONE and a public policy, “any unauthenticated user with your function URL can invoke your function”.^[74] Require authentication or throttling on public paths and alarm on invocation rate: every call is billed.
• Secrets. AWS recommends Secrets Manager, not environment variables, for credentials.^[75]
• DLQs hold full payloads. Apply the source data’s retention, encryption and access rules (PrivacyOps).
• Snapshots and uniqueness. With SnapStart, unique values generated during initialization can be reused across environments; generate them later.^[46]
• Runtime currency. Major runtime upgrades are “a customer-driven operation”.^[49]
• Account guardrails belong to CloudSecOps; permission design to IAMOps.

Cost model. Lambda bills per request and per GB-second, and “Duration charges apply to code that runs in the handler of a function as well as initialization code”. As of October 2026 the US East (N. Virginia) x86 rates are $0.20 per million requests and $0.0000166667 per GB-second, with a monthly free tier of one million requests and 400,000 GB-seconds; provisioned concurrency is billed while configured, without free tier.^[12]

The function is not the whole bill. In Lab 2 the idempotency table (two writes per attempt at $0.625 per million on demand^[67]) and queue requests ($0.40 per million^[68]) add $53.78 to $58.80 of Lambda. Logs, the API layer, data transfer and VPC networking come on top; take them from your own bill.

Levers, in order of usual impact:

1. Fix slow dependencies and timeouts. Duration is the multiplier (Lab 2, step 2).
2. Cut wasted attempts: one retry layer, partial batch responses, cheap duplicate short-circuits.
3. Tune memory. More memory brings more CPU (one vCPU at 1,769 MB^[10]); measure with Power Tuning on a side-effect-free copy.^[64]
4. Batch: fewer invocations, bigger blast radius per failure.
5. Change the model for steady load. AWS positions Lambda Managed Instances as “ideal for steady-state, high-volume workloads” (EC2 prices plus a 15% management fee, as of October 2026);^[12] Google recommends instance-based billing for “steady, slowly varying” traffic.^[45]

Capacity is concurrency: the Regional quota (1,000 by default, lower for new accounts), 1,000 new environments per function every 10 seconds, and for SQS mappings five concurrent invocations at first, up to 300 more per minute, to 1,250; provisioned mode, which excludes maximum concurrency, scales to 100,000 (as of October 2026).^[10][24][37] Sustainability: nothing idles when nothing runs, but retries and oversized memory burn energy (GreenOps).

Monthly evidence that the model works: every DLQ has an owner and an alarm tested by firing; the last duplicate-storm game day produced zero double effects; redrives record a cause, an approver and the provider’s key retention; cost per event follows forecast; every runtime is at least 180 days from deprecation.^[49]

flowchart TB
  A["Alarm: DLQ depth or failed records above 0"]:::risk --> B{"Most messages fail with the same error?"}:::decision
  B -->|"yes"| C["Payload or code defect: pause the mapping if good traffic is also failing"]
  B -->|"no"| D{"Downstream errors or Lambda throttles?"}:::decision
  D -->|"downstream"| E["Lower maximum concurrency and let messages wait in the queue"]
  D -->|"throttles"| F["Check reserved and account concurrency and maxReceiveCount"]
  C --> G["Fix, test with a DLQ sample, deploy through the canary"]
  E --> G
  F --> G
  G --> H["Redrive at a fixed low rate and watch failed records and duplicates"]:::core
  H --> I{"DLQ empty and backlog age normal?"}:::decision
  I -->|"no"| B
  I -->|"yes"| J["Close: record the cause, add a test, adjust alarms"]:::core
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029

1. Classify (5 min). Ten DLQ samples all fail on the amount field and good orders still succeed, so do not pause the mapping (--no-enabled stops polling; messages then wait only until their retention ends^[76]).
2. Contain. Ask the partner to fix the field; classify the incident as data, not availability.
3. Fix. Accept and validate both forms of the amount; add the DLQ sample as a test; deploy through the canary.
4. Redrive. aws sqs start-message-move-task --source-arn <dlq-arn> --max-number-of-messages-per-second 20, a rate the provider can absorb;^[69] keys only hours old make any repeat harmless.
5. Verify. DLQ empty; FailedRecords at 0; 4,000 new charges and no repeated keys at the provider.
6. Close. Postmortem via IncidentOps; validate the contract at intake so bad payloads fail at the API.

• “The platform gives us exactly once.” Platform de-duplication is limited in time and scope (alternatives).
• Retrying everywhere. SDK, function, queue and caller retries multiply load during an incident;^[36] pick one layer.
• The DLQ as a bin. Without an owner, an alarm and retention longer than the source,^[27] it is delayed data loss.
• messageId as the idempotency key. It misses producer resends and redriven messages, which get new IDs.^[52]
• Synchronous function chains. A waiting caller is billed while it waits;^[12] use a queue or a workflow engine.
• Unlimited concurrency by default. A spike becomes a database outage; cap consumers of fragile dependencies.
• “Serverless is always cheaper.” Cost scales with attempts × duration × memory; steady, heavy work can be cheaper elsewhere.^[71]

Good fit: bursty traffic with idle periods (most functions in Azure’s production data were “invoked very infrequently”^[77]); event-driven glue between managed services; short units of work; teams that prefer owning code to hosts.

Poor fit, or not yet: steady high-volume work; work beyond platform limits (15 minutes on Lambda Event Functions, 90 for async and most event-source invocations on Managed Instances;^[10] 60 minutes per Cloud Run request^[41]); strict latency with no budget for warm capacity; side effects that cannot be idempotent; nobody on call. Hidden responsibilities: DLQ triage, runtime upgrades, quotas, cost per event.

flowchart TB
  A["New workload"] --> B{"Spiky or event-driven, with idle periods?"}:::decision
  B -->|"no: steady and heavy"| X1["Containers, instance-based billing or Lambda Managed Instances"]:::neutral
  B -->|"yes"| C{"Each unit of work fits platform time limits?"}:::decision
  C -->|"no"| X2["Jobs or a workflow engine (WorkflowOps)"]:::neutral
  C -->|"yes"| D{"Every side effect can be made idempotent?"}:::decision
  D -->|"no"| X3["Not yet: fix the contract first"]:::risk
  D -->|"yes"| E{"Latency target on every request?"}:::decision
  E -->|"yes"| F["Functions with warm capacity, or containers with minimum instances"]:::core
  E -->|"no"| G["Functions or serverless containers on demand"]:::core
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

The tree is this page’s judgement; adjust it with your own cost (Lab 2) and latency data.

This is this Atlas’s model, not an industry standard.

Preflight for a new event-driven function

• Invocation model identified; retry owner and limits written down
• Business idempotency key passed to every downstream side effect; the provider’s key retention known
• Failure destination or DLQ with an owner, an alarm and retention longer than the source
• Timeouts nested: downstream call < function timeout; SQS visibility ≥ 6 × function timeout
• Maximum concurrency on consumers of fragile dependencies; reserved concurrency ≥ that maximum
• Stream mappings: finite retries, maximum record age, bisect, partial batch responses

Production readiness

• Duplicate, poison-message and slow-dependency tests pass
• Canary alarms include failed records, not only Errors
• Peak concurrency ≤ 70% of what is available, or a quota increase requested
• Execution role scoped to exact actions; invoke permissions limited by source account or ARN; public paths authenticated

Monthly review

• DLQ counts, oldest ages and redrive records reviewed
• Cost per business event against forecast
• Runtimes over 180 days from deprecation, including versions behind aliases (notices list only $LATEST; use Trusted Advisor or the API^[49]); container-image functions checked by hand

Event-path design record (one per trigger, kept next to the code):

Function / trigger:          order-consumer  <-  SQS orders-queue (poll-based)
Business key:                order_id (sent downstream as Idempotency-Key)
Who retries, how long:       SQS after 180 s visibility, maxReceiveCount 5 (~15 min)
After the last attempt:      orders-dlq, 14 days, owner: payments team, alarm: depth > 0
Timeouts:                    provider call 5 s < function 30 s <= visibility 180 s / 6
Concurrency:                 mapping max 20, reserved 25; provider limit: ___ requests/s
Partial failures reported:   yes (ReportBatchItemFailures), FailedRecords metric
Expected volume / cost:      30 M events/month, 1.05 attempts/event, $3.75 per million
                             (Lab 2: Lambda + table + queue; logs and API layer extra)
Local record expiry:         15 days > DLQ retention (14 days)
Provider key retention:      ___ (Stripe API v1: may be pruned after 24 h)
Replay safety:               the shorter of the two; older redrives: look up the order
                             at the provider before charging

DLQ runbook card:

Alarm:           orders-dlq ApproximateNumberOfMessagesVisible > 0
First 5 minutes: sample 10 messages; group by error; is good traffic still succeeding?
Pause?           only if good traffic fails: update-event-source-mapping --no-enabled
Fix:             add the sample as a test; deploy through the canary
Redrive:         start-message-move-task at <= consumer capacity; approver: ______
Close:           cause, count, duplicates seen (expect 0), alarm or test added

Is ServerlessOps a recognised discipline? No. It is a niche label from 2016–2020; the practice is simply operating serverless applications. Use the label only if your organisation already does.

Does a FIFO queue make my consumer exactly-once? No. It prevents duplicate sends within a five-minute window;^[54] a consumer that times out after a side effect can still repeat it.

Should I keep functions warm with scheduled pings? Measure first: on Lambda, cold starts typically affect under 1% of invocations.^[23] Where a latency target requires it, use the platform’s warm capacity; Cloud Run treats minimum instances as “best-effort”.^[47]

Do the canary alarms judge only the new version? Not in the reference template: both are function-wide, so during the canary an old-version failure (a flaky provider, say) also rolls the release back, a conservative choice. To judge the new version alone, alarm on Errors per version (the ExecutedVersion dimension, which AWS suggests for two versions behind a weighted alias^[78]) and give FailedRecords a version dimension.

Can I safely redrive a ten-day-old DLQ message? Only if both your idempotency record and the provider still hold its key; otherwise look the order up at the provider first (keys expire).

• New compute types (as of October 2026). AWS now distinguishes Lambda Event Functions (“the name for the original Lambda function type”), Web Functions (several requests per environment) and MicroVMs (environments living up to 8 hours).^[22] The boundaries of “serverless” shift; the three event-path questions do not.
• Durable execution. Lambda durable functions run workflows “lasting up to a year”;^[22] treat them as a workflow engine (WorkflowOps).
• Telemetry standards. OpenTelemetry’s FaaS conventions are at “Development” status;^[61] attribute names may change.

• Day 1: read this page to the decision framework; run Labs 1 and 2.
• Week 1: deploy Lab 3 in a sandbox; break it with a poison message; redrive it.
• Month 1: design records for your real functions; failed-record alarms; a duplicate-storm game day.
• Advanced: stream semantics; durable execution (WorkflowOps); SLO design (SRE).
• Portfolio project: an idempotent consumer whose tests replay every message twice, plus a cost-per-event dashboard.

• Label: niche, mostly historical; the practice is operating serverless compute in production.
• Assume: at-least-once delivery, retries you did not write, events dropped after the last attempt, keys that expire.
• Do: business-key idempotency sent downstream; partial batch responses; owned failure destinations; nested timeouts; concurrency caps with matching reservations; canaries with failed-record alarms.
• Measure: backlog and DLQ age, throttle ratio, duplicate effects, whole-path cost per million events.
• Watch out for: infinite stream retries, metadata-only stream destinations, short DLQ retention, alarms on the wrong metric.
• Decide: spiky, short, idempotent work fits; steady, heavy or long-running work usually fits elsewhere.

Situation (illustrative). After a four-hour incident, 2 million orders are queued and the business wants them processed within the hour. The payment provider’s contract allows 300 requests per second; each charge takes about 0.25 s.

Option A: remove the caps. Drop maximum concurrency and the reservation so the consumer draws on the Regional pool (1,000 by default). 1,000 consumers at 0.25 s offer about 4,000 requests per second, over 13 times the contract, while the account’s other unreserved functions in that Region are throttled.^[11] Expect provider throttling, timeouts, retries multiplying load^[36] and a filling DLQ.

Option B: drain at the contract rate. Concurrency = rate × duration = 300 × 0.25 = 75 consumers.^[11] Two million orders at 300 per second take about 1.9 hours.

Decision. Choose B: maximum concurrency about 70 (below 75) and reserved concurrency raised to at least 70 in the same change, or Lambda throttles messages^[37] and healthy orders reach the DLQ after five receives. Tell the business it takes about two hours, or negotiate a temporary higher rate. A looks faster but probably finishes later, as failed charges cycle through retries and the DLQ. Uncertainty: the provider may tolerate more, and urgent orders may justify a priority queue with its own cap.

You should now be able to explain at-least-once delivery; design an idempotent consumer with a failure destination; configure timeouts and concurrency consistently; diagnose a stalled stream or a filling DLQ; and decide whether a workload fits serverless compute.

1. A consumer records each order_id in a table before charging. Why can customers still be charged twice?

2. A Kinesis consumer’s IteratorAge keeps rising on one shard while the others are fine. What do you check first?

3. Your function timeout is 60 s. What is the minimum SQS visibility timeout AWS recommends, and why?

4. A canary deployment passed with Errors at zero, yet the DLQ filled an hour later. Name two possible causes.

5. A workload processes 400 events per second at 200 ms each. Its dependency slows to 2 s. What happens to concurrency and cost?

6. A stream mapping’s on-failure destination is an SQS queue. What does the repair job find there?

ServerlessOps never became a discipline with a body or certification, and the label has faded. The work it named did not. Serverless platforms take away servers and hand you a contract written in retries, delivery guarantees, concurrency limits and per-attempt prices. Teams that read it carefully, making every effect idempotent, giving every failed event an owned destination, and budgeting concurrency against their slowest dependency, get less undifferentiated work and costs that follow demand. Teams that skip it get duplicate charges, silent losses and bills that grow without new customers.

Sources were opened on 2026-10-16, or on 2026-10-17 where the footnote says so. Limits, prices and product names marked “as of October 2026” go stale: re-verify them quarterly and on each Lambda, SAM or Powertools release; Lab 2 takes prices as inputs.

Changelog.

• 2026-10-16, r0: first draft (term status confirmed as niche; 13 parts; 6 diagrams; Labs 1–2 executed, Lab 3 not executed).
• 2026-10-17, r1: revised after the independent review: provider key expiry, metadata-only stream destinations, whole-path cost in Lab 2 (re-executed), concurrency in the judgement call, Lens dated 2017, SAM version publishing, 8 new sources, repetition cut.
• 2026-10-17, post-score polish: stream partial-batch retries, function-wide canary alarms (FAQ), S3 bucket-deletion precondition, re-check cadence; repeats trimmed.