IoTOps

IoTOps (IoT operations) is the work of running fleets of networked devices that have little or no user interface (sensors, meters, locks, cameras, controllers) from the factory to the scrap bin: giving each one an identity, onboarding it, keeping its firmware updatable without bricking it, watching its health within battery and bandwidth limits, and decommissioning it. The IETF’s IOT Operations working group describes the devices well: they are networked, “have a very limited end-user interface or no end-user interface at all”, and are deployed in numbers that “cannot easily be managed or maintained manually”.^[1]

In one line: every device in the field is known, reachable on its own terms, safely updatable and deliberately retired.

Three failures recur when nobody runs a device fleet as a system:

1. Devices that cannot be fixed. RFC 9019 calls a firmware update mechanism “a fundamental requirement for fixing vulnerabilities”.^[2] The Mirai botnet showed the cost of insecure defaults: it scanned Telnet ports 23 and 2323, tried a dictionary of 62 username and password pairs, and peaked at about 600,000 infected devices.^[3]
2. Updates that break the fleet. In August 2017 LockState told The Register that firmware for its 7i lock “had mistakenly been sent to some 6000i customers”. The locks lost their connection to the vendor’s servers,^[4] and at least 500 owners had to mail part of the lock back or wait for a replacement.^[5]
3. Devices that outlive their dependencies. Google gave Cloud IoT Core customers a year’s notice that from 16 August 2023 devices would be unable to connect to its bridges;^[6][7] Let’s Encrypt warned that some older devices “won’t ever trust” its new root “because they don’t get software updates”;^[8] US alarms faced the 2022 3G shutdowns.^[9]

• Not a defined discipline. The IETF uses “IOT Operations” (iotops) as a working-group name, chartered for “device onboarding and lifecycle management”.^[1] SecuriThings, a physical-security vendor, promoted “IoTOps” in 2021 as “a practice to operationally manage IoT devices in a consolidated, automated, and secure manner”;^[10][11] in October 2026 its home page says “Enterprise IoT Orchestration” and does not use the word.^[12]
• Not Azure IoT Operations. That is a Microsoft product, “a unified data plane for the edge” on Arc-enabled Kubernetes,^[13] managed with az iot ops.^[14] Other collisions (as of October 2026): Cisco IoT Operations Dashboard, a cloud service that provisions and manages industrial routers and gateways (EdgeOps territory);^[15] a monitoring service sold as “IoT-OPS”;^[16] and a 2021 article that expands IoTOps as “IoT operating systems”.^[17]
• Not edge computing. The IETF’s draft update of RFC 7228 separates “M-group” (microcontroller) from “J-group” (general-purpose, Linux-capable) devices.^[18] This page refines the Atlas’s device-class split (an editorial distinction): microcontroller and single-purpose embedded Linux devices (cameras, access controllers) belong here; gateways and general-purpose nodes that host workloads belong to EdgeOps.
• Not user devices, robots or vehicles: EndpointOps, RoboOps, FleetOps.
• Not OT security engineering. Operational technology (OT) such as industrial control systems has its own guidance (NIST SP 800-82, ISA/IEC 62443);^[19][20] the Atlas has no OT page, so this page covers device security baselines and SecOps routes the rest.

• Constrained-node engineering (2014). RFC 7228 classified devices by memory and energy, and the Constrained Application Protocol (CoAP) gave Class 1 devices a web protocol over UDP.^[21][22] MQTT 3.1.1 became an OASIS Standard in October 2014 and MQTT 5.0 in March 2019.^[23]
• Cloud device platforms. Managed brokers added stored device state (“shadow” on AWS, “twin” on Azure), bulk provisioning and OTA job services.^{[24][25][26][27]}
• Security baselines after Mirai. NIST published its IoT device core baseline in May 2020;^[28] ETSI EN 303 645 set consumer baselines (version 3.1.3, September 2024).^[29] In April 2026 NIST’s IR 8259 Rev. 1 extended manufacturer guidance to “pre-market through post-market activities”, including end of life.^[30][31]
• Regulation. The UK’s Product Security and Telecommunications Infrastructure (PSTI) regime for connectable consumer products took effect on 29 April 2024;^[32] the EU Cyber Resilience Act (CRA) entered into force on 10 December 2024, with reporting duties from 11 September 2026 and the main obligations from 11 December 2027.^[33]

Mental model: three planes and one budget. The identity plane answers who a device is and what it may do. The state plane holds desired versus reported configuration. The change plane moves firmware and settings through staged jobs. All three are limited by one budget: energy, airtime and money. Every operation is asynchronous: the device, not the operator, decides when it is awake.

flowchart TB
  SE["SecOps: baselines, OT routing"]:::neutral -->|"security baseline"| I
  PR["PrivacyOps: legal basis"]:::neutral -->|"data rules"| I
  I["IoTOps: constrained device fleet"]:::core
  I -->|"gateways, site nodes"| ED["EdgeOps"]:::neutral
  I -->|"health signals"| OB["ObservabilityOps"]:::neutral
  I -->|"device data"| DA["DataOps"]:::neutral
  I -->|"telematics"| FL["FleetOps"]:::neutral
  I -->|"device patterns: identity, OTA"| RO["RoboOps"]:::neutral
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Charter (this Atlas’s proposal, not a published standard): keep every device known, reachable, updatable and retirable for its support period, at an accepted cost per device.

1. Design for the device you cannot reach. AWS’s IoT Lens says: “Design for offline behavior” and track devices “that are not connected or communicating on a regular timeframe”.^[38]
2. No update path, no shipment. An update mechanism goes in before the first unit leaves the factory.^[2]
3. One identity per device, never shared, even across a test batch.
4. Every fleet change is staged and abortable, with a written abort rule.
5. Confirm only after a self-test that includes reconnecting.
6. Spend energy and airtime on purpose; every reporting interval is a budget decision.
7. Publish the support period and plan the end before launch.
8. Count the silent devices: fleet KPIs divide by the devices you expect to hear from.

Non-goals: analytics on device data, OT process safety, and user-device management.

Connectivity shapes everything. LoRaWAN Class A devices receive downlinks only in two short windows after they transmit, and must switch to Class B or C for firmware updates over the air;^[50] European sub-bands typically limit transmit time to 1% or 0.1%.^[51] Cellular LTE-M and NB-IoT devices use power saving mode (PSM), registered but not pageable while asleep, or extended discontinuous reception (eDRX), which trades some battery for reachability; the network sets or negotiates several timers.^[52]

R = responsible, A = accountable, C = consulted, I = informed. In a small company two people hold every column; the RACI still names who halts a rollout at 03:00.

stateDiagram-v2
  [*] --> Manufactured
  Manufactured --> Provisioned: identity injected
  Provisioned --> Commissioned: installed, onboarded
  Commissioned --> Operational
  Operational --> Updating: update job
  Updating --> Operational: confirmed or reverted
  Operational --> Quarantined: fault
  Quarantined --> Operational: fixed
  Operational --> Decommissioned: retired
  Quarantined --> Decommissioned: not recoverable
  Decommissioned --> Provisioned: new owner
  Decommissioned --> [*]

RFC 8576 adds that a device may be “removed and recommissioned to be used in a different system under a different owner”, and lists uncomfortable end states: a vendor that “no longer supports a device type”, or a device “simply forgotten but remains functional”.^[53]

• Day 0 (design): hardware class, bootloader and slots, secure key storage, identity scheme, energy budget, support period.
• Day 1 (rollout): factory provisioning, onboarding at the site, first check-in, inclusion in fleet KPIs.
• Day 2 (operation): updates, credential rotation, health review, quarantine, field service, decommissioning.

Fleet-wide destructive actions (factory reset, batch revocation, forced job cancellation) need a second approver.

flowchart TB
  subgraph DEV["Device"]
    KS["Key store"]:::core
    BL["Bootloader, slots A/B"]:::core
    AG["Device agent"]:::core
  end
  KS -->|"signs TLS handshake"| AG
  BL -->|"verifies, boots image"| AG
  CA["CA and provisioning"]:::core
  AG --> LK["Radio link or gateway"]:::neutral
  CA --> BR
  LK -->|"mutual TLS"| BR["Broker"]:::core
  BR --> Q["Ingest queue"]:::neutral
  BR --> SH["Shadow or twin"]:::core
  BR --> JB["Jobs and image store"]:::core
  LK -->|"image download"| JB
  Q --> TS["Time-series store"]:::neutral
  BR -->|"presence events"| FI["Fleet index and health"]:::decision
  SH --> FI
  JB --> FI
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029

Alternatives. A managed platform (AWS IoT Core, Azure IoT Hub) removes broker operations but ties identity, state and jobs to one provider; a self-hosted stack (Mosquitto, hawkBit, Ditto or ThingsBoard)^{[56][46][43][57]} keeps control and adds work; an LwM2M server offers a standard device-management object model.^[42] Gateways for Class 0 devices^[21] and LoRaWAN network servers become part of the platform you operate.

sequenceDiagram
  participant Op as Operator
  participant Svc as Job service
  participant Dev as Device agent
  participant Boot as Bootloader
  Op->>Svc: Create job for class rev A
  Svc->>Dev: Job notice at next check-in
  Dev->>Svc: Get job document
  Svc-->>Dev: Document with time-limited URL
  Note over Dev: Fetch image from store
  Dev->>Boot: Image in slot B, marked test
  Boot-->>Dev: Boots slot B
  Dev->>Svc: Self-test passed, reconnected
  Dev->>Boot: Confirm image
  Dev->>Svc: Report SUCCEEDED
  Svc-->>Op: Wave statistics

Happy path: the figure, when class, battery, signature and sequence number all pass.

Exception paths.

• Rejected before install: wrong class, bad signature or an older sequence number. RFC 9124 requires a monotonic sequence number “to prevent malicious actors from reverting a firmware update” and a Class ID for devices that “can accept the same firmware update without modification”.^[36] The device reports REJECTED, which AWS IoT Jobs can count towards an abort.^[27]
• Crash or hang after install: the unconfirmed image reverts at the next reset,^[35] and the device reports failure from the old image.
• Boots but cannot reconnect: the most dangerous case, because a device that confirms is then unreachable. Hence the self-test includes a round trip to the service (an editorial recommendation built on MCUboot’s “self-test routine” guidance).^[35]
• Silent: an IN_PROGRESS execution that goes quiet becomes TIMED_OUT when the timeout expires;^[58] a device that never fetched the job stays queued. Count both until proven otherwise.

1. Generate the key where it will live (ideally a secure element); the CA signs the device’s certificate signing request (CSR).
2. Choose an onboarding pattern: certificates registered at manufacture; by claim (a shared claim certificate is exchanged for a unique one at first connection); by trusted user (an installer’s app gets a claim certificate valid for five minutes);^[41] or late binding with FDO ownership vouchers.^[40] Azure’s DPS attests X.509, TPM or symmetric-key identities.^[26]
3. Treat claim credentials as hazardous. AWS says claim private keys “should be secured at all times, including on the device”, and to turn off the claim certificate if misuse is detected.^[41]
4. Bind permissions to the identity. The device may use only its own client ID and topics (see the reference implementation).
5. Plan trust-store updates. Roots expire too: DST Root CA X3, which cross-signed Let’s Encrypt’s chain, expired on 30 September 2021; IoT clients needed Let’s Encrypt’s ISRG Root X1 and, with OpenSSL, version 1.1.0 or later.^[8] Ship spare trust anchors and update them over the air.

Adapted from AWS’s reference flow;^[61] the rule is the firmware rule: test, then confirm.

1. Start early: an expiring-certificate audit (30 days by default) or a schedule opens a rotation job.^[60]
2. New key on the device, ideally in its secure element (AWS calls key rotation “optional but recommended”); the device sends a CSR over its current session.^[61]
3. Same rights: the CA signs it (on AWS, CreateCertificateFromCsr^[62]); attach the same policy and the same thing, which the IsAttached condition requires.^[63]
4. Store beside, not over: keep the old key and certificate until the new pair has worked.
5. Reconnect and report with the new certificate; on failure, reconnect with the old one and report the error.^[61]
6. Retire the old certificate only after success: deactivate, then revoke or delete it (self-hosted: add it to the CRL).
7. Chase stragglers: sleeping or failed devices stay on the audit list and retry. Lab 2 rehearses steps 2–4.

1. Pre-flight: signed image; manifest with class and sequence number; the previous image still on the device.
2. Eligibility: right class, enough battery, seen recently; the rest are deferred and counted.
3. Canary by stratum: devices from each hardware revision, radio type and region, with the abort rule checked per class (one job per class); the first 1% of a sorted list is not a canary (Lab 3).
4. Abort rules: in AWS IoT Jobs, abort criteria combine a failure type (FAILED, REJECTED, TIMED_OUT or ALL), a threshold percentage and a minimum number of devices notified before the job can abort.^[64] Keep that minimum no larger than the canary.
5. Rate and time: an exponential rate rises only after a set number of devices were notified or succeeded; maintenance windows (continuous jobs only) avoid busy hours; an in-progress timeout turns silence into TIMED_OUT.^[27][58]
6. Confirm after a self-test that includes reconnecting. Mender commits on first boot and rolls back if the device reboots before committing;^[44] RAUC warns that a clean write “does not mean that the system you just installed will really boot”.^[45]
7. Watch the deferred and the silent until the campaign closes.

• Commands to sleeping devices go into desired state, not queued messages: AWS keeps persistent MQTT sessions for 3,600 seconds by default,^[65] while a shadow holds the desired value until the device next connects.^[24]
• Presence: set the MQTT Last Will so the broker announces abnormal disconnects,^[23] and send status on a fixed interval, as AWS’s IoT Lens recommends.^[38]
• Reconnect with jittered backoff. Devices retrying on one schedule after an outage arrive together; Marc Brooker concludes “The solution isn’t to remove backoff. It’s to add jitter.”^[66] AWS IoT Core allows one connect per second per client ID.^[65]
• Count the handshake. A sensor that reconnects for every report repeats a TLS or DTLS handshake, certificate chain included, to send a few bytes. RFC 7925’s IoT profile requires constrained clients to support session resumption, which needs less bandwidth, and certificate-chain caching;^[67] the DTLS Connection ID avoids a new handshake when a sleeping device’s address changes.^[68]
• Keep low-power wide-area network (LPWAN) payloads small and binary, avoid downlinks and unnecessary confirmed uplinks.^[69]

1. Revoke the certificate and remove the device from groups, jobs and network allow-lists. A self-hosted broker needs the updated CRL and a reload; then check that the device is refused when it next connects.^[56]
2. Wipe credentials and user data; from December 2027 the CRA requires that users can “securely and easily remove on a permanent basis all data and settings”.^[70]
3. For resale or transfer, re-provision under the new owner (RFC 8576’s “reownership”).^[53]
4. For a product line, publish the end of support and ship a final update; RFC 8576 argues for an “authorization handover” so owners can add trust anchors and install firmware from other sources.^[53]

Laws bind the manufacturer placing the product on the market; standards and guidance bind only when a contract, conformity route or regulator adopts them. Check scope with ComplianceOps.

Platform lifecycle is a risk to manage. Microsoft’s Azure Updates notice AZ569914 (29 September 2026) retires Azure IoT Central on 20 September 2029 and points to IoT Hub, DPS and Azure Device Registry;^[72] a 2024 retirement message had been withdrawn as an error.^[73] Keep identity and update logic portable, with an exit plan.

Automate the change plane: eligibility filters, abort rules, expiry alerts and rotation jobs belong in code. AI has narrower uses: anomaly detection on fleet health needs a baseline per hardware class and firmware version, and an agent should propose, not push, firmware changes.

Assumptions (illustrative): 10,000 battery-powered LTE-M sensors, two hardware revisions (sensor-revA, sensor-revB), one AWS account and region. It maps to Azure IoT Hub with DPS and Device Update, or to Mosquitto with hawkBit.

1. Naming and identity. Thing name = MQTT client ID = certificate subject CN, for example sensor-0001. One certificate per device; thing groups per hardware revision and per rollout wave (sensor-revA-canary, sensor-revA-wave2 …).

2. Least-privilege device policy, after AWS’s “registered devices” example: iot:Connection.Thing.ThingName is resolved from the certificate the device presents, and IsAttached admits only registered things attached to it.^[63][74] In AWS policies * is not confined to one topic level,^[74] so shadow/* also covers named shadows, but only the device’s own.

Example status: JSON syntax-checked; not executed (requires an AWS account). Replace the region and 123456789012.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource":
        "arn:aws:iot:eu-west-1:123456789012:client/${iot:Connection.Thing.ThingName}",
      "Condition": { "Bool": { "iot:Connection.Thing.IsAttached": "true" } }
    },
    {
      "Effect": "Allow",
      "Action": "iot:Publish",
      "Resource": [
        "arn:aws:iot:eu-west-1:123456789012:topic/dt/sensor/${iot:Connection.Thing.ThingName}",
        "arn:aws:iot:eu-west-1:123456789012:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/*",
        "arn:aws:iot:eu-west-1:123456789012:topic/$aws/things/${iot:Connection.Thing.ThingName}/jobs/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": [
        "arn:aws:iot:eu-west-1:123456789012:topicfilter/$aws/things/${iot:Connection.Thing.ThingName}/shadow/*",
        "arn:aws:iot:eu-west-1:123456789012:topicfilter/$aws/things/${iot:Connection.Thing.ThingName}/jobs/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Receive",
      "Resource": [
        "arn:aws:iot:eu-west-1:123456789012:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/*",
        "arn:aws:iot:eu-west-1:123456789012:topic/$aws/things/${iot:Connection.Thing.ThingName}/jobs/*"
      ]
    }
  ]
}

Self-hosted equivalent. Mosquitto can require client certificates, use the certificate’s CN as the user name, and substitute it into access control list (ACL) patterns with %u.^[56] Three preconditions make that safe:

• A CA for devices only. Any CA in cafile can issue certificates “valid for connecting to your broker”, and any of them can claim any CN.^[56]
• A CRL. Without crlfile, a retired device’s unexpired certificate still connects; reload the broker (SIGHUP) after each CRL update.^[56]
• Read-only commands, with answers on a separate topic, as in the AWS policy, where commands arrive only through shadow and jobs.

Recent Mosquitto documentation suggests the mosquitto_acl_file plugin in place of acl_file; check your version.

Example status: compared line by line with the mosquitto.conf manual; not executed (no broker on the test host).

# mosquitto.conf (excerpt)
listener 8883
# CA that issues device certificates only, and its revocation list
cafile /etc/mosquitto/ca/device-ca.pem
crlfile /etc/mosquitto/ca/device-ca.crl
certfile /etc/mosquitto/certs/broker.pem
keyfile /etc/mosquitto/certs/broker.key
require_certificate true
use_identity_as_username true
acl_file /etc/mosquitto/acl

# /etc/mosquitto/acl : each device may use only topics under its own CN
pattern write dt/sensor/%u
pattern read cmd/sensor/%u/#
pattern write resp/sensor/%u

3. A firmware job with staged rate, abort rules and timeout. The job document’s fields are this design’s own (AWS imposes no schema); the presigned-URL placeholder is AWS’s syntax, filled in when a device asks for the document.^[54] Class and sequence number follow RFC 9124, but this is not an IETF SUIT (Software Updates for Internet of Things) manifest.^[36]

Example status: JSON syntax-checked; the create-job command passed parameter validation in AWS CLI 1.18.69 with all credentials disabled (no request sent); not executed against AWS.

{
  "operation": "firmware-update",
  "hwClass": "sensor-revA",
  "version": "2.4.1",
  "sequence": 1760745600,
  "sha256": "<64-hex-digest-of-the-image>",
  "url":
    "${aws:iot:s3-presigned-url-v2:https://s3.eu-west-1.amazonaws.com/example-fw-bucket/sensor-revA/2.4.1.bin}",
  "minBatteryPct": 30,
  "confirmAfter": ["selftest", "cloud-reconnect"]
}

{
  "maximumPerMinute": 200,
  "exponentialRate": {
    "baseRatePerMinute": 10,
    "incrementFactor": 2.0,
    "rateIncreaseCriteria": { "numberOfSucceededThings": 50 }
  }
}

{
  "criteriaList": [
    { "failureType": "FAILED", "action": "CANCEL",
      "thresholdPercentage": 5, "minNumberOfExecutedThings": 20 },
    { "failureType": "TIMED_OUT", "action": "CANCEL",
      "thresholdPercentage": 10, "minNumberOfExecutedThings": 20 }
  ]
}

# Files above saved as job-document.json, rollout.json, abort.json.
# Canary group: at least 20 devices (minNumberOfExecutedThings), and enough of each
# revision for a one-revision fault to cross the threshold.
aws iot create-job \
  --job-id fw-2-4-1-revA-wave1 \
  --targets arn:aws:iot:eu-west-1:123456789012:thinggroup/sensor-revA-canary \
  --document file://job-document.json \
  --target-selection SNAPSHOT \
  --presigned-url-config \
    roleArn=arn:aws:iam::123456789012:role/iot-fw-download,expiresInSec=3600 \
  --job-executions-rollout-config file://rollout.json \
  --abort-config file://abort.json \
  --timeout-config inProgressTimeoutInMinutes=60

4. Device-side rules. Accept only if hwClass matches, the sequence number is higher than the installed one and the battery is above minBatteryPct; verify hash and signature; install to the inactive slot as a test image; reboot. Then confirm, as below for embedded Linux with RAUC, whose rauc status mark-good and mark-bad signal success or failure to the bootloader.^[45] The marker holds the new image’s IMAGE_VERSION, which every image must set,^[75] so after a bootloader fallback the old slot is kept, never marked bad. On Zephyr with MCUboot, boot_write_img_confirmed() marks the running image OK so MCUboot keeps it.^[76]

Example status: syntax-checked; run against stub rauc, systemctl and checks in eight cases, from a clean pass to a fallback during a cloud outage (bash 5.0, 2026-10-18); not run on a device. selftest-local and cloud-roundtrip stand for your own checks.

#!/usr/bin/env bash
# confirm-boot.sh: runs once after every boot (e.g. a systemd unit) on a device
# with RAUC-managed A/B slots. Just before it reboots into a newly installed
# slot, the update agent writes that image's IMAGE_VERSION to $PENDING; it reports
# SUCCEEDED after a confirm, or FAILED from the old image after a fallback.
set -u
PENDING=${PENDING:-/data/ota/pending}
if [ ! -e "$PENDING" ]; then          # a slot confirmed earlier: keep it good
  rauc status mark-good               # (backends may count boot attempts)
  exit 0
fi
. /etc/os-release                     # sets IMAGE_VERSION of the running image
want=$(cat "$PENDING")
if [ -z "${IMAGE_VERSION:-}" ] || [ -z "$want" ]; then
  exit 1                              # unknown image: change nothing
fi
if [ "$IMAGE_VERSION" != "$want" ]; then
  # The new slot failed before this ran and the bootloader fell back to the old
  # one, which is running now: keep it good, never mark it bad.
  rauc status mark-good
  rm -f "$PENDING"
  exit 0
fi
for attempt in 1 2 3 4 5; do
  # local self-test, then connect with the device certificate and get a reply
  if selftest-local && cloud-roundtrip; then
    rauc status mark-good && rm -f "$PENDING" && exit 0
  fi
  sleep $((attempt * 60 + RANDOM % 60))   # jittered retries, 15-20 minutes
done
# Not confirmed in time (a long outage lands here too): give up the slot.
rauc status mark-bad                  # backend disables it or switches slot
rm -f "$PENDING"
systemctl reboot

Objective: see how sampling, batching and payload size drive battery life, freshness and message cost. Prerequisites: Python 3.8 or later; 15 minutes. Cost and risk: none (arithmetic).

1. Save the script as budget.py.
2. Run python3 budget.py.
3. Run python3 budget.py --reading-bytes 12 --batches 48,96 (binary readings instead of JSON).
4. Replace the defaults with currents measured on your own board with a power profiler. If the device reconnects for each report, include the TLS or DTLS handshake in --tx-s; the byte columns count the message only.

Example status: verified by execution (Python 3.8.10, 2026-10-18); inputs are illustrative assumptions, not measurements of any product. Price: AWS’s Europe (Ireland) example, $1 per million messages (as of October 2026).^[77]

#!/usr/bin/env python3
"""Battery, freshness and message budget for a battery-powered sensor fleet.

The device samples every --sample-min minutes and transmits a batch of
readings every k samples. Pure arithmetic: no network, no device access.
Every default is an illustrative assumption; measure your own board.
"""
import argparse
import math


def plan(k, a):
    samples = 24 * 60 / a.sample_min                    # per device per day
    reports = samples / k
    awake_s = samples * a.sense_s + reports * a.tx_s
    mas = (samples * a.sense_s * a.sense_ma + reports * a.tx_s * a.tx_ma
           + (86400 - awake_s) * a.sleep_ua / 1000)     # mA-seconds per day
    years = a.capacity_mah * (1 - a.derate) / (mas / 3600) / 365
    payload_kb = (a.header_bytes + k * a.reading_bytes) / 1024
    metered = math.ceil(payload_kb / 5)                 # 5 KB metering steps
    msgs = reports * 30 * a.fleet * metered             # fleet, per month
    return mas / 86.4, years, payload_kb, metered, msgs


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--batches", default="1,4,12,48,96", help="samples per report")
    p.add_argument("--sample-min", type=float, default=5)
    p.add_argument("--capacity-mah", type=float, default=2600)
    p.add_argument("--derate", type=float, default=0.30,
                   help="share of capacity lost to self-discharge, cold, pulses")
    p.add_argument("--sleep-ua", type=float, default=5.0)
    p.add_argument("--sense-ma", type=float, default=3.0)
    p.add_argument("--sense-s", type=float, default=0.2)
    p.add_argument("--tx-ma", type=float, default=120.0)
    p.add_argument("--tx-s", type=float, default=4.0, help="attach + send")
    p.add_argument("--header-bytes", type=int, default=200)
    p.add_argument("--reading-bytes", type=int, default=60)
    p.add_argument("--fleet", type=int, default=10000)
    p.add_argument("--usd-per-million", type=float, default=1.00)
    p.add_argument("--target-years", type=float, default=5.0)
    a = p.parse_args()
    if a.sample_min <= 0:
        raise SystemExit("--sample-min must be positive")
    print(f"{'k':>3} {'report':>7} {'avg uA':>7} {'years':>6} {'KB':>5} "
          f"{'metered':>7} {'msgs/month':>12} {'USD/month':>9}  flag")
    for k in [int(x) for x in a.batches.split(",")]:
        if k < 1:
            raise SystemExit("batch sizes must be >= 1")
        avg_ua, years, kb, metered, msgs = plan(k, a)
        usd = msgs / 1e6 * a.usd_per_million
        flag = "BELOW TARGET" if years < a.target_years else ""
        print(f"{k:>3} {k * a.sample_min:>5g}m {avg_ua:>7.1f} {years:>6.1f} "
              f"{kb:>5.1f} {metered:>7} {msgs:>12,.0f} {usd:>9,.2f}  {flag}")


if __name__ == "__main__":
    main()

Expected output (step 2, then step 3):

  k  report  avg uA  years    KB metered   msgs/month USD/month  flag
  1     5m  1606.9    0.1   0.3       1   86,400,000     86.40  BELOW TARGET
  4    20m   407.0    0.5   0.4       1   21,600,000     21.60  BELOW TARGET
 12    60m   140.3    1.5   0.9       1    7,200,000      7.20  BELOW TARGET
 48   240m    40.3    5.2   3.0       1    1,800,000      1.80
 96   480m    23.7    8.8   5.8       2    1,800,000      1.80

  k  report  avg uA  years    KB metered   msgs/month USD/month  flag
 48   240m    40.3    5.2   0.8       1    1,800,000      1.80
 96   480m    23.7    8.8   1.3       1      900,000      0.90

Validation: batching 48 readings per report cuts average current from about 1,607 µA to 40 µA, because the radio attaches 6 times a day instead of 288. At 96 readings the JSON payload (5.8 KB) crosses AWS’s 5 KB metering step,^[77] so halving the reports does not halve the bill until readings become 12-byte binary records. The price is freshness: data can be eight hours old, so alarms need an event-triggered send. Troubleshooting: years far above target usually mean the transmit time omits network attach; batch sizes must be >= 1 means a zero in --batches. Teardown: delete budget.py.

Objective: issue per-device certificates, verify chains, find certificates about to expire, rotate one, and see a root change from the device’s side. Prerequisites: bash and OpenSSL 1.1.1 or later; 10 minutes. Cost and risk: none; throwaway test keys in a temporary directory that the script deletes.

1. Save the script as lab2.sh and run bash lab2.sh.
2. Read the three device lines, the rotation pair, and the two verifications of the broker certificate.

Example status: verified by execution (OpenSSL 1.1.1f, bash 5.0, 2026-10-18); OpenSSL 3 output differs slightly in subject formatting (not executed with 3.x).

#!/usr/bin/env bash
# Lab 2: per-device identity with a private TEST CA. Never use these keys in production.
set -euo pipefail
WORK=$(mktemp -d)                       # everything stays in this directory
cd "$WORK"
echo "work dir: $WORK"

# 1. A test CA (stands in for your device-issuing CA or a cloud-registered CA).
openssl ecparam -name prime256v1 -genkey -noout -out ca.key
openssl req -x509 -new -key ca.key -sha256 -days 3650 \
  -subj "/O=Example Devices Test/CN=Example Device Test CA" -out ca.pem

# 2. Three devices: each makes its own key; the CA signs a CSR whose CN is the thing name.
issue() {  # $1 = device id, $2 = validity in days
  openssl ecparam -name prime256v1 -genkey -noout -out "$1.key"
  openssl req -new -key "$1.key" -subj "/CN=$1" -out "$1.csr"
  openssl x509 -req -in "$1.csr" -CA ca.pem -CAkey ca.key -CAcreateserial \
    -sha256 -days "$2" -out "$1.pem" 2>/dev/null
}
issue sensor-0001 730
issue sensor-0002 730
issue sensor-0003 20                    # simulates a certificate near expiry

# 3. Verify each chain and report certificates that expire within 30 days.
report() {  # $1 = certificate file
  cn=$(openssl x509 -in "$1" -noout -subject | sed 's/.*CN *= *//')
  chain=$(openssl verify -CAfile ca.pem "$1" | awk '{print $2}')
  if openssl x509 -in "$1" -noout -checkend $((30*24*3600)) >/dev/null; then
    exp="ok"
  else
    exp="EXPIRES <30d"
  fi
  echo "$1 CN=$cn chain=$chain expiry=$exp"
}
for c in sensor-000?.pem; do report "$c"; done

# 4. Rotation for sensor-0003: a new key and CSR "on the device", a new certificate.
#    Keep the old pair until the device has connected with the new one.
mv sensor-0003.key old-0003.key && mv sensor-0003.pem old-0003.pem
issue sensor-0003 730
report old-0003.pem; report sensor-0003.pem

# 5. A root change seen from the device: the broker's new certificate comes from a
#    new root; a device that trusts only the old root (ca.pem) rejects it.
openssl ecparam -name prime256v1 -genkey -noout -out root2.key
openssl req -x509 -new -key root2.key -sha256 -days 3650 \
  -subj "/CN=Example New Root" -out root2.pem
openssl ecparam -name prime256v1 -genkey -noout -out broker.key
openssl req -new -key broker.key -subj "/CN=broker.example.com" -out broker.csr
openssl x509 -req -in broker.csr -CA root2.pem -CAkey root2.key -CAcreateserial \
  -sha256 -days 365 -out broker.pem 2>/dev/null
openssl verify -CAfile ca.pem broker.pem || echo "=> rejected: old trust store"
cat ca.pem root2.pem > trust-updated.pem    # after an over-the-air anchor update
openssl verify -CAfile trust-updated.pem broker.pem

# 6. Teardown: remove only the directory this script created.
cd / && rm -rf -- "$WORK" && echo "removed $WORK"

Expected output (the temporary directory name will differ):

work dir: /tmp/tmp.<random>
sensor-0001.pem CN=sensor-0001 chain=OK expiry=ok
sensor-0002.pem CN=sensor-0002 chain=OK expiry=ok
sensor-0003.pem CN=sensor-0003 chain=OK expiry=EXPIRES <30d
old-0003.pem CN=sensor-0003 chain=OK expiry=EXPIRES <30d
sensor-0003.pem CN=sensor-0003 chain=OK expiry=ok
CN = broker.example.com
error 20 at 0 depth lookup: unable to get local issuer certificate
error broker.pem: verification failed
=> rejected: old trust store
broker.pem: OK
removed /tmp/tmp.<random>

Validation: sensor-0003 is flagged, as AWS IoT Device Defender’s “Device certificate expiring” check would flag it (default 30 days, configurable up to 3,652).^[60] In step 4 old and new certificates both verify, so the device can fall back until the new one has connected. Step 5 is the device’s view of a root change: with only the old root it rejects the broker, as un-updated clients did when DST Root CA X3 expired;^[8] after an anchor update it accepts it. Troubleshooting: set -e stops the script at a failing command; rerun with bash -x lab2.sh. Teardown: the script removes its own directory (after a failure, delete the printed one); delete lab2.sh.

Objective: see why the abort minimum must fit the canary, how a mixed wave dilutes a fault in one hardware class, and how a canary stratified by class catches it. Prerequisites: Python 3.8 or later; 20 minutes. Cost and risk: none (synthetic fleet with a fixed seed).

1. Save the script as rollout.py and run python3 rollout.py.
2. Run python3 rollout.py --min-done 5 and compare the first plan.
3. Run python3 rollout.py --seed 3, then python3 rollout.py --seed 3 --canary-per-class 8.
4. Run python3 rollout.py --seed 6 --canary-per-class 8 and look at plan 2.

Example status: verified by execution (Python 3.8.10, 2026-10-18), seeds 1–7, with and without --canary-per-class 8. The rule copies the shape of AWS IoT Jobs abort criteria but counts finished devices, where AWS counts notified ones; a per-class rule means one job per hardware class.^[64]

#!/usr/bin/env python3
"""Simulate a staged firmware rollout with eligibility and abort rules.

Synthetic fleet, deterministic seed, no network. The abort rule mirrors the
shape of AWS IoT Jobs abort criteria: abort when failures reach a threshold
percentage after a minimum number of devices have finished. With
--canary-per-class N the canary is the first N eligible devices of each
hardware class, and the rule is checked per class.
"""
import argparse
import random

WAVES = [0.01, 0.10, 0.50, 1.00]          # cumulative share of eligible devices


def make_fleet(n, rng):
    fleet = []
    for i in range(n):
        fleet.append({
            "id": f"dev-{i:04d}",
            "hw": "B" if rng.random() < 0.15 else "A",   # 15% hardware rev B
            "online": rng.random() > 0.03,                # seen in last 24 h
            "battery": int(rng.random() * 100),
        })
    return fleet


def outcome(dev, rng):
    """New firmware: fine on rev A, a driver fault on rev B (synthetic)."""
    r = rng.random()
    if dev["hw"] == "B":
        return "FAILED" if r < 0.60 else "SUCCEEDED"
    if r < 0.005:
        return "FAILED"
    return "TIMED_OUT" if r < 0.010 else "SUCCEEDED"


def run(fleet, classes, a):
    rng = random.Random(a.seed)
    eligible = [d for d in fleet if d["online"] and d["battery"] >= 30
                and d["hw"] in classes]
    offline = sum(not d["online"] for d in fleet)
    low = sum(d["online"] and d["battery"] < 30 for d in fleet)
    print(f"classes={sorted(classes)} eligible={len(eligible)} of {len(fleet)} "
          f"(deferred: offline={offline}, battery<30%={low})")
    canary = 0
    if a.canary_per_class:                # stratified canary goes first
        picked = [d for hw in sorted(classes)
                  for d in [e for e in eligible if e["hw"] == hw][:a.canary_per_class]]
        ids = {d["id"] for d in picked}
        eligible = picked + [d for d in eligible if d["id"] not in ids]
        canary = len(picked)
    tally = {hw: [0, 0] for hw in sorted(classes)}   # finished, failed
    start = 0
    for share in WAVES:
        end = max(start + 1, round(len(eligible) * share))
        if start == 0 and canary:
            end = canary
        for dev in eligible[start:end]:
            failed = outcome(dev, rng) in ("FAILED", "TIMED_OUT")
            tally[dev["hw"]][0] += 1
            tally[dev["hw"]][1] += failed
        groups = tally if canary else {
            "all": [sum(t[0] for t in tally.values()),
                    sum(t[1] for t in tally.values())]}
        wave = "canary" if start == 0 and canary else f"wave to {share:>4.0%}"
        hits = []
        for name, (done, bad) in groups.items():
            rate = 100 * bad / done if done else 0.0
            tag = f" rev {name}" if canary else ""
            print(f"  {wave}{tag}: finished={done:4d} failed={bad:3d} "
                  f"rate={rate:5.1f}%")
            if done >= a.min_done and rate >= a.threshold:
                hits.append(f"{tag} {rate:.1f}%")
        if hits:
            total = sum(t[0] for t in tally.values())
            print(f"  ABORT:{','.join(hits)} >= {a.threshold}% "
                  f"after {total} devices")
            return
        start = end
    print("  COMPLETE")


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--fleet", type=int, default=1000)
    p.add_argument("--threshold", type=float, default=5.0, help="percent")
    p.add_argument("--min-done", type=int, default=8)
    p.add_argument("--seed", type=int, default=7)
    p.add_argument("--canary-per-class", type=int, default=0,
                   help="0 = canary is the first 1%% of the list")
    a = p.parse_args()
    if a.canary_per_class < 0:
        raise SystemExit("--canary-per-class must be 0 or more")
    fleet = make_fleet(a.fleet, random.Random(a.seed))
    print("Plan 1: manifest without a hardware class (targets A and B)")
    run(fleet, {"A", "B"}, a)
    print("Plan 2: class-aware manifest (rev A only; rev B gets its own build)")
    run(fleet, {"A"}, a)


if __name__ == "__main__":
    main()

Expected output (step 1, then the second command of step 3, plan 1 only):

Plan 1: manifest without a hardware class (targets A and B)
classes=['A', 'B'] eligible=682 of 1000 (deferred: offline=41, battery<30%=277)
  wave to   1%: finished=   7 failed=  2 rate= 28.6%
  wave to  10%: finished=  68 failed=  7 rate= 10.3%
  ABORT: 10.3% >= 5.0% after 68 devices
Plan 2: class-aware manifest (rev A only; rev B gets its own build)
classes=['A'] eligible=573 of 1000 (deferred: offline=41, battery<30%=277)
  wave to   1%: finished=   6 failed=  0 rate=  0.0%
  wave to  10%: finished=  57 failed=  0 rate=  0.0%
  wave to  50%: finished= 286 failed=  3 rate=  1.0%
  wave to 100%: finished= 573 failed=  8 rate=  1.4%
  COMPLETE

Plan 1: manifest without a hardware class (targets A and B)
classes=['A', 'B'] eligible=689 of 1000 (deferred: offline=21, battery<30%=290)
  canary rev A: finished=   8 failed=  0 rate=  0.0%
  canary rev B: finished=   8 failed=  5 rate= 62.5%
  ABORT: rev B 62.5% >= 5.0% after 16 devices

Validation: in plan 1 the canary fails 2 of 7 devices but cannot abort below the minimum of 8; with --min-done 5 it aborts there. With seed 3 the minimum hides the canary’s failure again, and in the 10% wave rev A successes dilute rev B’s failures to 4.3%, so the job aborts after 344 devices. A canary of 8 per class, checked per class, stops it after 16. Seed 6 shows the price: one ordinary rev A failure in 8 devices is 12.5%, so small strata with a 5% threshold also stop good builds; size them against the normal failure rate. Plan 2 completes; its 318 deferred devices remain open work. Troubleshooting: other numbers on another Python version mean a changed random generator; --canary-per-class must be 0 or more means a negative value. Teardown: delete rollout.py.

1. Battery soil sensors on LoRaWAN (illustrative). 3,000 Class A sensors on a farm cooperative. Method: binary payloads, configuration queued until the next uplink, firmware updates only after switching a group to Class B or C.^[50] Success measure: battery forecast per firmware version within 10% of the field trend. Pitfall: treating downlinks as free; a transmitting gateway cannot receive.^[69]

2. A consumer smart lock sold in the UK and the EU (illustrative, sourced lessons). Method: unique credentials per device (ETSI 5.1-1, UK PSTI),^[29][32] a support period of at least five years (CRA, from December 2027),^[70] class-checked signed updates with revert, and a CRA reporting runbook. Lesson: LockState’s wrong-model update (see The problem it solves). Success measure: zero wrong-class installs; a reporting drill within 24 hours. Pitfall: treating the support period as marketing copy.

3. Cameras and access controllers across 200 sites (illustrative). The gap SecuriThings described: device maintenance falling “in a gap between IT and operational teams”.^[10] Method: one inventory across sites, firmware currency and password rotation tracked per device. Success measure: share of devices on supported firmware with rotated credentials. Pitfall: unclear ownership; here, the embedded Linux cameras stay on this page and site gateways go to EdgeOps.

4. Cellular alarms facing a network shutdown (sourced). Before AT&T’s 3G shutdown, scheduled for 22 February 2022, an alarm-industry group estimated that 1.5 million alarm customers still needed upgrades and about half a million medical-alert devices ran on 3G; an unnetworked fire alarm still sounds but cannot call the fire service.^[9] Method: record radio and network per device, track operator sunset notices quarterly, budget replacements years ahead. Success measure: no device on a network whose announced shutdown falls inside its support period.

Leading KPIs (this page’s classification) trigger a halt, a visit or a rotation job; lagging ones confirm outcomes. Set targets per hardware class from your own baseline; revisit them quarterly.

Worked calculation (illustrative fleet). 10,000 devices should report every 4 hours, with 2 hours’ grace; 9,412 were heard from in the last 6 hours: reachability 94.1%. A security release reached 8,830 devices: 8,700 confirmed, 55 reverted and reported failure, 75 timed out. Wave failure rate = (55 + 75) ÷ 8,830 = 1.5%. Only 2.4.1 now counts as supported, so firmware currency = 8,700 ÷ 10,000 = 87.0%. Report the 1,300 devices not current, split into deferred (low battery), silent and failed, each with an owner.

• Credentials. No universal defaults (ETSI provision 5.1-1; UK PSTI);^[29][32] one certificate per device, which AWS’s “Device certificate shared” audit rates critical when it finds sharing.^[59]
• Least privilege per device on topics and state; MUD lets a device declare the network access it needs.^[49] Close debug ports in production builds.^[28]
• Update integrity. Signed images, class checks and monotonic sequence numbers against downgrade.^[36] Going back is still possible. An unconfirmed image that reverts returns to the confirmed version, which is not a downgrade; an authorised rollback is a new signed decision, because the sequence number belongs to the manifest: “A firmware version may be rolled back by creating a new manifest for the old firmware version with a later sequence number”.^[36] NIST lists rollback by authorised entities, noting that some organisations prefer to rule it out;^[28] MCUboot’s optional rollback protection uses a security counter separate from the version.^[35]
• Vulnerability handling. Since 11 September 2026, manufacturers of products on the EU market, including products already sold, must report actively exploited vulnerabilities and severe incidents affecting product security through ENISA’s single reporting platform: early warning within 24 hours, notification within 72 hours, final report 14 days after a fix is available (vulnerabilities) or a month after the notification (incidents).^[78][37] From 11 December 2027 Annex I adds, where applicable, automatic security updates on by default with an opt-out, and a machine-readable software bill of materials (SBOM) of at least top-level dependencies;^[70] see DevSecOps and VulnerabilityOps.
• Privacy. Telemetry from homes and people can be personal data; legal basis belongs to PrivacyOps, data removal to the decommissioning playbook.

Platform cost formula (AWS, Europe (Ireland) example prices, as of October 2026). Messages cost $1 per million for the first billion and are “metered in 5 KB increments”; always-on connectivity costs $0.08 per million connection minutes, so 10,000 devices connected for 30 days use 432,000,000 minutes, or $34.56.^[77] Shadow operations are metered in 1 KB steps,^[77] so frequent, large state writes multiply metered operations, and each document is capped at 8 KB.^[65]

Device-to-cloud messaging cost = reports per device per month × devices × ⌈payload ÷ 5 KB⌉ × price per message (Lab 1 computes it). AWS also bills messages delivered to subscribers and, separately, rules engine use; messages sent through the Basic Ingest reserved topic carry no messaging charge.^[77] Connectivity (SIM data plans or LPWAN fees, from the operator) is its own line. One site visit at an assumed $150 costs more than a month of messaging for Lab 1’s fleet ($86.40), so reachability and the unrecoverable rate are cost metrics too.

Capacity. AWS IoT Core discards publish requests above 100 per second per connection and allows one CONNECT per second per client ID;^[65] duplicated client IDs and reconnect storms hit these first. Sustainability: battery replacements and devices bricked or abandoned early are waste; carbon accounting is in GreenOps.

flowchart TB
  A["Failures rise in a wave"]:::risk --> B["Halt rollout: cancel queued only"]:::core
  B --> C{"One class, region or link?"}:::decision
  C -->|yes| D["Exclude that stratum"]:::core
  C -->|no| E["Treat build as bad"]:::risk
  D --> F{"Devices reverted?"}:::decision
  E --> F
  F -->|yes| G["Fix build, new canary"]:::core
  F -->|"no, still silent"| H{"Reachable by another path?"}:::decision
  H -->|yes| I["Recovery job or reset"]:::core
  H -->|no| J["Field service, notify customers"]:::risk
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029

Example status: commands passed AWS CLI 1.18.69 parameter validation with credentials disabled; not executed against AWS.

1. Stop new installs. aws iot cancel-job --job-id fw-2-4-1-revA-wave2 --reason-code DEVICE_LOSS --comment "halted". Without --force only queued executions are cancelled; with it, in-progress ones too, and AWS warns to “ensure that each device executing a job which is canceled is able to recover to a valid state”.^[79] Do not force-cancel mid-write devices.
2. Size the damage. aws iot list-job-executions-for-job --job-id fw-2-4-1-revA-wave2 --status FAILED, then TIMED_OUT; join with the fleet index on hardware revision, radio, region and battery.
3. Classify. Reverted and reporting (recoverable); silent but within the expected interval (wait); silent past it (lost until proven otherwise).
4. Communicate. Field service and support get the affected list; if the release fixed an exploited flaw, the CRA clock (Security, privacy and compliance) still runs.
5. Recover. A fixed build goes to a new, stratified canary; lost devices need a visit or a postal swap.^[5]
6. Prevent. Add the missing check to eligibility, the class check or the self-test; record the review in IncidentOps.

• “We will add OTA after launch.” Without a bootloader, slots and keys in the first build, there is nothing to add it to.
• One credential per product line. One extracted key compromises every unit.
• The shadow as a database. It holds desired and last-reported state; history belongs in messages.^[25]
• Myth: QoS 2 makes delivery reliable end to end. MQTT QoS is between client and broker;^[23] sleeping devices and expired sessions still lose commands.

Use it when devices outnumber what people can manage by hand,^[1] stay in the field for years, sit where visits are expensive, or fall under consumer-product law. Do not build it for a dozen lab prototypes, for user devices (EndpointOps) or site servers (EdgeOps), or when a supplier operates the fleet as a service: govern the contract (support period, update service levels, data return). Class 0 devices “generally cannot be secured or managed comprehensively”;^[21] manage their gateway and plan to replace them.

flowchart TB
  Z{"Supplier-run, or a few lab units?"}:::decision -->|yes| Y["Do not build: govern the contract"]:::risk
  Z -->|no| A{"Devices updated remotely?"}:::decision
  A -->|no| B["Fixed-function: plan replacement"]:::risk
  A -->|yes| C{"Microcontroller class?"}:::decision
  C -->|yes| D["Microcontroller bootloader: test, confirm"]:::core
  C -->|"no, Linux"| E["A/B image updater"]:::core
  D --> F{"Team can run brokers 24x7?"}:::decision
  E --> F
  F -->|no| G["Managed platform plus exit plan"]:::core
  F -->|yes| H{"Data residency or lock-in critical?"}:::decision
  H -->|yes| I["Self-hosted stack"]:::core
  H -->|no| G
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029

This Atlas’s model, not an industry standard.

Before a device model ships

• Bootloader verifies signatures; two slots or a recovery image; revert drill passed
• Key generated on the device or in a secure element; rotation rehearsed
• Hardware class and sequence number checked before install
• Self-test reconnects to the service before confirming
• Energy and message budget reviewed with measured currents
• Spare trust anchor; trust store updatable over the air
• Support period, vulnerability contact and end-of-life plan published

Before every rollout

• Canary covers every hardware revision, radio and region in scope
• Abort minimum no larger than the canary; timeout set
• Deferred devices (offline, low battery) counted, with a follow-up date
• Halt owner on call; cancel procedure known (queued or in progress)

Decommissioning

• Certificate revoked (self-hosted: CRL updated); registry, groups and allow-lists cleaned
• Data and settings wiped; ownership transferred or device disposed of

DEVICE LIFECYCLE STATEMENT                         (one per model, reviewed yearly)
Model / hardware classes:     <model> / <class IDs>
Support period ends:          <YYYY-MM>     Basis: <law or contract, e.g. CRA, PSTI>
Security contact:             <URL or address>   Acknowledge within: <n days>
Update path:                  <bootloader, slots, signing key owner>
Connectivity and end dates:   <radio, operator, announced sunsets>
Platform and exit plan:       <service, notice period, export of identities>
End of life:                  <final update, data wipe, authorisation handover>

FIRMWARE ROLLOUT RECORD
Release / sequence / class:   <version> / <sequence> / <class ID>
Canary strata:                <revisions, radios, regions, count per stratum>
Abort rule:                   <failure types> >= <n>% after <m> devices
Timeout / maintenance window: <minutes> / <window>
Halt owner and second:        <name> / <name>
Result per wave:              <succeeded / failed / timed out / deferred>
Decision:                     continue | halt | rebuild        Signed: <name, date>

Is IoTOps a discipline, a job title or a product? None formally (see What IoTOps is not); search for IoT device management and firmware update.

Shadow, twin or digital twin? AWS’s shadow and Azure’s twin store a device’s desired and reported state;^[24][25] “digital twin” frameworks such as Eclipse Ditto model things more broadly.^[43]

How long should device certificates live? Long enough that a device on a shelf does not wake up expired, short enough that rotation stays routine; alert early^[60] and rehearse the rotation playbook on the bench.

Do microcontrollers really need two slots? They need a recovery strategy: RFC 9019 allows a second image or a recovery image that can fetch a new one.^[2] Without either, a bad image is a field visit.

Does a managed platform remove IoTOps? It removes broker operations, not identity, update, budget or lifecycle decisions, and it adds a dependency with its own end date.^[6][72]

• Standard manifests. The IETF SUIT manifest format is approved and in the RFC Editor queue (as of October 2026).^[81]
• Attestation. The remote attestation (RATS) architecture defines how a verifier judges evidence that a device is in an intended state;^[82] an IOTOPS working-group draft, “MUD-Based RATS Resources Discovery”, links MUD with attestation (as of October 2026).^[1]
• Regulated operations. With CRA reporting in force and the main obligations due in December 2027, this Atlas expects reporting drills and support-period records to become routine audit evidence.^[33]

By role. Beginners: the glossary, Labs 1 and 2. Practitioners: the reference implementation, Lab 3, the playbooks and the runbook. Leaders and architects: the lifecycle statement in Templates, the decision framework and the judgement call.

• Decide: support period, identity scheme, update mechanism, platform and exit plan, budget per device.
• Do: per-device credentials and rotation; signed, class-checked updates with test and confirm; stratified canaries with abort rules; desired state for sleeping devices; jittered reconnects.
• Measure: reachability, firmware currency, wave failure and unrecoverable rates, identity hygiene, battery risk, cost per device.
• Watch out for: shared claim keys, root changes, metering steps, network and platform retirements, forced cancellation.

Situation (illustrative). An actively exploited flaw affects 40,000 locks in the EU. A fixed build passed the bench. The product lead wants it everywhere today; the platform lead wants the usual canary.

Assumptions: devices revert unconfirmed images; a configuration change (disabling the exposed service) can go out through desired state first; the early warning is due within 24 hours either way.^[78]

Decision: B, after a configuration mitigation. Illustrative arithmetic: if the build strands 2% of devices, a 400-device canary costs 8 site visits and option A costs 800. Option A looks responsible, but bricked devices lose the path for the next fix too. Uncertainty: if exploitation causes physical harm and the bench covered every hardware revision, a shorter canary may be justified; record the reasoning either way.

After this page you can explain what IoTOps covers and why the label is niche; design an identity, rotation and update scheme; measure fleet health honestly; decide platform, rollout and lifecycle trade-offs.

1. Your dashboard shows 99% of connected devices are healthy. Why may that mislead?

2. A firmware wave’s abort rule needs 20 devices, but the canary has 12. What happens if all 12 fail?

3. Why should the self-test before confirming an image include reconnecting to the service?

4. A device reports every 5 minutes with a 6 KB JSON payload. Name two changes that cut its message cost.

5. How do you change a setting on a LoRaWAN Class A sensor that sleeps for 30 minutes?

6. Your platform provider announces a retirement in three years. What do you check first?

“IoTOps” is a working-group name and a short-lived marketing word, not a discipline. The work behind it is real and unforgiving: a device in a field, a wall or a lamp post must take every change without a human nearby, on a battery and a link someone else controls, for longer than many platforms and networks last. Teams that do it well decide the end before the beginning: one identity per device, an update path that can revert, rollouts that stop themselves, budgets set before release, and a support period they mean to keep.

Evidence per claim: research/iotops-ledger.md. Prices, quotas, platform behaviour and retirement dates change; re-check vendor pages before relying on them. All sources checked 2026-10-18.

Changelog. 2026-10-18, r0: first version (research, ledger, six diagrams, Labs 1–3 executed, AWS job parameters validated offline). 2026-10-18, r1: revised after independent review (reviews/iotops-r1-response.md): CRA and UK dates, Mosquitto CRL, rotation playbook, Labs 2–3 extended and re-run, cost, handshake, rollback, diagrams, EdgeOps refinement; five sources added. 2026-10-18, post-score polish (reviews/iotops-polish.md, reviews/iotops-polish-check.md): LwM2M bindings, PSTI passwords, Figure 3 edges, a fallback-safe confirm script, KPI kinds, repetition trimmed; three footnotes added.