EdgeOps

EdgeOps, as this Atlas uses the word, is the work of operating compute at many remote sites (stores, plants, depots, cell sites, ships) rather than in a few data centres: bringing nodes into service without an expert, changing software across thousands of sites in staged waves, keeping each site working when its link fails, and recovering sites without a visit.

Two facts separate it from running a data centre: sites lose connectivity as a matter of course, and nobody technical is there. A Cloud Native Computing Foundation (CNCF) working group says edge applications “must adapt to network connectivity that is unreliable or even unavailable (air gapped)”, and that management needs “zero touch provisioning of hardware and software” because on-site staff may be untrained, untrusted, minimal or absent.^[1] The page is for edge architects, platform teams, retail and industrial IT, and leaders deciding whether to put compute in the field.

Without deliberate edge operations, four things go wrong.

1. A central dependency fails and every site stops. Sign-in, configuration or images fetched from the cloud at start-up turn a WAN outage into a store outage. The target is static stability: “the overall system keeps working even when a dependency becomes impaired”.^[2]
2. A bad change reaches everything at once, and recovery needs hands, as the 2024 CrowdStrike outage showed for endpoints (EndpointOps; see the judgement call).
3. Sites drift. A site closed for a month returns three releases behind, with an expired certificate and a full disk.
4. Hardware sits in a back office. LF Edge calls such nodes “physically-accessible to hackers” and “often deployed on untrusted networks”.^[3]

“EdgeOps” is not a recognised discipline. We found no standards body, foundation, certification or book that defines it, and the label has competing meanings:

1. A product toolset: in Altair IoT Studio, “a set of tools for building, maintaining and continuously improving code in resource constrained devices”.^[4]
2. Vendor methodologies and platforms: ZPE Systems’ “infrastructure management methodology”;^[5] XenonStack’s lifecycle “tool”, “platform” and DevOps “strategy”, in one article;^[6] Dell’s NativeEdge, “an edge operations software platform”.^[7] A Dell-sponsored article calls edge operations a “new discipline” without defining it,^[8] and a third-party article says Dell promotes “DevEdgeOps”.^[9]
3. A company name: EdgeOps.AI sells machine-learning pipelines for edge hardware.^[10]
4. Consultancy definitions, such as “a comprehensive discipline that applies operational principles to distributed computing”.^[11]

Even “edge” is unsettled: NIST notes no consensus on how fog, mist, cloudlets and edge computing differ.^[12] The better-anchored vocabularies map onto Atlas pages:

EdgeOps here is not content-delivery edge (WebOps) and not “Kubernetes on small boxes”: a cluster is one possible site runtime. These boundaries are editorial distinctions, not industry rules.

• 2009: cloudlets, nearby “trusted, resource-rich” computers for mobile devices.^[15]
• 2014: a proposed ETSI group for Mobile-edge Computing, “IT and cloud-computing capabilities within the Radio Access Network”.^[16]
• 2018: NIST’s fog model^[12] and IEEE 1934-2018 (OpenFog architecture);^[14] a US restaurant chain and a US retailer described a Kubernetes cluster in every location (use case 1).^[17][18]
• 2023: CNCF edge-native principles;^[1] Dell NativeEdge.^[7]
• 2024–2026: KubeEdge graduates in the CNCF;^[19] ETSI GS MEC 003 V4.1.1;^[13] FIDO Device Onboard (FDO) 2.0 becomes a Proposed Standard.^[20]

Mental model: an edge fleet is many small data centres, each sometimes alone. Ask of every site: does it keep working alone, can I change it safely in batches, and can I get it back without going there?

The Atlas splits sites by device class and layer, an editorial distinction: one platform team often does several of these jobs.

flowchart BT
  D["`Devices: IoTOps
  Robots, vehicles:
  RoboOps, FleetOps`"]:::neutral
  S["`Sites: EdgeOps
  servers, gateways,
  site clusters
  (KubernetesOps mechanics)`"]:::core
  W["`Links: NetworkOps
  WAN, cellular, OOB`"]:::neutral
  C["`Centre: CloudOps
  GitOps, ReleaseOps`"]:::neutral
  O["`ObservabilityOps
  backends`"]:::neutral
  D -.->|gateways, shared patterns| S
  S -->|sites open every connection| W
  W -->|pull state, images| C
  W -.->|send telemetry| O
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

No body publishes an EdgeOps charter; this is the Atlas’s proposal. Mission: every site keeps serving through the failures it will meet, and every change reaches the fleet in batches that can be stopped and undone without a visit.

1. Sites are alone by default: nothing the core job needs is fetched centrally at run time.
2. Pull, never push: sites open outbound connections and fetch desired state (“Pulled Automatically” in OpenGitOps);^[23] the centre needs no route into a store. The out-of-band path calls home too.
3. Every change travels in waves, including OS, cluster and configuration changes.
4. Silence after a change is a signal: a site offline before a wave is unknown; one silent for T minutes after receiving the change is a suspected failure.
5. Images, not snowflakes: versioned, immutable images with a known previous version.
6. Zero-touch in, zero-touch back: onboarding and re-imaging need no expert; an out-of-band path exists for when both fail.
7. Assume the box will be stolen: secrets scoped to one site and revocable; know what a stolen box unlocks by itself.
8. Manage the fleet as a distribution: convergence, freshness and version spread, not anecdotes.

Non-goals: device firmware, WAN design, application logic. Example goals: 98% of sites healthy on a release within seven days; 90% of incidents resolved remotely; at most two OS versions; every site class drilled yearly.

The RACI below is this Atlas’s proposal. Site staff are hands, not operators: give them tasks as simple as “plug in, power on”.

A = accountable, R = responsible, C = consulted, I = informed.

• Day 0 (design): site classes, hardware, control-plane placement, written connectivity assumptions, golden image, wave rules.
• Day 1 (onboarding): imaged, TPM-enrolled and given onboarding credentials at a factory or depot, the node onboards itself on site and passes an acceptance check before carrying traffic.
• Day 2 (operate): waves; certificate and credential renewal; drills; hardware swaps; remote re-imaging.
• Retirement: crypto-erase, revoke the site’s credentials, remove it from waves, route hardware to reuse (GreenOps).

• One change path: every site change, and every promotion to the next wave, is a reviewed commit to the fleet repository. Write access to that repository or to the fleet manager reaches every site as root: protect the branch and require a second approver to promote a wave.
• Maintenance windows are data in each site profile, enforced by the upgrade tooling.
• Halt rules are written before the release: application owners name health signals; the platform team owns the gate.
• Cadence: weekly fleet review (stragglers, version spread); quarterly drill per site class; reviews of fleet-level incidents (IncidentOps).
• Exceptions expire: a site pinned out of the waves has an owner, a reason and an end date.

%%{init: {"flowchart": {"nodeSpacing": 14, "rankSpacing": 20, "padding": 8}}}%%
flowchart LR
  subgraph SITE["Untrusted site ×1,000s"]
    subgraph NODE["Node: A/B OS, TPM"]
      AG["`Fleet agent,
      runtime`"]:::core
      MIR["`Image
      cache`"]:::neutral
      COL["`Telemetry
      disk queue`"]:::neutral
    end
    OOB["`Out-of-band
    console`"]:::risk
  end
  subgraph CEN["Centre, via WAN"]
    FM["`Fleet
    manager`"]:::core
    REG["Registry"]:::neutral
    TEL["`Telemetry
    backend`"]:::neutral
    ONB["`Onboarding
    service`"]:::neutral
    ACC["`OOB access
    service`"]:::neutral
  end
  AG -->|desired state| FM
  MIR -->|layers| REG
  COL -->|when up| TEL
  NODE -->|once| ONB
  OOB -.->|calls home| ACC
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Trust boundaries. Credentials are per site, and the centre never connects in. Fleet’s agent-initiated registration shows the pull pattern: “the Manager will never make an outbound API request to the downstream clusters”.^[33]

The central decision, often made by default.

What a stretched cluster does in a partition. After a grace period (50 seconds upstream, 40 on EKS) the node controller marks a silent node Unknown.^[42][36] When the pods’ unreachable toleration expires, 300 seconds by default, it evicts them, but pods marked for deletion “may continue to run on the partitioned node”.^[43] For a pod pinned to its site, eviction moves nothing: it shows Terminating, keeps running and is replaced when the link returns. tolerationSeconds therefore only decides whether reconnection restarts it; omit it to wait indefinitely.^[43]

Zone rules rarely help a site. Upstream stops evictions only in a zone of at most 50 nodes with more than two, and at least 55%, unhealthy. A cut-off site labelled as its own topology.kubernetes.io/zone is entirely unreachable, so it is evicted at the normal rate unless every zone is down.^[44][45] One AWS page says otherwise;^[46] AWS’s failover guide agrees with upstream.^[36]

The trap is a reboot during the partition. The kubelet cannot ask the API server what to run, so after a power cut the site’s pods stay down until the link returns, and crictl cannot start them; only static pods restart.^[36] AWS observed all this with one configuration; test your own.

Example status: YAML syntax-checked; fragments, not applied. Pattern from the EKS Best Practices Guide.

# Stretched cluster: pin a site workload and decide when an outage may restart it
# kubelet flag at node registration:  --node-labels=topology.kubernetes.io/zone=store-0413
# (a zone per site helps topology spread and per-site reports; it does not stop evictions)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pos-sync
spec:
  replicas: 1
  selector:
    matchLabels: {app: pos-sync}
  template:
    metadata:
      labels: {app: pos-sync}
    spec:
      nodeSelector:
        topology.kubernetes.io/zone: store-0413    # never "fail over" to the cloud
      tolerations:
        - key: node.kubernetes.io/unreachable
          operator: Exists
          effect: NoExecute
          # outages under 30 min: no restart (default 300)
          tolerationSeconds: 1800
      containers:
        - name: app
          image: registry.example.com/store/pos-sync@sha256:<digest>
          imagePullPolicy: IfNotPresent

Default for user-edge sites: a cluster or host runtime per site, managed centrally; a stretched cluster only when sites may freeze during outages (decision framework).

Happy path. A release (application, OS image or cluster version) is signed, tested in a lab, then goes to a canary covering every model and site class, bakes, passes its gate and widens in waves. Amazon’s description of a typical pipeline is a useful model: “waves” of increasing size, baking at least one hour after each one-box stage and 12 hours after the first regional wave.^[21]

Exception path. A gate fails: stop promotion, revert the wave’s desired state and recover sites down the recovery ladder. Sites offline during the halt pull whatever desired state is current when they return, so a halt must change it, not only pause the tool. Gates fail in two ways. Fleet treats an offline cluster as NotReady “until it comes back online and successfully deploys the BundleDeployment”,^[47] so a NotReady gate halts healthy rollouts. Counting only reported failures is worse: bricked or boot-looping nodes never report. Exclude sites offline before the wave; count sites that received the change but have not checked in healthy by a deadline as suspected failures, judged against the class’s normal offline rate (Lab 1).

flowchart TB
  L["`Lab: every hardware model,
  disconnection test`"]:::neutral
  C["`Canary: a few sites,
  every model and class`"]:::core
  G1{"`Any canary failed,
  or silent after
  updating?`"}:::decision
  W1["`Wave 1: about 5%
  of sites, then bake`"]:::core
  G2{"`Failed + silent
  after update
  below limit?`"}:::decision
  W2["`Wave 2: about 25%,
  then bake`"]:::core
  G3{"`Still below
  the limit?`"}:::decision
  W3["`Wave 3: the rest,
  in batches`"]:::core
  S["`Sites offline earlier
  converge on return`"]:::neutral
  H["`Halt: revert state,
  recover sites`"]:::risk
  L --> C --> G1
  G1 -->|none after bake| W1 --> G2
  G1 -->|any| H
  G2 -->|yes| W2 --> G3
  G2 -->|no| H
  G3 -->|yes| W3 --> S
  G3 -->|no| H
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Playbook 1: zero-touch onboarding with FDO. FIDO Device Onboard “permits late binding of device credentials”, so one device can onboard to whichever owner bought it; version 2.0 extends it to “an arbitrary device”.^[20] Its ownership voucher is “extended with each owner while the device is offline”. The device opens every connection, and FDO then hands over to the management agent.^[20] Related standards are SZTP (RFC 8572) for network equipment, in NetworkOps,^[48] and BRSKI (RFC 8995), in IoTOps.^[49]

flowchart TB
  F["`1 Factory inserts
  credentials (DI);
  voucher created`"]:::neutral
  V["`2 Voucher reaches the
  owner; node still boxed`"]:::neutral
  R["`3 Owner service registers
  with rendezvous (TO0)`"]:::neutral
  P["`4 Staff plug it in; node
  asks rendezvous for
  its owner (TO1)`"]:::core
  T["`5 Trust established;
  secrets, config (TO2)`"]:::core
  A["`6 Fleet agent registers
  outbound; baseline applied`"]:::core
  Q{"`7 Acceptance
  check passes?`"}:::decision
  OK["`In service;
  added to its wave`"]:::core
  NO["`Quarantine: re-image
  remotely, else swap`"]:::risk
  F --> V --> R --> P --> T --> A --> Q
  Q -->|yes| OK
  Q -->|no| NO
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Playbook 2: updates that undo themselves. bootc upgrade “does not affect the running system; updates operate in an ‘A/B’ style by default”; bootc rollback queues the previous deployment, without /etc changes.^[22] With greenboot, a failing required.d check fails the boot, and after GREENBOOT_MAX_BOOT_ATTEMPTS the node returns to the previous deployment.^[50] Caution: the optional greenboot-default-health-checks package adds a required DNS check of repository URLs,^[50] which a WAN outage during the first boot after an update can fail; leave it out where the WAN may be down.

Playbook 3: partition-tolerant workloads. Give every remote dependency of the core job (identity, licences, configuration, images, payment endpoints, time, DNS) an offline behaviour: cached, queued, degraded or refused, so that losing them “does not compromise the static stability of your applications”.^[46] Queue data written offline with idempotency keys so a replay cannot apply it twice, reconcile conflicts after reconnecting, and keep a local time source: a drifting clock breaks certificate checks and ordering (money movement: FinTechOps). Prove it with a drill that includes a power cycle.

Playbook 4: telemetry that survives outages. The OpenTelemetry Collector’s exporter gives up on a batch after max_elapsed_time (300 seconds by default) and drops data when its queue is full; a persistent queue on disk survives restarts.^[32] Size it for your longest outage (reference implementation).

Playbook 5: decide what data leaves the site. LF Edge expects work such as “stripping PII” or “extracting information from streaming video” to run close to where the data originates.^[3] Per data stream, write down what the site filters or aggregates (events, not raw frames), how long raw data stays on local disk, and what never leaves (video of people, payment data: PrivacyOps). Size local storage for the longest outage; the backlog then drains only at the link’s spare capacity, so a day at 50 kB/s (4.32 GB) needs about two hours of a spare 5 Mbit/s.

None is mandatory for edge computing as such; contracts, sector rules or customers may make some binding.

As of October 2026.

Selection criteria, in order: behaviour during partitions (test it), footprint, upgrade path and version lag, onboarding support, dependence on one vendor’s control plane, and who answers the phone at 03:00.

Pulls, waves, health checks and rollbacks run without people; keep humans for promoting high-risk changes past the canary and for fleet-wide emergencies. AI anomaly detection can rank failing sites, but offline sites look anomalous, and no model should widen a wave without the gates a person faces. Inference at sites belongs to MLOps.

Illustrative design: 1,200 stores, each with one server running an image-based OS, single-node K3s and an OpenTelemetry Collector; a central Fleet manager and registry. One node means a hardware fault stops local compute until a swap; three servers triple the hardware for etcd quorum.^[34]

1. Site profile, one file per site; its labels drive waves, windows and reports.

Example status: YAML syntax-checked; an illustrative schema for your own tooling.

site: store-0413
region: eu-west
timezone: Europe/Dublin
hardware: {model: gen3-1u, nodes: 1, tpm: "2.0"}
connectivity:
  primary: broadband
  backup: lte
  out_of_band: cellular-console
labels:                      # copied onto the Fleet cluster object
  wave: wave2
  hw: gen3
  site-class: standard-store
maintenance_window: {days: [mon, tue, wed, thu], start: "02:00", end: "05:00"}
owners: {site_contact_role: store-manager, field_service: regional-it}

2. Application waves in Fleet. Each wave gets its own image tag (immutable in the registry, or use digests); promotion is a reviewed one-line commit. Fleet “does not add artificial delays during rollout” and its default maxUnavailable of 100% gates nothing,^[47] so the gate lives in the pipeline that writes this file; rolloutStrategy limits batch size and paused: true stops updates.^[59]

Example status: YAML syntax-checked; keys checked against the Fleet 0.16 fleet.yaml reference; not applied. Each customization repeats the full image map: deep-merging of per-target Helm values was not confirmed.

# fleet.yaml — in-store app bundle.
# Wave 3 has no customization and runs the root (baseline) values.
defaultNamespace: store-apps
helm:
  chart: ./chart
  values:
    image: {repository: registry.example.com/store/pos-sync, tag: "2026.09.4"}
targetCustomizations:
  - name: canary
    clusterSelector: {matchLabels: {wave: canary}}
    helm:
      values:
        image: {repository: registry.example.com/store/pos-sync, tag: "2026.10.2"}
  - name: wave1
    clusterSelector: {matchLabels: {wave: wave1}}
    helm:
      values:
        image: {repository: registry.example.com/store/pos-sync, tag: "2026.10.2"}
  - name: wave2                  # promoted after wave 1 baked
    clusterSelector: {matchLabels: {wave: wave2}}
    helm:
      values:
        image: {repository: registry.example.com/store/pos-sync, tag: "2026.10.2"}
rolloutStrategy:
  maxUnavailable: 10%            # offline sites count as NotReady: leave headroom
  maxUnavailablePartitions: 0
  autoPartitionSize: 10%         # partitions of about 120 sites
  maxNew: 50                     # default: at most 50 sites staged at once (pull storms)

3. Cluster upgrades inside each site, delivered by Fleet and versioned per wave the same way. Pin version: a channel upgrades “any time it resolves to a new release”,^[58] on a fleet every site at once. On 2026-10-18 the K3s stable channel resolved to v1.36.5+k3s1.^[60]

Example status: YAML syntax-checked; fields from the K3s automated-upgrade guide and the Plan reference;^[61] not applied.

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3s-server
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true                   # single node: no drain target, so only cordon
  nodeSelector:
    matchExpressions:
      - {key: node-role.kubernetes.io/control-plane, operator: In, values: ["true"]}
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  version: v1.36.5+k3s1          # one minor at a time; never skip
  window:
    days: [monday, tuesday, wednesday, thursday]
    startTime: "02:00"
    endTime: "05:00"
    timeZone: Europe/Dublin

4. A boot health check that looks only inside the site. If the local API or store service does not answer, the boot fails and greenboot eventually returns to the previous OS deployment.^[50]

Example status: bash -n syntax-checked; not executed (needs a bootc host with greenboot and K3s). Port and path are placeholders.

#!/usr/bin/env bash
# /etc/greenboot/check/required.d/50-site-ready.sh
# Local checks only: a WAN outage must never roll back a good image.
set -euo pipefail
for _ in $(seq 1 30); do
  k3s kubectl get --raw='/readyz' >/dev/null 2>&1 && break
  sleep 10
done
k3s kubectl get --raw='/readyz' >/dev/null
curl --fail --silent --max-time 5 http://127.0.0.1:30080/healthz >/dev/null

5. Telemetry buffer. Size it by calculation: at an illustrative 50 kB/s per signal, a day of outage is 50 kB/s × 86,400 s = 4.32 GB, so a 2 GiB cap covers about 11.9 hours. The cap is per queue file, with one file per exporter and signal,^[62][63] so metrics plus logs can use 4 GiB. Decide in advance what the site drops first, such as debug logs.

Example status: YAML syntax-checked; keys from the collector’s exporterhelper and file storage READMEs; not run. otlp is the deprecated alias of otlp_grpc, kept here for older collectors.^[64] max_size may not exist in older versions.^[62]

extensions:
  file_storage/wan:
    directory: /var/lib/otelcol/file_storage
    create_directory: true
    max_size: 2147483648        # 2 GiB
receivers:
  otlp:
    protocols:
      grpc: {endpoint: 127.0.0.1:4317}
exporters:
  otlp:
    endpoint: telemetry.example.com:4317
    sending_queue:
      storage: file_storage/wan
      queue_size: 20000          # batches on disk
    retry_on_failure:
      max_elapsed_time: 0        # default 300s: data dropped after five minutes of outage
service:
  extensions: [file_storage/wan]
  pipelines:
    metrics: {receivers: [otlp], exporters: [otlp]}
    logs: {receivers: [otlp], exporters: [otlp]}

6. Fleet report. Reporting convergence (healthy ÷ recently checked-in sites) flatters a fleet with many silent sites; true convergence does not. The wave gate counts only sites that pulled the change (pulled_at).

Example status: JSON syntax-checked; this exact file was used for the executed run below.

{
  "target_version": "2026.10.2",
  "generated_at": "2026-10-18T06:00:00Z",
  "sites": [
    {"site": "store-0001", "wave": "canary", "version": "2026.10.2", "health": "ok",
     "last_checkin": "2026-10-18T05:58:10Z", "pulled_at": "2026-10-14T02:00:05Z"},
    {"site": "store-0002", "wave": "canary", "version": "2026.10.2", "health": "ok",
     "last_checkin": "2026-10-18T05:59:02Z", "pulled_at": "2026-10-14T02:00:40Z"},
    {"site": "store-0107", "wave": "wave1", "version": "2026.10.2", "health": "ok",
     "last_checkin": "2026-10-18T05:57:44Z", "pulled_at": "2026-10-16T02:01:10Z"},
    {"site": "store-0108", "wave": "wave1", "version": "2026.10.2", "health": "degraded",
     "last_checkin": "2026-10-18T05:56:30Z", "pulled_at": "2026-10-16T02:03:20Z"},
    {"site": "store-0109", "wave": "wave1", "version": "2026.09.4", "health": "ok",
     "last_checkin": "2026-10-15T21:12:00Z", "pulled_at": null},
    {"site": "store-0412", "wave": "wave2", "version": "2026.09.4", "health": "updating",
     "last_checkin": "2026-10-18T04:06:00Z", "pulled_at": "2026-10-18T04:05:30Z"},
    {"site": "store-0413", "wave": "wave2", "version": "2026.08.1", "health": "ok",
     "last_checkin": "2026-09-30T10:00:00Z", "pulled_at": null},
    {"site": "store-0414", "wave": "wave2", "version": "2026.10.2", "health": "failed",
     "last_checkin": "2026-10-18T05:50:00Z", "pulled_at": "2026-10-18T04:02:00Z"},
    {"site": "store-0415", "wave": "wave2", "version": "2026.10.2", "health": "ok",
     "last_checkin": "2026-10-18T05:59:40Z", "pulled_at": "2026-10-18T04:04:10Z"}
  ]
}

Example status: executed on 2026-10-18 with Python 3.8.10 against the JSON above; output below is verbatim.

"""Summarise edge-fleet check-ins: convergence, wave gates, stragglers.

Usage: python3 fleet_report.py fleet-status.json [stale_hours] [silent_minutes]
Standard library only (Python 3.8+).
"""
import json
import sys
from collections import Counter, defaultdict
from datetime import datetime, timedelta


def parse(ts):
    return datetime.strptime(ts, "%Y-%m-%dT%H:%M:%SZ")


def age(delta):
    return "%dd %dh %02dm" % (delta.days, delta.seconds // 3600,
                              delta.seconds % 3600 // 60)


def main(path, stale_hours=24, silent_minutes=60):
    with open(path) as f:
        data = json.load(f)
    now = parse(data["generated_at"])
    target = data["target_version"]
    sites = data["sites"]
    fresh = [s for s in sites
             if now - parse(s["last_checkin"]) <= timedelta(hours=stale_hours)]
    healthy = [s for s in sites if s["version"] == target and s["health"] == "ok"]

    print("sites in scope:           %d" % len(sites))
    print("reporting (<= %dh):       %d" % (stale_hours, len(fresh)))
    print("true convergence:         %.1f%% (healthy on %s / in scope)"
          % (100.0 * len(healthy) / len(sites), target))
    print("reporting convergence:    %.1f%% (healthy on target / reporting)"
          % (100.0 * len([s for s in healthy if s in fresh]) / len(fresh)))
    print("version spread:           %s" % dict(Counter(s["version"] for s in sites)))

    # Wave gate. Sites that never pulled the change are unknown and excluded.
    # A site that pulled it, then stayed silent past the deadline, is suspect.
    rows, suspects = defaultdict(Counter), []
    for s in sites:
        if s["pulled_at"] is None:
            continue
        silence = now - parse(s["last_checkin"])
        if s["version"] == target and s["health"] == "ok":
            state = "healthy"
        elif s["health"] in ("failed", "degraded"):
            state = s["health"]
        elif silence > timedelta(minutes=silent_minutes):
            state = "silent"
        else:
            state = "updating"
        rows[s["wave"]]["received"] += 1
        rows[s["wave"]][state] += 1
        if state in ("failed", "silent"):
            suspects.append((s, state, silence))
    print("wave    received healthy degraded updating failed silent  suspect")
    for wave, r in sorted(rows.items()):
        print("%-7s %8d %7d %8d %8d %6d %6d  %d/%d"
              % (wave, r["received"], r["healthy"], r["degraded"], r["updating"],
                 r["failed"], r["silent"], r["failed"] + r["silent"], r["received"]))
    print("suspected failures (pulled the change, then failed or went silent):")
    for s, state, silence in suspects:
        print("  %s  %-6s  last check-in %s ago" % (s["site"], state, age(silence)))
    print("offline before their wave (unknown, not counted), oldest first:")
    for s in sorted((s for s in sites if s["pulled_at"] is None and s not in fresh),
                    key=lambda s: s["last_checkin"]):
        print("  %s  %s  silent for %s"
              % (s["site"], s["version"], age(now - parse(s["last_checkin"]))))


if __name__ == "__main__":
    args = [int(a) for a in sys.argv[2:4]]
    main(sys.argv[1], *args)

sites in scope:           9
reporting (<= 24h):       7
true convergence:         44.4% (healthy on 2026.10.2 / in scope)
reporting convergence:    57.1% (healthy on target / reporting)
version spread:           {'2026.10.2': 6, '2026.09.4': 2, '2026.08.1': 1}
wave    received healthy degraded updating failed silent  suspect
canary         2       2        0        0      0      0  0/2
wave1          2       1        1        0      0      0  0/2
wave2          3       1        0        0      1      1  2/3
suspected failures (pulled the change, then failed or went silent):
  store-0412  silent  last check-in 0d 1h 54m ago
  store-0414  failed  last check-in 0d 0h 10m ago
offline before their wave (unknown, not counted), oldest first:
  store-0413  2026.08.1  silent for 17d 20h 00m
  store-0109  2026.09.4  silent for 2d 8h 48m

Read it like an operator: in wave 2 one site reported a failure and one has been silent for almost two hours since it started updating, which is how a boot loop looks from the centre. Revert wave 2’s tag; do not promote wave 3. The 24-hour freshness view still counts store-0412 as reporting; only the post-update deadline catches it.

Lab 1 was executed. Labs 2 and 3 need disposable virtual machines and were not executed; their expected output comes from the documentation cited inline.

• Objective: compare three wave gates on a fleet that is never fully online, and see why canaries must be stratified (IoTOps Lab 3 goes further).
• Prerequisites: Python 3.8+; no network, cost or risk. About 20 minutes.

1. Save the script as waves.py and run python3 waves.py. Its “bad C” release bricks every model C node it reaches; those nodes never report.
2. Change DAILY_OFFLINE to 0.01 and 0.08, and the canary size from 12 to 30; rerun each time.

Example status: executed on 2026-10-18 with Python 3.8.10; output below is verbatim.

"""Simulate a staged rollout to an edge fleet where some sites are offline.

Standard library only. Deterministic: change SEED to see other runs.
"""
import random

SEED = 7
SITES = 1200
MODELS = {"A": 0.70, "B": 0.20, "C": 0.10}   # hardware mix
DAILY_OFFLINE = 0.03       # chance a site is offline on a given day
CLOSED_SHARE = 0.01        # sites closed for the whole period (seasonal)
DROP_AFTER_UPDATE = DAILY_OFFLINE / 24   # unrelated drop-off within the hour
WAVES = [("canary", 12, 2), ("wave1", 60, 2), ("wave2", 300, 1), ("wave3", None, 0)]
# (name, size or None for "the rest", bake days before the next wave opens)
RULES = {"notready": "NotReady <= 2 % of the wave",
         "reported": "failures reported by sites",
         "two-part": "reported + silent after update"}


def build_fleet(rng):
    fleet = []
    for i in range(SITES):
        r, acc, model = rng.random(), 0.0, "A"
        for name, share in MODELS.items():
            acc += share
            if r < acc:
                model = name
                break
        fleet.append({"id": i, "model": model,
                      "closed": rng.random() < CLOSED_SHARE,
                      "updated_day": None})
    return fleet


def assign_waves(fleet, rng, stratified):
    order = fleet[:]
    rng.shuffle(order)
    if stratified:  # put one site of every model first, then the rest
        firsts = []
        for m in MODELS:
            site = next(s for s in order if s["model"] == m and not s["closed"])
            firsts.append(site)
        order = firsts + [s for s in order if s not in firsts]
    waves, start = {}, 0
    for name, size, _ in WAVES:
        end = len(order) if size is None else start + size
        waves[name] = order[start:end]
        start = end
    return waves


def rollout(fleet, waves, rng, days=21):
    opened, day_open, history = [], 0, []
    schedule = {}
    for name, _, bake in WAVES:
        schedule[name] = day_open
        day_open += bake
    for day in range(days):
        for name, sites in waves.items():
            if schedule[name] == day:
                opened.extend(sites)
        for s in opened:
            online = not s["closed"] and rng.random() >= DAILY_OFFLINE
            if s["updated_day"] is None and online:
                s["updated_day"] = day
        done = sum(1 for s in fleet if s["updated_day"] is not None)
        history.append((day, done))
    return history


def canary_catches_model_c(rng, stratified, trials=2000):
    hits = 0
    for _ in range(trials):
        fleet = build_fleet(rng)
        waves = assign_waves(fleet, rng, stratified)
        if any(s["model"] == "C" for s in waves["canary"]):
            hits += 1
    return hits / trials


def halts(rule, wave, size, received, silent):
    """Gate for one wave. A node that is bricked or boot-looping never reports."""
    reported_failures = 0
    if rule == "notready":   # counts every site that is not Ready, for any reason
        return (size - received) + silent > int(0.02 * size)
    if rule == "reported":   # failures among reporting sites only
        return reported_failures > 0
    # two-part rule: ignore sites offline before the wave, count silence after it
    limit = 0 if wave == "canary" else max(1, int(0.01 * received))
    return reported_failures + silent > limit


def release(waves, opening, rule, bad_model, rng):
    """Open waves in order until a gate halts; return (where, sites broken)."""
    broken_total = 0
    for name, sites in waves.items():
        received = [s for s in sites if s["updated_day"] == opening[name]]
        broken = [s for s in received if s["model"] == bad_model]
        dropped = sum(1 for s in received
                      if s["model"] != bad_model and rng.random() < DROP_AFTER_UPDATE)
        broken_total += len(broken)
        if halts(rule, name, len(sites), len(received), len(broken) + dropped):
            return name, broken_total
    return "none", broken_total


def main():
    rng = random.Random(SEED)
    fleet = build_fleet(rng)
    waves = assign_waves(fleet, rng, stratified=True)
    history = rollout(fleet, waves, rng)
    print("day  sites_updated  convergence")
    for day, done in history:
        if day in (0, 2, 4, 5, 6, 7, 10, 14, 20):
            print("%3d  %13d  %10.1f%%" % (day, done, 100.0 * done / SITES))
    opening = {}
    day_open = 0
    for name, _, bake in WAVES:
        opening[name] = day_open
        day_open += bake
    print("wave    sites  offline on opening day")
    for name, sites in waves.items():
        late = sum(1 for s in sites if s["updated_day"] != opening[name])
        print("%-6s %6d  %22d" % (name, len(sites), late))
    stragglers = [s for s in fleet if s["updated_day"] is None]
    print("never updated after 21 days: %d sites (all closed: %s)"
          % (len(stragglers), all(s["closed"] for s in stragglers)))
    print("release  gate rule                       halts at  model C sites broken")
    for label, bad in (("good", None), ("bad C", "C")):
        for rule, text in RULES.items():
            where, broken = release(waves, opening, rule, bad, random.Random(SEED + 2))
            print("%-8s %-31s %-9s %d" % (label, text, where, broken))
    for strat in (False, True):
        p = canary_catches_model_c(random.Random(SEED + 1), strat)
        label = "stratified" if strat else "random    "
        print("canary of 12, %s: contains a model C site in %.1f%% of runs"
              % (label, 100 * p))


if __name__ == "__main__":
    main()

• Expected output (Python 3.8.10, unchanged parameters):

day  sites_updated  convergence
  0             12         1.0%
  2             71         5.9%
  4            360        30.0%
  5           1169        97.4%
  6           1193        99.4%
  7           1193        99.4%
 10           1193        99.4%
 14           1193        99.4%
 20           1193        99.4%
wave    sites  offline on opening day
canary     12                       0
wave1      60                       1
wave2     300                      12
wave3     828                      29
never updated after 21 days: 7 sites (all closed: True)
release  gate rule                       halts at  model C sites broken
good     NotReady <= 2 % of the wave     wave2     0
good     failures reported by sites      none      0
good     reported + silent after update  none      0
bad C    NotReady <= 2 % of the wave     canary    1
bad C    failures reported by sites      none      103
bad C    reported + silent after update  canary    1
canary of 12, random    : contains a model C site in 71.4% of runs
canary of 12, stratified: contains a model C site in 100.0% of runs

• Validation: nothing fails in the good release, yet a 2% NotReady gate halts it at wave 2 (12 sites offline, limit 6). Counting only reported failures lets the bad release through every wave, breaking 103 sites; the two-part rule passes the good release and stops the bad one at the canary. The random canary sits near 1 − 0.9¹² = 71.8%; the seven never-updated sites are the closed ones.
• Troubleshooting: StopIteration means a model has no open site (tiny fleets only); lower CLOSED_SHARE.
• Teardown: delete waves.py.

• Objective: upgrade a site cluster declaratively, one minor version, only inside its maintenance window.
• Prerequisites: a disposable Ubuntu VM with 2 vCPU and 2 GB RAM (the K3s server minimum),^[24] outbound internet, sudo; about 45 minutes; no cost beyond the VM. Do not run on a machine you care about.

Example status: bash -n syntax-checked; not executed. Versions are the K3s releases of 2026-09-30.^[60]

# 1. Install a pinned, older K3s (as a site would be after a year in the field)
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.9+k3s1 sh -
sudo k3s kubectl get nodes            # expect one Ready node, VERSION v1.35.9+k3s1
# 2. Install the system-upgrade-controller at a pinned release
SUC=https://github.com/rancher/system-upgrade-controller/releases/download/v0.20.2
sudo k3s kubectl apply -f "$SUC/crd.yaml" -f "$SUC/system-upgrade-controller.yaml"
# 3. Label the node like a canary site
sudo k3s kubectl label node "$(hostname)" wave=canary

4. Save the reference implementation’s Plan as plan.yaml, add wave: canary under a new matchLabels key of its nodeSelector, set your timeZone and a window that already ended today, and apply it.
5. Run sudo k3s kubectl -n system-upgrade get plans,jobs. Expected: the plan resolves its version but no job appears, because jobs “will not be created outside the time window”.^[58]
6. Move the window to include the current time and apply again. Expected: a job runs, the node is cordoned, K3s restarts, and sudo k3s kubectl get nodes shows v1.36.5+k3s1.

• Validation: sudo k3s kubectl -n system-upgrade get plan k3s-server -o jsonpath='{.status.latestVersion}' prints the pinned version, and the node is schedulable again.
• Troubleshooting: a failed job with the node left cordoned (unschedulable) usually means a downgrade, which the upgrade image refuses.^[58] Skipping a minor is not blocked and may break the cluster, so check the version first; then fix it, or delete the plan and uncordon the node.
• Teardown: /usr/local/bin/k3s-uninstall.sh,^[65] then delete the VM.

• Objective: observe a stretched cluster when a site loses the WAN, and when the site also reboots.
• Prerequisites: two disposable VMs on one network, central (K3s server) and site (K3s agent, joined with K3S_URL and the token in /var/lib/rancher/k3s/server/node-token; agents connect outbound to the server);^[66] about 90 minutes. Lab only: the firewall rules below cut traffic.

Example status: YAML and bash -n syntax-checked; not executed. Expected behaviour is documented for upstream Kubernetes and EKS, not verified on K3s for this page.

# site-app.yaml — pinned to the site node, as a store service would be
apiVersion: apps/v1
kind: Deployment
metadata:
  name: site-app
spec:
  replicas: 1
  selector:
    matchLabels: {app: site-app}
  template:
    metadata:
      labels: {app: site-app}
    spec:
      nodeSelector: {kubernetes.io/hostname: site}
      containers:
        - name: web
          image: nginx:1.27
          ports: [{containerPort: 80, hostPort: 8080}]

# On central: deploy, then cut the site off
# (the rules live on central, so they survive a site reboot)
sudo k3s kubectl apply -f site-app.yaml
SITE_IP=192.0.2.20                      # replace with the site VM's address
sudo iptables -I INPUT -s "$SITE_IP" -j DROP
sudo iptables -I OUTPUT -d "$SITE_IP" -j DROP
watch -n 10 sudo k3s kubectl get nodes,pods -o wide
# On site: the container keeps serving locally
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/
sudo k3s crictl ps --name web
# Later, on site: simulate the power cut, then check again after boot
sudo systemctl reboot
# Finally, on central: heal the partition
sudo iptables -D INPUT -s "$SITE_IP" -j DROP
sudo iptables -D OUTPUT -d "$SITE_IP" -j DROP

• Expected output: within about a minute site shows NotReady;^[42] the local curl still returns 200. About five minutes later central shows the pod Terminating and a Pending replacement while the container keeps running on the site.^[43] After the reboot, expect no running container until the partition heals.^[36]
• Validation: after healing, site returns to Ready and exactly one site-app pod runs.
• Variation: make site a single-node K3s server with its own control plane and repeat the cut: the app keeps running and restarts after the reboot. That difference is the control-plane decision.
• Troubleshooting: if nothing changes, check the rules were inserted on central (sudo iptables -L INPUT -n | head).
• Teardown: remove the rules, run /usr/local/bin/k3s-agent-uninstall.sh on site and /usr/local/bin/k3s-uninstall.sh on central,^[65] then delete both VMs.

1. Restaurant and store fleets (documented, 2018 snapshots). Chick-fil-A ran “tens of containers over thousands of physically distributed clusters”;^[17] its slides ask for nodes “Simple enough for a non-technologist to install” that can be re-imaged remotely; its in-house deployment system, also called Fleet, is unrelated to Rancher’s.^[67] Target made each store cluster “its own standalone mini-cloud” and let teams choose groups of stores to deploy to first.^[18] Lesson: cluster per site, central desired state and staged groups of sites emerged early, independently. Pitfall: this scale “breaks” some of the native tooling, says Chick-fil-A’s abstract;^[17] both firms built their own deployment systems. Success metric: releases reach every store without visits.

2. Line-side inference in a plant (illustrative). Cameras feed a defect-detection model on a gateway in the plant’s operational technology (OT) network. Method: a three-node site cluster, because the line cannot stop for a hardware fault; waves timed to shift changes; a one-way data flow agreed with OT security under ISA/IEC 62443.^[54] Pitfall: the OT firewall may block the agent’s outbound pull; agree that path before the first wave, and keep raw video on site (Playbook 5). Success metric: no line stops caused by site compute.

3. Ships, mines and remote depots (illustrative). Links are metered and sometimes absent for days. Method: pre-stage images in idle periods; use site caches and K3s’s peer-to-peer mirror;^[68] make waves long; size telemetry buffers for the longest outage. Pitfall: a site back after three weeks may need intermediate versions. Success metric: days from reconnection to the current release, within the link budget.

Worked example 1: canary size. If a defect hits only hardware model C (10% of sites), a random canary of n sites includes a model C site with probability 1 − 0.9ⁿ: 71.8% for n = 12, as Lab 1 reproduced. Reaching 95% needs n ≥ ln 0.05 ÷ ln 0.9 ≈ 28.4, so 29 sites; a stratified canary with one site per model reaches 100% with three.

Worked example 2: latency. ITU-T planning values give 5 µs per km of optical fibre.^[69] A data centre 1,000 km away costs 10 ms per round trip before routing, queueing or processing: too far for a 10 ms control loop, fine for a dashboard.

• Physical access is the threat model. A disk key sealed in the TPM to “some combination of PCRs 7, 11, and 14 (if shim/MOK is used)”^[70] protects a disk taken out of the box, but a box stolen whole boots the same signed chain and unlocks itself (a PIN is off by default),^[70] leaving the running system to attack. Network-bound unlock (Clevis and Tang) opens the disk only on the site network,^[71] but its Tang server must stay reachable during WAN outages, so it usually sits on site and can be stolen too; TPM plus PIN needs staff at every boot. Always lock the console and boot order, keep site credentials remotely revocable, and plan firmware protection and recovery (NIST SP 800-193).^[53]
• Treat the out-of-band path as a master key: per-site credentials, multi-factor sign-in, recorded sessions and a central switch that disables it.
• Prove a node’s state before trusting it with remote attestation.^[51]
• Secure the update path. The Update Framework defends against rollback, indefinite-freeze and mix-and-match attacks,^[52] all plausible against sites offline for long periods.
• Plan identity for the outage. On EKS local clusters, IAM “isn’t available during network disconnects”, so AWS documents “an alternative authentication mechanism using x.509 certificates”.^[39] Treat such credentials as break-glass: per site, rotated after each use.
• Mind privileged machinery. The K3s upgrade job runs with host namespaces, CAP_SYS_BOOT and the host root mounted read-write;^[58] whoever can write a Plan controls the node. Peer-to-peer image sharing assumes equal trust between nodes.^[68]
• Compliance. Keeping data on site may simplify data-transfer questions, but rules depend on jurisdiction (PrivacyOps, ComplianceOps). OT security has no Atlas page; start from SecOps.

Costs multiply by sites, so price one site class per year with the worksheet below. Pricing models differ: Azure Local is “priced per physical core”,^[41] while EC2 instances are “included in the price of AWS Outposts”.^[39] Bandwidth (illustrative): 300 MB of new image layers to 1,200 sites is 360 GB per release; site caches pull once, and Fleet’s partitions and maxNew cap simultaneous pulls.^[47] Capacity: choose one node or N+1 per site class from the cost of a site outage, not habit. Hardware lifetime and embodied carbon belong to GreenOps.

Template status: illustrative; use your own figures.

PER-SITE COST WORKSHEET (one site class, per year)
Hardware      price per node ____ x nodes ____ / service life in years ____   = ____
Spares        spare units ____ x unit price ____ / sites covered ____ / life ____ = ____
Visits        visits per site per year ____ x cost per visit ____             = ____
Links         primary ____ + backup ____ + out-of-band ____                   = ____
Licences      per node or per core ____ x count ____                          = ____
Energy        average watts ____ x 8.76 = kWh ____ x price per kWh ____       = ____
Releases      MB per release ____ x releases ____ x price per GB, if metered  = ____
Total per site per year                                                       = ____
Payback       visits avoided by remote recovery ____ x cost per visit ____    = ____

Check automatically the leading indicators (freshness, version spread, expired exceptions, certificate renewals) and the lagging ones (fleet-level incidents, visits, halted waves); test the design with scheduled drills.

Illustrative scenario. An OS image reaches wave 1 (60 sites). Within an hour 9 sites stop checking in, and a store reports “the till screen keeps restarting”.

1. Stop the spread (minutes 0–10). Revert wave 1’s image version in the fleet repository so reconnecting sites pull the old desired state; declare an incident (IncidentOps).
2. Treat the 9 as failed until proven otherwise: they received the image, then went silent. Compare them with sites routinely offline at this hour, and look for a shared model, image or class.
3. Climb the recovery ladder (figure below), recording which rung worked where.
4. Verify check-in, the previous image and application health at each site.
5. Prevent: add the missing model to the canary and the failing check to the lab; write the post-incident review.

flowchart TB
  A["1 Automatic: failed health checks boot the previous image"]:::core
  B["2 Remote: management agent rolls back or re-images"]:::core
  C["3 Out-of-band: console or cellular path, boot menu"]:::neutral
  D["4 Guided hands: staff swap in a pre-imaged spare"]:::neutral
  E["5 Visit: technician repairs or replaces"]:::risk
  A -->|still failing| B
  B -->|agent unreachable| C
  C -->|not recoverable remotely| D
  D -.->|spare fails or none on site| E
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Put compute at sites when work must continue through WAN outages, needs answers faster than distance allows, produces more data than the link carries, or must stay on site for legal reasons. Do not when a thin client to a central service would do, outages are tolerable, or nobody will fund the hidden duties: spares, onboarding logistics, waves, drills, physical security and refresh.

%%{init: {"flowchart": {"nodeSpacing": 14, "rankSpacing": 30, "padding": 8}}}%%
flowchart TB
  Q1{"`Must it keep working
  through WAN outages,
  or need low latency
  or local data?`"}:::decision
  N1["`Run it centrally:
  CloudOps`"]:::neutral
  Q2{"`Is it a single-purpose
  or microcontroller
  device?`"}:::decision
  Q3{"`Must the site
  restart or change
  workloads while
  disconnected?`"}:::decision
  N2["`Device patterns:
  IoTOps`"]:::neutral
  S1["`Cluster or runtime
  per site, managed
  centrally`"]:::core
  S2["`Stretched cluster
  acceptable; drill
  power cuts in outages`"]:::core
  Q4{"`Can you staff
  hardware lifecycle
  and spares?`"}:::decision
  O1["`Own the hardware:
  stock spares, agree
  swap times`"]:::core
  M1["`Provider-owned
  hardware, such as
  Outposts: needs a
  reliable link`"]:::risk
  Q1 -->|no| N1
  Q1 -->|yes| Q2
  Q2 -->|yes| N2
  Q2 -->|no| Q3
  Q3 -->|yes| S1
  Q3 -->|no| S2
  S1 --> Q4
  S2 --> Q4
  Q4 -->|yes| O1
  Q4 -->|no| M1
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

To compare options, score each 1–5 on autonomy during outages (35%), effort per site (25%), node security (20%) and cost per site (20%), this Atlas’s weights, and record the result in the decision record.

This Atlas’s model, not an industry standard.

• Days 1–30: see the fleet. Inventory sites, hardware, versions, links and owners; write site profiles; measure freshness and true convergence. Quick win: an expiry report for certificates and credentials.
• Days 31–60: make change safe. Pin versions; define waves and a stratified canary; move one application to pull-based delivery; write the dependency inventory for one site class. Milestone: one release shipped in waves with recorded gates.
• Days 61–90: prove autonomy. Drill with a power cycle on a lab site, then a real one in a quiet window; buffer telemetry; draft the recovery ladder; stock spares. Milestone: the first site class passes its drill.

Before a node ships

• Image recorded; TPM enrolled; disk unlock policy chosen (TPM only, network-bound or PIN); console and boot order locked
• FDO voucher (or vendor equivalent) delivered to the owner service
• Site profile committed with wave, hardware and maintenance window

Before each release wave

• Canary covers every hardware model and architecture, image, site class and link type
• Halt rule written: failures plus silence after the change; sites offline before it excluded
• Previous version still available; rollback tested on each model
• Batch size limits registry load

Disconnection drill (per site class)

• Core job works with the WAN cut (or has a documented fallback)
• Power cycle during the outage: services return without the WAN
• Break-glass login works locally
• Telemetry from the outage arrives after reconnection

Monthly review

• Stragglers older than seven days have a plan
• Every node restarted inside its certificate renewal window; no certificate under 30 days
• Exceptions reviewed; expired ones closed

Template status: illustrative; adapt names and numbers.

WAVE PLAN
Release:                         Change type: app | OS image | cluster | config
Canary sites (one per model/class/link):
Wave sizes and bake times:  canary __ (bake __h)  wave1 __% (__h)  wave2 __% (__h)  rest
Halt rule: failed + silent __ min after receiving > __ per wave (offline before: excluded)
Rollback: previous version ____  method ____  tested on models ____
Owner on call:                   Communication channel:

PARTITION-TOLERANCE WORKSHEET (one row per remote dependency)
Offline behaviour: cache | queue | degrade | refuse
Dependency  Core job?  Offline behaviour  Max outage covered  Tested on (date)
Identity    ____       ____               ____ h              ____
Licences    ____       ____               ____ h              ____
Images      ____       ____               ____ h              ____
Time, DNS   ____       ____               ____ h              ____

ARCHITECTURE DECISION RECORD: CONTROL-PLANE PLACEMENT
Context: site classes, link quality, outage history, staff on site
Options: cluster per site | stretched | stretched with autonomy | provider-owned hardware
Scores (autonomy 35%, effort 25%, security 20%, cost 20%):
Drill result (WAN cut plus power cycle):
Decision, consequences, review date:

Comparisons sit where they are used: neighbours, edge, fog and cloud, control-plane patterns.

Is EdgeOps a job title or a certification? Not a standard one; the label is used mostly by vendors.

Do we need Kubernetes at the edge? No. A container runtime with a pull agent, or virtual machines on an edge OS, can be enough; Kubernetes pays off when sites run several services and the team already operates it (KubernetesOps).

How is EdgeOps different from IoTOps? By device class, an editorial line both pages share: general-purpose nodes that host workloads here; microcontroller and single-purpose embedded devices in IoTOps.

Can a site skip versions after a long outage? Not Kubernetes minors,^[58] so keep intermediate images for returning sites.

Are spares worth stocking? Usually: a pre-imaged spare that staff can swap is the cheapest rung below remote recovery.

• Upstream partition behaviour is unchanged (October 2026): the pull request and issue that AWS cites were “closed without changes being merged”,^[36] in September 2025.^[73] Design for today’s behaviour.
• Attestation-gated onboarding: FDO’s late binding^[20] plus remote attestation,^[51] so secrets reach only nodes in a known state; the pieces are standard, the integration product-specific.

• Prerequisites: Linux, networking, containers; KubernetesOps for cluster mechanics.
• One day: the LF Edge taxonomy and CNCF edge-native principles;^[3][1] Lab 1.
• One week: Lab 2; the K3s certificate and upgrade guides; Fleet’s rollout page.^[47]
• Thirty days: Lab 3; an image-based OS with a health check; a partition-tolerance worksheet for a real workload.
• Specialise and show: onboarding and attestation, fleet delivery, OT or telecom MEC; a wave simulator tuned to your fleet, a drill report, a recovery ladder with measured times per rung.

Example status: commands from the K3s and Kubernetes documentation; not executed here.

k3s certificate check --output table                 # expiry of node certificates
kubectl -n system-upgrade get plans,jobs             # upgrade progress per node
kubectl get nodes -L wave,topology.kubernetes.io/zone # waves and site zones
kubectl get events -A --field-selector reason=CertificateExpirationWarning
crictl ps                                            # on the node; API unreachable
bootc status                                         # booted, staged, rollback images

Situation (illustrative). A remotely exploitable flaw in the store gateway’s web component is published, with reports of active exploitation. The fixed image is ready; normal waves take about five days (Lab 1).

Reasoning. A looks responsible and is the trap: it turns a security risk at some sites into an availability risk at all of them. The 2024 CrowdStrike outage (a content update, not a patch) was a fleet-wide change whose recovery needed someone to “insert the USB key into an affected device”,^[74] and CrowdStrike’s remedy was “an updated deployment ring process”.^[75] Default: C when a reversible mitigation exists, then B; A only if exploitation is under way at your sites and the change cannot affect boot. Uncertainty: whether attackers can reach your sites, and whether the fix touches boot; settle both in the first hour.

You can now explain the label’s ambiguity, place a site’s control plane, design waves and gates for a partly offline fleet, size a canary, plan onboarding and recovery without visits, and measure a fleet honestly.

1. A rollout halts at wave 2 with 3.5% of sites NotReady and no failures reported. What happened, and what do you change?

2. A team proposes one stretched cluster for 400 stores because “there is only one control plane to upgrade”. What test settles it?

3. Model C is 5% of sites. How large must a random canary be to include one with 90% probability, and what is the better design?

4. A K3s node ran 13 months without a restart and now fails to authenticate. Why, and what prevents it fleet-wide?

5. Your company calls its store-platform team “EdgeOps”. What should a new joiner know about the term?

Under a vendor-flavoured label sits a demanding practice: running many small data centres that are often alone and never staffed. Three habits carry most of it. Design each site to keep working when the link is gone, and test that with a power cut, not a slide. Move every change in stratified waves whose gates know which silences mean failure. Build the recovery ladder so a visit is the last rung. Tools change; these habits keep technicians at home.

Re-check before reuse: the K3s channel and system-upgrade-controller release (Lab 2); Fleet’s minor version (rollout defaults); KubeEdge’s vendored Kubernetes; the two AWS pages that disagree on zone evictions; the Collector’s otlp alias. Code links are pinned to the tag or commit read.

Changelog: 2026-10-18 — first version (research, ledger, draft, diagrams, labs). 2026-10-18 — r1 revision after the independent review: offline-aware gates (labs re-run), eviction mechanics, provider- versus customer-owned edge, theft model, IoTOps boundary. 2026-10-18 — r2 revision after the final score: local data, edge/fog/cloud terms, multi-architecture images, cost worksheet, phone-width figures, a wave-2 gate, pinned code links, fewer restatements. 2026-10-18, post-score polish: figures 2, 5 and 6; origins before 2018; a metric source; code wrapped.