RoboOps

RoboOps is the operations discipline for fleets of autonomous robots (autonomous mobile robots, or AMRs, in warehouses and hospitals; inspection and delivery robots; robot arms in cells) after the demo works: watching every robot, handling the moments when one cannot continue on its own, changing software, maps and settings safely across sites, and making each failure rarer. Unlike servers, the system moves in a space shared with people: a bad release can cause a collision, not just an outage.

The community behind the term puts its core idea in one sentence: “the key to operating robots at scale is to follow the lifecycle of failures”.^[1] Robots will meet autonomy exceptions, situations “where the robot cannot continue operating safely nor efficiently”; operations means expecting them, resolving them fast and making them rarer.^[1]

For fleet operators, site managers, robotics engineers who ship to robots in the field, and architects choosing fleet software. Beginners can stop after the glossary; engineers should do the labs; leaders should read the decision framework and the judgement call.

A five-robot pilot is run by the engineers who built it: they know each robot’s quirks, update by hand and walk over when one gets stuck. The Robot Operations Group describes what scale does: 1–5 robots is the prototype phase, 5–50 a limited field deployment, and between 50 and 5,000 “most of what worked before stops working”.^[1]

Physical consequences make this more than an efficiency problem: Ocado said a 2021 fire at its Erith warehouse “appears to have been caused by the collision of three bots on the grid”.^[7] Robot fleets fail like distributed systems and like machines.

• Not a standard. The standards it relies on are under methods, standards and frameworks.
• Not ROS. The Robot Operating System (ROS) is open-source robot middleware; many commercial robots do not expose it.
• Not robotic process automation (RPA). Software “bots” that click through business applications are operated by an automation team (AutomationOps); this page is about physical machines.
• Not safety engineering. Designing and validating safety functions (protective stops, speed and separation monitoring) is the job of the robot manufacturer and the integrator under the applicable standards. RoboOps keeps those functions intact and evidenced in daily operation.
• Editorial distinction: this Atlas separates RoboOps from EdgeOps and IoTOps by autonomy, motion and safety: if the device can move itself or a load near people, the questions on this page apply. No standard defines that boundary.

The label’s centre of gravity is commercial: ROG was co-founded by InOrbit’s CEO, InOrbit sponsors it, sells RobOps software and hosts the 2026 conference.^[6][8][13] Treat vendor definitions as positions, not consensus.

Mental model: three loops at three speeds. The safety loop runs in milliseconds inside the robot; you never bypass it. The exception loop runs in minutes: detect, assist, resume or recover. The improvement loop runs in weeks: record, analyse, fix, release, check the rate fell. Many failures come from mixing loops, such as overriding a protective stop for convenience or rushing a weekly fix out as an emergency patch.

flowchart TB
  E["`EdgeOps
  remote sites, disconnection`"]:::neutral
  I["`IoTOps
  constrained devices, firmware`"]:::neutral
  M["`MLOps
  perception models`"]:::neutral
  R["`RoboOps
  autonomy exceptions, safe updates,
  fleet telemetry, remote assistance`"]:::core
  O["`ObservabilityOps
  telemetry pipelines`"]:::neutral
  N["`IncidentOps
  incident command`"]:::neutral
  F["`FleetOps
  vehicle fleets`"]:::neutral
  S["`SupplyChainOps
  flow of goods`"]:::neutral
  E -->|"rollout and offline patterns"| R
  I -->|"device identity, OTA"| R
  M -->|"models to run"| R
  R -->|"hands telemetry to"| O
  R -->|"escalates incidents to"| N
  R -.-|"sibling: vehicles"| F
  R -.-|"keeps goods moving for"| S
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

The Robot Operations Manifesto groups the work into four pillars: monitoring and observability, configuration management, safety, security and auditing, and interventions, with interoperability and orchestration across all four.^[1] It draws on the Failure Reporting, Analysis and Corrective Action System (FRACAS) from reliability engineering.^[1] The charter below is this Atlas’s proposal, built on those pillars; ROG publishes no charter of this form.

Mission (proposed). Keep a robot fleet productive and safe at scale by turning every autonomy exception into either a fast recovery or a permanent fix.

Non-goals. Zero interventions (a target that encourages hiding them); remote driving as the normal recovery; replacing the manufacturer’s or integrator’s safety engineering.

ROG’s manifesto asks operators to think of “a fleet of robots as a distributed system, with compute/storage/networking on each robot and often in the cloud”.^[1] The building blocks are the robot (safety controller and sensors, autonomy stack, on-board computer, operations agent, batteries), the site (fleet manager, Wi-Fi, maps and zones, chargers, doors and lifts, people) and operations (console, telemetry, release service, simulation CI, ticketing).

ROS 2 essentials for operators. Where a fleet runs ROS 2, a node should “do one logical thing”;^[24] topics carry continuous streams such as sensor data;^[25] services are short requests and “should never be used for longer running processes”;^[26] actions are long-running goals with feedback and cancellation, such as driving to a waypoint.^[27] Two details matter most to operators:

• Quality of Service (QoS). ROS 2 “can be as reliable as TCP or as best-effort as UDP”.^[16] A best-effort publisher and a reliable subscriber do not connect at all; defaults are keep-last 10, reliable and volatile, while the sensor-data profile uses best effort.^[16] A dashboard subscribing with the wrong profile sees nothing, which looks like a dead sensor.
• Security is opt-in. ROS 2 security (SROS2, built on the Data Distribution Service’s DDS-Security) uses a keystore and per-node enclaves, enabled through ROS_SECURITY_KEYSTORE, ROS_SECURITY_ENABLE=true and ROS_SECURITY_STRATEGY=Enforce; without them, nodes communicate unauthenticated. Once enforced, every operations tool (recorder, dashboard bridge, command line via ROS_SECURITY_ENCLAVE_OVERRIDE) needs its own enclave, and permissions files can limit each to the topics it needs.^[28]

As of October 2026 the supported ROS 2 distributions are Lyrical Luth (released 22 May 2026, end of life May 2031), Kilted Kaiju (to December 2026), Jazzy Jalisco (to May 2029) and Humble Hawksbill (to May 2027);^[29] ROS 1 ended on 31 May 2025.^[30] Recommended Gazebo pairings are Humble–Fortress, Jazzy–Harmonic, Kilted–Ionic and Lyrical–Jetty.^[31] Plan upgrades by date.

Safety standards split duties between manufacturers, integrators and users (see standards by robot class), and RoboOps should follow the same split. For industrial mobile robots (IMRs) in the United States, ANSI/A3 R15.08-2 covers “integrating, configuring, and customizing an IMR or fleet of IMRs into a site”,^[32] and ANSI/A3 R15.08-3-2026 covers users, including, according to a trade report of A3’s announcement, “risk assessment, management of change, employee training and periodic reviews and inspections”;^[11] a second report lists the same duties.^[33] (This research read reports, not the standard.) On this page those duties appear as the site-readiness checklist, change classes A–C, training in the lifecycle and periodic inspection on Day 2. The operating organisation adds roles of its own.

The remote assistant gives first response within the actions the runbook allows; the field technician does physical recovery, stop resets and hardware swaps; the site lead owns local priorities, layout changes and the site risk assessment; the release owner owns builds, gates and rollback; the safety officer owns risk assessments and near-miss investigations; the fleet operations manager owns metrics, staffing and vendors.

R = responsible, A = accountable, C = consulted, I = informed (illustrative). Robots-as-a-service contracts often move the release and remote-assistance rows to the vendor; the site’s “I” entries then become contractual requirements.

The governance core is ROG’s lifecycle of failures loop.^[1]

flowchart TB
  A["Autonomy exception detected"]:::risk
  B["`Resolve now
  remote assist or field visit`"]:::core
  C["`Record
  exception log + black-box snapshot`"]:::core
  S{"Safety-relevant?"}:::decision
  Q["`Quarantine the robot;
  safety officer investigates
  it as a near miss`"]:::risk
  D{"Recurring?"}:::decision
  E["`Analyse root cause
  weekly review`"]:::core
  F["`Corrective action
  software, map, site or procedure`"]:::core
  G["`Staged release
  with gates`"]:::core
  H{"Did the rate fall?"}:::decision
  K["Keep for trend analysis"]:::neutral
  Z["Close the corrective action"]:::core
  A --> B --> C --> S
  S -->|yes| Q --> E
  S -->|no| D
  D -->|no| K
  D -->|yes| E --> F --> G --> H
  H -->|no| E
  H -->|yes| Z
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029

Change classes (proposed). Class A changes cannot affect motion (dashboards, telemetry fields): normal review. Class B changes can (navigation parameters, maps, zones, speeds, new tasks): release board plus a check of the risk assessment. Class C changes touch safety configuration (protective fields, safety-rated speeds): only through the manufacturer’s or integrator’s safety process, never through the fleet’s OTA pipeline on operations authority alone.

The architecture is vendor-neutral: its trust boundaries matter more than the products.

flowchart TB
  RC["`OPS: remote-assistance
  console`"]:::ops
  FM["`SITE: fleet manager
  missions, traffic`"]:::site
  IA["`SITE: doors, lifts,
  chargers`"]:::site
  SF["`ROBOT: safety controller
  safety-rated, local only`"]:::risk
  AS["`ROBOT: autonomy stack
  ROS 2 or vendor`"]:::robot
  AG["`ROBOT: ops agent and
  black-box recorder`"]:::robot
  TL["`OPS: telemetry and
  exception log`"]:::ops
  SC["OPS: simulation CI"]:::ops
  RL["`OPS: release and
  OTA service`"]:::ops
  RC -->|assist actions| FM
  FM <--> IA
  FM <-->|orders, state| AS
  SF -.->|stop overrides| AS
  SF -.->|stop status, read-only| AG
  AS --> AG
  AG -->|health, snapshots| TL
  TL -->|exception queue| RC
  TL --> SC
  SC --> RL
  RL -->|signed release| AG
  classDef robot fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef site fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef ops fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029

Trust boundaries. (1) The safety layer is local and has no remote write path. (2) Remote staff act through the fleet manager, which enforces the operating mode; VDA 5050 3.0.0, for example, forbids orders and actions in MANUAL and only allows cancelOrder as an instant action in INTERVENED.^[9] (3) Robots and their middleware are never reachable from the public internet: a 2018 scan found “over 100 publicly-accessible hosts” running a ROS 1 master, which “trusts all nodes that connect to it”.^[34] (4) Releases are signed, and the robot verifies them before installing.

Trade-offs. Vendor choices are weighed in the decision framework. Continuous logging gives the richest evidence at a cost in storage, bandwidth and privacy; snapshots are cheaper but miss slow-building problems unless health metrics also stream.

Trigger: blocked path, lost localization, failed pick, a door that did not open. Outputs: a robot back in service and an exception record. Feedback: the weekly review.

sequenceDiagram
  participant R as Robot
  participant F as Fleet manager
  participant A as Assistant
  participant T as Technician
  R->>F: Blocked, snapshot saved
  F->>A: Ticket with camera view
  alt Solvable remotely
    A->>F: Confirm detour
    F->>R: Updated order
  else Not solvable remotely
    A->>T: Dispatch to robot
    Note over T: Recovers on site
    T->>A: Back in AUTOMATIC
    A->>F: Return to service
    F->>R: New order
  end
  R->>F: Driving again

Exception path. If no allowed action fits, the robot is in a protective or emergency stop, or the camera shows damage, smoke or people at risk: quarantine it, recover it on site, and return it only when the cause is understood. Several robots with the same class within minutes is an incident (IncidentOps): usually a site change or a release, not bad luck.

Remote assistance, not remote driving. In Waymo’s fleet response, the vehicle asks questions, remote agents answer or propose a path, and the automated driver “is in control of the vehicle at all times”; if stopping is safest it stops.^[35] The lesson transfers from vehicles (FleetOps): motion decisions stay on the robot.

Playbook: recover after an emergency stop (template). (1) Find out why it was pressed. (2) Make the area safe; leave the robot stopped. (3) Inspect robot and load on site. (4) Reset locally per the manufacturer’s procedure. (5) Move it in manual mode to a known position. (6) Return it to automatic mode, quarantined until the record is reviewed. (7) Treat any unexplained stop as a near miss.

These rows rest on publishers’ summaries, ISO catalogue entries and trade reports; conformance needs the standards’ texts and usually a qualified safety engineer. OSHA has no robot-specific regulation; its technical manual chapter predates the 2025 ISO revision.^[23][40]

Retired tools are part of the landscape: AWS RoboMaker (2018–2025) is gone.^[51] Keep simulation and fleet tooling portable (containers, MCAP, exportable data) so a vendor decision does not strand your test suite.

Automate actions that are frequent, reversible and do not change motion near people: tickets, attaching snapshots, quarantine after repeated faults, sending a robot to charge, grouping exceptions. Changes to what robots do in shared space (maps, speeds, zones, releases) need human approval (principle 6 above).

AI is entering the operations layer: Boston Dynamics advertises vision-language models in Orbit,^[47] and RobOpsCon 2026 features agentic software for robot operations.^[13] Treat a generated root cause as a hypothesis for the weekly review. AI inside a safety function is regulated territory: the EU Machinery Regulation adds provisions for it.^[36]

Assumptions (illustrative scenario): a 40-robot mobile-robot fleet on ROS 2 Jazzy at two sites, one remote-assistance desk, releases built in GitHub Actions. Three artefacts make the operating model concrete: an exception record, a release policy and a CI workflow. The schemas are this page’s proposals, not a standard.

Example status: syntax-checked by the Atlas validator; not executed (illustrative data).

{
  "exception_id": "EX-2026-10-05-0042",
  "robot": "amr-01",
  "site": "site-a",
  "raised_at": "2026-10-05T09:40:12Z",
  "operating_mode": "AUTOMATIC",
  "class": "serious",
  "category": "localization",
  "symptom": "mislocalized near dock 3; robot holding position",
  "resolution": "onsite",
  "robot_stopped_minutes": 25,
  "snapshot": "s3://fleet-evidence/site-a/amr-01/2026-10-05T09-40-12Z/",
  "release": {"software": "nav-4.12.1", "map": "site-a-map-37",
              "config": "site-a-cfg-19"},
  "safety_relevant": false,
  "linked_exceptions": ["EX-2026-10-05-0040", "EX-2026-10-05-0041"],
  "corrective_action": "CA-118"
}

The release policy treats software, map and configuration as one unit and names the gates that stop a rollout.

Example status: syntax-checked by the Atlas validator; the policy format is illustrative and not tied to a product.

release: nav-4.13.0+site-a-map-37+site-a-cfg-20
change_class: B            # can affect motion: release board + risk-assessment check
install_window: "robot docked and charging, no active order"
cohorts:
  - name: canary
    robots: [amr-07, amr-12]          # one per hardware revision at site-a
    min_hours: 48                     # wall-clock hours
    gates: {failed_boots: 0, fatal_exceptions: 0}   # too few hours for rate gates
  - name: site-a
    robots: all-at-site-a
    hold_back: 4                      # stay on the previous release as a control
    min_robot_hours: 400              # about 12 protective stops expected at baseline
    gates:                            # fail above 1.5 x the expected baseline count
      protective_stops_per_100h: {baseline: 2.9, max_ratio: 1.5}
      serious_exceptions_per_100h: {baseline: 4.0, max_ratio: 1.5}
  - name: site-b
    robots: all-at-site-b
    hold_back: 4
    min_robot_hours: 400
    gates:                            # baselines from site-b's own history
      protective_stops_per_100h: {baseline: 3.4, max_ratio: 1.5}
      serious_exceptions_per_100h: {baseline: 4.6, max_ratio: 1.5}
rollback: "switch boot slot (A/B) and restore previous map and config"

Why gates are sized in robot-hours. At 2.9 protective stops per 100 robot-hours, two canary robots expect two or three stops in 48 hours, depending on shifts: a “no increase” rule would fail an unchanged release about half the time. At 400 robot-hours (about 12 expected stops) with a limit of 1.5 times the expected count, an unchanged release trips the gate about 5% of the time and a doubled rate about 89% (Poisson arithmetic). Canaries catch crashes and install faults; site cohorts catch rate changes; held-back robots give a concurrent comparison when baselines drift with season or staffing.

The CI workflow builds and tests the ROS 2 packages on every change, using the recommended base container of action-ros-ci.^[49]

Example status: syntax-checked; not executed (needs a GitHub repository with a ROS 2 package). The action is pinned to the commit of release 0.4.8 and the job token is read-only; replace <digest> with the image digest you tested.

name: robot-software-ci
on: [pull_request]
permissions:
  contents: read
jobs:
  build_and_test:
    runs-on: ubuntu-latest
    container:
      # pin the image you tested: the ubuntu-noble-latest tag moves
      image: rostooling/setup-ros-docker@sha256:<digest>
    steps:
      - name: Build and run unit and launch tests
        uses: ros-tooling/action-ros-ci@3a640b10f09b756dabe556dac5413aba369f71b0  # 0.4.8
        with:
          package-name: fleet_nav_bringup
          target-ros2-distro: jazzy

Integration tests written with launch_testing run under colcon test in the same job.^[52] For scenario tests, Gazebo can run without a GUI: gz sim -s -r --headless-rendering --iterations 20000 <world.sdf> runs only the server, starts the simulation immediately and stops after a fixed number of iterations (flags checked in the Jetty and Harmonic sources).^[53] Each serious exception class should add one scenario: the dock where amr-01 lost localization becomes a world file and a test.

Objective: see how much a definition changes the headline autonomy metric. Prerequisites: Python 3.8 or later; no robot, cost or risk; 20 minutes. The 13 interventions are illustrative data for four robots over a week (448 operating hours).

Example status: executed on 2026-10-18 with Python 3.8.10; the output below is from that run.

robot,start,cls,resolved,minutes,cause
amr-01,2026-10-05T07:12,minor,remote,1,manual mission added
amr-01,2026-10-05T09:40,serious,remote,4,mislocalized near dock
amr-01,2026-10-05T09:47,serious,remote,3,mislocalized near dock
amr-01,2026-10-05T09:52,serious,onsite,18,mislocalized near dock
amr-02,2026-10-06T11:05,serious,remote,5,path blocked by pallet
amr-02,2026-10-07T14:30,fatal,onsite,42,navigation stack crash
amr-03,2026-10-05T16:20,minor,remote,1,map annotation edit
amr-03,2026-10-08T08:02,serious,remote,6,path blocked by pallet
amr-03,2026-10-09T13:15,serious,remote,4,lift door timeout
amr-04,2026-10-06T10:10,serious,remote,3,path blocked by pallet
amr-04,2026-10-06T10:16,serious,remote,4,path blocked by pallet
amr-04,2026-10-10T15:45,fatal,onsite,55,drive fault after protective stop
amr-04,2026-10-11T09:30,minor,remote,1,manual mission added

"""Compute intervention metrics for a robot fleet from an intervention log (CSV).

Minor interventions (normal use) can be excluded; interventions on one robot that start
within cluster_min minutes of the previous one can be grouped.
MTBI = operating hours / counted events.
"""
import csv
import sys
from collections import Counter
from datetime import datetime

# 7 days x 16 h per robot
OPERATING_HOURS = {"amr-01": 112, "amr-02": 112, "amr-03": 112, "amr-04": 112}


def load(path):
    with open(path, newline="") as f:
        rows = list(csv.DictReader(f))
    for r in rows:
        r["start"] = datetime.strptime(r["start"], "%Y-%m-%dT%H:%M")
        r["minutes"] = int(r["minutes"])
    return sorted(rows, key=lambda r: (r["robot"], r["start"]))


def group(rows, cluster_min):
    """Merge a robot's interventions that start within cluster_min of the previous one."""
    events, open_event, last_start = [], {}, {}
    for r in rows:
        robot = r["robot"]
        gap = None
        if robot in last_start:
            gap = (r["start"] - last_start[robot]).total_seconds()
        if gap is not None and gap <= cluster_min * 60:
            open_event[robot]["members"].append(r)   # same underlying problem
        else:
            open_event[robot] = {"robot": robot, "members": [r]}
            events.append(open_event[robot])
        last_start[robot] = r["start"]
    return events


def report(rows, cluster_min, count_minor):
    hours = sum(OPERATING_HOURS.values())
    kept = [r for r in rows if count_minor or r["cls"] != "minor"]
    events = group(kept, cluster_min) if cluster_min else [{"members": [r]} for r in kept]
    n = len(events)
    onsite = sum(1 for e in events
                 if any(m["resolved"] == "onsite" for m in e["members"]))
    lost = sum(m["minutes"] for e in events for m in e["members"])
    return n, hours / n if n else float("inf"), 100.0 * n / hours, onsite, lost


def main():
    rows = load(sys.argv[1] if len(sys.argv) > 1 else "interventions.csv")
    hours = sum(OPERATING_HOURS.values())
    print("fleet operating hours:", hours)
    print("%-34s %6s %9s %12s %7s %9s" % (
        "definition", "events", "MTBI (h)", "per 100 h", "onsite", "lost min"))
    for label, cluster, minor in [("all interventions, no grouping", 0, True),
                                  ("exclude minor, no grouping", 0, False),
                                  ("exclude minor, group within 15 min", 15, False)]:
        n, mtbi, rate, onsite, lost = report(rows, cluster, minor)
        print("%-34s %6d %9.1f %12.2f %7d %9d" % (label, n, mtbi, rate, onsite, lost))
    causes = Counter(r["cause"] for r in rows if r["cls"] != "minor")
    print("top causes (non-minor):")
    for cause, n in causes.most_common(3):
        print("  %s=%d" % (cause, n))


if __name__ == "__main__":
    main()

1. Save the data as interventions.csv and the script as mtbi.py in an empty folder.
2. Run python3 mtbi.py.

Expected output:

fleet operating hours: 448
definition                         events  MTBI (h)    per 100 h  onsite  lost min
all interventions, no grouping         13      34.5         2.90       3       147
exclude minor, no grouping             10      44.8         2.23       3       144
exclude minor, group within 15 min      7      64.0         1.56       3       144
top causes (non-minor):
  path blocked by pallet=4
  mislocalized near dock=3
  navigation stack crash=1

Validation: the same week reads as 34.5, 44.8 or 64.0 hours MTBI. Published studies state such choices: one inspection deployment excluded minor interventions,^[14] and a forest-robot study merged safety-operator commands within 10-second windows.^[21] The top cause points at the pallet staging area, not the navigation software. Troubleshooting: FileNotFoundError means the CSV is not in the working folder; ValueError means a malformed timestamp. Teardown: delete the folder.

Objective: find how many remote assistants a fleet needs, and why fan-out alone overstates it. Prerequisites: Python 3.8 or later; no cost or risk; 30 minutes. Model (assumptions): each robot works for a random time averaging NT = 90 minutes, then stops with an exception; it waits for a free assistant, who needs on average IE = 4 minutes; both times are exponential. Fan-out is (IE + NT) / IE.^[22]

Example status: executed on 2026-10-18 with Python 3.8.10 (seeded, deterministic); output below is from that run.

"""Simulate a remote-assistance desk: robots stop on an autonomy exception and wait
for an operator.

Each robot works for an exponentially distributed time (mean NT minutes), raises an
exception, waits for a free operator, needs IE minutes of operator attention
(exponential), then resumes.
"""
import heapq
import random
import sys


def simulate(robots, operators, nt, ie, minutes, seed):
    rng = random.Random(seed)
    events = [(rng.expovariate(1 / nt), "exception", r) for r in range(robots)]
    heapq.heapify(events)
    free, queue, waits, busy, stopped = operators, [], [], 0.0, 0.0
    while events:
        t, kind, robot = heapq.heappop(events)
        if t > minutes:
            break
        if kind == "exception":
            queue.append((t, robot))
        else:                                     # an operator finished helping a robot
            free += 1
            heapq.heappush(events, (t + rng.expovariate(1 / nt), "exception", robot))
        while free and queue:
            raised, r = queue.pop(0)
            service = rng.expovariate(1 / ie)
            waits.append(t - raised)
            stopped += (t - raised) + service
            busy += service
            free -= 1
            heapq.heappush(events, (t + service, "resolved", r))
    waits.sort()
    p90 = waits[int(0.9 * (len(waits) - 1))] if waits else 0.0
    return {"exceptions": len(waits), "util": busy / (operators * minutes),
            "mean_wait": sum(waits) / len(waits) if waits else 0.0, "p90_wait": p90,
            "productive": 1 - stopped / (robots * minutes)}


def main():
    robots = int(sys.argv[1]) if len(sys.argv) > 1 else 40
    # NT: mean minutes between exceptions; IE: mean operator minutes per exception
    nt = float(sys.argv[2]) if len(sys.argv) > 2 else 90.0
    ie = float(sys.argv[3]) if len(sys.argv) > 3 else 4.0
    minutes = 30 * 16 * 60  # 30 days of 16-hour operation
    fan_out = (ie + nt) / ie
    print("fan-out (IE+NT)/IE = %.1f robots per operator (upper bound)" % fan_out)
    print("%-9s %10s %6s %10s %9s %11s" % (
        "operators", "exceptions", "util", "mean wait", "p90 wait", "productive"))
    for ops in (1, 2, 3, 4):
        s = simulate(robots, ops, nt, ie, minutes, seed=11)
        print("%-9d %10d %5.0f%% %8.1f m %7.1f m %10.1f%%" % (
            ops, s["exceptions"], 100 * s["util"], s["mean_wait"], s["p90_wait"],
            100 * s["productive"]))


if __name__ == "__main__":
    main()

1. Save as desk.py; run python3 desk.py (40 robots, NT 90, IE 4).
2. Run python3 desk.py 40 45 4 (exceptions twice as frequent).

Expected output (step 1):

fan-out (IE+NT)/IE = 23.5 robots per operator (upper bound)
operators exceptions   util  mean wait  p90 wait  productive
1               7256   100%     64.3 m    97.9 m       57.0%
2              11702    81%      5.2 m    14.1 m       90.7%
3              12312    57%      0.8 m     3.0 m       94.9%
4              12327    43%      0.1 m     0.0 m       95.6%

Validation: two assistants cover 20 robots each, below the fan-out of 23.5, yet the p90 wait is 14 minutes because random arrivals queue at 81% utilisation; three bring it to about three minutes. In step 2 fan-out drops to 12.2 and four assistants give a 5.5-minute p90 wait (79% utilisation). Interaction effort is mostly cognitive (remembering robot state, interface modes and robot abilities) and rises with fan-out,^[22] so the model is optimistic. Troubleshooting: ValueError means a non-numeric argument; ZeroDivisionError means NT or IE was 0. Teardown: delete the file.

Objective: keep the last minutes of key topics in memory and save them automatically when any component reports an error. Prerequisites: a test machine or simulator with ROS 2 Jazzy or later and the diagnostics packages; never a robot in production; 45 minutes. Risk: none to motion; the cache is double-buffered, so memory use can reach twice --max-cache-size (about 400 MB here) plus overhead.^[54]

Example status: syntax-checked; not executed (no ROS 2 installation in this environment). Commands follow the rosbag2 Jazzy documentation and source.^[17][54][55]

# Terminal 1: recorder in snapshot mode, 200 MB per buffer, MCAP (the default format)
ros2 bag record --snapshot-mode --max-cache-size 200000000 \
  --topics /diagnostics /tf /tf_static /scan /odom /cmd_vel -o blackbox
# Terminal 2: the trigger node below
python3 blackbox_trigger.py
# Terminal 3: inject a fake fault, wait for the second snapshot, inspect the closed files
python3 inject_fault.py
sleep 25
ls blackbox
ros2 bag info blackbox/blackbox_0.mcap
ros2 bag info blackbox/blackbox_1.mcap

"""Call the rosbag2 snapshot service when any /diagnostics status reaches ERROR or STALE,
and again POST_S seconds later, so that the record also shows the aftermath."""
import time

import rclpy
from diagnostic_msgs.msg import DiagnosticArray, DiagnosticStatus
from rclpy.node import Node
from rosbag2_interfaces.srv import Snapshot

COOLDOWN_S = 60.0  # at most one triggered snapshot per minute
POST_S = 20.0      # a snapshot holds only earlier data; a second one shows the aftermath


def level_value(level):
    # byte fields arrive as bytes in rclpy
    return level if isinstance(level, int) else ord(level)


class BlackBoxTrigger(Node):
    def __init__(self):
        super().__init__("blackbox_trigger")
        self.client = self.create_client(Snapshot, "/rosbag2_recorder/snapshot")
        self.create_subscription(DiagnosticArray, "/diagnostics", self.on_diagnostics, 10)
        self.last = 0.0
        self.post_timer = None

    def snapshot(self, why):
        self.client.call_async(Snapshot.Request())
        self.get_logger().warning("snapshot requested: %s" % why)

    def on_diagnostics(self, msg):
        error = level_value(DiagnosticStatus.ERROR)
        bad = [s for s in msg.status if level_value(s.level) >= error]
        if not bad or time.monotonic() - self.last < COOLDOWN_S:
            return
        if not self.client.service_is_ready():
            self.get_logger().error(
                "snapshot service not available; is the recorder in --snapshot-mode?")
            return
        self.last = time.monotonic()
        self.snapshot("%s: %s" % (bad[0].name, bad[0].message))
        self.post_timer = self.create_timer(POST_S, self.on_post_trigger)

    def on_post_trigger(self):
        self.post_timer.cancel()  # one-shot
        self.snapshot("post-trigger")


def main():
    rclpy.init()
    rclpy.spin(BlackBoxTrigger())


if __name__ == "__main__":
    main()

"""Publish one fake ERROR on /diagnostics to test the black-box trigger."""
import time

import rclpy
from diagnostic_msgs.msg import DiagnosticArray, DiagnosticStatus


def main():
    rclpy.init()
    node = rclpy.create_node("inject_fault")
    pub = node.create_publisher(DiagnosticArray, "/diagnostics", 10)
    time.sleep(1.0)  # allow discovery
    status = DiagnosticStatus(level=DiagnosticStatus.ERROR, name="drive_left",
                              message="test fault: motor overcurrent",
                              hardware_id="amr-test")
    pub.publish(DiagnosticArray(status=[status]))
    time.sleep(0.5)
    node.destroy_node()
    rclpy.shutdown()


if __name__ == "__main__":
    main()

Expected output: the trigger logs snapshot requested: drive_left: test fault: motor overcurrent and, 20 seconds later, snapshot requested: post-trigger. ls blackbox shows .mcap files but no metadata.yaml: each snapshot closes one file, and rosbag2 writes metadata.yaml only when the recorder stops.^[55] blackbox_0.mcap holds messages from before the fault, blackbox_1.mcap those from the 20 seconds after it. Validation: levels (OK 0, WARN 1, ERROR 2, STALE 3) come from diagnostic_msgs;^[56] the service name and type from the recorder source;^[54] the second file holds only newer messages because the recorder clears the buffer it has written.^[55] Troubleshooting: “snapshot service not available” means the recorder lacks --snapshot-mode or uses another --node-name; “Could not find metadata in bag directory” means ros2 bag info was given the directory while the recorder runs; an empty file usually means different topic names or a QoS mismatch. Teardown: stop all terminals (Ctrl+C); the recorder then writes metadata.yaml, so ros2 bag info blackbox works; delete blackbox. In production: set retention and camera access, and upload at the dock.

In the inspection deployment, a process manager “intermittently causing all ROS nodes to shut down when operators reconnected” caused the fatal interventions at one site and was suspected in three at the other: weeks of operation surfaced a bug that short tests had not.^[14]

Worked calculation (illustrative). A site runs 40 robots for 16 hours on 30 days: 19,200 robot-hours. It logs 768 serious or fatal exceptions after grouping repeats: MTBI = 19,200 ÷ 768 = 25 hours; rate = 768 ÷ 19,200 × 100 = 4.0 per 100 robot-hours. If 85% are resolved remotely in a mean 6 minutes and the rest take a mean 40 minutes on site, stopped time is about 768 × (0.85 × 6 + 0.15 × 40) ≈ 8,525 minutes, or 0.7% of the 1,152,000 scheduled robot-minutes. A target of “MTBI 30 hours” means cutting about 128 exceptions a month, so the weekly review should start with the class that contributes most of them.

Leading and lagging. Exception rate and protective stops lead; throughput lags. “Uptime” alone is a vanity metric: a robot can be powered, connected and useless.

• Safety. Change classes B and C and near-miss reviews carry the safety duties into daily operation. In VDA 5050 3.0.0, safetyState reports activeEmergencyStop (MANUAL, REMOTE or NONE) and fieldViolation;^[9] 2.1.0 reports eStop, with an extra AUTOACK value, for example for a bumper hit.^[15] Map both to count stops across vendors.
• Security. Segment robot networks; never expose middleware to the internet;^[34] enable SROS2 or the vendor’s equivalent, with an enclave per operations tool;^[28] sign releases; use role-based access control, as ROG recommends,^[1] with named accounts. VDA 5050 leaves protocol security to broker configuration, and 3.0.0 notes for its certificate-update action that “the sender of the instantAction cannot be verified”,^[9] so the MQTT broker needs TLS, authentication and per-topic access rules.
• Privacy. Robot cameras record staff, patients and customers. Decide what the black box records, who may view it, retention and redaction before go-live; jurisdictions differ (PrivacyOps).
• Compliance. Map each site to its regime: in the EU, besides the Machinery Regulation (see the standards table), the Cyber Resilience Act covers hardware and software “products with digital elements”, with reporting duties from 11 September 2026 and main obligations from 11 December 2027.^[58] Ask vendors for their security-support period and vulnerability handling (more in IoTOps). This is not legal or safety advice.

Recurring costs are people (assistants, technicians), downtime, field visits, spare parts, connectivity and data. Useful unit costs are cost per mission and cost per exception (handling time × loaded staff cost, plus the robot’s lost time). Illustrative: in the worked calculation an exception takes 0.85 × 6 + 0.15 × 40 = 11.1 staff-minutes, if staff are busy for the whole stop. At an assumed 60 per staff-hour (any currency), that is about 11 per exception and 8,500 a month; MTBI 30 hours (128 fewer exceptions) saves about 1,400 a month before the robots’ recovered output. Capacity often runs out at chargers, Wi-Fi, lifts and the assistance desk before computers. Upload snapshots at the dock rather than streaming video. This research found no sourced data on robot-fleet carbon footprints; energy and battery lifecycle are the obvious levers, and carbon accounting is taught in GreenOps.

Check the loop, not just the robots: serious exceptions with a snapshot (target near 100%); corrective actions closed on time; repeat rate of classes marked “fixed”; drills held; releases that passed every gate; and an audit trail of who changed which map, configuration or release on which robot.^[1]

Traffic between fleets. Robots from one vendor know of each other and avoid conflicts;^[39] two fleets sharing a crossing or lift can each wait for the other indefinitely (deadlock). VDA 5050 3.0.0 makes deadlock resolution a fleet-control duty but leaves traffic logic out of scope; its tools are released path segments, at whose end a robot stops, and RELEASE zones entered only when fleet control grants access.^[9] Open-RMF has fleet managers negotiate conflicts on a shared traffic schedule and warns that a fleet with no interface “will likely result in deadlocks”.^[60] Give each shared crossing and lift one traffic authority.

flowchart TB
  A["Release candidate passes simulation CI"]:::core
  B["`Canary cohort
  one robot per hardware revision`"]:::core
  C{"`No failed boots or
  fatal exceptions in 48 h?`"}:::decision
  D["`Site cohort
  a few robots held back`"]:::core
  E{"`Rates within limits
  after 400 robot-hours?`"}:::decision
  F["Next site"]:::core
  G["`Contain: halt rollout,
  roll back or limit speed`"]:::risk
  H["`Notify the safety officer;
  log a near miss if stops rose`"]:::risk
  I["`Root cause
  snapshots and control robots`"]:::core
  A --> B --> C
  C -->|yes| D --> E
  E -->|yes| F
  C -->|no| G
  E -->|no| G
  G --> H --> I
  I -->|fixed| A
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029

Situation (illustrative). Release nav-4.13.0 passed its two-robot canary (hard faults only) and was installed at site A on Monday, except on four robots then in maintenance. On Tuesday, after 420 robot-hours, the site gate trips: 26 protective stops (6.2 per 100 robot-hours) against a baseline of 2.9 and a limit of 18. No injuries.

1. Contain first. Freeze the rollout. At site A, roll back to the previous slot, map and configuration, or apply a pre-approved lower speed limit in the affected zones until the rollback is done. A doubled stop rate in shared space is a near-miss signal: the safety layer is catching something the release changed, and no dashboard shows whether the assumptions behind the protective fields still hold.
2. Notify. Tell the safety officer and site lead; log a near miss; keep the snapshots.
3. Diagnose. 70% of stops are at two crossings where people push carts; snapshots show robots now approach faster before slowing. The four robots still on nav-4.12.1 had 4 stops in 130 robot-hours: too few hours to prove anything alone, but consistent with the release, not the site.
4. Classify. Faster approach near people is a class B change the release notes did not declare and the risk assessment did not list. Confirm the rate returns to baseline after the rollback.
5. Correct. Set crossing speeds in map zones, add a cart-traffic simulation scenario and the crossings to the risk review, and re-release through the board as class B.
6. Learn. Diff motion parameters automatically so no release hides a class B change; canary each site layout, not only each hardware revision.

• “Zero interventions” targets. Teams stop logging them. Track rates by class and reward reporting.
• The software e-stop. A dashboard’s “stop all robots” button is a network command, not an emergency stop; label it honestly.
• Hand-tuned robots. A parameter changed on one robot to “unstick” it becomes an unknown configuration in the next incident.
• Fleet-wide Friday releases. Robots work shifts; release when you can watch the first hours of a cohort.
• Myth: simulation passes, so the field will. Floors, light, people and Wi-Fi differ; simulation catches regressions, cohorts catch reality.

Use these practices when robots run unattended for long periods, across more than one site or vendor, or in shared space with people. For a handful of supervised robots, the vendor’s tools and a simple log are enough. Robots-as-a-service contracts move much of the work to the provider, but you still need written metric definitions, evidence after incidents and a say in releases.

flowchart TB
  S["`For every option, weigh
  cost per exception,
  data export and exit terms`"]:::neutral
  Q1{"More than about 10 robots or unattended shifts?"}:::decision
  Q2{"More than one robot vendor on a site?"}:::decision
  Q3{"Is operating robots your product?"}:::decision
  A1["`Vendor tools plus
  an exception log;
  revisit as the fleet grows`"]:::neutral
  A2["`Vendor fleet manager
  plus your own exception loop`"]:::core
  A3["`Interoperability layer
  VDA 5050 or Open-RMF`"]:::core
  A4["`Build or extend a platform
  open source plus your own agent`"]:::core
  S --> Q1
  Q1 -->|no| A1
  Q1 -->|yes| Q2
  Q2 -->|no| Q3
  Q2 -->|yes| A3
  Q3 -->|no| A2
  Q3 -->|yes| A4
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029

The thresholds are judgement calls, not industry figures. One vendor is simpler and accountable but locks the site in, so weigh exit terms as heavily as features; an interoperability layer admits several vendors at the cost of integration work.

This is the Atlas’s model, not an industry standard. In this Atlas’s judgement, a fleet heading past ROG’s 50-robot mark needs level 3.^[1]

Site readiness (before go-live)

• Risk assessment signed by integrator and site; lists routes, speeds, crossings and tasks
• Wi-Fi survey along every route; behaviour on link loss tested
• Emergency-stop locations agreed; recovery procedure drilled
• Exception classes, allowed remote actions and escalation contacts written
• Black-box recording, retention and camera access agreed

Release (every cohort)

• Change class decided (A, B or C); risk assessment checked for class B
• Simulation scenarios for known exception classes pass
• Install only when robots are docked and idle; previous slot, map and configuration kept
• Gates, minimum robot-hours, held-back control robots and an owner for the first hours named

Weekly operations review

• Exceptions by class and site, with definitions unchanged
• Every serious exception has a snapshot and an owner
• Unexplained protective or emergency stops reviewed as near misses
• Corrective actions due this week closed or escalated

Periodic inspection (manufacturer’s schedule)

• Emergency stops, protective scanners and bumpers cleaned and function-tested; results recorded
• Batteries and chargers checked for damage, heat and capacity loss; suspect units taken out of service

Starter exception taxonomy (proposed). The category (the exception record’s category field) decides the first response; severity (minor, serious, fatal) is recorded separately.

SITE RUNBOOK: <site>                                    owner: <site lead>   version: <n>
Robots: <count, models, hardware revisions>      Fleet manager: <product/version>
Allowed remote actions: confirm static obstacle | approved detour | cancel mission |
                        send to parking or charger | snapshot | quarantine
Never remotely: reset MANUAL-type e-stop | change protective fields | untested config
Escalation: assistant -> field technician (<15 min on site) -> site lead -> safety officer
Incident when: >= 3 robots same class in 30 min | any contact with a person | smoke/heat
Contacts: <names, phones, vendor support, hours>

EXCEPTION REVIEW (weekly)                         period: <dates>   site: <site>
Definitions: MTBI excludes minor; repeats within 15 min grouped (unchanged? yes/no)
Top classes (count, per 100 h, trend): 1. <class>  2. <class>  3. <class>
Safety-relevant events: <list, near-miss ids>
Decisions: <root cause, owner, corrective action, release target>
Actions due / closed / overdue: <n> / <n> / <n>

ADR: <title>                                      date: <YYYY-MM-DD>   status: <proposed>
Context: <fleet, sites, vendors, constraint>
Options: <vendor fleet manager | interoperability layer | own platform>
Decision and reason: <...>
Exit plan: <data export format, MCAP evidence, contract terms>
Safety impact: <class A/B/C, risk-assessment change>

Do we need ROS to do RoboOps? No. Many robots expose only a vendor API or VDA 5050. ROS tools help when you control the software.

Can a remote operator reset an emergency stop? Only if the system was designed and assessed for it; VDA 5050 distinguishes stops acknowledged on the robot (MANUAL) from facility stops acknowledged remotely (REMOTE).^[9] Default to on-site recovery.

Is MTBI comparable between vendors? Only with identical, written definitions of an intervention, grouping and operating time.

What should a black box record? Health, localization, commands, safety state and the sensors that explain a stop, for a few minutes around each trigger; cameras only if policy allows.

Who owns safety when we only operate robots? Manufacturer and integrator for the design; the site, as user, for safe use, change, training and periodic inspection (roles). Contracts should say so.

• Interoperability standards. InOrbit calls OpenRobOps the first ISO 21423 reference implementation, a vendor claim from a CEO who sits in the ISO working group.^[10]
• Fleets as data sources. RobOpsCon 2026 themes include heterogeneous fleets with humanoids, data pipelines for training robot models and decentralized on-robot decisions.^[13] Field data feeds MLOps; keep privacy rules attached.

• Label: niche (RobOps/RoboOps); the practice of running robot fleets is real and growing.
• Core loop: accept exceptions, resolve them fast, make them rarer.
• Never: route safety functions through the ops pipeline; reset stops remotely by default; release fleet-wide at once.
• Always: written metric definitions, snapshots for serious exceptions, cohorts with gates, quarantine before return to service.
• Measure: serious exceptions per 100 robot-hours, time to resume, assistant utilisation, protective stops per release.
• Decide: vendor tools for small fleets; your own exception loop at scale; an interoperability layer for mixed vendors.

Situation (illustrative). A navigation bug makes robots at three sites stop at lift doors about twice a shift; assistants clear each case in four minutes. A fix passed simulation this afternoon. Peak season starts tomorrow.

Option A: push tonight to all 120 robots. Peak starts without the bug, if the fix works. But it has no field hours: if it changes approach behaviour near lifts, peak starts with an unknown regression.

Option B: canary tonight, run degraded at peak. Two robots per site get the fix overnight; the rest keep the bug. Cost: about 114 robots × 2 stops × 4 minutes ≈ 910 assistant-minutes per 8-hour shift, so roughly two extra assistants per shift, plus queueing headroom (Lab 2).

Decision. B, with a gate review after the first peak shift. Six canary robots over one shift can show the lift-door stops vanish (a large effect), not that protective stops did not rise, so the second night goes to one site, not all 120. The bug is costly but understood and safe; the fix is unproven in a motion-relevant area. Uncertainty: if stopped robots block people or emergency access at lifts, the balance shifts toward A with extra site staff.

You should be able to explain autonomy exceptions and the lifecycle of failures; distinguish remote assistance from teleoperation and safety stops from operational pauses; define comparable fleet metrics; size an assistance desk; and design a staged release for robots.

1. Why is a dashboard’s “stop all robots” button not an emergency stop?

2. Two vendors report MTBI of 40 and 70 hours. What do you ask before comparing?

3. Each robot needs help every 120 minutes and each case takes 5 minutes. What is the fan-out, and should one person watch that many?

4. Which changes need a risk-assessment review before release?

5. A two-robot canary ran for 48 hours with no rise in protective stops. Is the release ready for every site?

RoboOps is a small label for a large shift. As robots leave cages and pilots, the hard problems stop being “can the robot do the task?” and become “who notices when it cannot, how fast does it get going again, and how do we make sure the next release does not put someone in its path?” The Robot Operations Group’s answer, follow the lifecycle of failures, holds up, but only with things the label does not provide: safety functions kept out of reach of operations software, metrics defined before they are compared, a staffed and modelled assistance desk, and releases that respect cohorts and risk assessments. Treat the word as optional and the practices as mandatory.

All sources were opened on 2026-10-18. docs.ros.org blocks automated access, so ROS 2 documentation is cited from its source repository; www.iso.org and A3 pages also refused access, so ISO catalogue entries are cited from committee.iso.org and the A3 FAQ from an Internet Archive copy. No standard’s text was read. Re-check dated facts each May (ROS releases) and when a standard or VDA 5050 version changes.

Changelog.

• 2026-10-18, r0: first draft (term status confirmed as niche; 13 parts; 6 diagrams; Labs 1–2 executed, Lab 3 syntax-checked only).
• 2026-10-18, r1: after the independent review: R15.08-3-2026, ISO 21423:2026 and standards by robot class added; runbook contains first; gates sized in robot-hours; Lab 3 corrected; VDA 5050 2.1.0, SROS2 enclaves, the CRA, periodic inspection and diagram fixes added; repeated comparison table and FAQ cut.
• 2026-10-18, post-score polish: OSHA and Ocado wording fixed; second R15.08-3 report; exception taxonomy, cross-fleet traffic, cost example, site-b gates and a decision-tree exit check added; code wrapped to 90 characters; repeats trimmed.
• 2026-10-18, post-polish check: Lab 2 interaction-effort paraphrase corrected to Olsen and Goodrich’s “interface modes”; vendor-versus-interoperability trade-off restored under the decision framework; standards-table link added.