--- title: "RoboOps" slug: roboops category: cloud term_status: niche evidence_status: verified summary: "RoboOps (more often spelled RobOps) is the practice of running fleets of autonomous robots in production: resolving autonomy exceptions, monitoring, managing configuration and updates, and keeping people safe when a software fault can move a machine." audience: [Robot fleet operators, Robotics and field engineers, Site and operations managers, Architects, Beginners] last_verified: 2026-10-18 key_takeaways: - "The label is niche. The Robot Operations Group (meeting since 2020) and InOrbit spell it RobOps; Wipro sells a managed service called RoboOps. No standard defines it, but the work of running robot fleets is real." - "Plan for autonomy exceptions, not perfect autonomy: accept that robots get stuck, resolve each case quickly, and drive the rate down over time (the Robot Operations Group's 'lifecycle of failures')." - "Safety functions belong to the robot's safety-rated controls, not to the fleet dashboard. Remote staff assist; they do not override protective stops, and a software 'pause all' is not an emergency stop." - "Write metric definitions down before comparing numbers: on one illustrative log, mean time between interventions ranges from 34.5 to 64.0 hours depending only on what counts as an intervention." - "Fan-out, the number of robots one person can supervise, is an upper bound. In Lab 2, 40 robots need three remote assistants, not two, to keep the 90th-percentile wait near three minutes." - "Ship software, maps and configuration as one versioned release by cohort, with A/B rollback, a risk-assessment check for motion changes and rate gates sized in robot-hours: a two-robot canary catches crashes, not a doubled stop rate." - "Standards moved in 2026: VDA 5050 reached 3.0.0, ISO 21423 (fleet interoperability) was published, and in the US ANSI/A3 R15.08-3 set out users' duties for mobile robots. The interface standards say they are not safety standards." related: [edgeops, iotops, fleetops, mlops, observabilityops, incidentops, releaseops, supplychainops] --- ## What RoboOps is ### In 30 seconds **RoboOps** is the operations discipline for fleets of autonomous robots (autonomous mobile robots, or AMRs, in warehouses and hospitals; inspection and delivery robots; robot arms in cells) after the demo works: watching every robot, handling the moments when one cannot continue on its own, changing software, maps and settings safely across sites, and making each failure rarer. Unlike servers, **the system moves in a space shared with people**: a bad release can cause a collision, not just an outage. The community behind the term puts its core idea in one sentence: "the key to operating robots at scale is to follow the lifecycle of failures".[^rogmanifesto] Robots will meet **autonomy exceptions**, situations "where the robot cannot continue operating safely nor efficiently"; operations means expecting them, resolving them fast and making them rarer.[^rogmanifesto] ::: note Spelling and status "RobOps" is the spelling used by the Robot Operations Group, by InOrbit and at the RobOpsCon conference;[^roghome][^inorbitrobops] Wipro sells a managed service called "RoboOps".[^wipro] This page keeps the manifest spelling **RoboOps** and treats both as one label (an unrelated 2021 Robot Framework library also uses the name).[^pypiroboops] No standard or certification defines it; a 2020 trade-press profile of the group noted that "there currently are no accepted standards" for the processes it addresses.[^oitzman2020] ::: **For** fleet operators, site managers, robotics engineers who ship to robots in the field, and architects choosing fleet software. Beginners can stop after the [glossary](#glossary-and-mental-model); engineers should do the [labs](#hands-on-labs); leaders should read the [decision framework](#decision-framework) and the [judgement call](#a-judgement-call-push-the-fix-to-every-robot-tonight-or-run-degraded). ### At a glance | Item | Summary | |---|---| | Term status | **Niche.** A small community since 2020; no standard or certification. | | Goal | Robots that are productive and safe at fleet scale, with exceptions resolved quickly and made rarer over time. | | In scope | Exceptions and remote assistance; fleet telemetry and black-box logging; configuration, maps and over-the-air (OTA) updates; simulation and continuous integration (CI); multi-vendor interfaces; operational safety and security. | | Out of scope | Designing safety functions; generic edge servers ([EdgeOps](../edgeops/)); constrained devices ([IoTOps](../iotops/)); vehicle fleets ([FleetOps](../fleetops/)); training perception models ([MLOps](../mlops/)). | | Prerequisites | Basic Linux and networking; helpful: [EdgeOps](../edgeops/) and [IncidentOps](../incidentops/). | | Key outputs | An [exception taxonomy](#templates) and log, a remote-assistance runbook, a black-box recording policy, a staged release process, fleet metrics with written definitions. | ### The problem RoboOps solves A five-robot pilot is run by the engineers who built it: they know each robot's quirks, update by hand and walk over when one gets stuck. The Robot Operations Group describes what scale does: 1–5 robots is the prototype phase, 5–50 a limited field deployment, and between 50 and 5,000 "most of what worked before stops working".[^rogmanifesto] | Without deliberate robot operations | What it looks like | |---|---| | Nobody owns the stuck robot | It waits in an aisle until someone notices; throughput drops unexplained. | | Hand-made configuration | Each robot differs slightly; a fix on one breaks another. | | No evidence after a failure | The robot rebooted; the logs that explain the near miss are gone. | | Updates without gates | A release that passed in the lab doubles protective stops at a site with different floors. | | Metrics nobody trusts | "Interventions" and "uptime" mean different things per vendor. | Physical consequences make this more than an efficiency problem: Ocado said a 2021 fire at its Erith warehouse "appears to have been caused by the collision of three bots on the grid".[^register2021] Robot fleets fail like distributed systems *and* like machines. ### What RoboOps is not - **Not a standard.** The standards it relies on are under [methods, standards and frameworks](#methods-standards-and-frameworks). - **Not ROS.** The Robot Operating System (ROS) is open-source robot middleware; many commercial robots do not expose it. - **Not robotic process automation (RPA).** Software "bots" that click through business applications are operated by an automation team ([AutomationOps](../automationops/)); this page is about physical machines. - **Not safety engineering.** Designing and validating safety functions (protective stops, speed and separation monitoring) is the job of the robot manufacturer and the integrator under the applicable standards. RoboOps keeps those functions intact and evidenced in daily operation. - **Editorial distinction:** this Atlas separates RoboOps from EdgeOps and IoTOps by **autonomy, motion and safety**: if the device can move itself or a load near people, the questions on this page apply. No standard defines that boundary. ## Where the label comes from and where it sits ### Origins and evolution | When | Event | Why it matters | |---|---|---| | 2019 | A conversation at the RoboBusiness show starts what becomes the Robot Operations Group (ROG)[^oitzman2020] | Operators lacked a forum | | Apr 2020 | ROG meets regularly; founders from InOrbit and Brain Corp; 54 individuals by October[^oitzman2020] | The label gets a community | | Sep 2020 | ROG releases the Robot Operations Manifesto[^inorbit2020] | "Autonomy exceptions" and four pillars | | 2022 | First RobOpsCon, Santa Cruz, California[^roghome] | Talks recorded and published | | Mar 2026 | VDA 5050 version 3.0.0 published on GitHub[^vda5050] | New major version of the main fleet interface | | Sep 2026 | InOrbit releases OpenRobOps (Apache 2.0), described as a reference implementation of the then-upcoming ISO 21423[^oro2026] | Open-source fleet operations | | Sep–Oct 2026 | ANSI/A3 R15.08-3-2026 for users of mobile robots announced;[^r15083] ISO 21423:2026 published on 14 October[^iso21423] | Users' duties and fleet interfaces get standards | | Oct 2026 | Third RobOpsCon (22 October, Mountain View); ROG reports a community of 400[^rogcon26] | Still a small community | The label's centre of gravity is commercial: ROG was co-founded by InOrbit's CEO, InOrbit sponsors it, sells RobOps software and hosts the 2026 conference.[^oitzman2020][^inorbit2020][^rogcon26] Treat vendor definitions as positions, not consensus. ### Glossary and mental model | Term | Meaning | Why it matters | |---|---|---| | Autonomy exception | A situation where the robot "cannot continue operating safely nor efficiently" on its own[^rogmanifesto] | The unit of work for operations | | Intervention | Any change to the robot's state by a person; one study splits them into minor, serious and fatal[^autoinspect] | What most autonomy metrics count | | Remote assistance | A remote person gives the robot information or a decision; the robot keeps control of its own motion | Motion decisions stay on the robot | | Teleoperation | A person drives the robot directly through a remote link | Latency and link loss become safety issues | | Protective stop / emergency stop | Safety-rated stops built into the robot or cell | Reported by fleet software, never re-implemented in it | | Operating mode | Who controls the robot: VDA 5050 3.0.0 defines AUTOMATIC, SEMIAUTOMATIC, INTERVENED, MANUAL, SERVICE, TEACH_IN and STARTUP;[^vda5050] version 2.1.0 has no INTERVENED or STARTUP and spells TEACHIN[^vda21] | Orders in the wrong mode are rejected | | Fleet manager (fleet control) | Software that assigns missions, manages traffic and talks to many robots | The integration point | | QoS profile | ROS 2 delivery rules per topic (reliability, durability, history, deadline, liveliness)[^rosqos] | Mismatches silently drop data | | Bag, MCAP, snapshot mode | A topic recording (`ros2 bag`), its default open file format, and a memory ring buffer written only on request[^rosbag2][^mcap] | A black box without filling disks | | Simulation (digital twin) | A virtual robot and site, for example in Gazebo, for testing changes[^gzreleases] | Repeatable regression tests | | A/B update | Two system slots: install into the inactive one, boot it, fall back if it fails[^raucscen] | Recoverable OTA without a site visit | | MTBI | Mean time between interventions: autonomous operating time per counted intervention[^autoinspect][^forest2025] | Headline autonomy metric; definition-sensitive | | Fan-out | Number of robots one person can supervise: (interaction effort + neglect time) / interaction effort[^olsen2003] | Upper bound for remote-ops staffing | **Mental model: three loops at three speeds.** The *safety loop* runs in milliseconds inside the robot; you never bypass it. The *exception loop* runs in minutes: detect, assist, resume or recover. The *improvement loop* runs in weeks: record, analyse, fix, release, check the rate fell. Many failures come from mixing loops, such as overriding a protective stop for convenience or rushing a weekly fix out as an emergency patch. ### How RoboOps relates to other Ops | Page | Shared ground | This page's focus | Go there for | |---|---|---|---| | [EdgeOps](../edgeops/) | Remote fleets, disconnection, staged rollouts | Machines that move; safety and autonomy exceptions | Gateways and edge servers, lightweight Kubernetes | | [IoTOps](../iotops/) | Device identity, OTA, telemetry | Robots with rich compute and actuators | Constrained devices, firmware, MQTT device platforms | | [FleetOps](../fleetops/) | The word "fleet", remote monitoring | Robot autonomy and exceptions | Vehicles, drivers, dispatch, fuel and running costs | | [MLOps](../mlops/) | Models on the robot, field data | Operating the robot that runs the model | Training, evaluating and serving perception models | | [ObservabilityOps](../observabilityops/) | Logs, metrics, traces | What to record on a robot and when | Telemetry pipelines and platforms | | [IncidentOps](../incidentops/) | Incident roles and reviews | Incidents with a physical side | Incident command, communications, MTTR definitions | | [ReleaseOps](../releaseops/) | Staged rollout and rollback | Cohorts by site and hardware; risk review | Release strategy and feature flags in general | ::: figure Where RoboOps sits among neighbouring Atlas pages ```mermaid flowchart TB E["`EdgeOps remote sites, disconnection`"]:::neutral I["`IoTOps constrained devices, firmware`"]:::neutral M["`MLOps perception models`"]:::neutral R["`RoboOps autonomy exceptions, safe updates, fleet telemetry, remote assistance`"]:::core O["`ObservabilityOps telemetry pipelines`"]:::neutral N["`IncidentOps incident command`"]:::neutral F["`FleetOps vehicle fleets`"]:::neutral S["`SupplyChainOps flow of goods`"]:::neutral E -->|"rollout and offline patterns"| R I -->|"device identity, OTA"| R M -->|"models to run"| R R -->|"hands telemetry to"| O R -->|"escalates incidents to"| N R -.-|"sibling: vehicles"| F R -.-|"keeps goods moving for"| S classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: EdgeOps contributes rollout and offline patterns, IoTOps device identity and OTA, and MLOps the models robots run, all into RoboOps (autonomy exceptions, safe updates, fleet telemetry, remote assistance). RoboOps hands telemetry to ObservabilityOps and escalates incidents to IncidentOps. Dotted lines link siblings: FleetOps (vehicles) and SupplyChainOps (the flow of goods the robots keep moving). ::: ## Principles and operating model ### Charter and principles The Robot Operations Manifesto groups the work into four pillars: **monitoring and observability**, **configuration management**, **safety, security and auditing**, and **interventions**, with interoperability and orchestration across all four.[^rogmanifesto] It draws on the Failure Reporting, Analysis and Corrective Action System (FRACAS) from reliability engineering.[^rogmanifesto] The charter below is **this Atlas's proposal**, built on those pillars; ROG publishes no charter of this form. **Mission (proposed).** Keep a robot fleet productive and safe at scale by turning every autonomy exception into either a fast recovery or a permanent fix. | Principle | What it means in practice | Evidence or reason | |---|---|---| | 1. Safety outranks throughput | No operational shortcut disables, bypasses or "temporarily" relaxes a safety function | ROG calls safety "the most critical element";[^rogmanifesto] OSHA's inspection guide checks that safeguards "are not disabled or removed"[^osha] | | 2. Expect exceptions | Design assistance, recovery and escalation before go-live | "accepting that autonomy exceptions will occur"[^rogmanifesto] | | 3. Every robot is reproducible | Software, maps, configuration and calibration are versioned per robot, with an audit trail and rollback | ROG's configuration-management pillar[^rogmanifesto] | | 4. Evidence first | Record the seconds before and after every exception automatically | A snapshot holds only what preceded its trigger; a second one records the aftermath ([Lab 3](#lab-3-advanced-a-black-box-recorder-for-a-ros-2-robot)) | | 5. Motion changes are safety changes | A new route, zone, speed or task triggers a review of the site risk assessment | OSHA's inspection guide checks for task-based risk assessments for "new or modified" tasks before they are performed[^osha] | | 6. A human is always reachable | Automation complements people; it does not remove the escalation path | Runbook automation "can complement but usually not completely replace human interventions"[^rogmanifesto] | | 7. Define before you measure | Every metric has a written definition and data source | See [Lab 1](#lab-1-beginner-compute-mtbi-from-an-intervention-log) | **Non-goals.** Zero interventions (a target that encourages hiding them); remote driving as the normal recovery; replacing the manufacturer's or integrator's safety engineering. ### Fundamentals: the robot as a distributed system ROG's manifesto asks operators to think of "a fleet of robots as a distributed system, with compute/storage/networking on each robot and often in the cloud".[^rogmanifesto] The building blocks are the **robot** (safety controller and sensors, autonomy stack, on-board computer, operations agent, batteries), the **site** (fleet manager, Wi-Fi, maps and zones, chargers, doors and lifts, people) and **operations** (console, telemetry, release service, simulation CI, ticketing). **ROS 2 essentials for operators.** Where a fleet runs ROS 2, a *node* should "do one logical thing";[^rosnodes] *topics* carry continuous streams such as sensor data;[^rostopics] *services* are short requests and "should never be used for longer running processes";[^rosservices] *actions* are long-running goals with feedback and cancellation, such as driving to a waypoint.[^rosactions] Two details matter most to operators: - **Quality of Service (QoS).** ROS 2 "can be as reliable as TCP or as best-effort as UDP".[^rosqos] A best-effort publisher and a reliable subscriber do not connect at all; defaults are keep-last 10, reliable and volatile, while the sensor-data profile uses best effort.[^rosqos] A dashboard subscribing with the wrong profile sees nothing, which looks like a dead sensor. - **Security is opt-in.** ROS 2 security (SROS2, built on the Data Distribution Service's DDS-Security) uses a keystore and per-node enclaves, enabled through `ROS_SECURITY_KEYSTORE`, `ROS_SECURITY_ENABLE=true` and `ROS_SECURITY_STRATEGY=Enforce`; without them, nodes communicate unauthenticated. Once enforced, every operations tool (recorder, dashboard bridge, command line via `ROS_SECURITY_ENCLAVE_OVERRIDE`) needs its own enclave, and permissions files can limit each to the topics it needs.[^sros2] As of October 2026 the supported ROS 2 distributions are Lyrical Luth (released 22 May 2026, end of life May 2031), Kilted Kaiju (to December 2026), Jazzy Jalisco (to May 2029) and Humble Hawksbill (to May 2027);[^rosreleases] ROS 1 ended on 31 May 2025.[^noeticeol] Recommended Gazebo pairings are Humble–Fortress, Jazzy–Harmonic, Kilted–Ionic and Lyrical–Jetty.[^gzros] Plan upgrades by date. ### Roles, ownership and RACI Safety standards split duties between **manufacturers**, **integrators** and **users** (see [standards by robot class](#comparing-safety-standards-by-robot-class)), and RoboOps should follow the same split. For industrial mobile robots (IMRs) in the United States, ANSI/A3 R15.08-2 covers "integrating, configuring, and customizing an IMR or fleet of IMRs into a site",[^a3r1508] and ANSI/A3 R15.08-3-2026 covers users, including, according to a trade report of A3's announcement, "risk assessment, management of change, employee training and periodic reviews and inspections";[^r15083] a second report lists the same duties.[^r15083ran] (This research read reports, not the standard.) On this page those duties appear as the [site-readiness checklist](#checklists), change classes A–C, training in the lifecycle and periodic inspection on Day 2. The operating organisation adds roles of its own. The **remote assistant** gives first response within the actions the runbook allows; the **field technician** does physical recovery, stop resets and hardware swaps; the **site lead** owns local priorities, layout changes and the site risk assessment; the **release owner** owns builds, gates and rollback; the **safety officer** owns risk assessments and near-miss investigations; the **fleet operations manager** owns metrics, staffing and vendors. | Activity | Remote assistant | Field technician | Site lead | Release owner | Safety officer | |---|---|---|---|---|---| | Resolve an exception remotely (allowed actions only) | R | I | I | C | – | | Recover a robot after an emergency stop | I | R | A | – | C | | Release software to a site cohort | I | I | C | R/A | C | | Change a map, zone or speed limit | – | C | A | R | C | | Update the site risk assessment | – | C | R | C | A | | Investigate a near miss | C | C | C | C | R/A | R = responsible, A = accountable, C = consulted, I = informed (illustrative). Robots-as-a-service contracts often move the release and remote-assistance rows to the vendor; the site's "I" entries then become contractual requirements. ### Lifecycle: Day 0, Day 1, Day 2 | Phase | Work | Exit criteria | |---|---|---| | Day 0: plan and build | Site survey (floors, ramps, lighting, Wi-Fi coverage along routes); maps and zones; risk assessment with the integrator; exception taxonomy; black-box and data policy; simulation of the site | Signed risk assessment; written exception classes; simulated missions pass | | Day 1: commission and pilot | Install and commission; a small cohort runs supervised; remote assistance in shadow mode; baseline metrics | Agreed MTBI and assistance-time baseline; runbook tested in a drill | | Day 2: operate and improve | Exception loop; weekly corrective-action review; staged releases; layout changes through change control; periodic safety-function and battery inspections; refresher training | Exception rate trending down; no open safety actions past due | | Retire | Remove robots from service; erase site data and credentials; battery handling and recycling as local rules require | Asset register closed; access revoked | ### Operating model and governance The governance core is ROG's **lifecycle of failures** loop.[^rogmanifesto] ::: figure The lifecycle of failures: from exception to corrective action ```mermaid flowchart TB A["Autonomy exception detected"]:::risk B["`Resolve now remote assist or field visit`"]:::core C["`Record exception log + black-box snapshot`"]:::core S{"Safety-relevant?"}:::decision Q["`Quarantine the robot; safety officer investigates it as a near miss`"]:::risk D{"Recurring?"}:::decision E["`Analyse root cause weekly review`"]:::core F["`Corrective action software, map, site or procedure`"]:::core G["`Staged release with gates`"]:::core H{"Did the rate fall?"}:::decision K["Keep for trend analysis"]:::neutral Z["Close the corrective action"]:::core A --> B --> C --> S S -->|yes| Q --> E S -->|no| D D -->|no| K D -->|yes| E --> F --> G --> H H -->|no| E H -->|yes| Z classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 ``` Text equivalent: an autonomy exception is detected, resolved now (remote assistance or a field visit) and recorded with a black-box snapshot. A safety-relevant one leads to quarantine and a near-miss investigation by the safety officer, which feeds root-cause analysis. Otherwise a one-off stays in the log for trend analysis, and a recurring one goes to root-cause analysis at the weekly review, a corrective action (software, map, site or procedure), a staged release with gates, and a check that the rate fell: if not, analysis starts again; if so, the action is closed. ::: | Cadence | Forum | Decisions | |---|---|---| | Every shift | Handover | Open exceptions, robots out of service, quarantines | | Weekly | Corrective-action review | Root causes, owners, release candidates | | Per release | Release board | Go, hold, or narrow the cohort | | Monthly | Site review | Targets, staffing, vendor escalations | | Quarterly | Safety review | Near misses, drills, training, audits | **Change classes (proposed).** *Class A* changes cannot affect motion (dashboards, telemetry fields): normal review. *Class B* changes can (navigation parameters, maps, zones, speeds, new tasks): release board plus a check of the risk assessment. *Class C* changes touch safety configuration (protective fields, safety-rated speeds): only through the manufacturer's or integrator's safety process, never through the fleet's OTA pipeline on operations authority alone. ## Architecture and workflow ### Reference architecture The architecture is vendor-neutral: its trust boundaries matter more than the products. ::: figure Reference architecture for an operated robot fleet ```mermaid flowchart TB RC["`OPS: remote-assistance console`"]:::ops FM["`SITE: fleet manager missions, traffic`"]:::site IA["`SITE: doors, lifts, chargers`"]:::site SF["`ROBOT: safety controller safety-rated, local only`"]:::risk AS["`ROBOT: autonomy stack ROS 2 or vendor`"]:::robot AG["`ROBOT: ops agent and black-box recorder`"]:::robot TL["`OPS: telemetry and exception log`"]:::ops SC["OPS: simulation CI"]:::ops RL["`OPS: release and OTA service`"]:::ops RC -->|assist actions| FM FM <--> IA FM <-->|orders, state| AS SF -.->|stop overrides| AS SF -.->|stop status, read-only| AG AS --> AG AG -->|health, snapshots| TL TL -->|exception queue| RC TL --> SC SC --> RL RL -->|signed release| AG classDef robot fill:#e8f3f2,stroke:#16756f,color:#122029 classDef site fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef ops fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 ``` Text equivalent: node labels name the layer (OPS for operations, SITE, ROBOT). The telemetry and exception log fills the remote-assistance console's exception queue; the console sends assist actions to the site's fleet manager, which talks to doors, lifts and chargers and exchanges orders and state with each robot's autonomy stack. The robot's safety controller (safety-rated, local only) can stop the autonomy stack and reports stop status, read-only, to the operations agent and black-box recorder; nothing in operations sends it commands. The agent sends health and snapshots to the exception log, which feeds simulation CI, which gates the release and OTA service, which sends signed releases back to the agent. ::: | Component | Design notes | |---|---| | Operations agent | Applies configuration and releases, supports rollback, leaves an audit trail (ROG's configuration-management pillar)[^rogmanifesto] | | Black-box recorder | Keeps the last minutes of key topics in memory and writes them on a trigger, as `ros2 bag` snapshot mode does[^rosbag2] | | Remote-assistance console | Exception queue from the exception log, camera and map views; actions go through the fleet manager's API, never around it | | Release and OTA service | Signed artefacts; software, maps and configuration released together, by cohort | **Trust boundaries.** (1) The safety layer is local and has no remote write path. (2) Remote staff act through the fleet manager, which enforces the operating mode; VDA 5050 3.0.0, for example, forbids orders and actions in `MANUAL` and only allows `cancelOrder` as an instant action in `INTERVENED`.[^vda5050] (3) Robots and their middleware are never reachable from the public internet: a 2018 scan found "over 100 publicly-accessible hosts" running a ROS 1 master, which "trusts all nodes that connect to it".[^demarinis2018] (4) Releases are signed, and the robot verifies them before installing. **Trade-offs.** Vendor choices are weighed in the [decision framework](#decision-framework). Continuous logging gives the richest evidence at a cost in storage, bandwidth and privacy; snapshots are cheaper but miss slow-building problems unless health metrics also stream. ### End-to-end workflow: an exception from detection to closure **Trigger:** blocked path, lost localization, failed pick, a door that did not open. **Outputs:** a robot back in service and an exception record. **Feedback:** the weekly review. ::: figure Handling one autonomy exception: remote path and field path ```mermaid sequenceDiagram participant R as Robot participant F as Fleet manager participant A as Assistant participant T as Technician R->>F: Blocked, snapshot saved F->>A: Ticket with camera view alt Solvable remotely A->>F: Confirm detour F->>R: Updated order else Not solvable remotely A->>T: Dispatch to robot Note over T: Recovers on site T->>A: Back in AUTOMATIC A->>F: Return to service F->>R: New order end R->>F: Driving again ``` Text equivalent: the robot reports that it is blocked and a snapshot was saved; the fleet manager opens a ticket with camera and map views for the remote assistant. If remote resolution works, the assistant confirms a detour and the fleet manager sends an updated order. If not, a field technician recovers the robot on site and reports it back in automatic mode; the assistant returns it to service and the fleet manager sends a new order. Either way, the robot reports that it is driving again. ::: **Exception path.** If no allowed action fits, the robot is in a protective or emergency stop, or the camera shows damage, smoke or people at risk: quarantine it, recover it on site, and return it only when the cause is understood. Several robots with the same class within minutes is an incident ([IncidentOps](../incidentops/)): usually a site change or a release, not bad luck. **Remote assistance, not remote driving.** In Waymo's fleet response, the vehicle asks questions, remote agents answer or propose a path, and the automated driver "is in control of the vehicle at all times"; if stopping is safest it stops.[^waymo2024] The lesson transfers from vehicles ([FleetOps](../fleetops/)): motion decisions stay on the robot. | Assist actions usually safe to allow remotely | Actions to keep local or forbidden | |---|---| | Confirm an obstacle is static; choose a pre-approved alternative route | Reset an emergency stop that requires acknowledgement on the robot (VDA 5050 `MANUAL` type)[^vda5050] | | Cancel the mission; send to a parking spot or charger | Disable or shrink protective fields | | Trigger a snapshot; mark the robot for quarantine | Teleoperate out of line of sight beyond what the risk assessment covers | | Relocalize only where the robot confirms the pose against its sensors | Push an untested configuration to "unstick" one robot | ## Practices, frameworks and tools ### Core practices and playbooks | Practice | Trigger | Owner | Output | Verification | |---|---|---|---|---| | Exception triage | Exception raised | Remote assistant | Closed record with class and cause | Time to resume; reopen rate | | Black-box capture | Exception, protective stop, request | Release owner | Snapshot linked to the record | Share of serious records with a snapshot | | Quarantine | Repeated exceptions, damage, unknown state | Remote assistant | Robot blocked from missions | Field sign-off before return | | Staged release | New software, map or configuration | Release owner | Cohort plan, gate results | Exception rate per cohort | | Map and zone change control | Layout change, new racking | Site lead | Versioned map, risk check | Localization exceptions after change | | Drills | Quarterly | Safety officer | Drill report | Time to safe state | | Periodic inspection | Manufacturer's schedule, risk assessment | Field technician | Tested emergency stops, scanners and bumpers; battery and charger health record | No overdue tests; failed units out of service | | Scenario regression | Each serious exception class | Robotics engineers | Simulation test case | Runs in CI on every change | **Playbook: recover after an emergency stop (template).** (1) Find out why it was pressed. (2) Make the area safe; leave the robot stopped. (3) Inspect robot and load on site. (4) Reset locally per the manufacturer's procedure. (5) Move it in manual mode to a known position. (6) Return it to automatic mode, quarantined until the record is reviewed. (7) Treat any unexplained stop as a near miss. ### Methods, standards and frameworks | Type | Item | Status (as of October 2026) | |---|---|---| | Regulation (EU) | Machinery Regulation (EU) 2023/1230 | Machinery placed on the market before 20 January 2027 follows the Machinery Directive; adds provisions for AI-powered safety functions and cyber-safety[^eumachinery] | | Interface | VDA 5050, fleet control to mobile robot over MQTT and JSON | Version 3.0.0 (2026); 2.1.0 (2024) uses other field names; "optional and non-binding"; not a safety standard; no cybersecurity specification[^vda5050][^vda21] | | Interface | ISO 21423:2026, communication between IMRs, their fleet managers and site equipment such as lifts and doors | Published 14 October 2026; "This document is not safety-related"; excludes medical, consumer, entertainment, military and public-road uses[^iso21423] | | Interface | MassRobotics AMR Interoperability Standard | JSON schema with example sender and receiver[^massrobotics] | | Open source | Open-RMF, multi-fleet traffic and shared doors and lifts | Managed by the Open Source Robotics Alliance since 2024[^openrmf][^rmfbook] | | Practice | Robot Operations Manifesto | Community document, no conformance criteria[^rogmanifesto] | #### Comparing safety standards by robot class | Robot class | Standards (as of October 2026) | Duties covered | |---|---|---| | Industrial robot arms and cells | ISO 10218-1:2025 and -2:2025, replacing the 2011 editions and absorbing the collaborative content of ISO/TS 15066; Part 2 adds cybersecurity requirements pertaining to robot safety[^a3iso10218][^awiso10218] | Manufacturer, integrator; in the US, ANSI/A3 R15.06 adds a Part 3 for users[^r15083] | | Industrial mobile robots in the US | ANSI/RIA R15.08-1-2020, ANSI/A3 R15.08-2 (2023) and ANSI/A3 R15.08-3-2026[^a3r1508][^r15083] | Manufacturer, integrator, user | | Driverless industrial trucks, including AMRs | ISO 3691-4:2023, under revision; ISO lists "autonomous mobile robot" among its examples and excludes remotely controlled trucks[^isoscope] | Safety requirements and their verification | | Service robots in hospitals and public spaces | The standards above are written for industrial robots and trucks; ISO 13482:2014 covers personal care robots, excludes robots as medical devices and is being revised[^isoscope] | Ask the manufacturer which standards it claims | These rows rest on publishers' summaries, ISO catalogue entries and trade reports; conformance needs the standards' texts and usually a qualified safety engineer. OSHA has no robot-specific regulation; its technical manual chapter predates the 2025 ISO revision.[^osha][^a3iso10218] ### Tooling landscape | Category | Examples (verified October 2026) | Selection criteria and risks | |---|---|---| | Robot middleware | ROS 2[^rosreleases] | Distribution end-of-life dates; vendor support | | Simulation | Gazebo (Jetty and Harmonic are long-term support releases; Gazebo Classic ended in January 2025)[^gzreleases][^gzclassic] | Fidelity of the sensors you depend on; headless runs in CI | | Recording and replay | `ros2 bag` with MCAP;[^rosbag2][^mcap] Foxglove for visualisation and fleet data[^foxglove] | Storage, upload bandwidth, camera privacy | | Health reporting | ROS `diagnostics` (`/diagnostics`, aggregator, remote logging)[^diagnostics] | Every driver must publish meaningful status | | Fleet operations platforms | InOrbit;[^inorbitrobops] Formant;[^formant] Boston Dynamics Orbit;[^bdorbit] OpenRobOps (Apache 2.0)[^oro2026] | Vendor coverage, on-premises option, data export, exit plan | | Multi-fleet coordination | Open-RMF;[^openrmf] VDA 5050 implementations[^vda5050] | Which vendors implement which version | | OTA and system update | RAUC (A/B slots, boot confirmation)[^raucbasic] | Bootloader support, watchdog, space for two slots | | CI for robot software | `action-ros-ci`,[^actionrosci] `industrial_ci`[^industrialci] | Match the robot's distribution and base image | **Retired tools are part of the landscape:** AWS RoboMaker (2018–2025) is gone.[^robomaker] Keep simulation and fleet tooling portable (containers, MCAP, exportable data) so a vendor decision does not strand your test suite. ### Automation and AI Automate actions that are frequent, reversible and do not change motion near people: tickets, attaching snapshots, quarantine after repeated faults, sending a robot to charge, grouping exceptions. Changes to what robots do in shared space (maps, speeds, zones, releases) need human approval (principle 6 above). AI is entering the operations layer: Boston Dynamics advertises vision-language models in Orbit,[^bdorbit] and RobOpsCon 2026 features agentic software for robot operations.[^rogcon26] Treat a generated root cause as a hypothesis for the weekly review. AI inside a **safety function** is regulated territory: the EU Machinery Regulation adds provisions for it.[^eumachinery] ## RoboOps in practice ### Reference implementation **Assumptions (illustrative scenario):** a 40-robot mobile-robot fleet on ROS 2 Jazzy at two sites, one remote-assistance desk, releases built in GitHub Actions. Three artefacts make the operating model concrete: an exception record, a release policy and a CI workflow. The schemas are this page's proposals, not a standard. _Example status: syntax-checked by the Atlas validator; not executed (illustrative data)._ ```json { "exception_id": "EX-2026-10-05-0042", "robot": "amr-01", "site": "site-a", "raised_at": "2026-10-05T09:40:12Z", "operating_mode": "AUTOMATIC", "class": "serious", "category": "localization", "symptom": "mislocalized near dock 3; robot holding position", "resolution": "onsite", "robot_stopped_minutes": 25, "snapshot": "s3://fleet-evidence/site-a/amr-01/2026-10-05T09-40-12Z/", "release": {"software": "nav-4.12.1", "map": "site-a-map-37", "config": "site-a-cfg-19"}, "safety_relevant": false, "linked_exceptions": ["EX-2026-10-05-0040", "EX-2026-10-05-0041"], "corrective_action": "CA-118" } ``` The release policy treats software, map and configuration as one unit and names the gates that stop a rollout. _Example status: syntax-checked by the Atlas validator; the policy format is illustrative and not tied to a product._ ```yaml release: nav-4.13.0+site-a-map-37+site-a-cfg-20 change_class: B # can affect motion: release board + risk-assessment check install_window: "robot docked and charging, no active order" cohorts: - name: canary robots: [amr-07, amr-12] # one per hardware revision at site-a min_hours: 48 # wall-clock hours gates: {failed_boots: 0, fatal_exceptions: 0} # too few hours for rate gates - name: site-a robots: all-at-site-a hold_back: 4 # stay on the previous release as a control min_robot_hours: 400 # about 12 protective stops expected at baseline gates: # fail above 1.5 x the expected baseline count protective_stops_per_100h: {baseline: 2.9, max_ratio: 1.5} serious_exceptions_per_100h: {baseline: 4.0, max_ratio: 1.5} - name: site-b robots: all-at-site-b hold_back: 4 min_robot_hours: 400 gates: # baselines from site-b's own history protective_stops_per_100h: {baseline: 3.4, max_ratio: 1.5} serious_exceptions_per_100h: {baseline: 4.6, max_ratio: 1.5} rollback: "switch boot slot (A/B) and restore previous map and config" ``` **Why gates are sized in robot-hours.** At 2.9 protective stops per 100 robot-hours, two canary robots expect two or three stops in 48 hours, depending on shifts: a "no increase" rule would fail an unchanged release about half the time. At 400 robot-hours (about 12 expected stops) with a limit of 1.5 times the expected count, an unchanged release trips the gate about 5% of the time and a doubled rate about 89% (Poisson arithmetic). Canaries catch crashes and install faults; site cohorts catch rate changes; held-back robots give a concurrent comparison when baselines drift with season or staffing. The CI workflow builds and tests the ROS 2 packages on every change, using the recommended base container of `action-ros-ci`.[^actionrosci] _Example status: syntax-checked; not executed (needs a GitHub repository with a ROS 2 package). The action is pinned to the commit of release 0.4.8 and the job token is read-only; replace `` with the image digest you tested._ ```yaml name: robot-software-ci on: [pull_request] permissions: contents: read jobs: build_and_test: runs-on: ubuntu-latest container: # pin the image you tested: the ubuntu-noble-latest tag moves image: rostooling/setup-ros-docker@sha256: steps: - name: Build and run unit and launch tests uses: ros-tooling/action-ros-ci@3a640b10f09b756dabe556dac5413aba369f71b0 # 0.4.8 with: package-name: fleet_nav_bringup target-ros2-distro: jazzy ``` Integration tests written with `launch_testing` run under `colcon test` in the same job.[^rosintegration] For scenario tests, Gazebo can run without a GUI: `gz sim -s -r --headless-rendering --iterations 20000 ` runs only the server, starts the simulation immediately and stops after a fixed number of iterations (flags checked in the Jetty and Harmonic sources).[^gzcli] Each serious exception class should add one scenario: the dock where `amr-01` lost localization becomes a world file and a test. ### Hands-on labs #### Lab 1 (beginner): compute MTBI from an intervention log **Objective:** see how much a definition changes the headline autonomy metric. **Prerequisites:** Python 3.8 or later; no robot, cost or risk; 20 minutes. The 13 interventions are illustrative data for four robots over a week (448 operating hours). _Example status: executed on 2026-10-18 with Python 3.8.10; the output below is from that run._ ```text robot,start,cls,resolved,minutes,cause amr-01,2026-10-05T07:12,minor,remote,1,manual mission added amr-01,2026-10-05T09:40,serious,remote,4,mislocalized near dock amr-01,2026-10-05T09:47,serious,remote,3,mislocalized near dock amr-01,2026-10-05T09:52,serious,onsite,18,mislocalized near dock amr-02,2026-10-06T11:05,serious,remote,5,path blocked by pallet amr-02,2026-10-07T14:30,fatal,onsite,42,navigation stack crash amr-03,2026-10-05T16:20,minor,remote,1,map annotation edit amr-03,2026-10-08T08:02,serious,remote,6,path blocked by pallet amr-03,2026-10-09T13:15,serious,remote,4,lift door timeout amr-04,2026-10-06T10:10,serious,remote,3,path blocked by pallet amr-04,2026-10-06T10:16,serious,remote,4,path blocked by pallet amr-04,2026-10-10T15:45,fatal,onsite,55,drive fault after protective stop amr-04,2026-10-11T09:30,minor,remote,1,manual mission added ``` ```python """Compute intervention metrics for a robot fleet from an intervention log (CSV). Minor interventions (normal use) can be excluded; interventions on one robot that start within cluster_min minutes of the previous one can be grouped. MTBI = operating hours / counted events. """ import csv import sys from collections import Counter from datetime import datetime # 7 days x 16 h per robot OPERATING_HOURS = {"amr-01": 112, "amr-02": 112, "amr-03": 112, "amr-04": 112} def load(path): with open(path, newline="") as f: rows = list(csv.DictReader(f)) for r in rows: r["start"] = datetime.strptime(r["start"], "%Y-%m-%dT%H:%M") r["minutes"] = int(r["minutes"]) return sorted(rows, key=lambda r: (r["robot"], r["start"])) def group(rows, cluster_min): """Merge a robot's interventions that start within cluster_min of the previous one.""" events, open_event, last_start = [], {}, {} for r in rows: robot = r["robot"] gap = None if robot in last_start: gap = (r["start"] - last_start[robot]).total_seconds() if gap is not None and gap <= cluster_min * 60: open_event[robot]["members"].append(r) # same underlying problem else: open_event[robot] = {"robot": robot, "members": [r]} events.append(open_event[robot]) last_start[robot] = r["start"] return events def report(rows, cluster_min, count_minor): hours = sum(OPERATING_HOURS.values()) kept = [r for r in rows if count_minor or r["cls"] != "minor"] events = group(kept, cluster_min) if cluster_min else [{"members": [r]} for r in kept] n = len(events) onsite = sum(1 for e in events if any(m["resolved"] == "onsite" for m in e["members"])) lost = sum(m["minutes"] for e in events for m in e["members"]) return n, hours / n if n else float("inf"), 100.0 * n / hours, onsite, lost def main(): rows = load(sys.argv[1] if len(sys.argv) > 1 else "interventions.csv") hours = sum(OPERATING_HOURS.values()) print("fleet operating hours:", hours) print("%-34s %6s %9s %12s %7s %9s" % ( "definition", "events", "MTBI (h)", "per 100 h", "onsite", "lost min")) for label, cluster, minor in [("all interventions, no grouping", 0, True), ("exclude minor, no grouping", 0, False), ("exclude minor, group within 15 min", 15, False)]: n, mtbi, rate, onsite, lost = report(rows, cluster, minor) print("%-34s %6d %9.1f %12.2f %7d %9d" % (label, n, mtbi, rate, onsite, lost)) causes = Counter(r["cause"] for r in rows if r["cls"] != "minor") print("top causes (non-minor):") for cause, n in causes.most_common(3): print(" %s=%d" % (cause, n)) if __name__ == "__main__": main() ``` 1. Save the data as `interventions.csv` and the script as `mtbi.py` in an empty folder. 2. Run `python3 mtbi.py`. **Expected output:** ```text fleet operating hours: 448 definition events MTBI (h) per 100 h onsite lost min all interventions, no grouping 13 34.5 2.90 3 147 exclude minor, no grouping 10 44.8 2.23 3 144 exclude minor, group within 15 min 7 64.0 1.56 3 144 top causes (non-minor): path blocked by pallet=4 mislocalized near dock=3 navigation stack crash=1 ``` **Validation:** the same week reads as 34.5, 44.8 or 64.0 hours MTBI. Published studies state such choices: one inspection deployment excluded minor interventions,[^autoinspect] and a forest-robot study merged safety-operator commands within 10-second windows.[^forest2025] The top cause points at the pallet staging area, not the navigation software. **Troubleshooting:** `FileNotFoundError` means the CSV is not in the working folder; `ValueError` means a malformed timestamp. **Teardown:** delete the folder. #### Lab 2 (intermediate): size the remote-assistance desk **Objective:** find how many remote assistants a fleet needs, and why fan-out alone overstates it. **Prerequisites:** Python 3.8 or later; no cost or risk; 30 minutes. **Model (assumptions):** each robot works for a random time averaging NT = 90 minutes, then stops with an exception; it waits for a free assistant, who needs on average IE = 4 minutes; both times are exponential. Fan-out is (IE + NT) / IE.[^olsen2003] _Example status: executed on 2026-10-18 with Python 3.8.10 (seeded, deterministic); output below is from that run._ ```python """Simulate a remote-assistance desk: robots stop on an autonomy exception and wait for an operator. Each robot works for an exponentially distributed time (mean NT minutes), raises an exception, waits for a free operator, needs IE minutes of operator attention (exponential), then resumes. """ import heapq import random import sys def simulate(robots, operators, nt, ie, minutes, seed): rng = random.Random(seed) events = [(rng.expovariate(1 / nt), "exception", r) for r in range(robots)] heapq.heapify(events) free, queue, waits, busy, stopped = operators, [], [], 0.0, 0.0 while events: t, kind, robot = heapq.heappop(events) if t > minutes: break if kind == "exception": queue.append((t, robot)) else: # an operator finished helping a robot free += 1 heapq.heappush(events, (t + rng.expovariate(1 / nt), "exception", robot)) while free and queue: raised, r = queue.pop(0) service = rng.expovariate(1 / ie) waits.append(t - raised) stopped += (t - raised) + service busy += service free -= 1 heapq.heappush(events, (t + service, "resolved", r)) waits.sort() p90 = waits[int(0.9 * (len(waits) - 1))] if waits else 0.0 return {"exceptions": len(waits), "util": busy / (operators * minutes), "mean_wait": sum(waits) / len(waits) if waits else 0.0, "p90_wait": p90, "productive": 1 - stopped / (robots * minutes)} def main(): robots = int(sys.argv[1]) if len(sys.argv) > 1 else 40 # NT: mean minutes between exceptions; IE: mean operator minutes per exception nt = float(sys.argv[2]) if len(sys.argv) > 2 else 90.0 ie = float(sys.argv[3]) if len(sys.argv) > 3 else 4.0 minutes = 30 * 16 * 60 # 30 days of 16-hour operation fan_out = (ie + nt) / ie print("fan-out (IE+NT)/IE = %.1f robots per operator (upper bound)" % fan_out) print("%-9s %10s %6s %10s %9s %11s" % ( "operators", "exceptions", "util", "mean wait", "p90 wait", "productive")) for ops in (1, 2, 3, 4): s = simulate(robots, ops, nt, ie, minutes, seed=11) print("%-9d %10d %5.0f%% %8.1f m %7.1f m %10.1f%%" % ( ops, s["exceptions"], 100 * s["util"], s["mean_wait"], s["p90_wait"], 100 * s["productive"])) if __name__ == "__main__": main() ``` 1. Save as `desk.py`; run `python3 desk.py` (40 robots, NT 90, IE 4). 2. Run `python3 desk.py 40 45 4` (exceptions twice as frequent). **Expected output** (step 1): ```text fan-out (IE+NT)/IE = 23.5 robots per operator (upper bound) operators exceptions util mean wait p90 wait productive 1 7256 100% 64.3 m 97.9 m 57.0% 2 11702 81% 5.2 m 14.1 m 90.7% 3 12312 57% 0.8 m 3.0 m 94.9% 4 12327 43% 0.1 m 0.0 m 95.6% ``` **Validation:** two assistants cover 20 robots each, below the fan-out of 23.5, yet the p90 wait is 14 minutes because random arrivals queue at 81% utilisation; three bring it to about three minutes. In step 2 fan-out drops to 12.2 and four assistants give a 5.5-minute p90 wait (79% utilisation). Interaction effort is mostly cognitive (remembering robot state, interface modes and robot abilities) and rises with fan-out,[^olsen2003] so the model is optimistic. **Troubleshooting:** `ValueError` means a non-numeric argument; `ZeroDivisionError` means NT or IE was 0. **Teardown:** delete the file. #### Lab 3 (advanced): a black-box recorder for a ROS 2 robot **Objective:** keep the last minutes of key topics in memory and save them automatically when any component reports an error. **Prerequisites:** a test machine or simulator with ROS 2 Jazzy or later and the `diagnostics` packages; never a robot in production; 45 minutes. **Risk:** none to motion; the cache is double-buffered, so memory use can reach twice `--max-cache-size` (about 400 MB here) plus overhead.[^rosbag2record] _Example status: syntax-checked; not executed (no ROS 2 installation in this environment). Commands follow the rosbag2 Jazzy documentation and source._[^rosbag2][^rosbag2record][^rosbag2src] ```bash # Terminal 1: recorder in snapshot mode, 200 MB per buffer, MCAP (the default format) ros2 bag record --snapshot-mode --max-cache-size 200000000 \ --topics /diagnostics /tf /tf_static /scan /odom /cmd_vel -o blackbox # Terminal 2: the trigger node below python3 blackbox_trigger.py # Terminal 3: inject a fake fault, wait for the second snapshot, inspect the closed files python3 inject_fault.py sleep 25 ls blackbox ros2 bag info blackbox/blackbox_0.mcap ros2 bag info blackbox/blackbox_1.mcap ``` ```python """Call the rosbag2 snapshot service when any /diagnostics status reaches ERROR or STALE, and again POST_S seconds later, so that the record also shows the aftermath.""" import time import rclpy from diagnostic_msgs.msg import DiagnosticArray, DiagnosticStatus from rclpy.node import Node from rosbag2_interfaces.srv import Snapshot COOLDOWN_S = 60.0 # at most one triggered snapshot per minute POST_S = 20.0 # a snapshot holds only earlier data; a second one shows the aftermath def level_value(level): # byte fields arrive as bytes in rclpy return level if isinstance(level, int) else ord(level) class BlackBoxTrigger(Node): def __init__(self): super().__init__("blackbox_trigger") self.client = self.create_client(Snapshot, "/rosbag2_recorder/snapshot") self.create_subscription(DiagnosticArray, "/diagnostics", self.on_diagnostics, 10) self.last = 0.0 self.post_timer = None def snapshot(self, why): self.client.call_async(Snapshot.Request()) self.get_logger().warning("snapshot requested: %s" % why) def on_diagnostics(self, msg): error = level_value(DiagnosticStatus.ERROR) bad = [s for s in msg.status if level_value(s.level) >= error] if not bad or time.monotonic() - self.last < COOLDOWN_S: return if not self.client.service_is_ready(): self.get_logger().error( "snapshot service not available; is the recorder in --snapshot-mode?") return self.last = time.monotonic() self.snapshot("%s: %s" % (bad[0].name, bad[0].message)) self.post_timer = self.create_timer(POST_S, self.on_post_trigger) def on_post_trigger(self): self.post_timer.cancel() # one-shot self.snapshot("post-trigger") def main(): rclpy.init() rclpy.spin(BlackBoxTrigger()) if __name__ == "__main__": main() ``` ```python """Publish one fake ERROR on /diagnostics to test the black-box trigger.""" import time import rclpy from diagnostic_msgs.msg import DiagnosticArray, DiagnosticStatus def main(): rclpy.init() node = rclpy.create_node("inject_fault") pub = node.create_publisher(DiagnosticArray, "/diagnostics", 10) time.sleep(1.0) # allow discovery status = DiagnosticStatus(level=DiagnosticStatus.ERROR, name="drive_left", message="test fault: motor overcurrent", hardware_id="amr-test") pub.publish(DiagnosticArray(status=[status])) time.sleep(0.5) node.destroy_node() rclpy.shutdown() if __name__ == "__main__": main() ``` **Expected output:** the trigger logs `snapshot requested: drive_left: test fault: motor overcurrent` and, 20 seconds later, `snapshot requested: post-trigger`. `ls blackbox` shows `.mcap` files but no `metadata.yaml`: each snapshot closes one file, and rosbag2 writes `metadata.yaml` only when the recorder stops.[^rosbag2src] `blackbox_0.mcap` holds messages from before the fault, `blackbox_1.mcap` those from the 20 seconds after it. **Validation:** levels (OK 0, WARN 1, ERROR 2, STALE 3) come from `diagnostic_msgs`;[^diagstatus] the service name and type from the recorder source;[^rosbag2record] the second file holds only newer messages because the recorder clears the buffer it has written.[^rosbag2src] **Troubleshooting:** "snapshot service not available" means the recorder lacks `--snapshot-mode` or uses another `--node-name`; "Could not find metadata in bag directory" means `ros2 bag info` was given the directory while the recorder runs; an empty file usually means different topic names or a [QoS](#fundamentals-the-robot-as-a-distributed-system) mismatch. **Teardown:** stop all terminals (Ctrl+C); the recorder then writes `metadata.yaml`, so `ros2 bag info blackbox` works; delete `blackbox`. **In production:** set retention and camera access, and upload at the dock. ### Use cases and scenarios | Scenario | Context | Method | Trade-offs | Success metric | |---|---|---|---|---| | Multi-vendor warehouse (illustrative) | Mobile robots from two vendors share aisles | One fleet control over VDA 5050; robots report `operatingMode` and `safetyState`[^vda5050] | Versions 2.1.0 and 3.0.0 name fields differently; traffic logic is outside the specification | Serious exceptions per 100 robot-hours by vendor | | Hospital delivery and cleaning (illustrative) | Two fleets share lifts and doors | Open-RMF, first built for healthcare, coordinates fleets and building infrastructure[^openrmf][^rmfbook] | Another system to operate; lift integration with facilities; industrial safety standards may not apply ([by robot class](#comparing-safety-standards-by-robot-class)) | Lift-wait time; missions completed | | Long-term autonomous inspection (real) | A Spot quadruped inspecting a test facility (49 days) and a fusion reactor hall (35 days) | Interventions logged by severity; MTBI 78 h then 140 h, excluding minor ones[^autoinspect] | Clustered failures hide a single cause | MTBI by class; longest unassisted run | | Dense grid robots (real) | Ocado's grocery-carrying bots on shared storage grids (Erith processes about 120,000 orders a week) | Andover, 2019: fire traced to a first-generation battery charger; plastic lids removed, smoke detectors added. Erith, 2021: fire after a three-bot collision, contained to under 1% of the grid[^lmanager2019][^register2021] | Fire engineering and robot design dominate; ops must detect early | Time from fault to isolation; area affected | In the inspection deployment, a process manager "intermittently causing all ROS nodes to shut down when operators reconnected" caused the fatal interventions at one site and was suspected in three at the other: weeks of operation surfaced a bug that short tests had not.[^autoinspect] ## Measure, govern, pay ### Metrics and KPIs | KPI | Definition / formula | Data source | Pitfall | Review | |---|---|---|---|---| | MTBI | Autonomous operating hours ÷ counted interventions, with counting and grouping rules stated | Exception log | Definition-sensitive ([Lab 1](#lab-1-beginner-compute-mtbi-from-an-intervention-log)) | Weekly | | Exception rate | Serious + fatal exceptions per 100 robot-hours, by class | Exception log | Merged classes hide severity | Weekly, per release | | Time to resume | Minutes from exception to automatic mode again (median, p90) | Fleet manager | Averages hide field visits | Weekly | | Remote resolution share | Exceptions closed without a site visit ÷ all | Exception log | Rewards risky fixes; pair with reopen rate | Monthly | | Assistant utilisation | Exceptions × mean handling time ÷ staffed minutes | Console | Waits grow fast above about 80% ([Lab 2](#lab-2-intermediate-size-the-remote-assistance-desk)) | Daily | | Productive robot time | 1 − stopped minutes ÷ scheduled minutes | Fleet manager | Say whether charging counts | Weekly | | Protective stops | Per 100 robot-hours, by location | Safety state | More people nearby also raises it; needs hundreds of robot-hours to compare | Per release | | Release health | Failed installs, rollbacks, exception-rate change per cohort | Release service | Cohorts on different sites or shifts | Per release | **Worked calculation (illustrative).** A site runs 40 robots for 16 hours on 30 days: 19,200 robot-hours. It logs 768 serious or fatal exceptions after grouping repeats: MTBI = 19,200 ÷ 768 = 25 hours; rate = 768 ÷ 19,200 × 100 = 4.0 per 100 robot-hours. If 85% are resolved remotely in a mean 6 minutes and the rest take a mean 40 minutes on site, stopped time is about 768 × (0.85 × 6 + 0.15 × 40) ≈ 8,525 minutes, or 0.7% of the 1,152,000 scheduled robot-minutes. A target of "MTBI 30 hours" means cutting about 128 exceptions a month, so the weekly review should start with the class that contributes most of them. **Leading and lagging.** Exception rate and protective stops lead; throughput lags. "Uptime" alone is a vanity metric: a robot can be powered, connected and useless. ### Safety, security, privacy and compliance - **Safety.** Change classes B and C and near-miss reviews carry the safety duties into daily operation. In VDA 5050 3.0.0, `safetyState` reports `activeEmergencyStop` (`MANUAL`, `REMOTE` or `NONE`) and `fieldViolation`;[^vda5050] 2.1.0 reports `eStop`, with an extra `AUTOACK` value, for example for a bumper hit.[^vda21] Map both to count stops across vendors. - **Security.** Segment robot networks; never expose middleware to the internet;[^demarinis2018] enable SROS2 or the vendor's equivalent, with an enclave per operations tool;[^sros2] sign releases; use role-based access control, as ROG recommends,[^rogmanifesto] with named accounts. VDA 5050 leaves protocol security to broker configuration, and 3.0.0 notes for its certificate-update action that "the sender of the instantAction cannot be verified",[^vda5050] so the MQTT broker needs TLS, authentication and per-topic access rules. - **Privacy.** Robot cameras record staff, patients and customers. Decide what the black box records, who may view it, retention and redaction before go-live; jurisdictions differ ([PrivacyOps](../privacyops/)). - **Compliance.** Map each site to its regime: in the EU, besides the Machinery Regulation (see the [standards table](#methods-standards-and-frameworks)), the Cyber Resilience Act covers hardware and software "products with digital elements", with reporting duties from 11 September 2026 and main obligations from 11 December 2027.[^eucra] Ask vendors for their security-support period and vulnerability handling (more in [IoTOps](../iotops/)). This is not legal or safety advice. ### Cost, capacity and sustainability Recurring costs are people (assistants, technicians), downtime, field visits, spare parts, connectivity and data. Useful unit costs are **cost per mission** and **cost per exception** (handling time × loaded staff cost, plus the robot's lost time). **Illustrative:** in the [worked calculation](#metrics-and-kpis) an exception takes 0.85 × 6 + 0.15 × 40 = 11.1 staff-minutes, if staff are busy for the whole stop. At an assumed 60 per staff-hour (any currency), that is about 11 per exception and 8,500 a month; MTBI 30 hours (128 fewer exceptions) saves about 1,400 a month before the robots' recovered output. Capacity often runs out at chargers, Wi-Fi, lifts and the assistance desk before computers. Upload snapshots at the dock rather than streaming video. This research found no sourced data on robot-fleet carbon footprints; energy and battery lifecycle are the obvious levers, and carbon accounting is taught in [GreenOps](../greenops/). ### Assurance of the operating model Check the loop, not just the robots: serious exceptions with a snapshot (target near 100%); corrective actions closed on time; repeat rate of classes marked "fixed"; drills held; releases that passed every gate; and an audit trail of who changed which map, configuration or release on which robot.[^rogmanifesto] ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Remediation | Prevention | |---|---|---|---|---| | Robots lose localization in one area | Layout changed (racking, pallets), reflective surfaces, light | Compare map with the black box scans | Update the map through change control | Site-change notification to robotics team | | A dashboard shows "no data" but the robot works | QoS mismatch between tool and publisher;[^rosqos] with SROS2 enforced, a tool without an enclave or topic permission[^sros2] | `ros2 topic info -v ` prints each endpoint's QoS profile[^topicinfo] | Match the subscriber's profile; give the tool its own enclave | Standard QoS profiles per topic class | | Robots stall at the same spot | Wi-Fi dead zone, failed roaming | Correlate stalls with signal logs | Fix coverage; let robots finish orders while offline (VDA 5050 robots keep their order up to the last released node)[^vda5050] | Survey routes before go-live | | Missions queue but robots idle | Charger or lift adapter fault | Fleet manager events for infrastructure | Fail over to other chargers; manual lift mode | Monitor adapters as first-class services | | Robot boot-loops after an update | Bad image, missing dependency | Boot logs; slot status | Boot the previous slot | A/B with boot confirmation and a watchdog[^raucbasic] | | Protective stops rise after a release | Changed speeds, sensor filtering or path planning | Snapshots at stop locations; held-back control robots | Contain first: roll back or limit speed; log a near miss | Rate gates with enough robot-hours | **Traffic between fleets.** Robots from one vendor know of each other and avoid conflicts;[^rmfbook] two fleets sharing a crossing or lift can each wait for the other indefinitely (deadlock). VDA 5050 3.0.0 makes deadlock resolution a fleet-control duty but leaves traffic logic out of scope; its tools are released path segments, at whose end a robot stops, and `RELEASE` zones entered only when fleet control grants access.[^vda5050] Open-RMF has fleet managers negotiate conflicts on a shared traffic schedule and warns that a fleet with no interface "will likely result in deadlocks".[^rmfcore] Give each shared crossing and lift one traffic authority. ### Worked runbook: protective stops double after a release ::: figure Staged rollout with gates and the rollback path ```mermaid flowchart TB A["Release candidate passes simulation CI"]:::core B["`Canary cohort one robot per hardware revision`"]:::core C{"`No failed boots or fatal exceptions in 48 h?`"}:::decision D["`Site cohort a few robots held back`"]:::core E{"`Rates within limits after 400 robot-hours?`"}:::decision F["Next site"]:::core G["`Contain: halt rollout, roll back or limit speed`"]:::risk H["`Notify the safety officer; log a near miss if stops rose`"]:::risk I["`Root cause snapshots and control robots`"]:::core A --> B --> C C -->|yes| D --> E E -->|yes| F C -->|no| G E -->|no| G G --> H --> I I -->|fixed| A classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 ``` Text equivalent: a candidate that passes simulation CI goes to a canary cohort (one robot per hardware revision); with no failed boots or fatal exceptions in 48 hours, to one site, with a few robots held back; with rates within limits after 400 robot-hours, to the next site. A failed gate means containment (halt, roll back or limit speed), then notifying the safety officer and logging a near miss if stops rose, then root-cause analysis; a fixed candidate re-enters simulation CI. ::: **Situation (illustrative).** Release `nav-4.13.0` passed its two-robot canary (hard faults only) and was installed at site A on Monday, except on four robots then in maintenance. On Tuesday, after 420 robot-hours, the site gate trips: 26 protective stops (6.2 per 100 robot-hours) against a baseline of 2.9 and a limit of 18. No injuries. 1. **Contain first.** Freeze the rollout. At site A, roll back to the previous slot, map and configuration, or apply a pre-approved lower speed limit in the affected zones until the rollback is done. A doubled stop rate in shared space is a near-miss signal: the safety layer is catching something the release changed, and no dashboard shows whether the assumptions behind the protective fields still hold. 2. **Notify.** Tell the safety officer and site lead; log a near miss; keep the snapshots. 3. **Diagnose.** 70% of stops are at two crossings where people push carts; snapshots show robots now approach faster before slowing. The four robots still on `nav-4.12.1` had 4 stops in 130 robot-hours: too few hours to prove anything alone, but consistent with the release, not the site. 4. **Classify.** Faster approach near people is a class B change the release notes did not declare and the risk assessment did not list. Confirm the rate returns to baseline after the rollback. 5. **Correct.** Set crossing speeds in map zones, add a cart-traffic simulation scenario and the crossings to the risk review, and re-release through the board as class B. 6. **Learn.** Diff motion parameters automatically so no release hides a class B change; canary each site layout, not only each hardware revision. ### Anti-patterns and myths - **"Zero interventions" targets.** Teams stop logging them. Track rates by class and reward reporting. - **The software e-stop.** A dashboard's "stop all robots" button is a network command, not an emergency stop; label it honestly. - **Hand-tuned robots.** A parameter changed on one robot to "unstick" it becomes an unknown configuration in the next incident. - **Fleet-wide Friday releases.** Robots work shifts; release when you can watch the first hours of a cohort. - **Myth: simulation passes, so the field will.** Floors, light, people and Wi-Fi differ; simulation catches regressions, cohorts catch reality. ## Deciding and adopting ### When to use and when not to use Use these practices when robots run **unattended for long periods**, across **more than one site or vendor**, or in **shared space with people**. For a handful of supervised robots, the vendor's tools and a simple log are enough. Robots-as-a-service contracts move much of the work to the provider, but you still need written metric definitions, evidence after incidents and a say in releases. ### Decision framework ::: figure Choosing an operating model for a robot fleet ```mermaid flowchart TB S["`For every option, weigh cost per exception, data export and exit terms`"]:::neutral Q1{"More than about 10 robots or unattended shifts?"}:::decision Q2{"More than one robot vendor on a site?"}:::decision Q3{"Is operating robots your product?"}:::decision A1["`Vendor tools plus an exception log; revisit as the fleet grows`"]:::neutral A2["`Vendor fleet manager plus your own exception loop`"]:::core A3["`Interoperability layer VDA 5050 or Open-RMF`"]:::core A4["`Build or extend a platform open source plus your own agent`"]:::core S --> Q1 Q1 -->|no| A1 Q1 -->|yes| Q2 Q2 -->|no| Q3 Q2 -->|yes| A3 Q3 -->|no| A2 Q3 -->|yes| A4 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 ``` Text equivalent: for every option, weigh cost per exception, data export and exit terms. If the fleet has no more than about ten robots and no unattended shifts, use the vendor's tools plus an exception log, and revisit as the fleet grows. Otherwise, if more than one robot vendor shares a site, add an interoperability layer such as VDA 5050 or Open-RMF. With a single vendor, if operating robots is not your product, use the vendor's fleet manager plus your own exception loop; if it is, build or extend a platform from open-source parts and your own agent. ::: The thresholds are judgement calls, not industry figures. One vendor is simpler and accountable but locks the site in, so weigh exit terms as heavily as features; an interoperability layer admits several vendors at the cost of integration work. ### Maturity model This is **the Atlas's model**, not an industry standard. In this Atlas's judgement, a fleet heading past ROG's 50-robot mark needs level 3.[^rogmanifesto] | Level | Observable behaviour | Evidence | Next step | |---|---|---|---| | 1. Heroic | Engineers fix robots by hand; no exception log | Knowledge in people's heads | Log every intervention with a class | | 2. Logged | Exceptions recorded; manual updates | Weekly counts | Write metric definitions; add snapshots | | 3. Managed | Remote-assistance runbook; staged releases; maps under change control | MTBI and rate by class; gate results | Make serious exceptions into simulation tests | | 4. Improving | Weekly corrective-action loop closes; rates fall release over release | Trend of serious exceptions per 100 h | Multi-site cohorts; drills; supplier scorecards | | 5. Scaled | Multi-vendor orchestration; staffing modelled; audits routine | Fan-out and utilisation tracked; audit trail complete | Revisit with each new robot type | ### Adoption roadmap: 30/60/90 days | Days | Actions | Milestone | |---|---|---| | 1–30 | Exception classes and log; metric definitions; allowed remote actions; black-box recording on a test robot; Wi-Fi survey of routes | First weekly exception review held | | 31–60 | Snapshots on all robots; staged-release policy with gates; map change control; emergency-stop recovery drill | First release through cohorts with gate evidence | | 61–90 | Staffing model (Lab 2) from real data; simulation scenarios for the top three exception classes; quarterly safety review; supplier scorecard | Rate of the top exception class falls; audit trail complete | ## Toolkit ### Checklists **Site readiness (before go-live)** - [ ] Risk assessment signed by integrator and site; lists routes, speeds, crossings and tasks - [ ] Wi-Fi survey along every route; behaviour on link loss tested - [ ] Emergency-stop locations agreed; recovery procedure drilled - [ ] Exception classes, allowed remote actions and escalation contacts written - [ ] Black-box recording, retention and camera access agreed **Release (every cohort)** - [ ] Change class decided (A, B or C); risk assessment checked for class B - [ ] Simulation scenarios for known exception classes pass - [ ] Install only when robots are docked and idle; previous slot, map and configuration kept - [ ] Gates, minimum robot-hours, held-back control robots and an owner for the first hours named **Weekly operations review** - [ ] Exceptions by class and site, with definitions unchanged - [ ] Every serious exception has a snapshot and an owner - [ ] Unexplained protective or emergency stops reviewed as near misses - [ ] Corrective actions due this week closed or escalated **Periodic inspection (manufacturer's schedule)** - [ ] Emergency stops, protective scanners and bumpers cleaned and function-tested; results recorded - [ ] Batteries and chargers checked for damage, heat and capacity loss; suspect units taken out of service ### Templates **Starter exception taxonomy (proposed).** The category (the exception record's `category` field) decides the first response; severity (minor, serious, fatal) is recorded separately. | Category | Examples | Default remote action | Escalate when | |---|---|---|---| | Path blocked | Pallet, cart or parked robot | Confirm static obstacle; approved detour | A person stays in the path | | Localization | Lost pose after a layout change | Relocalize only if the robot confirms | Repeats in one area | | Infrastructure | Door, lift or charger not responding | Retry via the fleet manager | Same device twice in a shift | | Cross-fleet traffic | Two fleets waiting for each other | Cancel one mission via the traffic authority | Recurs at one crossing or lift | | Task or load | Failed pick or drop; shifted load | Cancel; send to parking | Load unstable or damaged | | Software or link | Stack crash, boot loop, stale telemetry | Snapshot; quarantine | Several robots on one release or area | | Safety stop | Protective or emergency stop | Snapshot and quarantine only | Always: field recovery; near miss if unexplained | | Battery or hardware | Drive fault; heat or smoke | Charger only without heat or smoke | Heat or smoke: clear the area | ```text SITE RUNBOOK: owner: version: Robots: Fleet manager: Allowed remote actions: confirm static obstacle | approved detour | cancel mission | send to parking or charger | snapshot | quarantine Never remotely: reset MANUAL-type e-stop | change protective fields | untested config Escalation: assistant -> field technician (<15 min on site) -> site lead -> safety officer Incident when: >= 3 robots same class in 30 min | any contact with a person | smoke/heat Contacts: ``` ```text EXCEPTION REVIEW (weekly) period: site: Definitions: MTBI excludes minor; repeats within 15 min grouped (unchanged? yes/no) Top classes (count, per 100 h, trend): 1. 2. 3. Safety-relevant events: Decisions: Actions due / closed / overdue: / / ``` ```text ADR: date: <YYYY-MM-DD> status: <proposed> Context: <fleet, sites, vendors, constraint> Options: <vendor fleet manager | interoperability layer | own platform> Decision and reason: <...> Exit plan: <data export format, MCAP evidence, contract terms> Safety impact: <class A/B/C, risk-assessment change> ``` ### FAQ **Do we need ROS to do RoboOps?** No. Many robots expose only a vendor API or VDA 5050. ROS tools help when you control the software. **Can a remote operator reset an emergency stop?** Only if the system was designed and assessed for it; VDA 5050 distinguishes stops acknowledged on the robot (`MANUAL`) from facility stops acknowledged remotely (`REMOTE`).[^vda5050] Default to on-site recovery. **Is MTBI comparable between vendors?** Only with identical, written definitions of an intervention, grouping and operating time. **What should a black box record?** Health, localization, commands, safety state and the sensors that explain a stop, for a few minutes around each trigger; cameras only if policy allows. **Who owns safety when we only operate robots?** Manufacturer and integrator for the design; the site, as user, for safe use, change, training and periodic inspection ([roles](#roles-ownership-and-raci)). Contracts should say so. ## Going further ### Advanced topics and emerging direction - **Interoperability standards.** InOrbit calls OpenRobOps the first ISO 21423 reference implementation, a vendor claim from a CEO who sits in the ISO working group.[^oro2026] - **Fleets as data sources.** RobOpsCon 2026 themes include heterogeneous fleets with humanoids, data pipelines for training robot models and decentralized on-robot decisions.[^rogcon26] Field data feeds [MLOps](../mlops/); keep privacy rules attached. ### Learning path | Stage | Do this | |---|---| | 1 day | Read the Robot Operations Manifesto;[^rogmanifesto] do Labs 1 and 2 | | 1 week | ROS 2 and Gazebo in a VM; record and replay a bag; Lab 3 | | 30 days | Shadow a fleet operations desk; write an exception taxonomy and site runbook | | Advanced | Study ISO 10218, ISO 3691-4 or R15.08 (Part 3 for users) with a safety engineer; implement VDA 5050 or Open-RMF in simulation | | Portfolio | A simulation scenario that reproduces a real exception, with the fix and the before/after rate | ## Wrap-up ### Executive recap and cheat sheet - **Label:** niche (RobOps/RoboOps); the practice of running robot fleets is real and growing. - **Core loop:** accept exceptions, resolve them fast, make them rarer. - **Never:** route safety functions through the ops pipeline; reset stops remotely by default; release fleet-wide at once. - **Always:** written metric definitions, snapshots for serious exceptions, cohorts with gates, quarantine before return to service. - **Measure:** serious exceptions per 100 robot-hours, time to resume, assistant utilisation, protective stops per release. - **Decide:** vendor tools for small fleets; your own exception loop at scale; an interoperability layer for mixed vendors. ### A judgement call: push the fix to every robot tonight, or run degraded? **Situation (illustrative).** A navigation bug makes robots at three sites stop at lift doors about twice a shift; assistants clear each case in four minutes. A fix passed simulation this afternoon. Peak season starts tomorrow. **Option A: push tonight to all 120 robots.** Peak starts without the bug, if the fix works. But it has no field hours: if it changes approach behaviour near lifts, peak starts with an unknown regression. **Option B: canary tonight, run degraded at peak.** Two robots per site get the fix overnight; the rest keep the bug. Cost: about 114 robots × 2 stops × 4 minutes ≈ 910 assistant-minutes per 8-hour shift, so roughly two extra assistants per shift, plus queueing headroom ([Lab 2](#lab-2-intermediate-size-the-remote-assistance-desk)). **Decision.** B, with a gate review after the first peak shift. Six canary robots over one shift can show the lift-door stops vanish (a large effect), not that protective stops did not rise, so the second night goes to one site, not all 120. The bug is costly but understood and safe; the fix is unproven in a motion-relevant area. **Uncertainty:** if stopped robots block people or emergency access at lifts, the balance shifts toward A with extra site staff. ### Learning objectives and mastery check You should be able to **explain** autonomy exceptions and the lifecycle of failures; **distinguish** remote assistance from teleoperation and safety stops from operational pauses; **define** comparable fleet metrics; **size** an assistance desk; and **design** a staged release for robots. 1. Why is a dashboard's "stop all robots" button not an emergency stop? ::: answer Show answer for question 1 It is a network command to software that is not safety-rated. Emergency stops are safety functions validated under the applicable standards; operations software reports them and never replaces them. ::: 2. Two vendors report MTBI of 40 and 70 hours. What do you ask before comparing? ::: answer Show answer for question 2 What counts as an intervention, how repeats are grouped, and which operating time is used: Lab 1's week reads 34.5 to 64.0 hours. ::: 3. Each robot needs help every 120 minutes and each case takes 5 minutes. What is the fan-out, and should one person watch that many? ::: answer Show answer for question 3 (5 + 120) ÷ 5 = 25 robots. No: it is an upper bound with no queueing headroom, and interaction effort rises with fan-out.[^olsen2003] Staff so that utilisation stays well below 100% and check the p90 wait, as in Lab 2. ::: 4. Which changes need a risk-assessment review before release? ::: answer Show answer for question 4 Any that can change motion or tasks near people: maps, zones, speeds, routes, new tasks (class B). OSHA's inspection guide checks for task-based risk assessments for new or modified tasks before they are performed.[^osha] ::: 5. A two-robot canary ran for 48 hours with no rise in protective stops. Is the release ready for every site? ::: answer Show answer for question 5 Not on that evidence. At 2.9 stops per 100 robot-hours the canary expected two or three stops, too few to see even a doubling reliably. It shows that the release boots and runs; rate gates need a site cohort with a few hundred robot-hours and a control. ::: ### Conclusion RoboOps is a small label for a large shift. As robots leave cages and pilots, the hard problems stop being "can the robot do the task?" and become "who notices when it cannot, how fast does it get going again, and how do we make sure the next release does not put someone in its path?" The Robot Operations Group's answer, follow the lifecycle of failures, holds up, but only with things the label does not provide: safety functions kept out of reach of operations software, metrics defined before they are compared, a staffed and modelled assistance desk, and releases that respect cohorts and risk assessments. Treat the word as optional and the practices as mandatory. ## Sources and further reading All sources were opened on 2026-10-18. docs.ros.org blocks automated access, so ROS 2 documentation is cited from its source repository; www.iso.org and A3 pages also refused access, so ISO catalogue entries are cited from committee.iso.org and the A3 FAQ from an Internet Archive copy. No standard's text was read. Re-check dated facts each May (ROS releases) and when a standard or VDA 5050 version changes. [^rogmanifesto]: [The Robot Operations Manifesto](https://robotoperationsgroup.org/manifesto) — Robot Operations Group, undated (released September 2020). Checked 2026-10-18. [^roghome]: [Robot Operations Group](https://robotoperationsgroup.org/) — home page (robops.org redirects here). Checked 2026-10-18. [^rogcon26]: [RobOpsCon 2026](https://robotoperationsgroup.org/robopscon26) — Robot Operations Group, event of 2026-10-22. Checked 2026-10-18. [^inorbit2020]: [Going ROG: Why Robot Operations Got a Manifesto](https://www.inorbit.ai/blog/going-rog-why-robot-operations-got-a-manifesto) — Florian Pestoni, InOrbit, 2020-09-01. Checked 2026-10-18. [^inorbitrobops]: [What is RobOps?](https://www.inorbit.ai/robops) — InOrbit, undated. Checked 2026-10-18. [^oitzman2020]: [New Robot Operations Group Is Solving Fleet Management Problems](https://www.automatedwarehouseonline.com/new-robot-operations-group-is-solving-fleet-management-problems/) — Mike Oitzman, Automated Warehouse, 2020-10-23. Checked 2026-10-18. [^wipro]: [RoboOps: Managed Operations for Enterprise Physical AI](https://www.wipro.com/innovation/emerging-technologies/physical-ai-at-wipro/offerings/roboops-managed-operations-for-enterprise-physical-ai/) — Wipro, undated (© 2026). Checked 2026-10-18. [^oro2026]: [InOrbit.AI releases OpenRobOps ISO 21423 reference implementation](https://www.therobotreport.com/inorbit-ai-releases-openrobops-iso-21423-reference-implementation/) — The Robot Report, 2026-09-15. Checked 2026-10-18. [^robomaker]: [AWS RoboMaker shuts down after failing to gain traction](https://www.therobotreport.com/aws-robomaker-shuts-down-after-failing-to-gain-traction/) — Steve Crowe, The Robot Report, 2025-09-10. Checked 2026-10-18. [^rosreleases]: [Distributions (Releases.rst)](https://github.com/ros2/ros2_documentation/blob/rolling/source/Releases.rst) — ROS 2 documentation source, rolling branch. Checked 2026-10-18. [^noeticeol]: [ROS Noetic End-of-Life: May 31, 2025](https://discourse.openrobotics.org/t/ros-noetic-end-of-life-may-31-2025/43160) — Open Robotics Discourse, 2025-04-10. Checked 2026-10-18. [^rosnodes]: [Nodes](https://github.com/ros2/ros2_documentation/blob/rolling/source/ROS-Framework/About-Nodes.rst) — ROS 2 documentation source. Checked 2026-10-18. [^rostopics]: [Topics](https://github.com/ros2/ros2_documentation/blob/rolling/source/ROS-Framework/interfaces/About-Topics.rst) — ROS 2 documentation source. Checked 2026-10-18. [^rosservices]: [Services](https://github.com/ros2/ros2_documentation/blob/rolling/source/ROS-Framework/interfaces/About-Services.rst) — ROS 2 documentation source. Checked 2026-10-18. [^rosactions]: [Actions](https://github.com/ros2/ros2_documentation/blob/rolling/source/ROS-Framework/interfaces/About-Actions.rst) — ROS 2 documentation source. Checked 2026-10-18. [^rosqos]: [Quality of Service settings](https://github.com/ros2/ros2_documentation/blob/rolling/source/ROS-Framework/interfaces/topics/About-Quality-of-Service-Settings.rst) — ROS 2 documentation source. Checked 2026-10-18. [^sros2]: [Setting up security](https://github.com/ros2/ros2_documentation/blob/rolling/source/Developer-Tools/Launch/Security/Introducing-ros2-security.rst) and [Setting access controls](https://github.com/ros2/ros2_documentation/blob/rolling/source/Developer-Tools/Launch/Security/Access-Controls.rst) — ROS 2 documentation source. Checked 2026-10-18. [^rosintegration]: [Writing basic integration tests with launch_testing](https://github.com/ros2/ros2_documentation/blob/rolling/source/Developer-Tools/Debugging/Testing/Integration.rst) — ROS 2 documentation source. Checked 2026-10-18. [^rosbag2]: [rosbag2 README (jazzy branch)](https://github.com/ros2/rosbag2/blob/jazzy/README.md) — ROS 2 project, updated 2026-10-14. Checked 2026-10-18. [^rosbag2record]: [ros2 bag record verb (jazzy)](https://github.com/ros2/rosbag2/blob/jazzy/ros2bag/ros2bag/verb/record.py) and [Snapshot.srv](https://github.com/ros2/rosbag2/blob/jazzy/rosbag2_interfaces/srv/Snapshot.srv) — ROS 2 project source. Checked 2026-10-18. [^rosbag2src]: [sequential_writer.cpp](https://github.com/ros2/rosbag2/blob/jazzy/rosbag2_cpp/src/rosbag2_cpp/writers/sequential_writer.cpp), [info.cpp](https://github.com/ros2/rosbag2/blob/jazzy/rosbag2_cpp/src/rosbag2_cpp/info.cpp) and [circular_message_cache.cpp](https://github.com/ros2/rosbag2/blob/jazzy/rosbag2_cpp/src/rosbag2_cpp/cache/circular_message_cache.cpp) — rosbag2 source, jazzy branch. Checked 2026-10-18. [^topicinfo]: [ros2 topic info verb (jazzy)](https://github.com/ros2/ros2cli/blob/jazzy/ros2topic/ros2topic/verb/info.py) — ros2cli source. Checked 2026-10-18. [^mcap]: [MCAP](https://mcap.dev/) — MCAP project home page. Checked 2026-10-18. [^diagnostics]: [diagnostics](https://github.com/ros/diagnostics) — ROS diagnostics repository, ros2 branch README. Checked 2026-10-18. [^diagstatus]: [DiagnosticStatus.msg](https://github.com/ros2/common_interfaces/blob/rolling/diagnostic_msgs/msg/DiagnosticStatus.msg) — ROS 2 common_interfaces. Checked 2026-10-18. [^gzreleases]: [Gazebo Releases](https://gazebosim.org/docs/latest/releases/) — Open Robotics. Checked 2026-10-18. [^gzros]: [Installing Gazebo with ROS](https://gazebosim.org/docs/latest/ros_installation/) — Open Robotics. Checked 2026-10-18. [^gzclassic]: [Gazebo Classic](https://classic.gazebosim.org/) — Open Robotics (end-of-life notice). Checked 2026-10-18. [^gzcli]: [gz sim command-line options (sim_main.cc, gz-sim10)](https://github.com/gazebosim/gz-sim/blob/gz-sim10/src/cmd/sim_main.cc) and [cmdsim.rb.in (gz-sim8)](https://github.com/gazebosim/gz-sim/blob/gz-sim8/src/cmd/cmdsim.rb.in) — Gazebo source. Checked 2026-10-18. [^actionrosci]: [action-ros-ci](https://github.com/ros-tooling/action-ros-ci) — ros-tooling, README (latest release 0.4.8, 2026-05-15). Checked 2026-10-18. [^industrialci]: [industrial_ci](https://github.com/ros-industrial/industrial_ci) — ROS-Industrial. Checked 2026-10-18. [^raucbasic]: [RAUC basics: boot confirmation and fallback](https://rauc.readthedocs.io/en/latest/basic.html) — RAUC documentation. Checked 2026-10-18. [^raucscen]: [RAUC scenarios: symmetric root-FS slots](https://rauc.readthedocs.io/en/latest/scenarios.html) — RAUC documentation. Checked 2026-10-18. [^vda5050]: [VDA 5050 version 3.0.0 (VDA5050_EN.md)](https://github.com/VDA5050/VDA5050/blob/main/VDA5050_EN.md) — VDA and VDMA, GitHub release 2026-03-18. Checked 2026-10-18. [^vda21]: [VDA 5050 version 2.1.0 (VDA5050_EN.md)](https://github.com/VDA5050/VDA5050/blob/2.1.0/VDA5050_EN.md) — VDA and VDMA, GitHub release 2024-08-19. Checked 2026-10-18. [^iso21423]: [ISO 21423:2026, Robotics — Industrial mobile robots — Communications and interoperability](https://committee.iso.org/standard/86749.html) — ISO catalogue entry (abstract; published 2026-10-14). Checked 2026-10-18. [^massrobotics]: [AMR Interoperability Standard](https://github.com/MassRobotics-AMR/AMR_Interop_Standard) — MassRobotics. Checked 2026-10-18. [^openrmf]: [Open-RMF](https://www.open-rmf.org/) — Open-RMF project (Open Source Robotics Alliance). Checked 2026-10-18. [^rmfbook]: [Programming Multiple Robots with ROS 2: Introduction](https://osrf.github.io/ros2multirobotbook/intro.html) — Open Robotics. Checked 2026-10-18. [^rmfcore]: [RMF Core Overview](https://osrf.github.io/ros2multirobotbook/rmf-core.html) — Programming Multiple Robots with ROS 2, Open Robotics. Checked 2026-10-18. [^a3iso10218]: [Updated ISO 10218: Answers to Frequently Asked Questions](https://web.archive.org/web/20251006040157/https://www.automate.org/robotics/blogs/updated-iso-10218-faq) — A3, 2025 (Internet Archive copy of 2025-10-06; live page blocks automated access). Checked 2026-10-18. [^awiso10218]: [A3 Updates Industrial Robot Safety Standards](https://www.automationworld.com/factory/safety/news/55271049/a3-updates-industrial-robot-safety-standards) — Automation World, 2025-03-26. Checked 2026-10-18. [^a3r1508]: [ANSI/A3 R15.08-2 Safety Standard for Industrial Mobile Robot Systems and Applications Now Available](https://s24.q4cdn.com/622300748/files/doc_news/ANSIA3-R15.08-2-Safety-Standard-for-Industrial-Mobile-Robot-Systems-and-Applications-Now-Available-2023.pdf) — A3 news release, 2023-10-09. Checked 2026-10-18. [^r15083]: [New ANSI Standard Completes Industrial Mobile Robot Safety Series](https://www.automationworld.com/factory/robotics/news/55408868/association-for-advancing-automation-new-ansi-standard-completes-industrial-mobile-robot-safety-series) — Automation World, 2026-09-30. Checked 2026-10-18. [^r15083ran]: [A3 releases final safety standard for industrial mobile robots](https://roboticsandautomationnews.com/2026/10/02/industry-body-a3-releases-final-part-of-safety-standard-for-industrial-mobile-robots/105474/) — Sam Francis, Robotics & Automation News, 2026-10-02. Checked 2026-10-18. [^isoscope]: [ISO 3691-4:2023](https://committee.iso.org/standard/83545.html) and [ISO 13482:2014](https://committee.iso.org/standard/53820.html) — ISO catalogue entries (abstracts and revision status). Checked 2026-10-18. [^osha]: [OSHA Technical Manual, Section IV Chapter 4: Industrial Robot Systems and Industrial Robot System Safety](https://www.osha.gov/otm/section-4-safety-hazards/chapter-4) — US OSHA, undated (written before the 2025 ISO 10218 revision). Checked 2026-10-18. [^eumachinery]: [Machinery](https://single-market-economy.ec.europa.eu/sectors/mechanical-engineering/machinery_en) — European Commission, Internal Market, Industry, Entrepreneurship and SMEs. Checked 2026-10-18. [^eucra]: [Cyber Resilience Act](https://digital-strategy.ec.europa.eu/en/policies/cyber-resilience-act) — European Commission, updated 2026-09-07. Checked 2026-10-18. [^waymo2024]: [Fleet response: Lending a helpful hand to Waymo's autonomously driven vehicles](https://waymo.com/blog/2024/05/fleet-response/) — Waymo, 2024-05-21. Checked 2026-10-18. [^autoinspect]: [AutoInspect: Towards Long-Term Autonomous Industrial Inspection](https://arxiv.org/abs/2404.12785) — Staniaszek et al., arXiv, 2024-04-19. Checked 2026-10-18. [^forest2025]: [Building Forest Inventories with Autonomous Legged Robots](https://arxiv.org/abs/2506.20315) — Mattamala et al., arXiv, 2025-06-25. Checked 2026-10-18. [^olsen2003]: [Metrics for Evaluating Human-Robot Interactions](https://faculty.cs.byu.edu/~mike/mikeg/papers/OlsenGoodrichPERMIS2003.pdf) — Dan R. Olsen and Michael A. Goodrich, PerMIS 2003. Checked 2026-10-18. [^demarinis2018]: [Scanning the Internet for ROS: A View of Security in Robotics Research](https://arxiv.org/abs/1808.03322) — DeMarinis et al., arXiv, 2018-07-23. Checked 2026-10-18. [^register2021]: [Impromptu game of Robot Wars sparks fire in warehouse at UK e-tailer Ocado](https://www.theregister.com/2021/07/19/ocado_robot_crash/) — Simon Sharwood, The Register, 2021-07-19 (quotes Ocado's statement). Checked 2026-10-18. [^lmanager2019]: [Ocado fire caused by battery charger fault](https://www.logisticsmanager.com/ocado-fire-caused-by-battery-charger-fault/) — Logistics Manager, 2019-04-29. Checked 2026-10-18. [^bdorbit]: [Orbit robot fleet management software](https://bostondynamics.com/products/orbit/) — Boston Dynamics. Checked 2026-10-18. [^formant]: [Formant](https://formant.io/) — Formant home page. Checked 2026-10-18. [^foxglove]: [Foxglove](https://foxglove.dev/) — Foxglove home page. Checked 2026-10-18. [^pypiroboops]: [robotframework-roboops](https://pypi.org/project/robotframework-roboops/) — PyPI project record (read through PyPI's JSON API), releases of January 2021. Checked 2026-10-18. **Changelog.** - 2026-10-18, r0: first draft (term status confirmed as niche; 13 parts; 6 diagrams; Labs 1–2 executed, Lab 3 syntax-checked only). - 2026-10-18, r1: after the independent review: R15.08-3-2026, ISO 21423:2026 and standards by robot class added; runbook contains first; gates sized in robot-hours; Lab 3 corrected; VDA 5050 2.1.0, SROS2 enclaves, the CRA, periodic inspection and diagram fixes added; repeated comparison table and FAQ cut. - 2026-10-18, post-score polish: OSHA and Ocado wording fixed; second R15.08-3 report; exception taxonomy, cross-fleet traffic, cost example, site-b gates and a decision-tree exit check added; code wrapped to 90 characters; repeats trimmed. - 2026-10-18, post-polish check: Lab 2 interaction-effort paraphrase corrected to Olsen and Goodrich's "interface modes"; vendor-versus-interoperability trade-off restored under the decision framework; standards-table link added.