--- title: "DisasterRecoveryOps" slug: disasterrecoveryops category: reliability term_status: overlapping-synonym evidence_status: verified summary: "DisasterRecoveryOps is a rare label for IT disaster recovery run as an operational practice: agreeing how fast each service must return and how much data it may lose, then deciding when to invoke recovery, failing over, failing back and proving it all in drills." audience: [Beginners, Engineers, SREs, Architects, Continuity and risk managers, Leaders] last_verified: 2026-10-17 key_takeaways: - "The label is rare; the practice is IT disaster recovery, described in NIST SP 800-34, ISO/IEC 27031 and the cloud providers' frameworks." - "The RTO clock covers detection and the decision to invoke, not only the technical failover; time the decision in every drill." - "High availability and replication are not disaster recovery: corruption and bad changes replicate, so keep point-in-time copies." - "Name who may invoke, set a decision deadline per service, and pre-authorise invocation when the criteria are met." - "Prefer data-plane actions, pre-recorded endpoints and DR-only credentials kept outside the failure domain; make unavoidable control-plane calls from the recovery Region." - "Failback is a second, planned migration: reconcile lost writes, resynchronise, then switch back without data loss." related: [backupops, sre, incidentops, riskops, cloudops, dbops, storageops, responseops] --- ## What DisasterRecoveryOps is ### In 30 seconds **DisasterRecoveryOps is IT disaster recovery run as a continuous operational practice.** For each service you agree two numbers with the business: how long it may be down (the recovery time objective, RTO) and how much recent data it may lose (the recovery point objective, RPO). You build a recovery capability that meets them, decide in advance who may declare a disaster and when, rehearse the failover and the return (failback), and measure what the rehearsals achieve. AWS calls a disaster any event that "prevents a workload or system from fulfilling its business objectives in its primary deployed location", whether natural, technical or human.[^awsdr][^awswhat] ### At a glance | Question | Short answer | |---|---| | Goal | Services that come back within their agreed RTO and RPO, proven by timed exercises | | Scope here | Objectives, strategy choice, invocation, failover, failback, drills, DR governance | | Owned by neighbours | Backup mechanics, incident command, the continuity programme ([neighbour table](#how-disasterrecoveryops-compares-with-other-ops)) | | Prerequisites | Basic infrastructure and incident knowledge; [BackupOps](../backupops/) helps | | Key outputs | Tiered RTO/RPO register, invocation policy, runbooks, drill reports, failback plans | | Term status | **Overlapping synonym**: rare label for the established practice of IT disaster recovery | ### The problem it solves Having a DR plan is not the same as being able to recover. Four problems recur: 1. **The clock runs before anyone acts.** A one-hour RTO must cover detecting, notifying, escalating, assessing, declaring a disaster *and* recovering.[^awsdetect] 2. **Nobody wants to press the button.** Failover costs some availability and, with asynchronous replication, some data. If stakeholders decline to invoke while the RTO is at risk, AWS advises re-evaluating the plans.[^awsdetect] 3. **The standby has drifted.** Capacity, quotas, images and configuration in the recovery location age silently; "the only error recovery that works is the path you test frequently".[^awstest] 4. **Replication copies the damage.** Deletions and corruption replicate too, so a highly available design can still lose data without a point-in-time copy.[^awsha] ### What it is not: boundaries and naming - **Not high availability.** Availability "focuses on components of the workload, whereas disaster recovery focuses on discrete copies of the entire workload".[^awsha] - **Not backup.** Copies of data and tested restores belong to [BackupOps](../backupops/); this page owns bringing the *service* back. - **Not business continuity.** NIST separates the business continuity plan (keeping business processes going) from the disaster recovery plan, which "only addresses information system disruptions that require relocation".[^nist34] AWS and Google describe DR as a subset of business continuity;[^awsbcp][^gcpdr] the programme belongs to [RiskOps](../riskops/). - **Not emergency management,** where "disaster recovery" means community recovery (FEMA's National Disaster Recovery Framework).[^fema] - **Not a new discipline.** No source found on 2026-10-16 uses "DisasterRecoveryOps". "DROps" appears in one managed-service provider's article as a framework that "coordinates people, processes, and technology for swift incident response".[^opsio] The backup vendor Commvault promotes a neighbouring label, "Resilience Operations (ResOps)" (July 2026), measured by "Mean Time to Clean Recovery (MTCR)".[^commvault] Both are vendor framing, not evidence of an established term. ::: note Term status: overlapping synonym The practice behind the label is long established: US federal contingency-planning guidance goes back at least to FIPS 87 (1981), which NIST SP 800-34 superseded in 2002;[^nist34v1] ISO/IEC 27031 (second edition, 2025) covers ICT readiness for business continuity,[^iso27031] ISO 22301 sets requirements for business continuity management systems,[^iso22301] and DRI International maintains a professional body of knowledge and certifications.[^dri] This page teaches that practice with the emphasis the manifest assigns, invocation, failover and failback: an **editorial distinction** of this Atlas, not an industry boundary. ::: ## Where it comes from and where it sits ### Origins and evolution NIST SP 800-34 Rev. 1 (2010) defines the vocabulary still in use: business impact analysis (BIA), RTO, RPO, maximum tolerable downtime (MTD) and alternate sites from cold to mirrored.[^nist34] Public cloud changed the economics more than the logic, since recovery can be scripted and rehearsed often, and regulation now demands tests, as the EU's Digital Operational Resilience Act (DORA) has since 17 January 2025.[^dora] ### Glossary and mental model These are the Atlas-wide definitions. For RPO, NIST names a point in time and AWS the time back to it: the same objective. | Term | Meaning | Why it matters | |---|---|---| | RTO | NIST: how long a system may stay down before the impact on dependent processes and the MTD is unacceptable. AWS: the maximum acceptable delay from interruption to restored service[^nist34][^awsbcp] | Drives strategy and cost; includes deciding | | RPO | NIST: the point in time, before the disruption, to which data can be recovered. AWS: the maximum acceptable time since the last data recovery point[^nist34][^awsbcp] | A time, not an amount of data[^gcpdr] | | MTD | Total outage the system owner accepts for a business process; the RTO must normally be shorter[^nist34] | Business ceiling above the RTO | | MBCO | Minimum business continuity objective, an ICT continuity objective in ISO/IEC 27031[^iso27031] | Allows a reduced first recovery | | BIA | Business impact analysis of downtime and data loss[^nist34] | Source of RTO and RPO | | Invocation | The formal decision to execute the DR plan | Needs a named authority and criteria | | Failover, switchover | Failover: unplanned move that can lose unreplicated writes. Switchover: planned, synchronised move with no data loss[^aurora] | Switchovers are how you test and fail back | | Failback | Return to the primary after recovery[^azdr] | A second migration | | Fencing, split brain | Fencing stops the old primary accepting writes; without it both sides may accept writes, a split brain[^aurora] | Divergent data must be [reconciled](#reconciling-lost-writes) | | Tabletop exercise | Discussion-based walk-through; nothing is deployed[^nist34] | Tests the decision cheaply | | Game day | Simulated event in a production-like environment, testing systems, processes and people[^gameday] | Tests the whole path | | Static stability | The system keeps working when a dependency is impaired[^staticstab] | Pre-provision rather than scale in a crisis | | Data plane / control plane | Running resources versus the machinery that changes them[^staticstab] | Prefer data-plane actions for failover[^awsopts] | **Mental model: the recovery-time budget.** People and systems share the RTO: detecting, escalating, assessing and declaring happen before any system moves, so a drill timed from the declaration measures only the second half. The figure uses the illustrative [Lab 2](#hands-on-labs) drill. ::: figure The RTO budget: in the Lab 2 drill, deciding took longer than failing over ```mermaid flowchart TB START(["T+0: impact, the 60-minute RTO clock starts"]) subgraph people["People: 31 minutes"] direction LR D["Detect: 6 min"] --> X["`Escalate, assess, declare: 25 min`"] end subgraph systems["Systems: 33 minutes"] direction LR P["Prepare, fence: 2"] --> F["Fail over: 19"] --> V["Verify: 12"] end STOP(["T+64: service verified, RTO missed by 4"]) START --> people people -->|"declared at T+31; deadline was T+22"| systems systems --> STOP classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class D,X risk class P,F,V core class START,STOP neutral ``` Text equivalent: The 60-minute RTO clock starts at impact (T+0). People spend 31 minutes, 6 detecting and 25 escalating, assessing and declaring, so the declaration comes at T+31, nine minutes after the T+22 decision deadline. Systems then spend 33 minutes: 2 to prepare and fence, 19 to fail over, 12 to verify. Service is verified at T+64, missing the RTO by 4 minutes. ::: ### How DisasterRecoveryOps compares with other Ops Recovery is split across Atlas pages. High availability reacts automatically and backup restores on request; disaster recovery starts with an invocation decision, which this page owns with the objectives and the orchestration. | Neighbour | What stays there | What this page adds | |---|---|---| | [BackupOps](../backupops/) | Copies, retention, immutability, restore tests | Service recovery; the Atlas-wide RTO/RPO definitions | | [DBOps](../dbops/) | Engine internals, point-in-time recovery | When and in what order to promote, and who decides | | [StorageOps](../storageops/) | Media, replication mechanics | Why replication alone is not DR | | [SRE](../sre/) | SLO design, error budgets, postmortem culture | Objectives for rare, large events | | [IncidentOps](../incidentops/) | Incident roles, comms, MTTR caveats | The invocation decision inside an incident | | [ResponseOps](../responseops/) | Containment, forensics | Recovery to a clean, segregated environment | | [RiskOps](../riskops/) | Programme, crisis management, risk appetite | ICT recovery that meets the programme's needs | | [CloudOps](../cloudops/) | Account structure, guardrails | Recovery-Region readiness | ## Principles and operating model ### Charter and principles ::: note A proposed charter (this Atlas's proposal, not a published standard) **Mission:** every service returns within the RTO and RPO the business agreed and paid for, with evidence. **Non-goals:** zero downtime and loss for everything; DR that costs more than the loss; replacing the continuity programme. ::: 1. **Objectives come from the business and carry a price.** NIST draws a cost-balance curve: shorter RTOs cost more.[^nist34] 2. **The clock includes the decision.** Name the invocation authority, write the activation criteria, and pre-authorise invocation when they are met. 3. **Recover to a known-good point, not just the latest.** If the fault is in the data or configuration, failover moves it too. 4. **Prefer data-plane actions.** AWS recommends "only data plane operations as part of your failover operation".[^awsopts] Make unavoidable control-plane calls, such as promoting a database, in the recovery Region, and rehearse them. Keep endpoints, credentials and runbooks outside the failure domain. 5. **Few recovery paths, exercised often.** Untested paths fail.[^awstest] 6. **Failback is a second migration.** Plan, rehearse and approve it like the failover. 7. **Evidence or it did not happen.** Every drill and invocation leaves a timed record. ### Roles, ownership and RACI NIST gives the ultimate authority to activate a plan to a senior management official, such as the CIO.[^nist34] The RACI below assumes that authority is delegated per service tier (R responsible, A accountable, C consulted, I informed). | Activity | Invocation authority | DR coordinator | Service owner | Platform and DB engineers | Incident commander | |---|---|---|---|---|---| | Set objectives from the BIA; choose and fund the strategy | C | R | A | C | I | | Maintain runbooks and drift checks | I | A | C | R | I | | Run drills and report | I | A/R | C | R | C | | Decide to invoke | A | R (recommends) | C | C | R (escalates) | | Execute failover | I | A | C | R | C | | Communicate during the event | I | C | C | I | A/R | | Approve and run failback | A | R | C | R | I | The continuity manager is consulted on objectives and informed of every invocation. Communications follow [IncidentOps](../incidentops/), but Microsoft counts a communication plan as part of every DR plan;[^azdr] DR adds three messages: recovery invoked, which data may be missing and how it will be reconciled, and when failback will happen. ### Lifecycle: Day 0, Day 1, Day 2 - **Day 0, design.** Run the BIA, assign tiers and objectives, map dependencies, choose a strategy per tier and record it in a register that tooling can check ([Lab 1](#hands-on-labs)). - **Day 1, build and prove.** Build the recovery environment as code, write the invocation policy and runbooks, and run the first planned switchover; a strategy that has never run is a hypothesis. - **Day 2, operate.** Drill on a schedule, check drift after every change, review objectives yearly. Run real invocations through the three phases of NIST's information system contingency plan (ISCP), applied here to DR: activation and notification, recovery, and reconstitution (validation, then deactivation of the plan).[^nist34] ### Operating model and governance - **Invocation policy.** One per service ([reference implementation](#reference-implementation)). NIST's example ISCP activation criteria include damage extent, criticality and an expected outage longer than the RTO.[^nist34] - **Exercise cadence by impact.** NIST suggests a tabletop for low-impact systems, a functional exercise with recovery from backup for moderate, and a full-scale exercise with failover to the alternate location for high-impact systems.[^nist34] - **Exercises that include people.** Google's disaster recovery testing (DiRT) runs real and fictitious incidents across the company to test technical systems, processes and people.[^gcpblog] - **Regulated sectors (check your jurisdiction).** DORA requires EU financial entities under its full ICT risk framework to test continuity and response and recovery plans "at least yearly", with switchovers to redundant capacity for entities other than microenterprises; Article 16 gives small and exempted entities a simplified framework.[^dora] The US HIPAA Security Rule requires a disaster recovery plan ([methods](#methods-standards-and-frameworks)).[^hipaa] These are examples, not legal advice. ## Architecture and workflow ### Building blocks of a recovery A recovery location is only as complete as its weakest layer; check each for every tier-1 service. | Layer | What must exist in the recovery location | Common gap | |---|---|---| | Data | Replicas for a low RPO **and** point-in-time copies for corruption, for every store the service needs[^awsha] | Replication only; backups in the same site or account; one store forgotten | | Compute and configuration | Infrastructure as code, images, secrets and parameters | Images or config never copied; quotas too low[^awstest] | | Traffic control | DNS or global load balancing switched with data-plane actions[^awsopts] | Console-only procedures; long DNS caches; record changes that depend on one Region's control plane[^r53accel] | | Identity and access | DR-only credentials, keys and CI/CD access that work when the primary is down ([reference implementation](#reference-implementation)) | Sign-in, token or key service tied to the failed Region[^aws2025] | | Observability | Monitoring and logs for the recovery location itself | Dashboards only in the primary | | People | Runbooks, contacts and chat bridges outside the failure domain | Runbook in a wiki hosted in the failed Region | **Several stores, several recovery points.** Orders in a database, documents in object storage, events in a queue: each store replicates with its own mechanism and lag (AWS lists S3 replication, Aurora global databases and DynamoDB global tables separately[^awsopts]), so after failover an order may exist without its uploaded invoice. This Atlas's rules: the effective RPO is the worst lag of any store the service needs, and [lost writes](#reconciling-lost-writes) are reconciled in every store. Mechanics belong to [DBOps](../dbops/) and [StorageOps](../storageops/). **Recovery order across services.** Bring services back in dependency order: identity, DNS, secrets and networking; data stores; the stateful middleware built on them; applications; traffic last. Site Recovery and VMware recovery plans sequence machines,[^asr][^vlsr] and ARC Region switch runs ordered steps,[^arcswitch] but only your dependency map says what the order is. This Atlas's rules of thumb for middleware and suppliers: - **Caches** often start cold: size the database for the surge, or warm them before traffic moves. - **Queues and streams** can lose or repeat in-flight messages: make consumers idempotent and let producers replay from the system of record. - **Search indexes and other derived stores** can be rebuilt from the system of record; time the rebuild in a drill, because it can outlast the RTO. - **Suppliers and SaaS** recover on their own terms: ask for their objectives and test evidence, and keep a manual workaround. DORA requires in-scope financial entities to keep tested exit plans for ICT services that support critical or important functions (Article 28).[^dora] ### Reference architecture: four recovery strategies Strategies run from cheap and slow to expensive and fast. AWS names backup and restore, pilot light, warm standby and multi-site active/active, with hot standby as a full-capacity variant;[^awsopts] Google speaks of cold, warm and hot patterns,[^gcpdr] NIST of cold, warm, hot and mirrored sites.[^nist34] | Strategy | Running in the recovery location | RTO is driven by | RPO is driven by | Main watch-out | |---|---|---|---|---| | Backup and restore | Nothing but copies of data and code | Rebuilding everything | Backup frequency | Restore is a control-plane action; AWS suggests scheduled restores to the DR Region[^awsopts] | | Pilot light | Data stores replicating; app servers "switched off"[^awsopts] | Switching on and scaling | Replication lag | Needs capacity to launch during the disaster | | Warm standby | Scaled-down, fully working copy that can take traffic at once[^awsopts] | Scaling to full size | Replication lag | Auto scaling is a control-plane dependency[^awsopts] | | Hot standby | Full capacity, not serving traffic | Traffic switch and promotion | Replication lag | Paying for idle capacity | | Active/active | Every Region serves traffic | Routing away from the failed Region | Replication lag; backups for corruption | Write conflicts; most complex and costly[^awsopts] | For the loss of one data centre, AWS notes that a well-architected multi-AZ workload may need only backup and restore; Region-level strategies are for regional disasters or regulation.[^awsopts] ::: figure Reference architecture: warm standby across two Regions ```mermaid flowchart TB U["Users"] --> R["`**Traffic control** DNS or global load balancer`"] K["`**Kept outside both Regions** runbook, endpoints, DR credentials`"] -.-> R R -->|"normal"| PA["`**Primary Region** app tier, full size`"] R -.->|"after invocation"| SA["`**Recovery Region** app tier, scaled down`"] PA --> PD["`**Primary data** database writer, object store`"] SA --> SD["`**Recovery data** a replica of each store; RPO = the worst store lag`"] PD -.-|"async, per store"| SD PD --> PB["`**Point-in-time backups** another account and Region`"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class PA,PD,SA,SD core class R,K decision class U,PB neutral ``` Text equivalent: Users reach the service through traffic control (DNS or a global load balancer). Normally traffic goes to the primary Region's full-size application tier and data stores; after invocation, to the recovery Region's scaled-down tier and a replica of each store. Each store replicates asynchronously with its own lag, so the service's RPO is the worst of them. Point-in-time backups go to another account and Region; runbook, endpoints and DR credentials sit outside both Regions. ::: ### End-to-end workflow: invoke, fail over, fail back **Happy path.** The order matters: fence writes, promote the data tier, then move traffic. ::: figure Invocation and failover: who hands what to whom ```mermaid %%{init: {"sequence": {"actorMargin": 24, "width": 120, "messageMargin": 30, "boxMargin": 6, "noteMargin": 6, "diagramMarginX": 8}}}%% sequenceDiagram participant IC as Incident lead participant AU as Authority participant CO as DR lead participant PL as Platform IC->>AU: Impact, duration, readiness AU->>CO: Invoke DR (logged) CO->>PL: Fence the old primary CO->>PL: Promote the standby PL-->>CO: New writer confirmed CO->>PL: Shift traffic PL-->>CO: Traffic shifted CO->>AU: Smoke tests green AU->>IC: Announce, plan failback ``` Text equivalent: The incident lead briefs the authority on impact, expected duration and standby readiness; the authority invokes DR, logged. The DR lead has the platform team fence writes and promote the standby database, waits for the new writer, then has traffic shifted. After green smoke tests the DR lead reports to the authority, who asks the incident lead to announce the recovery and plan failback. ::: **Exception paths.** - **The fault is in the data or configuration.** Replication has copied it: restore a clean point in time or roll back.[^awsha] - **The standby is not ready.** Quotas, capacity or drift block the failover: recover the most critical flows and escalate.[^awstest] - **The old primary may still accept writes.** Fencing means making sure it cannot. If you can reach it, turn on the application's write flag. If you cannot, no single control fences it: routing Off moves DNS traffic away,[^arc] but not batch jobs, queue consumers, the old Region's own application tier or clients with cached DNS. Have every client write through an endpoint that follows the new primary (Aurora's global writer endpoint, Cloud SQL's write endpoint[^cloudsqldr]), stop the old Region's writers once it answers (scale them to zero or revoke their database credentials), and treat what they wrote meanwhile as [lost writes](#reconciling-lost-writes). Aurora's own fencing is best-effort; it warns that "Failovers are also susceptible to split-brain issues" and advises taking applications offline first.[^aurora] - **The access path shares the failure.** In AWS's October 2025 us-east-1 event, STS returned errors there, Identity Center instances configured there could not sign users in, and some console sign-ins failed in other Regions, while DynamoDB global tables kept serving elsewhere.[^aws2025] Keep scripted access that avoids the failed Region. **Failback.** After an unplanned failover, unreplicated writes are missing from the new primary until they are [reconciled](#reconciling-lost-writes). Aurora then recommends a switchover "As a zero-data-loss method to fail back to the original primary Region after a failover".[^aurora] Azure warns that failback "can be complex to plan and implement".[^azdr] ::: figure Life after failover: the states before normal operation returns ```mermaid stateDiagram-v2 state "Running in recovery" as RIR state "Rebuilding primary" as RB state "Resynchronising" as RS state "Reconciling lost writes" as RC state "Ready to fail back" as RD state "Failing back (switchover)" as FB state "Normal operation" as N [*] --> RIR: failover complete RIR --> RS: primary back, old data saved RIR --> RB: primary site lost RB --> RS: rebuilt from code RS --> RC: replication caught up RC --> RD: data owners sign off RC --> N: keep recovery Region as primary RD --> FB: approved window FB --> N: RPO 0 N --> [*] ``` Text equivalent: After failover the service runs in the recovery location. A returning primary resynchronises once its unreplicated data is saved; a lost site is rebuilt from code first. Once replication has caught up, lost writes are reconciled. Then the recovery Region stays primary, or, after data owners sign off, the service switches back without data loss in an approved window. ::: #### Reconciling lost writes This Atlas's method, built on the database documentation: 1. **Preserve first.** Resynchronising overwrites the old primary: Aurora builds it a new storage volume after trying to snapshot the old one,[^aurora] and PostgreSQL's `pg_rewind` leaves it "analogous to a base backup" of the new primary.[^pgrewind] Copy Aurora's snapshot to a manual one: it expires with the old cluster's backup retention period.[^aurora] 2. **Diff a restored copy, never production,** against the new primary by key and update time, for writes after the last replicated commit, in every store the service needs. 3. **Classify.** Rows only in the old copy are lost writes; rows changed on both sides are conflicts. Do not expect the database to merge them: Informix, for example, keeps lost transactions in a file but cannot reapply them, because conflicting updates may have happened on the new primary.[^informix] 4. **Replay through the application,** idempotently, so business rules and downstream events run; data owners settle conflicts. 5. **Record** what was replayed, what was accepted as lost and who signed off ([integrity](#quality-security-and-compliance)). ## Practices, standards and tools ### Core practices and playbooks | Practice | Trigger | Owner | Output | Verification | |---|---|---|---|---| | BIA and tiering | Yearly; new service | Service owner | Tier, RTO, RPO, MTD | Compatible with dependencies[^rel13bp01] | | Dependency mapping | New dependency | Platform team | Hard dependencies: identity, DNS, data stores, CI/CD, suppliers | Each recovers within the dependant's RTO ([Lab 1](#hands-on-labs)) | | Runbooks as code | Every change | Platform team | Scripted failover with gates and an abort path | Dry run in CI; real run in drills | | Drift control | Every change to the primary | Platform team | Parity of config, images, quotas | Automated drift detection[^awstest][^rel13bp04] | | Timed drills | Schedule by tier | DR coordinator | Drill report with achieved RTO and RPO | [Lab 2](#hands-on-labs) | | Invocation tabletop | Twice a year | Invocation authority | Timed decision | Decision latency within the deadline | ### Methods, standards and frameworks | Source | Kind | What it contributes | |---|---|---| | NIST SP 800-34 Rev. 1 (2010)[^nist34] | US federal guidance; FISMA mandates the ISCPs it describes | Seven-step process, BIA, RTO/RPO/MTD, plan types (BCP, ISCP, DRP and others), exercise levels | | ISO 22301:2019, amended 2024[^iso22301] | Certifiable management-system requirements | The continuity programme DR sits inside | | ISO/IEC 27031:2025[^iso27031] | International guidance | ICT readiness for business continuity; MBCO, RPO and RTO | | DRI International Professional Practices[^dri] | Professional body of knowledge, certification | Programme development and assessment | | AWS Well-Architected REL13, Google and Microsoft guidance[^rel13bp01][^gcpdr][^azdr] | Vendor recommendations | Cloud strategies, testing and automation | | EU DORA, Articles 11, 12, 16 and 28[^dora] | Regulation for EU financial entities (simplified regime for some small entities) | Yearly testing, segregated restoration, redundant capacity, tested exit plans | | US HIPAA Security Rule[^hipaa] | Regulation for US health data | Contingency plan with a required disaster recovery plan | Mandatory rules come only from regulation, contracts or a chosen certification; vendor frameworks are recommendations. ### Tooling landscape Start from the job, then the product. Examples span clouds, on-premises virtualisation and Kubernetes (checked October 2026). | Job | Examples (verified) | Selection questions | |---|---|---| | Replicate and recover servers | AWS Elastic Disaster Recovery (block-level replication, pilot light).[^awsopts] Azure Site Recovery: Azure VMs and on-premises VMware, Hyper-V or physical servers, to Azure or a second site; drills that leave replication running[^asr] | Drills without touching production? Failback? | | On-premises virtual machines | VMware Live Site Recovery: ordered recovery plans, tested on a temporary copy of the data;[^vlsr] planned migration (synchronised) or disaster recovery (no protected site needed); separate test and run privileges[^vlsrrun] | Who may run a plan, not only test it? | | Disaster recovery as a service (DRaaS) | VMware Live Recovery Cloud, a "just-in-time" SaaS: VMs replicate to cloud storage and are recovered on VMware Cloud on AWS, also usable as an isolated environment for ransomware recovery[^vlrc] | Is provisioning the recovery site inside the RTO? | | Database-native recovery | Aurora global database: switchover (RPO 0), failover (RPO typically seconds).[^aurora] Cloud SQL Enterprise Plus advanced DR: replica failover (may lose lagging writes), zero-loss switchover, a write endpoint that follows the primary.[^cloudsqldr] DynamoDB global tables[^awsopts] | Fate of unreplicated writes and the old primary? | | Kubernetes workloads | Velero restores cluster resources and volumes into a cluster that reads the same object storage; its AWS and Azure plugins do not support migrating data between Regions, so cross-Region recovery needs its File System Backup[^velero] | Can volume data reach the recovery Region? | | Traffic control | Route 53 DNS failover and ARC routing controls;[^awsopts] Route 53 accelerated recovery for DNS changes (opt-in, November 2025);[^r53accel] Azure Front Door and Traffic Manager[^azdr] | A data-plane switch that works with the primary down? | | Recovery orchestration | ARC Region switch: plans of ordered steps, a data plane in each Region, manual or alarm-triggered,[^arcswitch] optional approval steps[^arcapproval] | Covers your resources? Who can trigger it? | | Drill tooling | AWS Fault Injection Service scenarios such as AZ power interruption and cross-Region connectivity loss[^fis] | Blast radius limited? Can you stop it? | Time-sensitive: ARC readiness check is closed to new customers; AWS recommends Region switch instead.[^arcreadiness] Lock-in matters most for orchestration tools, which end up holding your runbooks. ### Automation and AI Automate the *execution*; keep the *decision* human wherever a false alarm is expensive. AWS advises that automatically initiated failover "should be used with caution" and notes that "Manually initiated failover is therefore often used", with the steps themselves automated.[^awsopts] It lists console "click ops" and automatic recovery "in situations that present a high risk of data loss or unavailability" as anti-patterns, and asks for an abort mechanism.[^rel13bp05] Full automation suits stateless and active/active services. An AI agent that runs failover steps needs the gates, logs, permissions and abort path of a script. ## DisasterRecoveryOps in practice ### Reference implementation An illustrative tier-1 service, "checkout", on AWS: an Aurora global database and ARC routing controls across two Regions; names, numbers and ARNs are placeholders. AWS is one representative platform: a Cloud SQL replica failover[^cloudsqldr] or a VMware recovery plan[^vlsrrun] would keep the same policy, gates and drills. Four artefacts: a tier matrix (AWS suggests a similar one[^rel13bp01]), an invocation policy, a gated failover script and its least-privilege role. | Tier (example) | RTO | RPO | Strategy | Exercise | |---|---|---|---|---| | Tier 1: revenue and safety | 1 hour | 5 minutes | Warm or hot standby | Quarterly switchover; yearly failback | | Tier 2: internal critical | 8 hours | 1 hour | Pilot light | Half-yearly functional drill | | Tier 3: everything else | 72 hours | 24 hours | Backup and restore | Yearly restore into the recovery Region | **The decision deadline.** This Atlas's term for the latest point after impact at which an invocation can still meet the RTO: *deadline = RTO − slowest prepare-execute-verify time among recent drills − safety margin*. For checkout the slowest of the last three drills took 33 minutes ([Lab 2](#hands-on-labs)); the 5-minute margin is a judgement that a real event runs slower than a drill (widen it while drills are few): 60 − 33 − 5 = 22 minutes. At the deadline, an outage of unknown length counts as "longer than the RTO". Activation reuses the SLO burn-rate alert ([SRE](../sre/#practice-4-alerting-on-burn-rate)); lag comes from the secondary's own metric, as the primary may not answer (Aurora: `AuroraGlobalDBRPOLag`).[^aurora] _Example status: YAML syntax-checked by the Atlas validator; illustrative values._ ```yaml # Invocation policy for one service (illustrative values; adapt them) service: checkout tier: tier-1 # RPO: the worst lag of any store the service needs objectives: {rto_minutes: 60, rpo_minutes: 5} authority: # a role with deputies, never one named person accountable: head-of-platform-on-call deputies: [vp-engineering, cto] # the DR coordinator may invoke alone at the deadline if all invoke_when hold pre_authorised: true # RTO 60 - slowest of the last three drills 33 - margin 5 (a judgement) decision_deadline_minutes: 22 activation_criteria: # any one starts an assessment - SLO burn-rate alert for checkout in the primary Region for 10 minutes - provider reports a regional event affecting a hard dependency - primary database writer unreachable from two zones invoke_when: # all must hold - outage expected to outlast the RTO, or its length unknown at the deadline - fault is not replicated (no corruption, bad deploy or stolen credentials) - recovery Region drift check green within 24 hours - >- lag below rpo_minutes on the secondary's own metric, for every store (unreadable = worst recent value), or the owner accepts more loss in writing do_not_invoke_when: - the same fault is present in the recovery Region fence: primary_reachable: application write flag on and confirmed before promotion primary_unreachable: # routing Off alone steers only DNS traffic - every client writes through the global writer endpoint - old Region's routing control Off - its app tier, jobs and consumers scaled to zero as soon as it answers - writes accepted there meanwhile reconciled as lost writes abort_when: - recovery smoke tests fail twice - the old primary is reachable, still accepts writes and cannot be fenced failback: method: planned switchover after reconciliation across all stores approver: service-owner earliest: 24 hours of stable running in the recovery Region evidence: drill and invocation logs kept for 3 years ``` The script moves one service in either direction, so failback needs no edits. It defaults to a dry run, uses endpoints and ARNs recorded in advance (AWS recommends a local copy of the ARC cluster endpoints[^arccli]) and stops at human gates. A switchover calls `switchover-global-cluster` in the current primary's Region, as the Aurora guide shows and the API reference recommends;[^aurora][^apifailover] a failover calls `failover-global-cluster --allow-data-loss` in the target Region and reads the new writer from `IsWriter`.[^aurorafailcli][^auroradesc] _Example status: syntax-checked with `bash -n`; dry runs executed on 2026-10-17, and the retry, escalation and gate paths exercised with a stub `aws` command; no AWS call was executed. To confirm in a game day: the RDS API in the recovery Region accepts the failover call while the primary Region is impaired._ ```bash #!/usr/bin/env bash # Move one application's writer and traffic between its two Regions. # Usage: region_failover.sh to-recovery|to-primary switchover|failover \ # [--execute] # Default: a dry run that prints the commands. # Never add --safety-rules-to-override. set -euo pipefail DIRECTION="${1:-}"; MODE="${2:-}"; EXECUTE="${3:-}" usage() { echo "usage: $0 to-recovery|to-primary switchover|failover [--execute]" >&2 exit 2 } # Recorded in advance for both directions: never look these up in a disaster. ACCOUNT="111122223333" GLOBAL_CLUSTER="orders-global" CLUSTER_USE1="arn:aws:rds:us-east-1:${ACCOUNT}:cluster:orders-use1" CLUSTER_USW2="arn:aws:rds:us-west-2:${ACCOUNT}:cluster:orders-usw2" PANEL="arn:aws:route53-recovery-control::${ACCOUNT}:controlpanel/EXAMPLE" RC_USE1="${PANEL}/routingcontrol/use1" RC_USW2="${PANEL}/routingcontrol/usw2" CLUSTER_ENDPOINTS=( # "region url" pairs copied from DescribeCluster "us-west-2 https://host-aaaaaa.us-west-2.example.com/v1" "eu-west-1 https://host-bbbbbb.eu-west-1.example.com/v1" "ap-northeast-1 https://host-cccccc.ap-northeast-1.example.com/v1" ) case "$DIRECTION" in to-recovery) FROM=us-east-1 TO=us-west-2 TARGET="$CLUSTER_USW2" RC_OFF="$RC_USE1" RC_ON="$RC_USW2" ;; to-primary) FROM=us-west-2 TO=us-east-1 TARGET="$CLUSTER_USE1" RC_OFF="$RC_USW2" RC_ON="$RC_USE1" ;; *) usage ;; esac [[ "$MODE" == switchover || "$MODE" == failover ]] || usage run() { if [[ "$EXECUTE" == "--execute" ]]; then "$@" else printf 'DRY RUN:'; printf ' %s' "$@"; printf '\n'; fi } confirm() { # records intent; it does not enforce a second approver if [[ "$EXECUTE" != "--execute" ]]; then echo "GATE (skipped in dry run): $1"; return 0 fi read -r -p "GATE: $1 Type yes to continue: " answer [[ "$answer" == "yes" ]] || { echo "aborted at gate" >&2; exit 3; } } # 1. Human gates: the authority's decision, then the fence that fits the case. confirm "Invocation approved by the named authority and logged?" confirm "Fenced? Write flag on, or if unreachable, clients on the global endpoint?" # 2. Promote the standby. Switchover: call the current primary's Region, no data # loss. Failover: call the target Region; unreplicated writes are lost. if [[ "$MODE" == switchover ]]; then run aws rds switchover-global-cluster --region "$FROM" \ --global-cluster-identifier "$GLOBAL_CLUSTER" \ --target-db-cluster-identifier "$TARGET" else run aws rds failover-global-cluster --region "$TO" --allow-data-loss \ --global-cluster-identifier "$GLOBAL_CLUSTER" \ --target-db-cluster-identifier "$TARGET" fi if [[ "$EXECUTE" == "--execute" ]]; then writer="" for _ in $(seq 1 60); do # wait up to 30 minutes for the new writer writer=$(aws rds describe-global-clusters --region "$TO" \ --global-cluster-identifier "$GLOBAL_CLUSTER" --output text \ --query 'GlobalClusters[0].GlobalClusterMembers[?IsWriter].DBClusterArn') \ || writer="" # an API error must not end the script before the escalation [[ "$writer" == "$TARGET" ]] && break sleep 30 done if [[ "$writer" != "$TARGET" ]]; then echo "writer not promoted; escalate" >&2; exit 4 fi fi # 3. Shift traffic with the ARC data plane, trying each cluster endpoint. switched=false for entry in "${CLUSTER_ENDPOINTS[@]}"; do region="${entry%% *}"; url="${entry#* }" if run aws route53-recovery-cluster update-routing-control-states \ --region "$region" --endpoint-url "$url" \ --update-routing-control-state-entries \ "RoutingControlArn=$RC_OFF,RoutingControlState=Off" \ "RoutingControlArn=$RC_ON,RoutingControlState=On"; then echo "routing change accepted via $region endpoint"; switched=true; break fi echo "endpoint in $region failed; trying the next one" >&2 done if ! $switched; then echo "no cluster endpoint accepted the change; escalate" >&2; exit 5 fi confirm "Write flag cleared in $TO and smoke tests green?" echo "done: record timestamps in the drill log" ``` **Who may run it, and with what.** The script runs under an IAM role that can lose data and move production traffic, so the role allows only what the script calls, for one service: three RDS actions on its global database and two clusters, and the routing actions on its own control panel. ARC safety rules block unsafe routing changes, so the third statement denies overriding them, using the condition key AWS documents.[^arciam] Action names follow AWS's service reference.[^sar] _Example status: JSON syntax-checked; not attached to a real account._ ```json { "Version": "2012-10-17", "Statement": [{ "Sid": "PromoteOnlyThisGlobalDatabase", "Effect": "Allow", "Action": ["rds:FailoverGlobalCluster", "rds:SwitchoverGlobalCluster", "rds:DescribeGlobalClusters"], "Resource": [ "arn:aws:rds::111122223333:global-cluster:orders-global", "arn:aws:rds:us-east-1:111122223333:cluster:orders-use1", "arn:aws:rds:us-west-2:111122223333:cluster:orders-usw2" ] }, { "Sid": "SwitchOnlyThisControlPanel", "Effect": "Allow", "Action": ["route53-recovery-cluster:UpdateRoutingControlStates", "route53-recovery-cluster:GetRoutingControlState"], "Resource": [ "arn:aws:route53-recovery-control::111122223333:controlpanel/EXAMPLE/*" ] }, { "Sid": "NeverOverrideSafetyRules", "Effect": "Deny", "Action": "route53-recovery-cluster:UpdateRoutingControlState*", "Resource": "*", "Condition": { "Bool": {"route53-recovery-cluster:AllowSafetyRulesOverrides": "true"} } }] } ``` - **Who assumes it.** The DR coordinator and named deputies, through DR-only identities that can only assume this role, with MFA required, as ARC's IAM guidance recommends.[^arciam] Overriding safety rules is a separate break-glass step. - **Where the credentials live.** AWS's ARC guidance: long-lived credentials used only for DR tasks, kept "in an on-premises physical safe or a virtual vault", with everyday work on federated sign-in.[^arcbp] Keep the vault outside the failure domain, alert on and rotate after every use (this Atlas's practice), and test the path, token service included, from outside the primary Region.[^aws2025] - **What the gates do.** The `confirm` prompts record intent; anyone with the role can type "yes". Enforce approval with two people (one opens the vault, another runs the script) or an ARC Region switch manual-approval block, which waits for someone with the approval role but does not run in ungraceful mode.[^arcapproval] ### Hands-on labs Labs 1 and 2 need only Python 3.8+. Lab 3 needs a non-production AWS environment and was **not executed** for this page. #### Lab 1 (beginner): check a DR portfolio against its objectives **Objective:** find services that cannot meet their RTO with a safety margin, can lose more data than the RPO, depend on slower services, or have stale drills. **Prerequisites:** Python 3.8+, 30 minutes, no cloud account. **Status:** executed with Python 3.8.10, last on 2026-10-17; the output below is the actual output. 1. Save the portfolio as `workloads.json` and the checker as `dr_plan_check.py`. 2. Run `python3 dr_plan_check.py workloads.json`, then `echo $?`. 3. Fix `identity`: switch it to replication (`"data": {"mechanism": "replication", "worst_lag_min": 1}`), cut `execute` to 25 minutes and set `last_drill` to `"2026-10-01"`. Then pre-authorise the decision: set `decide` to 15 for `identity` and `checkout-api`. Run again. _Example status: JSON and Python syntax-checked; executed._ ```json { "as_of": "2026-10-16", "margin_min": 5, "max_days_since_drill": {"tier-1": 90, "tier-2": 180, "tier-3": 365}, "workloads": [ {"name": "checkout-api", "tier": "tier-1", "rto_min": 60, "rpo_min": 5, "steps_min": {"detect": 5, "decide": 20, "execute": 25, "verify": 10}, "data": {"mechanism": "replication", "worst_lag_min": 2}, "depends_on": ["payments-db", "identity"], "last_drill": "2026-08-20"}, {"name": "payments-db", "tier": "tier-1", "rto_min": 30, "rpo_min": 1, "steps_min": {"detect": 3, "decide": 5, "execute": 5, "verify": 5}, "data": {"mechanism": "replication", "worst_lag_min": 0.5}, "depends_on": [], "last_drill": "2026-09-30"}, {"name": "identity", "tier": "tier-1", "rto_min": 60, "rpo_min": 15, "steps_min": {"detect": 5, "decide": 20, "execute": 45, "verify": 10}, "data": {"mechanism": "backup", "interval_min": 60, "copy_delay_min": 15}, "depends_on": [], "last_drill": "2026-03-02"}, {"name": "reporting", "tier": "tier-3", "rto_min": 1440, "rpo_min": 1440, "steps_min": {"detect": 30, "decide": 60, "execute": 480, "verify": 60}, "data": {"mechanism": "backup", "interval_min": 1440, "copy_delay_min": 240}, "depends_on": ["checkout-api"], "last_drill": null} ] } ``` ```python """Check a DR portfolio: time budget, data-loss window, dependencies, drill age. Usage: python3 dr_plan_check.py workloads.json (exit 1 if any check fails)""" import json import sys from datetime import date DECIDE_WARN_SHARE = 0.25 # warn when deciding takes over a quarter of the RTO def worst_loss(data): if data["mechanism"] == "replication": return data["worst_lag_min"] # backup: the interval plus the time to reach the copy you restore from return data["interval_min"] + data["copy_delay_min"] def check(portfolio): as_of = date.fromisoformat(portfolio["as_of"]) margin = portfolio["margin_min"] by_name = {w["name"]: w for w in portfolio["workloads"]} failures = 0 for w in portfolio["workloads"]: problems, warnings = [], [] estimate = sum(w["steps_min"].values()) need = estimate + margin if need > w["rto_min"]: problems.append( f"needs {need} min with margin > RTO {w['rto_min']}") share = w["steps_min"]["decide"] / w["rto_min"] if share > DECIDE_WARN_SHARE: warnings.append(f"deciding uses {share:.0%} of the RTO") loss = worst_loss(w["data"]) if loss > w["rpo_min"]: problems.append( f"worst-case data loss {loss:g} min > RPO {w['rpo_min']} min") for dep in w["depends_on"]: dep_need = sum(by_name[dep]["steps_min"].values()) + margin if dep_need > w["rto_min"]: problems.append( f"dependency {dep} needs {dep_need} min > this RTO") limit = portfolio["max_days_since_drill"][w["tier"]] if w["last_drill"] is None: problems.append("never drilled") else: age = (as_of - date.fromisoformat(w["last_drill"])).days if age > limit: problems.append(f"last drill {age} days ago > {limit}") status = "FAIL" if problems else ("WARN" if warnings else "PASS") failures += bool(problems) head = f"{status:4} {w['name']:<13} {w['tier']}" print(f"{head} estimate {estimate} of {w['rto_min']} min") for p in problems + warnings: print(f" - {p}") print(f"{failures} of {len(by_name)} workloads fail") return failures if __name__ == "__main__": with open(sys.argv[1], encoding="utf-8") as f: sys.exit(1 if check(json.load(f)) else 0) ``` **Expected output (step 2),** exit code 1: ```text FAIL checkout-api tier-1 estimate 60 of 60 min - needs 65 min with margin > RTO 60 - dependency identity needs 85 min > this RTO - deciding uses 33% of the RTO PASS payments-db tier-1 estimate 18 of 30 min FAIL identity tier-1 estimate 80 of 60 min - needs 85 min with margin > RTO 60 - worst-case data loss 75 min > RPO 15 min - last drill 228 days ago > 90 - deciding uses 33% of the RTO FAIL reporting tier-3 estimate 630 of 1440 min - worst-case data loss 1680 min > RPO 1440 min - never drilled 3 of 4 workloads fail ``` **Validation.** After step 3 only `reporting` fails: it was never drilled, and a daily backup that takes four hours to reach its copy can lose 28 hours against a 24-hour RPO (the copy delay taught in [BackupOps](../backupops/#core-practices-and-playbooks)). An estimate equal to the RTO fails: it leaves no room for a slow day. **Troubleshooting.** `KeyError: 'worst_lag_min'` or `'copy_delay_min'` means a data entry lacks a field; `ValueError: Invalid isoformat string` means a date is not `YYYY-MM-DD`. **Teardown:** delete the two files. #### Lab 2 (intermediate): score a drill from its timeline **Objective:** turn a drill's timestamps into achieved RTO and RPO and see where the time went. **Prerequisites:** Python 3.8+, 20 minutes. **Status:** executed with Python 3.8.10, last on 2026-10-17. The drill log is illustrative. 1. Save `drill-log.json` and `drill_report.py`; run `python3 drill_report.py drill-log.json`. 2. Move `declared` to `09:22` and the three later events nine minutes earlier (`09:24`, `09:43`, `09:55`). Run again. _Example status: JSON and Python syntax-checked; executed._ ```json { "workload": "checkout-api", "rto_min": 60, "rpo_min": 5, "events": { "disruption_start": "2026-10-14T09:00:00+00:00", "detected": "2026-10-14T09:06:00+00:00", "declared": "2026-10-14T09:31:00+00:00", "failover_started": "2026-10-14T09:33:00+00:00", "traffic_switched": "2026-10-14T09:52:00+00:00", "service_verified": "2026-10-14T10:04:00+00:00" }, "last_replicated_commit": "2026-10-14T08:57:30+00:00", "deviations": [ "DNS record TTL was 300 s, runbook assumed 60 s", "instance quota in the recovery Region too low; raised during the drill" ] } ``` ```python """Score a DR drill from its event log: achieved RTO and RPO, and where the time went. Usage: python3 drill_report.py drill-log.json (exit 1 if the RTO or the RPO was missed)""" import json import sys from datetime import datetime PHASES = [("detect", "disruption_start", "detected"), ("decide", "detected", "declared"), ("prepare", "declared", "failover_started"), ("execute", "failover_started", "traffic_switched"), ("verify", "traffic_switched", "service_verified")] def minutes(a, b): delta = datetime.fromisoformat(b) - datetime.fromisoformat(a) return delta.total_seconds() / 60 def report(log): ev = log["events"] rto = minutes(ev["disruption_start"], ev["service_verified"]) rpo = minutes(log["last_replicated_commit"], ev["disruption_start"]) print(f"Drill: {log['workload']}") for name, start, end in PHASES: m = minutes(ev[start], ev[end]) print(f" {name:<8}{m:5.1f} min {m / rto:4.0%}") rto_ok, rpo_ok = rto <= log["rto_min"], rpo <= log["rpo_min"] margin = log["rto_min"] - rto print(f"Achieved RTO {rto:.1f} min vs target {log['rto_min']}: " f"{'MET' if rto_ok else 'MISSED'} (margin {margin:+.1f})") print(f"Achieved RPO {rpo:.1f} min vs target {log['rpo_min']}: " f"{'MET' if rpo_ok else 'MISSED'}") for d in log["deviations"]: print(f" deviation: {d}") return rto_ok and rpo_ok if __name__ == "__main__": with open(sys.argv[1], encoding="utf-8") as f: sys.exit(0 if report(json.load(f)) else 1) ``` **Expected output (step 1),** exit code 1: ```text Drill: checkout-api detect 6.0 min 9% decide 25.0 min 39% prepare 2.0 min 3% execute 19.0 min 30% verify 12.0 min 19% Achieved RTO 64.0 min vs target 60: MISSED (margin -4.0) Achieved RPO 2.5 min vs target 5: MET deviation: DNS record TTL was 300 s, runbook assumed 60 s deviation: instance quota in the recovery Region too low; raised during the drill ``` **Validation.** The phase shares add up to 100% (±1 for rounding). In step 2 the declaration meets the 22-minute [decision deadline](#reference-implementation): achieved RTO becomes 55.0 minutes, margin +5.0, exit code 0. **Troubleshooting.** On Python 3.8–3.10, timestamps ending in `Z` raise `ValueError`; write `+00:00`. A negative phase means events are out of order. **Teardown:** none; keep the log as evidence. #### Lab 3 (advanced): planned switchover and failback game day **Objective:** prove in a non-production copy of a tier-1 service that the team can move the writer and traffic to the recovery Region and back without data loss, timing each phase. **Prerequisites:** an Aurora global database in two Regions, ARC routing controls on Route 53 records, a client writing one numbered row per second, AWS CLI v2 and the [failover role](#reference-implementation). **Cost and risk:** hourly ARC and cross-Region replication charges ([cost](#cost-capacity-and-sustainability)); use staging. **Status:** not executed. 1. **Dry run.** `bash region_failover.sh to-recovery switchover` prints the gates, then `aws rds switchover-global-cluster --region us-east-1 ...` and the routing update. 2. **Execute.** Run with `--execute`, answer the gates, and log the time of each event. 3. **Fail back.** Once both clusters report `SynchronizationStatus` `connected`,[^auroradesc] run `region_failover.sh to-primary switchover --execute`. 4. **Score.** Feed the timestamps to [Lab 2](#hands-on-labs). **Expected output.** After step 2, `describe-global-clusters` shows the recovery-Region cluster with `IsWriter` true and `get-routing-control-state` returns `On` for its control. **Validation.** No acknowledged row is missing: a switchover synchronises before promoting.[^aurora] **Troubleshooting.** Switchover needs the same major and minor engine versions in both clusters;[^aurora] a safety rule can reject a routing update (ARC's example allows only one replica on), so change both controls in one call;[^arc] traffic still reaching the old Region usually means cached DNS. **Teardown:** confirm the original topology, stop the client, delete any ARC cluster created for the lab. ### Use cases and scenarios Three sourced events, three disaster shapes. The lessons are this Atlas's analysis of what *service* recovery could do. **1. Site loss: the OVHcloud Strasbourg fire (2021).** *Problem:* in March 2021 fire destroyed the SBG2 data centre and partly damaged SBG1. *Context:* some customers' backups on the same site were lost; in 2023 a French commercial court ordered OVHcloud to pay damages to two of them, and OVHcloud said it would appeal.[^ovh] Where copies belong is a [BackupOps](../backupops/#use-cases-and-scenarios) lesson. *Lesson here:* could the service be *rebuilt elsewhere* within its RTO, with images, configuration, secrets, DNS and capacity outside the site? *Success criterion:* a drill that assumes the site, and the tooling hosted there, are gone. **2. One building, a regional outage: Google Cloud europe-west9 (2023).** *Problem:* a water leak and fire forced one of the Region's three buildings to power down. *Context:* Google's internal regional Spanner database, which backs IAM and the Region's control planes, lost quorum because, through a misconfiguration, it had "two of its three replicas in two different clusters in the building that was powered down". Customers' multi-zone Regional Persistent Disk volumes with a replica there turned read-only, and regional IAM stayed offline to resynchronise policies 13 hours out of date.[^gcp2023] *Lesson:* the provider's regional control plane is a dependency zone redundancy cannot remove, so tier-1 services need a second Region and tooling outside the impaired one. *Success criterion:* a drill that removes a whole Region, control plane included. **3. A logical fault everywhere: the CrowdStrike update (2024).** *Problem:* a faulty content update released at 04:09 UTC on 19 July 2024 and reverted at 05:27 UTC crashed Windows hosts with sensor 7.11 or later that received it;[^cspir] Microsoft estimated 8.5 million devices.[^mscs] *Context:* each device had to boot from recovery media or into safe mode, and the WinPE route could require its BitLocker recovery key.[^msrecovery] *Lesson:* a recovery site running the same software shares the fault, so failover does not help; the plan needs out-of-band access to recovery secrets and capacity for mass manual remediation (host recovery: [SysOps](../sysops/#use-cases-and-scenarios)). | Disaster shape | Example | Does Region failover help? | What does | |---|---|---|---| | Site or building loss | OVHcloud 2021 | Yes, if copies and build inputs are elsewhere | A rehearsed rebuild outside the site | | Regional impairment | europe-west9 2023 | Yes, with a second Region | Independent Region and tooling | | Logical or software fault | CrowdStrike 2024 | No: the standby shares the fault | Clean restore, staged rollout, out-of-band keys | ## Measure, govern, pay ### Metrics and KPIs Count recoveries, not documents: "services with a DR plan" is a vanity metric. | KPI | Definition and formula | Data source | Pitfall | Cadence | |---|---|---|---|---| | Achieved RTO | Service verified − impact start | Drill and event logs | Starting the clock at the declaration hides decision time | Every drill and event | | Achieved RPO | Impact start − last recovered commit, in time, for the worst store | Database and replication logs | Average lag hides the worst case | Every drill and event | | Decision latency | Declared − detected; compare with the deadline minus detection time | Incident timeline | Not recorded in tabletops | Every drill and tabletop | | RTO margin | Target RTO − achieved RTO | Drill report | One good drill is not a trend | Quarterly | | Exercise coverage | Tier-1 services with a timed drill inside the policy window ÷ tier-1 services | DR register | Counting tabletops as failover tests | Monthly | | Objective conformance | Services passing the plan check ÷ all services | [Lab 1](#hands-on-labs) output | Estimates never compared with drill results | Monthly | | Failback proven | Tier-1 services that failed back without data loss in the last year ÷ tier-1 services | Drill reports | Teams stop after failover | Yearly | #### Worked calculation From the Lab 2 drill: achieved RTO = 10:04 − 09:00 = 64 minutes, margin −4. Decision latency = 09:31 − 09:06 = 25 minutes, against 22 − 6 = 16 allowed: a 9-minute overrun. Achieved RPO = 09:00:00 − 08:57:30 = 2.5 minutes. From Lab 1: two of three tier-1 services were drilled within 90 days (67% coverage); one of four passed every check (25% conformance). The fix is a faster decision, not a faster database. ### Quality, security and compliance - **Security parity.** The recovery location needs the primary's guardrails, patching and logging. AWS recommends an account per Region for the strongest isolation and cross-account backup copies against insider threats or account compromise.[^awsopts] - **Compromise and ransomware.** Coordinate with [ResponseOps](../responseops/) before invoking; failing over into an environment the attacker can reach repeats the incident. DORA requires restores on own systems to use systems "physically and logically segregated from the source ICT system".[^dora] - **Integrity and records.** DORA requires checks and reconciliations when recovering and readily accessible records of activities during disruptions;[^dora] NIST's reconstitution validates data and functionality before the plan is deactivated.[^nist34] - **Residency.** Microsoft advises reflecting sovereignty and residency obligations in DR plans;[^azdr] the legal analysis belongs to [PrivacyOps](../privacyops/). ### Cost, capacity and sustainability NIST frames the choice as balancing the cost of disruption against the cost of recovery, which rises as the RTO shrinks.[^nist34] Drivers are standby capacity, replication and data transfer (Aurora Global Database bills replicated write I/O to each secondary Region[^aurorapricing]), licences, tooling and drill time. At October 2026 prices an ARC routing-control cluster costs $2.50 per hour, about $21,900 a year (2.50 × 24 × 365), and a Region switch plan $70 a month, $840 a year.[^arcprice] Capacity is the hidden cost. Quotas in the recovery Region must allow production scale,[^awsopts] and scaling up during the disaster depends on control planes; pre-provisioned, statically stable capacity costs more but removes that dependency.[^awsopts][^staticstab] Idle standby also consumes energy; pilot light and warm standby trade RTO for a smaller idle footprint ([GreenOps](../greenops/) covers measuring it). ### Assurance of the operating model Review the DR register monthly: coverage, conformance, open drift findings and overdue plan reviews. A rising decision latency means the policy or the authority is unclear. DORA requires independent internal audit reviews of response and recovery plans for in-scope financial entities other than microenterprises.[^dora] Readiness tools help but do not replace drills; AWS points to Resilience Hub for tracking likely RTO and RPO.[^awsopts] ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnose | Fix now | Prevent | |---|---|---|---|---| | Nobody can run the failover | Access path shares the failure (sign-in, SSO, token service, CI/CD)[^aws2025] | Try the DR access path from outside the primary | DR-only credentials from the vault; scripted CLI path[^arcbp] | Test that path in every drill | | Standby cannot reach production scale | Quotas or capacity[^awstest] | Quota and capacity errors | Request increases; shed load | Pre-provision; drift-check quotas | | Traffic still reaches the old Region | Cached DNS; routing change not applied | Query resolvers; read the routing control state | Wait out the TTL; retry another cluster endpoint[^arccli] | Lower TTLs; use global endpoints[^aurora] | | Writes fail after promotion | Application still points at the old cluster endpoint[^aurora] | Connection errors to the demoted writer | Switch to the global writer endpoint | Use global endpoints everywhere | | Recent transactions missing | Asynchronous replication lag at failover | Compare with the old primary's point-of-failure snapshot[^aurora] | Reconcile with data owners | Alarm on lag against the RPO | | Two writers (split brain) | Old primary not fenced; jobs, consumers or cached DNS bypass the routing control | Writes landing in both Regions | Stop the old Region's writers; reconcile | Fence before promotion; global writer endpoint for every client[^aurora] | | Failover "worked" but data is corrupt | The fault was logical and replicated[^awsha] | Corruption present in both Regions | Restore a clean point in time | "Fault replicated?" check in the invocation policy | #### Worked runbook: regional impairment of checkout's primary Region ::: example Illustrative scenario Times are minutes after impact (T); values from the [reference implementation](#reference-implementation) and Lab 2. ::: 1. **T+0 to T+6, detect.** SLO burn alerts fire for checkout; the provider reports an event affecting the database service. The incident commander opens the incident ([IncidentOps](../incidentops/)). 2. **T+6 to T+15, assess.** The DR coordinator checks the invoke conditions: regional fault, not in the data; drift check green; secondary lag 40 seconds; recovery-Region quotas pre-raised. 3. **T+15, recommend.** To the authority: about 33 minutes to recover, up to 40 seconds of writes at risk. 4. **T+22, decision deadline.** Still no provider estimate, so the outage counts as outlasting the RTO; under the pre-authorisation the coordinator invokes and logs why. 5. **T+22 to T+24, prepare.** The degraded primary still answers, so the write flag fences it; the gates in `region_failover.sh to-recovery failover --execute` are confirmed. 6. **T+24 to T+43, execute.** The standby cluster is promoted; routing switches through the first reachable cluster endpoint. 7. **T+43 to T+55, verify.** Smoke tests and a cross-store consistency query (no order without its invoice) pass; the flag is cleared; recovery is announced. 8. **Abort** if smoke tests fail twice, or the old primary is reachable, still accepts writes and cannot be fenced: hold traffic in maintenance, escalate, never split writes. 9. **After.** Within 24 hours [reconcile lost writes](#reconciling-lost-writes); schedule failback as a switchover; file the timeline. ### Anti-patterns and myths - **"We are multi-AZ, so we have DR."** Zone redundancy covers a data centre, not a Region or the provider's regional control plane ([europe-west9](#use-cases-and-scenarios)).[^gcp2023] - **"Fail over automatically on any failed health check."** Automate the steps, not the high-risk decision.[^rel13bp05] - **"Never in production."** AWS lists never exercising failovers in production as an anti-pattern for DR testing.[^rel13bp03] - **"Zero RTO and RPO for everything."** Unrealistic or over-strict objectives make recovery costly and complicated.[^rel13bp01] - **"We will think about failback later."** Unrehearsed, the return risks data loss or strands the service. ## Deciding and adopting ### When to use and when not to use **Invest in invocation-grade DR** when the BIA shows that losing a site or Region would cost more than standing recovery, when regulation or contracts set recovery objectives, or when many critical services depend on the workload. **Keep it light, or skip it,** when the workload can be rebuilt from code and backups within its tolerance, when recovery would cost more than the loss, or when the business knowingly accepts a low-probability disaster; AWS treats both of the last two as legitimate.[^awsbcp] Alternatives: zone-level high availability, graceful degradation, a manual workaround, or transferring the risk, for example through insurance ([RiskOps](../riskops/)). **Hidden responsibilities** come with every strategy above backup and restore: standby cost, drift control, drills, failback. ### Decision framework **Strategy.** Choose the cheapest strategy whose *measured* recovery, decision time included, meets the objectives. The bands are this Atlas's starting heuristics, not a standard. | Objectives (example bands) | Starting strategy | Confirm with | |---|---|---| | RTO a day or more; RPO at least the backup interval | Backup and restore | A timed restore into the recovery location | | RTO several hours | Pilot light | Switch-on and scale-out timed under quota limits | | RTO about an hour; RPO minutes | Warm standby | Drill with decision time included | | RTO minutes | Hot standby | Data-plane switch with no scaling step | | Near-zero RTO and writes that tolerate a conflict policy | Active/active | Region-evacuation test plus a data-corruption restore | **Invocation.** The tree applies NIST's activation criterion[^nist34] and AWS's warning about replicated faults,[^awsha] plus two judgements of this Atlas: [unknown at the deadline](#reference-implementation) counts as outlasting the RTO, and an unready standby is worse than a degraded primary. ::: figure Invocation decision: fail over, restore, wait or escalate ```mermaid %%{init: {"flowchart": {"nodeSpacing": 22, "rankSpacing": 30}}}%% flowchart TB A{{"`Impact beyond what HA absorbs?`"}} -->|"no"| N1["`Run it as an incident, not DR`"] A -->|"yes"| B{{"`Fault replicated to the standby?`"}} B -->|"no"| C{{"`Outage likely to outlast the RTO?`"}} B -->|"yes"| N2["`Do not fail over: restore a clean point in time or roll back`"] C -->|"no"| W["`Wait and degrade; log the decision`"] W -.->|"re-assess"| C C -->|"`yes, or unknown at the deadline`"| D{{"`Standby ready? Drift, capacity, access`"}} D -->|"yes"| F{{"`Replication lag within the RPO?`"}} D -->|"no"| E["`Escalate: partial recovery or rebuild`"] F -->|"yes"| I["`**Invoke: fail over**`"] F -->|"no"| G{{"`Owner accepts the extra loss in writing?`"}} G -->|"yes"| I G -->|"no"| W2["`Do not invoke yet: log, re-assess`"] classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class A,B,C,D,F,G decision class N2,E risk class I core class N1,W,W2 neutral ``` Text equivalent: If high availability absorbs the impact, run an ordinary incident. If the fault is replicated to the standby, restore a clean point in time or roll back. If the outage is unlikely to outlast the RTO, wait, degrade, log and re-assess (the dashed loop). If it is likely to, or its length is unknown at the decision deadline, check the standby's drift, capacity and access; if it is not ready, escalate to partial recovery or a rebuild. If it is ready, invoke when lag is within the RPO or the owner accepts the extra loss in writing; otherwise do not invoke yet: log and re-assess. ::: ### Maturity model The Atlas's proposed model, not an official one; each exit criterion is measurable. | Level | Observable behaviour | Exit criterion | |---|---|---| | 1. Documented | A plan and backups exist; objectives are not agreed | RTO and RPO signed off for every tier-1 service | | 2. Defined | Objectives from a BIA, strategies chosen, runbooks written, authority named | Plan check passes; invocation policy for every tier-1 service | | 3. Exercised | Timed drills on schedule; achieved RTO, RPO and decision latency recorded | 100% tier-1 exercise coverage; two consecutive drills inside the RTO | | 4. Engineered | Scripted failover with a least-privilege role, drift detection, failback rehearsed, tooling independent of the primary | Failback proven for every tier-1 service; no console steps | | 5. Assured | Routine production switchovers; unannounced drill elements; evidence produced automatically | Sustain; review objectives yearly | ### Adoption roadmap: 30/60/90 days | Window | Actions | Milestone | |---|---|---| | Days 1–30 | Inventory top services; quick BIA; tiers and draft objectives; name authorities; register and plan check; one tabletop | A decision deadline for each tier-1 service | | Days 31–60 | Invocation policies and runbooks for tier 1; drift checks; failover role and DR access-path test; first planned switchover in staging | First timed drill report | | Days 61–90 | One tier-1 production switchover or functional drill; failback rehearsal in staging; KPI dashboard | Level 3 for one service; plan for the rest | Dependencies: infrastructure as code for the recovery location, a standby budget and the business owners' time. ## Toolkit ### Checklists **Preflight: before you rely on a strategy** - [ ] RTO, RPO and tier agreed by the service owner, compatible with every hard dependency - [ ] Point-in-time copies in another account and location, for every store - [ ] Recovery-location quotas and capacity cover production load - [ ] Endpoints, credentials and runbooks outside the primary failure domain - [ ] First planned switchover completed and timed **Invocation readiness and governance: monthly** - [ ] Drift check green for images, configuration and quotas; findings older than 30 days escalated - [ ] Failover role and DR credentials tested from outside the primary; vault accesses explained - [ ] Authority, deputies and contact routes current - [ ] Decision deadline recalculated from recent drills **During failover** - [ ] Invocation logged with time, reason and approver - [ ] Writes fenced before promotion (write flag; if unreachable, clients on the global writer endpoint, old writers stopped) - [ ] New writer confirmed before traffic moves - [ ] Smoke tests and a cross-store consistency query pass before reopening **Failback** - [ ] Primary resynchronised; lost writes reconciled and signed off by data owners - [ ] Switchover window approved; customers informed if needed - [ ] Post-failback drill report filed ### Templates **One-page DR plan (per service)** | Field | Content | |---|---| | Service, tier, owner | | | RTO, RPO, MTD (from the BIA) | | | Strategy and recovery location | | | Hard dependencies and their objectives | | | Invocation authority, deputies, decision deadline | | | Activation, invoke, do-not-invoke and abort criteria | | | Runbook link (stored outside the primary) and last dry run | | | Failback method and approver | | | Last drill: date, achieved RTO, achieved RPO, actions | | **Invocation decision record** | Field | Content | |---|---| | Time of impact, detection, decision | | | Criteria met (quote the policy) | | | Expected data loss accepted by | | | Alternatives considered (wait, restore, degrade) | | | Decision and approver | | **Drill report:** scenario; Lab 2 timeline; achieved RTO and RPO; decision latency against the deadline; deviations; actions with owners and dates; failback result. **BIA worksheet questions,** adapted from AWS's list:[^rel13bp01] maximum downtime and data loss before unacceptable impact; kinds of impact; whether lost data can be recreated; dependants' objectives; zone versus Region objectives; seasonal peaks; contractual penalties. ### FAQ **Is RTO the same as MTTR?** No. RTO is a target agreed in advance; MTTR is an average over many incidents with known statistical caveats ([IncidentOps](../incidentops/)). **How often should we test?** Scale by impact, within any regulatory minimum ([operating model](#operating-model-and-governance)); this Atlas suggests a quarterly timed drill per tier-1 service, plus tabletops for the decision. **We run on premises. Does any of this apply?** Yes, unchanged; even split brain recurs. In VMware Live Site Recovery, if the protected site comes back after a disaster recovery, virtual machines run on both sites until you run the plan again to power off the old ones.[^vlsrrun] **Do containers and Kubernetes change DR?** The logic stays; the inventory changes. Rebuild cluster objects from code ([GitOps](../gitops/)) and recover volume data separately, after checking that it can reach the recovery Region (Velero's AWS and Azure plugins do not migrate data between Regions[^velero]). Running clusters belongs to [KubernetesOps](../kubernetesops/). **Our provider runs the infrastructure. Is DR their job?** Partly: "DR isn't an automatic feature of Azure", says Microsoft; you map service capabilities to your own plan.[^azdr] **Does DR cover ransomware?** Only with [ResponseOps](../responseops/): containment and forensics first, then recovery to clean, segregated systems from a point before the compromise.[^dora] ## Going further ### Advanced topics and emerging direction - **Smaller blast radius first (established).** Zonal shift and autoshift move traffic off an impaired zone;[^arc] statically stable designs keep working without the control plane.[^staticstab] Both make Region-level invocations rarer. - **Residency-bound and exit-ready DR (emerging).** Microsoft describes cloud-to-site failover for a cloud exit, which Azure Site Recovery does not support natively.[^azdr] - **Continuous verification (growing).** Fault injection that cuts cross-Region connectivity and pauses replication[^fis] tests detection and invocation, not only mechanics. ### Learning path | Stage | What to do | |---|---| | Prerequisites | [BackupOps](../backupops/), [IncidentOps](../incidentops/), DNS, one cloud's Regions | | One-day primer | NIST SP 800-34 chapter 3,[^nist34] the AWS DR whitepaper,[^awsdr] Labs 1 and 2 | | One week | An invocation policy for one service, tested in a tabletop | | Thirty days | Lab 3 in staging: switchover, failback, drill report | | Specialise | [DBOps](../dbops/) (database failover), [ResponseOps](../responseops/) (cyber recovery), [RiskOps](../riskops/) (continuity), DRI International[^dri] | | Portfolio project | A DR register checked in CI and a dashboard of achieved RTO, RPO and decision latency | **Where to start by role.** Beginners: [In 30 seconds](#in-30-seconds), the [glossary](#glossary-and-mental-model) and Lab 1. Engineers: the [reference implementation](#reference-implementation), Labs 2 and 3 and the [worked runbook](#worked-runbook-regional-impairment-of-checkouts-primary-region). Architects: the [building blocks](#building-blocks-of-a-recovery) and the [decision framework](#decision-framework). Leaders: [when to use](#when-to-use-and-when-not-to-use), [cost](#cost-capacity-and-sustainability), the [maturity model](#maturity-model) and the [judgement call](#a-judgement-call-fail-over-now-or-wait-for-the-provider). ## Wrap-up ### Executive recap and cheat sheet - **Decide:** tiers and objectives with the business; one named invocation authority per service; a decision deadline. - **Build:** the cheapest strategy whose *measured* recovery meets the objectives; point-in-time copies beside replication; a least-privilege failover role; access and runbooks outside the failure domain. - **Practise:** timed drills by tier; tabletops for the decision; failback at least yearly for tier 1. - **Measure:** achieved RTO and RPO, decision latency, coverage, conformance, failback proven. - **Watch for:** replicated faults, drift, quotas, DNS caches, split brain. - **Next action:** compute the decision deadline for your three most critical services. ### A judgement call: fail over now, or wait for the provider? ::: decision Dilemma **Situation (illustrative).** T+20 into a regional event at your cloud provider. Checkout's RTO is 60 minutes, its RPO 5 minutes, its decision deadline 22 minutes. The secondary's lag metric reads 40 seconds, the standby passed last night's drift check, and the provider says the root cause has been found. The head of sales asks you to wait: failing over loses up to 40 seconds of orders and means a failback later. ::: **Option A, wait:** no data loss, no failback, no botched failover. **Option B, invoke at T+22:** meets the RTO if the drilled 33 minutes hold, at the cost of up to 40 seconds of writes to reconcile and a planned failback. **What the evidence says.** "Root cause found" is not "service restored". In AWS's October 2025 us-east-1 event, the DNS problem was identified about 50 minutes after it began and DNS restored after 2 hours 37 minutes, but new EC2 launches were normal only after 14 hours 2 minutes, and the event ended after 14 hours 32 minutes.[^aws2025] **Decision process.** The question is whether *your* service is back before T+60, after dependent backlogs clear; a status page cannot tell you, and the policy written in calm conditions says to invoke at T+22 and log why. Waiting looks attractive because its costs stay hidden until T+60. **Where the answer changes.** With a credible estimate well inside the RTO (mastery question 4), lag above the RPO, or a failover never rehearsed, waiting can be the better bet; record that reasoning too. Drill data, not the loudest voice, should set the deadline. ### Learning objectives and mastery check After this page you can **explain** RTO, RPO and the difference between HA, backup and DR; **design** a strategy per tier; **write** an invocation policy with a decision deadline; **run** and score a drill; and **evaluate** when not to fail over. 1. RTO 60 minutes; slowest recent drill 33 minutes from declaration to verified service; detection usually 6 minutes; margin 5 minutes. When is the decision deadline, and how long may deciding take? ::: answer Show answer for question 1 Deadline = 60 − 33 − 5 = 22 minutes after impact; after 6 minutes of detection, deciding may take 16. ::: 2. A faulty schema migration corrupted order records ten minutes ago. The recovery-Region replica is healthy and 2 seconds behind. Do you fail over? ::: answer Show answer for question 2 No. The corruption has replicated.[^awsha] Restore a clean point in time or roll back, with [BackupOps](../backupops/) and [DBOps](../dbops/). ::: 3. Why script failover with routing controls rather than edit weighted DNS records in the console? ::: answer Show answer for question 3 Record-weight changes are a control-plane operation, less resilient than data-plane routing controls,[^awsopts] whose cluster keeps at least three of five Regional endpoints available;[^arcapi] console-driven recovery is an AWS anti-pattern.[^rel13bp05] Even Route 53's opt-in accelerated recovery targets 60 minutes to regain DNS record changes if US East (N. Virginia) is unavailable,[^r53accel] far beyond a 22-minute deadline. ::: 4. Checkout (RTO 60, deadline T+22) is degraded. At T+15 the provider credibly estimates full recovery at T+40. Do you invoke at the deadline? ::: answer Show answer for question 4 No. The outage passes the deadline but not the RTO: waiting restores service at T+40 with no data loss, while failing over would finish near T+55 and lose writes. The criterion is an outage longer than the RTO;[^nist34] the deadline only says when an outage of *unknown* length counts as one. Keep re-assessing; invoke if the estimate slips. ::: 5. After an unplanned failover, finance reports about 30 seconds of orders missing. Where are they, and what do you do before failing back? ::: answer Show answer for question 5 Writes that had not replicated, possibly still in Aurora's point-of-failure snapshot. [Reconcile](#reconciling-lost-writes) with the data owners, then fail back with a switchover, which loses no data.[^aurora] ::: ### Conclusion DisasterRecoveryOps is a new name for an old, sound discipline: agree what each service needs, build the cheapest recovery that meets it, and rehearse. The sources agree on the mechanics; their warnings are about what surrounds them: the decision nobody owns, the standby nobody checked, the access path that failed with the primary, the lost writes nobody reconciled. A recovery capability is real only when someone may invoke it before the deadline, the path has run recently enough to trust, and the way back is as well understood as the way out. Prove all three with timestamps. ## Sources and further reading Product facts and prices are as of October 2026; claim-level evidence is in `research/disasterrecoveryops-ledger.md`. [^nist34]: [Contingency Planning Guide for Federal Information Systems, NIST SP 800-34 Rev. 1 (PDF)](https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-34r1.pdf) — M. Swanson et al., NIST, May 2010 (updated 2010-11-11). Checked 2026-10-16. [^nist34v1]: [SP 800-34, Contingency Planning Guide for Information Technology Systems](https://csrc.nist.gov/pubs/sp/800/34/final) — NIST CSRC, June 2002 (withdrawn 2010-05-31). Checked 2026-10-16. [^iso27031]: [ISO/IEC 27031:2025 Cybersecurity — Information and communication technology readiness for business continuity](https://webstore.iec.ch/en/publication/107542) — IEC Webstore, edition 2.0, 2025-05-16. Checked 2026-10-16. [^iso22301]: [ISO 22301:2019/Amd 1:2024 Security and resilience. Business continuity management systems. Requirements. Amendment 1](https://knowledge.bsigroup.com/products/security-and-resilience-business-continuity-management-systems-requirements-amendment-1-climate-action-changes) — BSI Knowledge, published 2024-02-23. Checked 2026-10-16. [^dri]: [Professional Practices](https://drii.org/resources/professionalpractices/EN) — DRI International, 2023 edition. Checked 2026-10-16. [^opsio]: [DROps: Disaster Recovery Operations Guide](https://opsiocloud.com/knowledge-base/what-is-drops-disaster-recovery-ops/) — Fredrik Karlsson, Opsio, 2025-08-28 (updated 2025-10-15). Checked 2026-10-16. [^commvault]: [How Resilience Operations (ResOps) Drives Enterprise Recoverability](https://www.commvault.com/blogs/how-resilience-operations-resops-drives-enterprise-recoverability) — Ben Herzberg, Commvault, 2026-07-06. Checked 2026-10-16. [^fema]: [National Disaster Recovery Framework](https://www.fema.gov/emergency-managers/national-preparedness/frameworks/recovery) — FEMA, updated 2025-05-14. Checked 2026-10-16. [^awsdr]: [Disaster Recovery of Workloads on AWS: Recovery in the Cloud](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-workloads-on-aws.html) — AWS whitepaper, 2021-02-12. Checked 2026-10-16. [^awswhat]: [What is a disaster?](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/what-is-a-disaster.html) — AWS whitepaper section. Checked 2026-10-16. [^awsha]: [High availability is not disaster recovery](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/high-availability-is-not-disaster-recovery.html) — AWS whitepaper section. Checked 2026-10-16. [^awsbcp]: [Business Continuity Plan (BCP)](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/business-continuity-plan-bcp.html) — AWS whitepaper section. Checked 2026-10-16. [^awsdetect]: [Detection](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/detection.html) — AWS whitepaper section. Checked 2026-10-16. [^awsopts]: [Disaster recovery options in the cloud](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html) — AWS whitepaper section. Checked 2026-10-16. [^awstest]: [Testing disaster recovery](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/testing-disaster-recovery.html) — AWS whitepaper section. Checked 2026-10-16. [^rel13bp01]: [REL13-BP01 Define recovery objectives for downtime and data loss](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_planning_for_recovery_objective_defined_recovery.html) — AWS Well-Architected Framework. Checked 2026-10-16. [^rel13bp03]: [REL13-BP03 Test disaster recovery implementation to validate the implementation](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_planning_for_recovery_dr_tested.html) — AWS Well-Architected Framework. Checked 2026-10-16. [^rel13bp04]: [REL13-BP04 Manage configuration drift at the DR site or Region](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_planning_for_recovery_config_drift.html) — AWS Well-Architected Framework. Checked 2026-10-16. [^rel13bp05]: [REL13-BP05 Automate recovery](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_planning_for_recovery_auto_recovery.html) — AWS Well-Architected Framework. Checked 2026-10-16. [^gameday]: [REL12-BP05 Conduct game days regularly](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_testing_resiliency_game_days_resiliency.html) — AWS Well-Architected Framework. Checked 2026-10-16. [^staticstab]: [Static stability using Availability Zones](https://builder.aws.com/content/3K9SiqiJ369TGkkGzqPM95BuYWI/static-stability-using-availability-zones) — Becky Weiss and Mike Furr, Amazon Builders' Library, 2019-12-05 (now on AWS Builder Center, which lists Mike Furr only). Checked 2026-10-16. [^arc]: [What is ARC?](https://docs.aws.amazon.com/r53recovery/latest/dg/what-is-route53-recovery.html) — AWS ARC Developer Guide. Checked 2026-10-16. [^arcswitch]: [Region switch in ARC](https://docs.aws.amazon.com/r53recovery/latest/dg/region-switch.html) — AWS ARC Developer Guide. Checked 2026-10-16. [^arcreadiness]: [Amazon Application Recovery Controller (ARC) readiness check availability change](https://docs.aws.amazon.com/r53recovery/latest/dg/arc-readiness-availability-change.html) — AWS documentation. Checked 2026-10-16. [^arcapi]: [Getting and updating routing control states using the ARC API (recommended)](https://docs.aws.amazon.com/r53recovery/latest/dg/routing-control.update.api.html) — AWS documentation. Checked 2026-10-16. [^arccli]: [List and update routing controls and states with the AWS CLI](https://docs.aws.amazon.com/r53recovery/latest/dg/getting-started-cli-routing.control-state.html) — AWS documentation. Checked 2026-10-16. [^arcprice]: [Amazon Application Recovery Controller pricing](https://aws.amazon.com/application-recovery-controller/pricing/) — AWS, as of October 2026. Checked 2026-10-16. [^arcbp]: [Best practices for routing control in ARC](https://docs.aws.amazon.com/r53recovery/latest/dg/route53-arc-best-practices.regional.html) — AWS documentation. Checked 2026-10-16. [^arciam]: [Identity-based policy examples for routing control in ARC](https://docs.aws.amazon.com/r53recovery/latest/dg/security_iam_id-based-policy-examples-routing.html) — AWS documentation. Checked 2026-10-16. [^arcapproval]: [Manual approval execution block](https://docs.aws.amazon.com/r53recovery/latest/dg/manual-approval-block.html) — AWS ARC Developer Guide. Checked 2026-10-16. [^sar]: [Simplified AWS service information for programmatic access](https://docs.aws.amazon.com/service-authorization/latest/reference/service-reference.html) — AWS Service Authorization Reference; `rds` and `route53-recovery-cluster` files (v1.4) read. Checked 2026-10-16. [^aurora]: [Using switchover or failover in Amazon Aurora Global Database](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-disaster-recovery.html) — Amazon Aurora User Guide. Checked 2026-10-16 and 2026-10-17. [^aurorafailcli]: [aws rds failover-global-cluster](https://docs.aws.amazon.com/cli/latest/reference/rds/failover-global-cluster.html) — AWS CLI 2.38.1 Command Reference. Checked 2026-10-16. [^apifailover]: [FailoverGlobalCluster](https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_FailoverGlobalCluster.html) — Amazon RDS API Reference. Checked 2026-10-16. [^aurorapricing]: [Amazon Aurora pricing](https://aws.amazon.com/rds/aurora/pricing/) — AWS, as of October 2026. Checked 2026-10-16. [^auroradesc]: [aws rds describe-global-clusters](https://docs.aws.amazon.com/cli/latest/reference/rds/describe-global-clusters.html) — AWS CLI Command Reference. Checked 2026-10-16. [^fis]: [Scenarios reference](https://docs.aws.amazon.com/fis/latest/userguide/scenario-library-scenarios.html) — AWS Fault Injection Service User Guide. Checked 2026-10-16. [^r53accel]: [Amazon Route 53 announces accelerated recovery for managing public DNS records](https://aws.amazon.com/about-aws/whats-new/2025/11/amazon-route-53-accelerated-recovery-managing-public-dns-records/) — AWS What's New, 2025-11-26. Checked 2026-10-16. [^aws2025]: [Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/101925/) — Amazon Web Services, October 2025. Checked 2026-10-16. [^gcpdr]: [Disaster recovery planning guide](https://docs.cloud.google.com/architecture/dr-scenarios-planning-guide) — Google Cloud Architecture Center, updated 2024-07-05. Checked 2026-10-16. [^gcpblog]: [Building resilient systems to weather the unexpected](https://cloud.google.com/blog/products/management-tools/sre-principles-in-practice-for-business-continuity) — Ben Treynor, Google Cloud Blog, 2020-06-11. Checked 2026-10-16. [^gcp2023]: [Multiple Google Cloud services in the europe-west9-a zone are impacted (incident report)](https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY) — Google Cloud Service Health, report posted 2023-06-23. Checked 2026-10-16. [^azdr]: [What are business continuity, high availability, and disaster recovery?](https://learn.microsoft.com/en-us/azure/reliability/disaster-recovery-overview) — Microsoft Learn, updated 2026-06-01. Checked 2026-10-16 and 2026-10-17. [^asr]: [About Azure Site Recovery](https://learn.microsoft.com/en-us/azure/site-recovery/site-recovery-overview) — Microsoft Learn, updated 2026-09-23. Checked 2026-10-16 and 2026-10-17. [^vlsr]: [Overview of VMware Live Site Recovery](https://techdocs.broadcom.com/us/en/vmware-cis/live-recovery/live-site-recovery/9-0/overview/overview-of-vmware-site-recovery-manager.html) — Broadcom TechDocs, version 9.0, updated 2026-07-14. Checked 2026-10-17. [^vlsrrun]: [Performing a Planned Migration or Disaster Recovery by Running a Recovery Plan](https://techdocs.broadcom.com/us/en/vmware-cis/live-recovery/live-site-recovery/9-0/how-do-i-protect-my-environment/creating-testing-and-running-recovery-plans/performing-a-planned-migration-or-disaster-recovery-by-running-a-recovery-plan.html) — Broadcom TechDocs, VMware Live Site Recovery 9.0, updated 2026-07-14. Checked 2026-10-17. [^vlrc]: [Using VMware Live Recovery Cloud](https://techdocs.broadcom.com/us/en/vmware-cis/live-recovery/live-cyber-recovery/saas/using-vmware-live-cyber-recovery.html) — Broadcom TechDocs, updated 2026-09-01. Checked 2026-10-17. [^cloudsqldr]: [About disaster recovery (DR) in Cloud SQL](https://docs.cloud.google.com/sql/docs/mysql/intro-to-cloud-sql-disaster-recovery) — Google Cloud, Cloud SQL for MySQL documentation, updated 2026-10-15. Checked 2026-10-17. [^velero]: [Cluster migration](https://velero.io/docs/v1.18/migration-case/) — Velero documentation, version 1.18 (a CNCF sandbox project). Checked 2026-10-17. [^pgrewind]: [pg_rewind](https://www.postgresql.org/docs/current/app-pgrewind.html) — PostgreSQL 18 documentation, The PostgreSQL Global Development Group. Checked 2026-10-17. [^informix]: [Lost-and-found transactions](https://help.hcl-software.com/hclinformix/15.0/admin/ids_admin_0870.html) — HCL Informix 15.0 Administrator's Guide, HCL Software. Checked 2026-10-17. [^dora]: [Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA)](https://eur-lex.europa.eu/eli/reg/2022/2554/oj/eng) — Official Journal of the EU, L 333, 2022-12-27 (EUR-Lex); Articles 11, 12, 16, 28 and 64 cited. Checked 2026-10-16; Article 28 checked 2026-10-17. [^hipaa]: [45 CFR 164.308 Administrative safeguards](https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-C/part-164/subpart-C/section-164.308) — Electronic Code of Federal Regulations, current text. Checked 2026-10-16. [^ovh]: [OVHcloud must pay damages for lost backup data](https://blocksandfiles.com/2023/03/23/ovh-cloud-must-pay-damages-for-lost-backup-data/) — Chris Mellor, Blocks and Files, 2023-03-23. Checked 2026-10-16. [^cspir]: [Preliminary Post Incident Review: Content Configuration Update Impacting the Falcon Sensor and the Windows Operating System](https://www.crowdstrike.com/en-us/blog/falcon-content-update-preliminary-post-incident-report/) — CrowdStrike, 2024-07-24 (updated 2024-07-25). Checked 2026-10-16. [^mscs]: [Helping our customers through the CrowdStrike outage](https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/) — David Weston, Microsoft, 2024-07-20. Checked 2026-10-16. [^msrecovery]: [New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints](https://techcommunity.microsoft.com/blog/intunecustomersuccess/new-recovery-tool-to-help-with-crowdstrike-issue-impacting-windows-endpoints/4196959) — Intune Support Team, Microsoft Community Hub, 2024-07-20 (tool updated 2024-07-22). Checked 2026-10-16. **Changelog.** 2026-10-16: first edition (two of three labs executed). 2026-10-16, r1 (independent review): "outlast the RTO" criterion; least-privilege role; switchover call; fencing; multi-store RPO; DORA from EUR-Lex. 2026-10-17, r2 (score 88): on-premises, DRaaS, Google Cloud and Kubernetes tooling; recovery order; DR communications; role entry points; decision-tree loop. 2026-10-17, r3 (score 91): layered fencing for an unreachable primary; a method for reconciling lost writes; a compact decision tree; NIST origins wording; fewer restatements.