--- title: "BackupOps" slug: backupops category: reliability term_status: overlapping-synonym evidence_status: verified summary: "BackupOps is a rare label for backup and recovery operations: keeping independent, protected copies of data and proving with scheduled restore tests that they can be restored recently enough and fast enough for the business." audience: [Backup and infrastructure engineers, SREs and platform engineers, IT and engineering managers, Auditors and risk owners, Beginners] last_verified: 2026-10-17 key_takeaways: - "BackupOps is not an established discipline: the label appears in a 2022 trade-press headline and on one consultancy page, with no authoritative definition. The practice it names, backup and recovery, is long established and set out by NIST, the UK NCSC and EU financial regulation." - "A backup is a claim and a restore is the proof. At GitLab in 2017 the backup job had been failing silently, and nobody owned testing it." - "Independence is the property that matters: a copy that shares a failure domain, credentials or encryption keys with the source can be lost with it. Replicas and many snapshots fail this test." - "Attackers go after backups and the backup servers themselves. Keep at least one copy that stolen credentials cannot delete, retain by time rather than by count, patch and isolate the backup system, protect the keys, and alert on attempts to delete." - "Measure what is proven recoverable, not job success: in this page's worked example a 99% job success rate sits next to 17% of assets proven recoverable." - "Immutability has a price. Locked copies cannot be removed early, even for cost or erasure requests, and a compliance-mode lock cannot be undone (in AWS Backup, once a grace period of at least 72 hours ends)." related: [disasterrecoveryops, storageops, dbops, responseops, sre, complianceops, kubernetesops, privacyops] --- ## What BackupOps is ### In 30 seconds: the definition **BackupOps** is an informal name for *backup and recovery operations*: deciding which data needs copies, making those copies independent of the systems they protect, keeping them safe from deletion, tampering and lost keys, and proving on a schedule that they can be restored recently enough and fast enough. Google's SRE book passes on the saying: "No one really wants to make backups; what people really want are restores."[^sre-di] This page therefore treats BackupOps as **restore engineering**: a copy counts only when a test has restored it. It serves engineers, risk owners and auditors; beginners can stop after this part. ::: note Term status: an overlapping synonym, not a discipline Searches on 2026-10-16 found the label only in a 2022 trade-press headline, about automating responses to failed backup jobs, and on one consultancy page (2025) that cites no sources.[^bocada][^opsio] No standard, professional body or certification defines BackupOps. The practice behind it is long established: the NIST Cybersecurity Framework, the UK NCSC and the EU's DORA all set expectations for backups and restore testing.[^csf2][^ncsccloud][^dora] ::: ### At a glance | Question | Answer | |---|---| | Goal | Data proven, by tests, to be restorable within the limits the business accepted | | Who | Backup or platform team, with data owners, application teams and security | | In scope | Policy, schedules, consistency, isolated and immutable copies, retention, keys, restore tests, evidence; SaaS and Kubernetes data | | Out of scope | Service failover, recovery time and point objectives (RTO, RPO) ([DisasterRecoveryOps](../disasterrecoveryops/)); snapshot mechanics ([StorageOps](../storageops/)); database recovery internals ([DBOps](../dbops/)); ransomware incidents ([ResponseOps](../responseops/)) | | Prerequisites | Basic OS and storage administration ([SysOps](../sysops/) helps) | | Outputs | Policies, independent copies, restore-test records, recoverability metrics | | Term status | Overlapping synonym for backup and recovery | ### The problem it solves Backups fail quietly, and the failure is discovered at the worst moment. Four documented patterns recur. - **Silent failure without an owner.** In January 2017 GitLab.com's primary database was wiped by mistake. The daily backup script had run `pg_dump` 9.2 against PostgreSQL 9.6 and failed, and its failure e-mails were rejected for lack of DMARC signing; recovery came from a six-hour-old LVM snapshot taken for staging. Why was the procedure never tested? "Because there was no ownership, as a result nobody was responsible for testing this procedure."[^gitlab2017] - **Attackers target backups.** CISA warns that "many ransomware variants attempt to find and subsequently delete or encrypt accessible backups".[^cisarw] In a Sophos-commissioned survey of 2,974 IT staff at organisations hit by ransomware (early 2024), 94% said attackers tried to compromise their backups and 57% of attempts succeeded (self-reported, so indicative only).[^sophos2024] - **Redundancy is not recoverability.** In 2024 a provisioning error by Google operators deleted the Australian customer UniSuper's Google Cloud VMware Engine private cloud, duplicate included; backups outside the deleted service limited the loss ([case 2](#use-cases-and-scenarios)).[^gcve2024][^unisuper] - **Obligations.** DORA requires most EU financial entities to document backup policies and test them periodically; HIPAA requires "retrievable exact copies" of electronic health data ([methods](#methods-standards-and-frameworks)).[^dora][^hipaa] With BackupOps, every data class has an owner, a policy, a copy beyond production credentials, and dated restore evidence. ### What BackupOps is not, and how backups compare A replica copies deletions as faithfully as good writes: "replication and redundancy are not recoverability". Nor is an archive a backup: "backups can be loaded back into an application, while archives cannot".[^sre-di] A snapshot is a backup only if it is independent: Amazon EBS snapshots are stored in Amazon S3,[^ebssnap] so they outlive the volume, but the same account credentials can delete them, as Code Spaces found in 2014 ([case 3](#use-cases-and-scenarios)). | | Backup | Snapshot | Replica | Archive | |---|---|---|---|---| | Protects against | Deletion, corruption, ransomware, site loss (if isolated) | Quick rollback of a volume or disk | Hardware or zone failure | Loss of records kept for legal reasons | | Point in time? | Yes, many | Yes, a few | No, current state only | Yes, selected | | Independent of source? | By design | Often not (same array, account or keys) | No: copies deletions | Usually | | Atlas page | This page | [StorageOps](../storageops/) | [StorageOps](../storageops/), [DisasterRecoveryOps](../disasterrecoveryops/) | [ComplianceOps](../complianceops/) | *Editorial distinction:* BackupOps restores **data**; [DisasterRecoveryOps](../disasterrecoveryops/) restores **services** (failover, failback, the decision to invoke) and owns the RTO and RPO definitions used here. Nor is BackupOps a product: a backup suite gives you jobs, not evidence that restores work. ## Where it comes from and where it sits ### Origins and evolution - **The 3-2-1 rule.** A 2012 paper for US-CERT attributes it to Peter Krogh's *The DAM Book* (2nd edition, 2009): three copies, on two media types, one offsite.[^cisa2012] - **The ransomware era.** In 2019 the NCSC added an "offline rule", because 3-2-1 "does not require any backup location to be offline";[^ncsc2019] vendors extended the rule (3-2-1-1-0, [methods](#methods-standards-and-frameworks)), and the NCSC published ransomware-resistant backup principles in November 2024.[^ncsconprem][^ncsccloud] Cloud services now offer root-proof retention locks and scheduled restore testing ([tooling](#tooling-landscape-as-of-october-2026)). ### Glossary and mental model | Term | Meaning | Example | |---|---|---| | Recovery point | A copy of data as it was at one moment | Last night's 02:00 dump | | RPO | The point in time to which data must be recovered after an outage[^nist34] ([DisasterRecoveryOps](../disasterrecoveryops/) sets it) | "At most 1 hour of orders lost" | | RTO | Longest acceptable delay before a service is restored (also set by [DisasterRecoveryOps](../disasterrecoveryops/)) | "Orders back within 4 hours" | | Restore time | Measured time to restore data; part of the service's RTO | 52 minutes in last test | | Crash- vs application-consistent | As after a power cut, or after the application flushed and paused writes | VSS writers, database dumps | | Immutable copy | Cannot be changed or deleted until retention ends (WORM) | Locked vault, S3 Object Lock | | Offline / air-gapped | Unreachable from production networks and credentials when idle | Tape on a shelf | **Mental model: four questions for every copy.** (1) **Independent?** Does it share a site, account, credentials or keys with the source? (2) **Intact?** Is it immutable and checksummed? (3) **Recent enough?** Is it newer than the RPO allows? (4) **Restorable in time?** Has a test restored it within the time budget? A copy that passes all four is *proven recoverable*; the rest are hopes. (This framing is the Atlas's own.) ### How BackupOps relates to other Ops | Neighbour | Shared ground | The neighbour's focus | BackupOps' focus | |---|---|---|---| | [DisasterRecoveryOps](../disasterrecoveryops/) | RPO, recovery | Service failover, RTO | Data copies and restores | | [StorageOps](../storageops/) | Snapshots, replication | Media, performance | Independent copies built on them | | [DBOps](../dbops/) | Dumps, write-ahead logs (WAL), point-in-time recovery (PITR) | Engine-level recovery | Cross-system policy, restore tests | | [ResponseOps](../responseops/) | Ransomware | Containment, forensics | A clean recovery point | | [SRE](../sre/) | Data integrity | SLOs, error budgets | Recoverability as a measure | | [ComplianceOps](../complianceops/) | Evidence, retention | Control evidence | Restore evidence | | [PrivacyOps](../privacyops/) | Erasure | Individual rights | Erased data kept "beyond use" | | [KubernetesOps](../kubernetesops/) | etcd | Cluster operations | Namespaces and volumes | ::: figure The recovery chain: what BackupOps owns and what it hands off ```mermaid flowchart TB S["StorageOps: snapshots, replication"] -->|"raw material"| B D["DBOps: dumps, WAL, PITR"] -->|"raw material"| B X["ResponseOps: ransomware incident"] -->|"asks for a clean point"| B B["BackupOps: independent copies, proven restores"] B -->|"restorable data"| R["DisasterRecoveryOps: service recovery"] B -->|"restore evidence"| C["ComplianceOps: audits"] B <-->|"erasure requests; re-applied after restores"| P["PrivacyOps: erasure"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class B core class S,D,X,R,C,P neutral ``` Text equivalent: StorageOps (snapshots, replication) and DBOps (dumps, write-ahead logs, point-in-time recovery) supply raw material to BackupOps, and ResponseOps asks it for a clean point during a ransomware incident. BackupOps hands restorable data to DisasterRecoveryOps and restore evidence to ComplianceOps. With PrivacyOps the link runs both ways: erasure requests come in, and BackupOps keeps erased data beyond use in its copies and re-applies erasures after restores. ::: ## Principles and operating model ### A proposed charter and principles *This charter is the Atlas's proposal; no official BackupOps manifesto exists.* **Mission:** every data asset that matters can be restored, recently enough and fast enough, by people who are not its usual operators, even when production and its credentials are compromised. 1. **The restore is the product.** A recovery point without a successful test is unproven. 2. **Count independent failure domains, not copies.** 3-2-1 is a floor. Ask what single event (a fire, a deleted account, a stolen password, a lost key) would destroy every copy. 3. **One copy that production credentials cannot delete.** Immutable, offline or held by a separate party; NIST SP 800-53 adds dual authorisation for deleting backups (CP-9(7)) and cryptographic protection (CP-9(8)).[^nist53] 4. **Retain by time, not by count.** The NCSC recommends storing backups "according to a fixed time period, rather than a fixed number of backups".[^ncsccloud] 5. **Keys and access must survive the disaster**, including the loss of all corporate identities.[^ncsccloud] 6. **Every asset has an owner, a policy and evidence**, the cheapest of GitLab's failures to prevent.[^gitlab2017] 7. **Proportionality.** Tier by business impact; data that can be rebuilt need not be backed up. **Non-goals:** high availability, service failover and long-term archiving, which other pages own. ### Building blocks | Block | Job | Design rule | |---|---|---| | Inventory and classification | Every data asset with its owner and tier (CMDB export, cloud tags) | Compared weekly with what is protected | | Protection policy | RPO, schedule, consistency, copies, retention and test frequency per class | Changed only with the data owner's approval | | Backup engine | Runs jobs and consistency hooks; deduplicates and encrypts | Its identity cannot destroy the isolated copy; a tool that needs delete rights (Velero) writes to a primary repository or to a locked bucket where deletes leave versions behind | | Primary repository | Fast everyday restores | Same site is acceptable: it is not the last line | | Isolated copy | Survives site, account or credential loss | Locked vault in another account or tenant with separate administrators, or offline media | | Keys and catalogue | Decrypt and find recovery points when production is gone | Recovery keys escrowed out of band; access reviewed | | Monitoring and evidence | Alerts on failure *and on absence of success* ("no success in 26 h") | Delivered outside production mail and chat | | Restore-test sandbox | Restores, validates and records results for audits | Isolated network; data deleted after each test | ### Roles, ownership and RACI R = responsible, A = accountable, C = consulted, I = informed. Adjust names to your organisation. | Activity | Data owner | Backup / platform team | Application team | Security | Risk / audit | |---|---|---|---|---|---| | Classify data; accept RPO and retention | A | C | R | C | I | | Configure jobs and consistency hooks | I | A | R | I | – | | Isolated and immutable copies, key custody | I | R | I | A | C | | Triage failed or missing jobs | I | A/R | C | I | – | | Run and validate restore tests | C | A | R | I | I | | Approve retention reductions and deletions | A | R | I | C (second approver) | I | | Review evidence and exceptions quarterly | A | R | C | C | C | ### Lifecycle of a recovery point Day 0 is design (classify, set policy, build the isolated copy); Day 1 onboards each system with a first verified restore; Day 2 is the loop of jobs, copies, tests and expiry, which each recovery point follows. ::: figure Lifecycle of one recovery point, with the failure exits ```mermaid stateDiagram-v2 direction TB [*] --> Created: job completes Created --> Verified: checksum and catalogue OK Created --> Failed: job or check error Verified --> Isolated: copied to independent vault Isolated --> Locked: retention lock applies Locked --> Proven: restore test passes Locked --> Suspect: restore test fails Suspect --> Failed: cannot be repaired Proven --> Expired: retention ends Locked --> Expired: retention ends (untested) Expired --> [*]: deleted by policy Failed --> [*]: alert owner, rerun ``` Text equivalent: a job creates a recovery point; a good checksum and catalogue entry make it verified, an error makes it failed. Verified points are copied to an independent vault and locked. A passing restore test makes a point proven; a failing one makes it suspect, and failed if it cannot be repaired. Proven and never-tested locked points expire and are deleted by policy; failed points alert the owner and trigger a rerun. Only a sample is tested, so tests must rotate across systems and ages. ::: ### Operating model and governance | Cadence | Activity | Output | |---|---|---| | Daily | Triage failed *and missing* jobs within one business day | Tickets; RPO breaches escalated | | Weekly | Compare inventory with protected assets | Unprotected assets assigned | | Monthly | Restore tests per tier; review delete and key permissions | Test records; access review | | Quarterly | Policy review with data owners; one application-level restore exercise | Policy and exception list | | On change | Shorter retention, unlocking or deleting needs two approvers | Change record | Exceptions are recorded with an owner, a compensating control and an expiry date. ## Architecture and workflow ### Reference architecture The central idea, on premises or in the cloud, is a **trust boundary**: the isolated copy lives where production identities cannot delete it, and a sandbox proves it works. ::: figure Reference architecture: primary copy for speed, isolated copy for survival ```mermaid flowchart TB subgraph PROD["Production account or site"] A["Apps, databases, volumes, SaaS"] -->|"quiesce, copy"| E["Backup engine and hooks"] E --> P[("Primary repository")] end subgraph ISO["Isolated recovery account or site"] V[("Locked vault: immutable copy")] -->|"scheduled restore"| T["Restore-test sandbox"] K["Escrowed recovery keys"] -.->|"decrypt"| T end P -->|"copy job, write-only identity"| V E -->|"job status"| M["Monitoring and evidence"] T -->|"test results"| M classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class A,E,P neutral class V,K,T core class M neutral ``` Text equivalent: in production, a backup engine with consistency hooks copies applications, databases, volumes and SaaS data into a primary repository. A copy job with a write-only identity moves recovery points across a trust boundary into a locked vault in an isolated recovery account or site. Scheduled restores go from the vault into a sandbox that decrypts with escrowed keys. Job status and test results flow to monitoring and an evidence store. ::: Design rules per component are in [Building blocks](#building-blocks). **Alternatives:** offline media isolate best but restore slowly; a second provider adds independence at the cost of egress and skills; small teams can run one locked vault in a separate account. ### Backup types and chains | Type | What each run copies | What a restore needs | Trade-off | |---|---|---|---| | Full | Everything | One recovery point | Slowest and largest run; simplest restore | | Incremental | Changes since the last backup of any type | The full plus every incremental since, in order | Fast runs; long chains are fragile | | Differential | Changes since the last full | The full plus the latest differential | Runs grow until the next full[^nist34] | | Synthetic full | Nothing new from the source: the repository merges the chain | One recovery point | Less load on production; the repository reads the chain and writes a new full[^veeamsynth] | | Continuous (log-based) | A stream of changes, such as database WAL | A base backup plus logs to the chosen moment | Fine-grained recovery; engine-specific ([DBOps](../dbops/)) | ::: figure One week of backups: what a restore from Wednesday's incremental depends on ```mermaid flowchart LR F["Sunday full"] ==> M["Mon incr."] ==> T["Tue incr."] ==> W["Wed incr."] F -.-> D["Wed differential"] F --> S["Thu synthetic full"] W --> S classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class F,M,T,W core class D,S neutral ``` Text equivalent: a Sunday full is followed by Monday, Tuesday and Wednesday incrementals, each depending on the one before; a restore from Wednesday's incremental needs all four (highlighted, joined by thick arrows) in order, and losing Tuesday's breaks the chain. A Wednesday differential depends only on the Sunday full. On Thursday the repository merges the chain into a synthetic full, so later restores need one recovery point. ::: ### End-to-end workflow **Happy path.** Schedule fires → pre-hook pauses writes → engine copies → post-hook resumes → checksum and catalogue → copy to the locked vault → success recorded → a scheduled test restores a sampled point, validates it and writes evidence → retention expires it. **Exception path.** A job fails or never starts → retry → page the owner before the RPO is breached. A failed restore test is an incident at the asset's tier: every point of that type is unproven until a new test passes. An unrequested deletion or retention change on the vault → a security alert for [ResponseOps](../responseops/), because attacks on backups can precede the main attack.[^ncsccloud] ## Practices, standards and tools ### Core practices and playbooks **Derive the schedule from the RPO, including copy delay.** Worst-case loss for a copy is roughly the interval between backups plus the time until a backup reaches the copy you would restore from. *Worked example:* a nightly job starts at 01:00 and reaches the isolated vault at 05:00. If everything else is lost at 04:59, the newest vault copy holds data from 01:00 the *previous* day: about 28 hours of loss against a 24-hour RPO. Back up more often or copy faster. **Size the restore against its budget.** Restore time is retrieval (cold tiers, offsite media) plus transfer plus recovery steps (log replay, application start) plus validation. *Illustrative calculation:* 4 TB at a measured 250 MB/s on one stream needs 16,000 seconds, about 4.4 hours, against a 2-hour budget. Four streams, if source, network and target sustain them, cut that to about 67 minutes; with 20 minutes of log replay and 15 of validation the restore fits, at 102 minutes. Cloud restores can hide time: a volume created from an EBS snapshot may be slow until its blocks arrive from S3, and even at the highest provisioned initialisation rate, 300 MiB/s per volume (as of October 2026), 4 TiB takes about 3.9 hours.[^ebsinit] Fast snapshot restore avoids the wait (the volume "is fully initialized at creation"), but it is enabled per snapshot and Availability Zone and billed for every minute it is on.[^ebsinit][^ebsfsr] Time each phase in tests at production size. **Make copies consistent.** A crash-consistent copy looks like the moment after a power cut. On Windows, Volume Shadow Copy Service writers guarantee "a consistent data set to back up".[^vss] Velero runs hooks in pods, such as `fsfreeze` around a volume snapshot; by default a failing hook is an error and a hook times out after 30 seconds.[^velerohooks] Velero's `fsfreeze` example runs a privileged sidecar,[^velerofreeze] which Pod Security "baseline" and "restricted" namespaces reject,[^pss] and if the post-hook never runs the volume stays frozen and writes hang. Prefer application-level quiescing or crash-consistent CSI snapshots where the application tolerates them. Velero's file-system backup is "less consistent than the snapshot approaches".[^velerofsb] For databases use the engine's tools: a `pg_dump` "represents a snapshot of the database at the time pg_dump began running".[^pgdump] **Build the isolated copy.** Use a retention-locked vault in a separate account, offline media or a second provider. The backup client should be able to create backups but not delete them, and should hold no credentials while idle.[^ncsc2019] Cloud locks come in two modes: *governance*, which privileged users can still change, and *compliance*, which nobody can change: AWS Backup Vault Lock after a grace period of at least 72 hours, S3 Object Lock at once.[^putlock][^vaultlock][^s3lock] **Design retention by time.** The NCSC's example keeps daily backups for a month and monthly backups for a year.[^ncsccloud] Count-based rules can be gamed: restic's documentation shows how an attacker who adds empty snapshots to an append-only repository can make a "keep three weekly" policy delete every legitimate one, and recommends the time-based `--keep-within`.[^restic-forget] Keep some points longer than an attacker's plausible dwell time. **Keep keys recoverable.** An attacker "doesn't need to actually delete the data itself if they can simply delete or modify the encryption key".[^ncsccloud] Escrow a recovery key out of band (the NCSC suggests paper or a QR code in a safe), restrict key deletion, and decrypt in every sandbox test. etcd snapshots hold every Kubernetes object, so the Kubernetes documentation advises encrypting them.[^etcd] **Test restores on a ladder.** Each rung finds failures the one below cannot: 1. *Integrity checks:* checksums; `restic check --read-data`, which reads every pack file;[^restic-check] `pg_verifybackup`, which "does not and cannot include every check" a running server performs.[^pgverify] 2. *Sample restores:* a file, a mailbox or a table, restored and compared. 3. *Full sandbox restores* with automated validation ([Lab 2](#hands-on-labs)). AWS Backup restore testing, for example, restores the latest or a random recovery point, accepts your validation result and deletes the test resources.[^restoretest][^valapi] 4. *Exercises:* the team rebuilds a service from backups against the clock, with [DisasterRecoveryOps](../disasterrecoveryops/). The SRE book: "the only test that should let you sleep at night is a full end-to-end test", automated and run continuously.[^sre-di] NIST SP 800-53 likewise calls for integrity testing and sample restores (CP-9(1), CP-9(2)).[^nist53] **Protect SaaS data deliberately.** In Microsoft's shared-responsibility model, whatever the deployment type, "you always retain" responsibility for your data.[^msrm] Microsoft 365 Backup keeps backups "within the protected services' data boundaries",[^m365] which buys fast restores, not independence from the provider or a compromised tenant administrator; decide per application whether to keep a copy outside it. **Back up Kubernetes at two levels.** etcd holds cluster state ([KubernetesOps](../kubernetesops/)). A tool such as Velero backs up namespaces and volume data, using CSI snapshots or Kopia-based file copies.[^etcd][^velerofsb] Its schedules keep backups for 30 days unless `ttl` says otherwise.[^veleroschedule] Velero's v1.18 documentation says it does not support immutable storage. On Amazon S3 with Object Lock, backups work, but deleting one leaves the old object versions behind. Azure works only with version-level immutability. Google Cloud bucket locks do not work.[^velerolimits] Velero's credential also needs delete rights, for expiry, deletion and Kopia maintenance.[^velero8724] [Artefact 3](#reference-implementation) compares three routes to a locked copy. **Back up the identity plane.** Directories, identity-provider configuration and privileged-access vaults are tier 1 data: other restores usually need them first (the Atlas's inference). Microsoft's forest recovery guide covers the case where all domain controllers fail;[^adfr] procedures belong to [IdentityOps](../identityops/). ### Methods, standards and frameworks | Source | Kind | What it asks of backups | |---|---|---| | NIST CSF 2.0, PR.DS-11 | Voluntary framework | "Backups of data are created, protected, maintained, and tested"[^csf2] | | NIST SP 800-53 Rev. 5, CP-9 | Control catalogue (US federal) | Protect backups; enhancements for testing, separate storage, dual authorisation, cryptography[^nist53] | | NIST SP 800-34 Rev. 1 | Federal guidance | Backup policy questions, types, offsite storage[^nist34] | | NIST SP 800-209 | Storage security guidance | Immutability; restore attack-recovery copies onto isolated staging[^nist209] | | NCSC principles (2024) | UK government guidance | Isolate and update the backup system; resist destruction, keep access, restore old versions, protect keys, alert[^ncsccloud][^ncsconprem] | | CISA #StopRansomware Guide | US government guidance | "Maintain offline, encrypted backups of critical data" and test them[^cisarw] | | DORA, Article 12 | EU regulation for financial entities | Backup policy by criticality, periodic testing, segregated systems for restores on own systems[^dora] | | HIPAA Security Rule §164.308(a)(7) | US regulation for covered entities | Backup plan required; testing addressable.[^hipaa] A 2025 proposal adds copies under 48 hours old, monthly sample restores and 72-hour restores; reportedly not final before July 2027[^hipaanprm][^paubox] | | 3-2-1 and 3-2-1-1-0 | Rules of thumb (the second is a vendor's) | Copies, media, offsite; plus immutable copy and verified restores[^cisa2012][^veeam321] | The regulations bind specific sectors in specific places; confirm what applies to you with your compliance team ([ComplianceOps](../complianceops/)). ### Tooling landscape (as of October 2026) Choose by the job, then by product, and re-check vendor documentation before relying on any row. | Category | Verified examples | Strengths | Watch-outs | |---|---|---|---| | Cloud-native services | AWS Backup (Vault Lock, logically air-gapped vaults, restore testing, malware scanning, EKS support); Azure Backup immutable vaults; Google Cloud backup vaults | Tag-based policy, managed locks | One provider's control plane[^vaultlock][^lagvault][^awsmalware][^awseks][^azimm][^gcpvault] | | Kubernetes | Velero (CNCF Sandbox since 2026; docs v1.18) | Namespaces, volumes, hooks, schedules | Consistency depends on hooks; immutability unsupported (S3 Object Lock works, with caveats); its credential can delete[^velerocncf][^velerohooks][^velerolimits][^velero8724] | | Enterprise suites | Veeam Backup & Replication 13 | Broad coverage, synthetic fulls, recovery verification | You patch, harden and isolate the server and repository[^veeamsynth][^veeam321][^kev] | | Open-source tools | restic | Scriptable; append-only server mode; full data checks | You build scheduling and monitoring[^restic-forget][^restic-check] | | Database-native | `pg_dump`, `pg_basebackup`, `pg_verifybackup` | Consistent, engine-aware | Roles need a separate dump[^pgdump] | | SaaS backup | Microsoft 365 Backup; third-party services | Restores at SaaS speed | Independence from provider and tenant admins[^m365] | **Selection criteria:** source coverage, consistency, locks and who can delete what, keys, restore speed, test automation, total cost, and exit (can you read backups without the vendor?). ### Automation and AI Automate the checkable: policy as code, tag-based selection that protects new resources by default, inventory diffs, machine-validated restore tests, and malware scanning (AWS Backup can have Amazon GuardDuty scan each point).[^awsmalware] NIST SP 800-53 asks for dual authorisation to delete or destroy backups (CP-9(7));[^nist53] the Atlas recommends two people for anything else that reduces protection: shorter retention, unlocking, or restoring over production. Language models can summarise job logs and draft runbooks; they cannot prove recoverability, and no AI agent should be able to delete or unlock backups. ## BackupOps in practice ### Reference implementation Three artefacts make the policy executable; names, accounts and ARNs are placeholders. **Artefact 1: a protection policy kept as code**, recording the data owner's decisions. _Example status: YAML syntax-checked; a policy document, not a tool configuration._ ```yaml # protection-policy.yaml: reviewed like code; the data owner approves changes version: 1 tiers: tier1: # revenue- or safety-critical data rpo_hours: 1 # from the business impact analysis isolated_rpo_hours: 28 # accepted worst case if only the vault survives restore_budget_minutes: 120 consistency: application # engine dump, VSS writer or freeze hook copies: - {where: primary-repository, every: hour, keep_days: 2} - {where: primary-repository, every: day, keep_days: 7} - {where: isolated-vault, every: day, lock: compliance, keep_days: 35} - {where: isolated-vault, every: month, lock: compliance, keep_days: 400} restore_test: {every_days: 30, level: full-sandbox} tier2: rpo_hours: 24 restore_budget_minutes: 480 consistency: crash-acceptable copies: - {where: isolated-vault, lock: governance, keep_days: 35} restore_test: {every_days: 90, level: sample} do_not_back_up: - {class: build-artifacts, reason: "rebuilt from source in under 30 minutes"} assets: - {id: orders-db, tier: tier1, owner: orders-team, engine: postgresql} - {id: k8s-prod, tier: tier2, owner: platform-team, engine: velero} ``` **Artefact 2: an AWS Backup plan** for tier 1 (without the monthly 400-day rule): hourly points kept two days, and a daily point copied to a locked vault in a separate recovery account. Cross-account copies need both accounts in one AWS Organizations organisation, cross-account backup enabled from its management account, a destination vault access policy, and a customer managed KMS key for resource types AWS Backup does not fully manage, such as Amazon RDS.[^xacct][^awsenc] `StartWindowMinutes` must be at least 60, so an hourly job can start up to an hour late; count that in the RPO.[^planapi] _Example status: JSON syntax-checked; not executed (requires an AWS account). Use with `aws backup create-backup-plan --backup-plan file://plan.json`._ ```json { "BackupPlanName": "tier1-orders", "Rules": [ { "RuleName": "hourly-local", "TargetBackupVaultName": "prod-primary", "ScheduleExpression": "cron(0 * ? * * *)", "StartWindowMinutes": 60, "CompletionWindowMinutes": 120, "Lifecycle": {"DeleteAfterDays": 2}, "RecoveryPointTags": {"tier": "1"} }, { "RuleName": "daily-isolated-copy", "TargetBackupVaultName": "prod-primary", "ScheduleExpression": "cron(30 1 ? * * *)", "StartWindowMinutes": 60, "CompletionWindowMinutes": 240, "Lifecycle": {"DeleteAfterDays": 7}, "CopyActions": [ { "DestinationBackupVaultArn": "arn:aws:backup:eu-west-1:444455556666:backup-vault:recovery-locked", "Lifecycle": {"DeleteAfterDays": 35} } ], "RecoveryPointTags": {"tier": "1"} } ] } ``` The recovery vault's access policy names the production account, not the whole organisation, so another compromised account cannot fill the locked vault with junk that must then be kept.[^xacct] _Example status: JSON syntax-checked; not executed. Apply in the recovery account with `aws backup put-backup-vault-access-policy --backup-vault-name recovery-locked --policy file://vault-policy.json`._ ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "CopyFromProductionOnly", "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::111122223333:root"}, "Action": "backup:CopyIntoBackupVault", "Resource": "*" } ] } ``` **Artefact 3: a Velero schedule** with freeze hooks, snapshot data moved off the cluster, 35-day retention, and backups that survive deletion of the schedule.[^veleroschedule][^velerohooks] Its location `recovery-bucket` is a bucket in the recovery account. Three routes lead to a locked copy; none was tested for this page: - **Object Lock on `recovery-bucket` itself**, which Velero's documentation says works on S3.[^velerolimits] A delete without a version ID only adds a delete marker.[^s3repl] A deleted backup must then be recovered from old versions by hand, and orphaned versions cost storage at least until their lock ends. Use compliance mode, because governance mode yields to anyone holding `s3:BypassGovernanceRetention`.[^s3lock] A bucket's default lock counts from when each version was written, but Kopia reuses stored data across backups.[^s3lock][^kopiafeat] So the lock on data that newer backups still use can end before those backups do (the Atlas's inference). Deleting a version requires `s3:DeleteObjectVersion`; keep it out of Velero's role, as Velero's documented AWS policy does.[^s3delete][^veleroaws] - **Replication into a locked bucket.** S3 does not replicate deletions if the rule is a current, Filter-based one with delete-marker replication left off.[^s3repl] Prove the copy by restoring from it with a second Velero whose location is `ReadOnly`.[^velerobsl] - **AWS Backup for Amazon EKS**, which supports logically air-gapped vaults (as of October 2026).[^awseks] If Velero stops between the hooks, unfreeze by hand with `kubectl exec -c fsfreeze -- /sbin/fsfreeze --unfreeze /data`. _Example status: YAML syntax-checked; not executed (requires a cluster with Velero and its node-agent, whose data-mover pods run as root,[^velerodm] a CSI snapshot class and a storage location named `recovery-bucket`)._ ```yaml apiVersion: velero.io/v1 kind: Schedule metadata: name: orders-nightly namespace: velero spec: schedule: "0 1 * * *" useOwnerReferencesInBackup: false # backups outlive the schedule template: includedNamespaces: ["orders"] snapshotVolumes: true snapshotMoveData: true # needs node-agent; moves snapshot data storageLocation: recovery-bucket # recovery account; see the text ttl: 840h0m0s # 35 days hooks: resources: - name: freeze-data-volume includedNamespaces: ["orders"] includedResources: ["pods"] labelSelector: matchLabels: {app: orders-api} pre: - exec: container: fsfreeze # privileged sidecar in the app pod command: ["/sbin/fsfreeze", "--freeze", "/data"] onError: Fail timeout: 30s post: - exec: container: fsfreeze command: ["/sbin/fsfreeze", "--unfreeze", "/data"] onError: Fail # a failed unfreeze must surface timeout: 30s ``` ### Hands-on labs Labs 1 and 2 were **executed** on 2026-10-16, and their scripts again on 2026-10-17 after line wrapping (Ubuntu 20.04, GNU tar 1.30, PostgreSQL 13.13, Python 3.8); Lab 3 was **not executed**. #### Lab 1 (beginner): full and incremental backups, and a restore that catches a silent miss **Objective:** see a backup chain work, verify a restore against checksums, and watch a restore test catch what jobs do not report. **Prerequisites:** Linux (or WSL) with GNU tar, `sha256sum` and `diff`. **Cost and risk:** none; everything stays in `./backup-lab`. **Time:** 20 minutes. _Example status: syntax-checked with `bash -n` and executed (outputs below)._ ```bash #!/usr/bin/env bash # Lab 1: full + incremental backups with GNU tar, then a verified restore. # Usage: ./backup-lab.sh init|full|change|incr|restore|corrupt|race set -euo pipefail LAB="${LAB:-$PWD/backup-lab}" SRC="$LAB/src"; REPO="$LAB/repo"; RESTORE="$LAB/restore" init() { # sample data rm -rf "$LAB"; mkdir -p "$SRC/docs" "$REPO" printf 'invoice 1\n' > "$SRC/docs/inv-001.txt" printf 'invoice 2\n' > "$SRC/docs/inv-002.txt" head -c 1048576 /dev/urandom > "$SRC/blob.bin" } full() { # level 0: everything; the snapshot file records what was seen tar --create --file="$REPO/full-0.tar" \ --listed-incremental="$REPO/state.snar" -C "$SRC" . } change() { # a day of work: modify, add, delete printf 'invoice 2 (corrected)\n' > "$SRC/docs/inv-002.txt" printf 'invoice 3\n' > "$SRC/docs/inv-003.txt" rm "$SRC/docs/inv-001.txt" } incr() { # level 1: only changes since level 0, plus evidence files tar --create --file="$REPO/incr-1.tar" \ --listed-incremental="$REPO/state.snar" -C "$SRC" . (cd "$SRC" && find . -type f -print0 | sort -z | xargs -0 sha256sum) \ > "$REPO/source-1.sha256" (cd "$REPO" && sha256sum full-0.tar incr-1.tar) > "$REPO/archives.sha256" tar --list --file="$REPO/incr-1.tar" } restore() { # verify archives, restore the chain oldest first, verify the result (cd "$REPO" && sha256sum --check archives.sha256) rm -rf "$RESTORE"; mkdir -p "$RESTORE" for a in full-0.tar incr-1.tar; do tar --extract --file="$REPO/$a" --listed-incremental=/dev/null -C "$RESTORE" done (cd "$RESTORE" && sha256sum --check --quiet "$REPO/source-1.sha256") diff -r "$SRC" "$RESTORE" n=$(find "$RESTORE" -type f | wc -l) echo "RESTORE VERIFIED: $n files match the source" } corrupt() { # simulate silent corruption of the incremental archive printf 'X' | dd of="$REPO/incr-1.tar" bs=1 seek=1024 conv=notrunc status=none } race() { # full, change and incremental within milliseconds of each other init; full; change; incr } case "${1:-}" in init|full|change|incr|restore|corrupt|race) "$1" ;; *) echo "usage: $0 init|full|change|incr|restore|corrupt|race" >&2; exit 2 ;; esac ``` 1. Save the script as `backup-lab.sh`, run `chmod +x backup-lab.sh`, then `./backup-lab.sh init` and `./backup-lab.sh full`. 2. Wait a few seconds, then run `change` and `incr`. The incremental lists only `./`, `./docs/`, `./docs/inv-002.txt` and `./docs/inv-003.txt`: the modified and new files; the deletion is recorded in directory metadata.[^gnutar] 3. Run `restore`. **Expected output:** `full-0.tar: OK`, `incr-1.tar: OK`, `RESTORE VERIFIED: 3 files match the source` (`inv-001.txt` stays deleted because GNU tar replays deletions).[^gnutar] 4. **Break the chain:** extract only `incr-1.tar` into an empty directory: two files, no `blob.bin`. 5. **Corrupt the media:** run `corrupt`, then `restore`. **Expected:** `incr-1.tar: FAILED`, exit status 1. 6. **Race the clock:** run `race`, then `restore`, several times. In our runs the incremental missed the edit to `inv-002.txt` in 7 to 20 of every 20 runs, depending on the machine; every `tar` exited 0, and only the restore check failed (`./docs/inv-002.txt: FAILED`). Our diagnosis: the kernel stamps file times from a clock that advances every few milliseconds, while tar records the full backup's start precisely, so an edit made moments later can look older. The GNU tar manual warns that incremental dumps "depend crucially on time stamps" (its examples: altered time stamps, a clock set backwards);[^gnutar] the same dependency explains this race. **Validation:** steps 3, 5 and 6 behave as described. **Troubleshooting:** `unrecognized option '--listed-incremental'` means a non-GNU tar. **Teardown:** `rm -rf backup-lab`. #### Lab 2 (intermediate): a scheduled restore test for a PostgreSQL database **Objective:** restore the *stored* copy into a scratch database, validate it against metadata written at backup time, and keep evidence. **Prerequisites:** a disposable, supported PostgreSQL (never production; we ran 13.13, unsupported since November 2025),[^pgversions] client tools of the same or newer major version, Python 3. **Cost and risk:** none locally. **Time:** 45 minutes. _Example status: syntax-checked and executed against a throwaway PostgreSQL 13.13 cluster (socket-only, created with `initdb`)._ ```bash # A throwaway, socket-only cluster (Debian/Ubuntu paths shown) PGBIN=/usr/lib/postgresql/13/bin # the major version we ran; use yours SOCK="$HOME/pglab/sock" mkdir -p "$SOCK" "$PGBIN/initdb" -D "$HOME/pglab/data" -U labadmin --auth=trust > /dev/null "$PGBIN/pg_ctl" -D "$HOME/pglab/data" -l "$HOME/pglab/pg.log" -w start -o \ "-c listen_addresses='' -c unix_socket_permissions=0700 -k $SOCK -p 54329" export PGHOST="$SOCK" PGPORT=54329 PGUSER=labadmin createdb shop psql -X -q -d shop -f setup-shop.sql ``` _Example status: executed (PostgreSQL 13.13)._ ```sql -- setup-shop.sql: fake data for the lab CREATE TABLE customers (id int PRIMARY KEY, email text NOT NULL UNIQUE); CREATE TABLE orders ( id int PRIMARY KEY, customer_id int NOT NULL REFERENCES customers(id), amount numeric(10,2) NOT NULL, created_at timestamptz NOT NULL DEFAULT now() ); INSERT INTO customers SELECT g, 'user' || g || '@example.com' FROM generate_series(1, 500) g; INSERT INTO orders (id, customer_id, amount) SELECT g, 1 + (g % 500), round((g * 7 % 10000) / 100.0, 2) FROM generate_series(1, 20000) g; ``` The backup script refuses client tools older than the server (GitLab's failure) and records a checksum and data fingerprint. _Example status: syntax-checked with `bash -n` and executed._ ```bash #!/usr/bin/env bash # pg-backup.sh: back up one database and write metadata for later restore tests. set -euo pipefail SRC_DB="${SRC_DB:?set SRC_DB}" REPO="${REPO:-$PWD/repo}" CHECK="SELECT count(*) || ':' || md5(string_agg( id || ',' || customer_id || ',' || amount, ';' ORDER BY id)) FROM orders" mkdir -p "$REPO" # Guard: client tools must not be older than the server (the GitLab 2017 failure) server_num=$(psql -X -At -d "$SRC_DB" -c 'SHOW server_version_num') server_major=$(( server_num / 10000 )) client_major=$(pg_dump --version | awk '{print $3}' | cut -d. -f1) if [ "$client_major" -lt "$server_major" ]; then echo "FAIL: pg_dump $client_major is older than server $server_major" >&2 exit 1 fi stamp=$(date -u +%Y%m%dT%H%M%SZ) dump="$REPO/$SRC_DB-$stamp.dump" fingerprint=$(psql -X -At -d "$SRC_DB" -c "$CHECK") # lab: database is quiet pg_dump --format=custom --file="$dump" "$SRC_DB" sha=$(sha256sum "$dump" | cut -d' ' -f1) fmt='{"db":"%s","created_at":"%s","sha256":"%s",' fmt+='"fingerprint":"%s","server_major":%d}\n' printf "$fmt" "$SRC_DB" "$(date -u +%FT%TZ)" "$sha" "$fingerprint" \ "$server_major" > "$dump.meta.json" echo "$dump" ``` _Example status: syntax-checked with `bash -n` and executed._ ```bash #!/usr/bin/env bash # restore-test.sh: restore a stored dump into a scratch database, validate it # against the metadata written at backup time, append an evidence record. set -euo pipefail DUMP="${1:?usage: $0 }" META="$DUMP.meta.json" SCRATCH_DB="restoretest_$(date -u +%Y%m%d%H%M%S)" CHECK="SELECT count(*) || ':' || md5(string_agg( id || ',' || customer_id || ',' || amount, ';' ORDER BY id)) FROM orders" field() { # one field from the metadata file python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))[sys.argv[2]])' \ "$META" "$1" } q() { psql -X -At -d "$SCRATCH_DB" -c "$1"; } status=PASSED; reason=ok # 1. Integrity of the stored copy, before spending time on a restore if [ "$(sha256sum "$DUMP" | cut -d' ' -f1)" != "$(field sha256)" ]; then status=FAILED; reason=checksum_mismatch elif ! pg_restore --list "$DUMP" > /dev/null 2>&1; then status=FAILED; reason=unreadable_archive fi # 2. Restore into a scratch database (dropped on exit) and time it t0=$(date +%s); t1=$t0 trap 'dropdb --if-exists "$SCRATCH_DB"' EXIT if [ "$status" = PASSED ]; then createdb "$SCRATCH_DB" if pg_restore --exit-on-error --no-owner --dbname="$SCRATCH_DB" "$DUMP"; then t1=$(date +%s) # 3. Validate data and structure; run one real application query actual=$(q "$CHECK") fks=$(q "SELECT count(*) FROM pg_constraint WHERE contype = 'f'") top=$(q "SELECT customer_id FROM orders GROUP BY 1 ORDER BY sum(amount) DESC LIMIT 1") if [ "$actual" != "$(field fingerprint)" ]; then status=FAILED; reason=fingerprint_mismatch elif [ "$fks" -lt 1 ] || [ -z "$top" ]; then status=FAILED; reason=validation_query fi else status=FAILED; reason=restore_error fi fi # 4. Evidence: one JSON line per test, kept with the backup records fmt='{"dump":"%s","tested_at":"%s","backup_created_at":"%s","restore_seconds":%d,' fmt+='"status":"%s","reason":"%s"}\n' printf "$fmt" "$(basename "$DUMP")" "$(date -u +%FT%TZ)" "$(field created_at)" \ $((t1 - t0)) "$status" "$reason" | tee -a "$(dirname "$DUMP")/evidence.jsonl" [ "$status" = PASSED ] ``` 1. Run `d=$(SRC_DB=shop ./pg-backup.sh)`, then `./restore-test.sh "$d"`. 2. **Expected output:** one JSON line ending `"status":"PASSED","reason":"ok"`, exit status 0, and no `restoretest_*` database left. 3. **Break it:** `truncate -s 60000 "$d"` and rerun. **Expected:** `"reason":"checksum_mismatch"`, exit status 1, and a harmless `NOTICE` from `dropdb --if-exists`. 4. Schedule the test; alert when the newest evidence line is older than the tier's interval. **Validation:** step 2 passes and step 3 fails as described. On a live database the pre-dump fingerprint can differ from the dump; validate invariants instead (row counts within a tolerance, newest `created_at` within the RPO, constraints present). **Troubleshooting:** `role "…" does not exist`: `pg_dump` omits roles; restore them from `pg_dumpall --globals-only` or keep `--no-owner`.[^pgdump] `Unix-domain socket path … is too long (maximum 107 bytes)`: use a shorter directory (we hit this). **Teardown:** `"$PGBIN/pg_ctl" -D "$HOME/pglab/data" -m fast stop`, then `rm -rf "$HOME/pglab" repo`. #### Lab 3 (advanced): a locked vault and a weekly restore test in AWS **Objective:** a **governance-mode** locked vault, one tagged resource backed up, and a weekly restore test you validate. **Prerequisites:** a *sandbox* AWS account, AWS CLI v2, an IAM role AWS Backup can assume, one small EBS volume tagged `backup-tier=lab`. **Cost and risk:** recovery-point storage and a charge per restore test.[^restoretest] **Never use compliance mode in a lab**: after its grace period nobody, including AWS, can delete the vault while it holds recovery points.[^vaultlock] _Example status: syntax-checked with `bash -n`; not executed (requires an AWS account). JSON files below are syntax-checked._ ```bash #!/usr/bin/env bash set -euo pipefail export AWS_REGION="eu-west-1" VAULT="lab-locked-vault" aws backup create-backup-vault --backup-vault-name "$VAULT" # Governance mode: no --changeable-for-days, so authorised users can unlock aws backup put-backup-vault-lock-configuration --backup-vault-name "$VAULT" \ --min-retention-days 7 --max-retention-days 35 aws backup describe-backup-vault --backup-vault-name "$VAULT" \ --query '{locked: Locked, min: MinRetentionDays, max: MaxRetentionDays}' PLAN_ID=$(aws backup create-backup-plan --backup-plan file://lab-plan.json \ --query BackupPlanId --output text) aws backup create-backup-selection --backup-plan-id "$PLAN_ID" \ --backup-selection file://lab-selection.json aws backup create-restore-testing-plan \ --restore-testing-plan file://lab-restore-plan.json aws backup create-restore-testing-selection \ --restore-testing-plan-name lab_weekly \ --restore-testing-selection file://lab-restore-selection.json ``` `lab-plan.json` keeps 7 days, inside the lock's limits (the vault rejects anything outside them): ```json { "BackupPlanName": "lab-daily", "Rules": [ { "RuleName": "daily-to-locked-vault", "TargetBackupVaultName": "lab-locked-vault", "ScheduleExpression": "cron(0 5 ? * * *)", "StartWindowMinutes": 60, "CompletionWindowMinutes": 180, "Lifecycle": {"DeleteAfterDays": 7} } ] } ``` `lab-selection.json` selects the tagged volume; tag conditions use the `aws:ResourceTag/` key format:[^awsassign] ```json { "SelectionName": "lab-tagged", "IamRoleArn": "arn:aws:iam::111122223333:role/LabBackupRestoreRole", "Resources": ["*"], "Conditions": { "StringEquals": [ {"ConditionKey": "aws:ResourceTag/backup-tier", "ConditionValue": "lab"} ] } } ``` `lab-restore-plan.json` picks a random recovery point from the last seven days:[^rtpapi] ```json { "RestoreTestingPlanName": "lab_weekly", "ScheduleExpression": "cron(0 6 ? * MON *)", "StartWindowHours": 4, "RecoveryPointSelection": { "Algorithm": "RANDOM_WITHIN_WINDOW", "IncludeVaults": ["*"], "RecoveryPointTypes": ["SNAPSHOT"], "SelectionWindowDays": 7 } } ``` `lab-restore-selection.json`: ```json { "RestoreTestingSelectionName": "lab_ebs", "ProtectedResourceType": "EBS", "IamRoleArn": "arn:aws:iam::111122223333:role/LabBackupRestoreRole", "ProtectedResourceArns": ["*"], "ProtectedResourceConditions": { "StringEquals": [{"Key": "backup-tier", "Value": "lab"}] }, "ValidationWindowHours": 2 } ``` 1. Run the script. **Expected:** the `describe-backup-vault` query prints `"locked": true` with `"min": 7` and `"max": 35`. 2. After the first backup and the next Monday, list the test's restore jobs: `aws backup list-restore-jobs --by-restore-testing-plan-arn `. 3. Validate the restored volume (attach it to a test instance and check a known file), then report: `aws backup put-restore-validation-result --restore-job-id --validation-status SUCCESSFUL --validation-status-message "file hash matched"`.[^valapi] **Validation:** your status appears on the restore job. **Troubleshooting:** a retention error means `DeleteAfterDays` is outside the lock's limits; if recent points are skipped, make `SelectionWindowDays` exceed the backup interval plus `StartWindowHours`.[^restoretest] **Teardown:** delete the restore-testing selection and plan and the backup selection and plan; `delete-backup-vault-lock-configuration`; `delete-recovery-point` for each point; `delete-backup-vault`. ### Use cases and scenarios The method for three estates is in the artefacts and labs: a PostgreSQL database (Lab 2), AWS accounts (Artefact 2, Lab 3) and a Kubernetes platform (Artefact 3). The four sourced cases show what happens without it. **Case 1 — silent failure, no owner (GitLab.com, 2017, sourced).** *What happened:* see [the problem](#the-problem-it-solves); disk snapshots were also off for the database servers, and copying the six-hour-old LVM snapshot over throttled disks took about 18 hours. *Follow-up work GitLab listed as under way:* Prometheus backup monitoring, automated testing of PostgreSQL backup recovery, hourly LVM snapshots and an owner for data durability.[^gitlab2017] *Success measure:* a passing automated restore in the last day. *Pitfall:* alerts that travel through the failing system. **Case 2 — the provider deletes the environment (UniSuper, 2024, sourced).** *What happened:* Google operators left a provisioning parameter blank, which gave the private cloud a fixed term and automatic deletion.[^gcve2024] The joint statement says duplication "in two geographies" went too and credits backups with an additional service provider;[^unisuper] Google's account describes one region and two zones, and credits untouched backups in Google Cloud Storage plus third-party backup software. *Lesson:* copies outside the deleted service and subscription saved the data; the accounts differ on whether copies outside Google were decisive. CISA suggests multi-cloud for cloud-to-cloud backups "in case all accounts under the same vendor are impacted".[^cisarw] *Success measure (the Atlas's recommendation):* a drill restoring only from copies outside the primary provider. **Case 3 — one console login destroys data and backups (Code Spaces, 2014, sourced).** *What happened:* an attacker with control-panel access deleted EBS snapshots, S3 buckets and machine images; the company said "most of our data, backups, machine configurations and offsite backups were either partially or completely deleted" and ceased trading.[^codespaces] *Lesson:* copies reachable with the same credentials are not independent. *Success measure:* no single identity can delete production and its last copy. **Case 4 — "offsite" in the same building (OVHcloud fire, Strasbourg, 2021, sourced).** One customer's contract called the backup "physically isolated from the infrastructure in which the VPS server is set up", yet both were in the same building; a French court awarded damages in 2023 (OVH appealed; outcome not checked).[^ovh] *Lesson:* verify where each copy is; the site-loss story is in [DisasterRecoveryOps](../disasterrecoveryops/). ## Measure, govern, pay ### Metrics and KPIs These KPIs are the Atlas's proposal, built on the four questions. | KPI | Formula | Data source | Pitfall | Cadence | |---|---|---|---|---| | Coverage | protected assets ÷ inventoried assets | Inventory, backup selections | Only as good as the inventory | Weekly | | RPO compliance | protected assets with newest *good* point within RPO ÷ protected assets | Catalogue | Measure the copy you would restore from | Daily | | Restore-test currency | assets with a passing test inside their tier's window ÷ protected assets | Evidence store | Easy tests on easy assets | Monthly | | Isolated-copy coverage | tier 1–2 assets with an immutable or offline copy ÷ tier 1–2 assets | Vault inventory | Must mean other credentials | Monthly | | Restore time vs budget | measured ÷ budgeted restore minutes | Test records | Small test data sets | Per test | | **Proven recoverable** | assets meeting all of the above ÷ all assets | Combination | Publish the list, not just the ratio | Monthly | | Job success rate | successful jobs ÷ all jobs | Backup engine | **Vanity**: silent misses still count as success | Watch only | The script computes them from an inventory export (illustrative data). _Example status: JSON and Python syntax-checked; executed with Python 3.8 (output below)._ ```json { "as_of": "2026-10-16T06:00:00Z", "test_window_days": {"1": 30, "2": 90, "3": 365}, "jobs_last_30d": {"succeeded": 1188, "failed": 12}, "assets": [ {"id": "orders-db", "tier": 1, "protected": true, "rpo_hours": 1, "last_good_point": "2026-10-16T05:15:00Z", "isolated_copy": true, "last_test": "2026-10-02", "test_passed": true, "restore_min": 52, "budget_min": 120}, {"id": "payments-db", "tier": 1, "protected": true, "rpo_hours": 1, "last_good_point": "2026-10-16T03:30:00Z", "isolated_copy": true, "last_test": "2026-09-30", "test_passed": true, "restore_min": 95, "budget_min": 120}, {"id": "crm-saas", "tier": 1, "protected": false, "rpo_hours": 24, "last_good_point": null, "isolated_copy": false, "last_test": null, "test_passed": false, "restore_min": null, "budget_min": 480}, {"id": "file-share", "tier": 2, "protected": true, "rpo_hours": 24, "last_good_point": "2026-10-15T22:00:00Z", "isolated_copy": true, "last_test": "2026-06-01", "test_passed": true, "restore_min": 300, "budget_min": 480}, {"id": "k8s-prod", "tier": 2, "protected": true, "rpo_hours": 24, "last_good_point": "2026-10-15T07:00:00Z", "isolated_copy": false, "last_test": "2026-09-20", "test_passed": true, "restore_min": 140, "budget_min": 240}, {"id": "wiki", "tier": 3, "protected": true, "rpo_hours": 168, "last_good_point": "2026-10-12T01:00:00Z", "isolated_copy": false, "last_test": null, "test_passed": false, "restore_min": null, "budget_min": 1440} ] } ``` ```python """Compute restore-centred backup KPIs from an asset inventory (JSON).""" import json import sys from datetime import date, datetime, timedelta def ts(s): return datetime.fromisoformat(s.replace("Z", "+00:00")) def pct(n, d): return f"{n}/{d} = {100 * n / d:.0f}%" if d else "n/a" def main(path): with open(path) as f: inv = json.load(f) now = ts(inv["as_of"]) window = {int(k): v for k, v in inv["test_window_days"].items()} assets = inv["assets"] protected = [a for a in assets if a["protected"]] def within_rpo(a): return now - ts(a["last_good_point"]) <= timedelta(hours=a["rpo_hours"]) rpo_ok = [a for a in protected if within_rpo(a)] tested = [a for a in protected if a["test_passed"] and a["last_test"] and (now.date() - date.fromisoformat(a["last_test"])).days <= window[a["tier"]]] critical = [a for a in assets if a["tier"] <= 2] isolated = [a for a in critical if a["protected"] and a["isolated_copy"]] in_budget = [a for a in tested if a["restore_min"] <= a["budget_min"]] proven = [a for a in in_budget if a in rpo_ok and (a["tier"] > 2 or a["isolated_copy"])] jobs = inv["jobs_last_30d"] all_jobs = jobs["succeeded"] + jobs["failed"] print("Job success rate (vanity): ", pct(jobs["succeeded"], all_jobs)) print("Coverage: ", pct(len(protected), len(assets))) print("RPO compliance (protected): ", pct(len(rpo_ok), len(protected))) print("Restore-test currency: ", pct(len(tested), len(protected))) print("Isolated copy (tier 1-2): ", pct(len(isolated), len(critical))) print("Proven recoverable (all): ", pct(len(proven), len(assets))) for a in assets: if a not in proven: print(" not proven:", a["id"]) return 0 if len(proven) == len(assets) else 1 if __name__ == "__main__": sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "inventory.json")) ``` ```text Job success rate (vanity): 1188/1200 = 99% Coverage: 5/6 = 83% RPO compliance (protected): 4/5 = 80% Restore-test currency: 3/5 = 60% Isolated copy (tier 1-2): 3/5 = 60% Proven recoverable (all): 1/6 = 17% not proven: payments-db not proven: crm-saas not proven: file-share not proven: k8s-prod not proven: wiki ``` **Worked calculation.** At 06:00 `payments-db`'s newest good point is 2.5 hours old against a 1-hour RPO. `file-share` was last tested 137 days ago, outside its 90-day window. `k8s-prod` passes RPO (23 of 24 hours) and its test but has no isolated copy. `crm-saas` is unprotected and `wiki` never tested. Only `orders-db` (45-minute-old point, tested 14 days ago, 52 of 120 minutes, isolated) is proven: **1 of 6, or 17%, while 99% of jobs succeeded.** Set targets per tier (for example 100% of tier 1, exceptions named and dated) and expect the number to fall when you first measure honestly. ### Security, privacy and compliance Backups hold sensitive data for years, and the backup system is itself a target: on 2026-10-16 CISA's Known Exploited Vulnerabilities catalogue listed at least 13 flaws in backup and data-protection products (Veeam, Veritas, Commvault, NAKIVO, Acronis, Arcserve), seven known to be used by ransomware.[^kev] The NCSC's on-premises principles begin "Make it possible to isolate your backup solution" and "Update your backup solution".[^ncsconprem] | Threat | Warning signal | Mitigation | |---|---|---| | Backup server or console exploited | Your product in the KEV catalogue; console logins from new hosts | Patch fast; MFA for destructive actions; separate admin accounts; management interface reachable only from controlled hosts[^ncsconprem][^kev] | | Stolen admin credentials delete backups | Mass deletions, lock or retention changes | Separate account and admins; locks; dual authorisation; out-of-band alerts[^ncsccloud][^nist53] | | Compromised client floods the store | Unusual volume or timestamps of new points | Create-only identity; time-based retention[^restic-forget] | | Encryption key deleted | Key scheduled for deletion | Escrowed recovery key; restricted key admins[^ncsccloud] | | Backups exfiltrated | Unexpected restore or read jobs | Encryption; few restore identities; log every restore | | Malware preserved in points | Scanner findings | Scan; restore to isolated staging first[^awsmalware][^nist209] | | Provider or operator error | Unexplained deletions | A copy outside the provider ([case 2](#use-cases-and-scenarios)) | **Privacy.** After a valid erasure request the UK ICO expects you to "take steps to ensure erasure from backup systems as well as live systems"; where backups cannot be overwritten at once, "the key issue is to put the backup data 'beyond use'".[^ico] *Atlas recommendation:* keep an erasure log and re-apply it after every restore, or a restore resurrects deleted personal data ([PrivacyOps](../privacyops/)). **Legal holds** stop points from expiring until removed, but only the points or object versions they are placed on (AWS Backup, S3 Object Lock).[^legalhold][^s3lock] **Residency:** a vault in another country may move personal data across borders. ### Cost, capacity and sustainability Costs come from stored volume (data × change rate × retention × copies), restores and egress, test compute and people. *Illustrative calculation:* 10 TB protected, 2% daily change, forever-incremental, 35 days: 10 + 10 × 0.02 × 35 = **17 TB** before compression and deduplication, and as much again in the vault: 3.4 times the protected data. Three traps: backups in a locked vault "cannot be deleted until their lifecycle completes", so retention mistakes become persistent costs;[^vaultlock] cold tiers have minimum periods (90 days in AWS Backup);[^lifecycle] and deleting an incremental snapshot "might not reduce your data storage costs".[^ebssnap] Data rebuildable within the RTO need not be backed up at all, which saves money and energy. ### Assurance of the operating model Auditors need the policy, coverage, dated restore-test records, the exception register and access reviews. Leading indicators are inventory drift and ageing evidence; the lagging one is a failed real restore. Quarterly, an outsider restores a random asset from the runbook alone. Services can automate checks (AWS Backup Audit Manager evaluates backups against controls you define).[^vaultlock] ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Fix and prevention | |---|---|---|---| | Jobs green, restored data incomplete | Missed changes; exclusions; inconsistent copy | Compare with source checksums | Periodic (synthetic) fulls; hooks; validate tests ([Lab 1](#hands-on-labs)) | | Backups stopped unnoticed | Alerts lost (GitLab's bounced)[^gitlab2017] | Age of newest good point | Alert on *absence* of success, outside production | | Restore far slower than the budget | Throttled disks, cold-tier retrieval, single stream | Time each phase in tests | Test at production size; keep a fast primary copy | | Copies exist but cannot be decrypted | Key deleted, or held only in the lost environment | Decrypt in the sandbox | Escrowed recovery keys; test decryption[^ncsccloud] | | Backups vanished with a schedule | Velero `useOwnerReferencesInBackup: true`[^veleroschedule] | Backup objects' owner references | Leave it `false` for schedules whose backups must outlive them | | Locked EC2 backup will not restore | AMI disabled; locks prevent deletion, not disablement[^vaultlock] | Recovery point status | Block AMI disabling with IAM policies | ### Worked runbook: recovering data after a ransomware attack [ResponseOps](../responseops/) leads the incident; BackupOps supplies clean, validated data. ::: figure Recovering data after ransomware: find a clean point before restoring at scale ```mermaid flowchart TB A["Incident declared; ResponseOps leads"] --> B["Legal holds on candidate points; freeze deletions"] B --> C["Confirm vault, catalogue and keys intact"] C --> D{"Start of compromise known?"} D -- "Yes" --> E["Candidate: last point before it"] D -- "No" --> E2["Candidate: newest point"] E --> F["Scan the candidate"] E2 --> F F --> G["Restore into isolated staging"] G --> H{"Clean and valid?"} H -- "No: next older point" --> F H -- "Yes" --> I["Promote by business priority"] I --> J["Re-apply erasures; reconcile lost data"] J --> K["Review: dwell time versus retention"] classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 class A,B risk class C,E,E2,F,G,I,J,K core class D,H decision ``` Text equivalent: after ResponseOps declares the incident, BackupOps places legal holds on candidate points, freezes deletions and confirms the vault, catalogue and keys are intact. The first candidate is the last point before a known start of compromise, or else the newest point. Each candidate is scanned and restored into isolated staging; if it is not clean and valid, the next older point is tried. Clean results are promoted by business priority; erasures are re-applied, lost data reconciled, and retention compared with dwell time. ::: | Step | Owner | Action | Check before moving on | |---|---|---|---| | 1 | Backup lead | Place legal holds on candidate points in the vault that holds them: a hold does not follow a copy, and continuous backups cannot be held, so note when their window ends;[^legalhold] a locked vault refuses lifecycle changes.[^vaultlock] Suspend other expiry and deletion jobs; revoke backup-admin sessions | No vault deletions since declaration | | 2 | Backup lead, security | Verify the vault from a clean device via out-of-band access[^ncsccloud] | Point counts match the last report | | 3 | Security | Establish the earliest indicator of compromise | Timeline agreed with ResponseOps | | 4 | Backup team | Choose the last point before it (if unknown, the newest) and scan it[^awsmalware] | No malware findings | | 5 | Backup and app teams | Restore into isolated staging, not onto production hosts[^nist209] | Validation passes; no persistence | | 6 | Incident commander | Promote by priority, identity systems first; rebuild hosts rather than clean them | RTO tracked with [DisasterRecoveryOps](../disasterrecoveryops/) | | 7 | Data owners | Reconcile data created after the point; re-apply erasures | Owner sign-off | | 8 | All | Did retention exceed dwell time? Did the isolated copy hold? | Actions in [IncidentOps](../incidentops/) | CISA advises looking for precursor "dropper" malware, because a ransomware event may signal an earlier, unresolved compromise;[^cisarw] restoring before it is found risks reinfection (the Atlas's inference). ### Anti-patterns and myths - **"The job is green, so we're covered."** It ran; that is all. *Instead:* report proven recoverable. - **"We replicate and snapshot, so we have backups."** Not if the copies share the source's path, array, account or credentials. *Instead:* point-in-time copies held elsewhere. - **"Restore testing is too expensive."** Less than discovering a gap during an outage. *Instead:* small automated tests, often. - **"Immutable means safe."** Locked copies can still be unrestorable (lost keys, disabled images) or hold malware. *Instead:* test and scan them. - **"Everyone owns backups."** Then nobody does. *Instead:* one accountable owner per asset. ## Deciding and adopting ### When to use and when not to use Anyone holding data they cannot recreate needs backup and recovery; the decisions are **how much**, **for which data** and **who owns it**. - **A dedicated backup function** pays off with many platforms, regulatory duties or a history of failed restores; small teams make backups an explicit platform-team duty, with the same policy and evidence. - **Do not back up** data rebuildable within the RTO (build artefacts, caches, derived tables); protect the source and test the rebuild. - **Do not apply long immutable retention** to data with erasure duties until you know how to keep it beyond use. - **Hidden responsibilities:** key custody, the erasure log, egress costs, the inventory and separate administrators for the vault. ### Decision framework ::: figure Choosing protection for one data asset ```mermaid flowchart TB A{"Rebuildable from a source of truth within the RTO?"} -- "Yes" --> N["No backup: protect the source and test the rebuild"] A -- "No" --> B{"Business impact tier?"} B -- "Tier 1" --> T1["Consistent copies, locked isolated vault, monthly full restore test"] B -- "Tier 2" --> T2["Daily copies, isolated vault, quarterly sample restore"] B -- "Tier 3" --> T3["Weekly copies, one offsite copy, yearly test"] T1 --> S{"Only one provider holds every copy?"} T2 --> S S -- "Yes" --> I["Add a copy outside it, or record the accepted risk"] S -- "No" --> P["Apply the policy; prove it with a first restore"] T3 --> P I --> P classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 class A,B,S decision class T1,T2,T3,P,I core class N neutral ``` Text equivalent: an asset rebuildable from a source of truth within the RTO is not backed up; its source and rebuild are protected and tested. Otherwise tier 1 gets consistent copies, a locked isolated vault and monthly full restore tests; tier 2 daily copies, an isolated vault and quarterly samples; tier 3 weekly copies, one offsite copy and a yearly test. For tiers 1 and 2, if one provider holds every copy, add one outside it or record the risk. Every path ends with a first restore. Settings are examples. ::: For **tool choice**, weight the selection criteria before any demo (for example restore speed 25%, locks 20%, coverage 20%, test automation 15%, exit 10%, cost 10%), then test the top two on your own data. ### Maturity model *This is the Atlas's model, not an industry standard.* | Level | Observable behaviour | Exit criteria | |---|---|---| | 0 Hope | Ad-hoc copies; no inventory; nobody has restored | Inventory and owners exist | | 1 Scheduled | Scheduled jobs; job success is the only metric | Missing-success alerts; all tier 1 covered | | 2 Owned | Policy per tier; coverage tracked; daily triage | Tier 1–2 isolated; each tier 1 asset restored once | | 3 Isolated and tested | Locked or offline copy; scheduled tests with evidence; keys escrowed | ≥ 90% of tier 1 proven; quarterly exercise held | | 4 Proven | Automated tests in all tiers; monthly KPIs; ransomware and provider-loss drills | Restore times within budget; exceptions shrinking | ### Adoption roadmap: 30/60/90 days | Window | Actions | Milestone | |---|---|---| | Days 1–30 | Inventory and owners; tier the top 20 assets; "no success in N hours" alerts; patch backup servers; restore one tier 1 asset by hand, timed | First proven-recoverable number published, however low | | Days 31–60 | Write the protection policy; create the isolated account and a governance-mode locked vault; copy tier 1 into it; escrow keys; automate Lab 2-style tests for two databases | Tier 1 isolated; restore tests scheduled | | Days 61–90 | Extend to tier 2 and SaaS; run a tabletop of the ransomware runbook; move the vault lock to compliance mode if the [judgement call](#a-judgement-call-lock-the-vault-in-compliance-mode) supports it; report KPIs monthly | Maturity level 3 for tier 1; exception register reviewed | ## Toolkit ### Checklists **Quick start (first week)** - [ ] Every data asset listed with an owner and a tier - [ ] Alerts on missing success, outside production channels - [ ] One tier 1 asset restored by hand, timed and validated - [ ] Known: who can delete backups, and who can delete the keys - [ ] Backup servers patched (check the KEV catalogue); MFA on destructive actions; consoles reachable only from admin hosts **Preflight: onboarding a new system** - [ ] Tier and RPO agreed with the data owner; "rebuild instead" considered - [ ] Consistency method chosen and tested - [ ] Copies land in the isolated vault within the lock's limits - [ ] Restore runbook written; first restore passed; erasure and residency answered **Before locking a vault** - [ ] Retention minimum and maximum match the policy; no "keep forever" points - [ ] Cost for the whole locked period approved; erasure handling documented - [ ] Governance mode run first; grace period understood (at least 72 hours in AWS Backup Vault Lock)[^putlock] - [ ] Account-closure consequences understood[^vaultlock] **Each restore test and each quarter** - [ ] Point chosen at random or by rotation; restored into a sandbox with the escrowed key - [ ] Validated with data checks and an application query; duration and evidence recorded - [ ] Quarterly: KPIs, delete and key permissions, exceptions and backup-server patch levels reviewed ### Templates **Protection policy (one page per data class)** ```text Data class: ____________________ Owner (accountable): ____________ Tier: 1 / 2 / 3 RPO: ____ h Restore budget: ____ min (the RTO comes from the DR plan) Rebuild instead? no / yes: how, and tested on ______ Consistency: engine dump / VSS writer / freeze hook / crash-consistent OK Primary copy: ______ (retain __ d) Isolated copy: ______ (lock: governance / compliance; retain __ d) Keys: KMS key ______ ; recovery key escrowed at ______ Restore test: every __ days; integrity / sample / full sandbox / exercise Personal data: yes / no; erasure handling ______ ; residency ______ Approved by: owner ______ backup lead ______ security ______ Date: ______ Review by: ______ ``` **Restore test record** ```text Asset / point: ______ / ______ (created ______) Tester / date: ______ / ______ Restore target: sandbox ______ (isolated: yes / no) Key: escrowed / primary Duration: ____ min against a budget of ____ min Validation: checksums __ | row counts __ | fingerprint __ | app query __ Result: PASSED / FAILED Failure reason: ______ Ticket: ______ Evidence: ______ ``` **Exception record** ```text Asset: ______ Requirement not met: ______ Risk accepted by (data owner): ______ Compensating control: ______ Expiry date: ______ Plan to close: ______ ``` **Architecture decision record: vault lock mode** ```text Context: tier; threat (ransomware, insider, provider error); regulation; erasure duties Options: (a) governance mode + separate account (b) compliance mode after the grace period (c) offline copy Decision: ______ Retention min/max: ______ Grace period: ______ Consequences: cost locked in for ____ days; erasure handled by ______; revisit on ______ ``` ### FAQ **Is "BackupOps" a job title or a team?** Rarely. Name an owner and write a policy before creating a team. **How often should we test restores?** By tier; this page's examples are monthly full restores for tier 1, quarterly samples for tier 2 and yearly tests for tier 3, plus automated integrity checks. DORA's Article 12 says testing "shall be undertaken periodically" without fixing an interval; check any technical standards that apply to you.[^dora] **Do we need to back up SaaS and cloud data?** Yes, or record why not. On AWS you are "responsible for managing resiliency of your data including backup, versioning, and replication strategies".[^awsresil] Native backup covers accidental deletion; an independent copy covers provider failure and tenant-admin compromise. **If the vault is immutable, do we still need encryption?** Yes: immutability protects integrity, encryption confidentiality. NIST lists cryptographic protection separately (CP-9(8)).[^nist53] **Is an etcd snapshot a Kubernetes backup?** It covers cluster state, not volume data or external databases; back up and test both ([KubernetesOps](../kubernetesops/)).[^etcd] ## Going further ### Advanced topics and emerging direction - **Isolated recovery environments.** NIST advises restoring cyber-attack recovery copies onto isolated staging,[^nist209] and DORA requires segregated systems for restores on an entity's own systems;[^dora] our expectation is that such environments become standing infrastructure. - **Recovery without the owning account.** AWS logically air-gapped vaults can be shared, or recovered through multi-party approval when the owning account is inaccessible,[^lagvault] which answers the NCSC's out-of-band access point.[^ncsccloud] ### Learning path | Stage | Activities | |---|---| | Prerequisites | Linux basics, storage concepts ([StorageOps](../storageops/)), one database engine | | 1 day | SRE data-integrity chapter and NCSC principles; Lab 1 | | 1 week | Lab 2; policies for three real assets; one timed manual restore | | 30 days | Lab 3 in a sandbox; automated tests and the KPI script; ransomware runbook tabletop | | Specialise | [DBOps](../dbops/), [KubernetesOps](../kubernetesops/), [ResponseOps](../responseops/), [DisasterRecoveryOps](../disasterrecoveryops/) | | Portfolio | A harness that restores random points nightly and publishes a proven-recoverable dashboard | ## Wrap-up ### Executive recap and cheat sheet - **Decide:** tiers, RPO and restore budget per tier, an owner per asset. - **Build:** consistent copies; one isolated, locked copy beyond production credentials; escrowed keys. - **Prove:** scheduled sandbox restores, validated and recorded; a human exercise each quarter. - **Measure:** coverage, RPO compliance, test currency, isolated copies and **proven recoverable**, not job success. - **Watch out for:** silent failures, count-based retention, keys in the blast radius, unpatched backup servers, replicas taken for backups. - **Next action:** restore one critical asset this week and time it. ### A judgement call: lock the vault in compliance mode? *Situation (illustrative):* a payments company keeps tier 1 backups in a separate AWS account under a governance-mode lock. Security wants compliance mode with 35-day minimum retention; finance fears locked costs, the privacy lead erasure requests. | | Governance mode, separate account, dual approval | Compliance mode after the grace period | |---|---|---| | Stops a stolen recovery-account admin | Only if that admin lacks lock permissions | Yes: nobody, including AWS, can change it[^vaultlock] | | Mistakes reversible | Yes | No: wrong retention becomes persistent cost | | Erasure requests | Early deletion with approval | Kept until expiry, beyond use[^ico] | | Residual risk | Insider or organisation-level compromise | Closing the account deletes contents after 90 days[^vaultlock] | **Assumptions:** ransomware is the main threat; the recovery account has separate administrators; retention is 35 days. **Decision process:** the attractive move, compliance mode for everything on day one, is wrong while the retention maths is unproven: a wrong maximum or a keep-forever point cannot be undone. A defensible path is governance mode for 60 days while retention, costs and tests settle, then compliance mode for tier 1 only, with erasure handling documented. **Uncertainty:** how likely an attacker is to reach the recovery account, and how a regulator weighs erasure against integrity. Revisit after any drill or incident. ### Learning objectives and mastery check You should now be able to **explain** what BackupOps covers; **design** a policy from an RPO; **build** an isolated, immutable copy; **operate** restore tests with evidence; and **evaluate** recoverability with metrics job success cannot game. 1. "Our database replicates synchronously to a second region, so we don't need backups." What do you answer? ::: answer Show answer for question 1 Replication copies deletions and corruption as faithfully as good writes: "replication and redundancy are not recoverability".[^sre-di] UniSuper's duplicate was deleted together with its private cloud ([case 2](#use-cases-and-scenarios)). Keep point-in-time copies outside the replication path. ::: 2. Hourly jobs have a 60-minute start window and take 20 minutes; copying to the vault takes 30 more. The RPO is one hour. Is it met if only the vault survives? ::: answer Show answer for question 2 No. Worst case is about 60 + 60 + 20 + 30 minutes, nearly three hours.[^planapi] Meet the RPO from the primary copy and have the data owner accept a longer one for the vault, recorded in the policy (Artefact 1's `isolated_rpo_hours`), or redesign (continuous backup, faster copies). ::: 3. Your retention rule is "keep the last 14 recovery points". Why might a security reviewer object? ::: answer Show answer for question 3 An attacker who can create points can flush the good ones with junk. Retain by time (the NCSC's advice; restic's `--keep-within`) and alert on unusual point creation.[^ncsccloud][^restic-forget] ::: 4. Investigators find the attacker entered 40 days before encrypting systems; isolated copies are kept for 30 days. What now, and what changes? ::: answer Show answer for question 4 Every isolated point may hold the foothold. Scan candidates, restore into isolated staging, rebuild hosts, and reconcile data from other sources.[^nist209][^cisarw] Afterwards keep some points (for example monthly) longer than plausible dwell times. ::: 5. You restore a customer database from three weeks ago; last week a valid erasure request was fulfilled. What must happen? ::: answer Show answer for question 5 Re-apply the erasure from your erasure log before the data is used. The ICO expects backup data to stay "beyond use";[^ico] a restore that brings it back into use defeats that. ::: ### Conclusion "BackupOps" is a new label on an old job, and it adds nothing the established practice lacks. What has changed is the threat: backups once guarded against failing disks and careless deletes; now attackers target them and the servers that run them, because a working restore lets a victim refuse to pay. The work moves from *making* copies to *proving* them: independent, locked, recent enough, and restored on a schedule by someone who records the result. GitLab and Code Spaces did not lack backup technology; they lacked a tested, independent copy and an owner whose job was to know it would work. UniSuper had copies outside the deleted service, and its story ended in recovery. Restore something important this week; it will teach you more than any number of green jobs. ## Sources and further reading Sources were opened on 2026-10-16, or on 2026-10-17 where marked; dates are the sources' own. Re-check living product documentation before relying on it; the KEV count and the AWS, Velero and Microsoft rows change fastest (suggested review: quarterly). [^sre-di]: [Data Integrity: What You Read Is What You Wrote](https://sre.google/sre-book/data-integrity/) — Raymond Blum and Rhandeev Singh, *Site Reliability Engineering* (O'Reilly, 2016). Checked 2026-10-16. [^gitlab2017]: [Postmortem of database outage of January 31](https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/) — GitLab, 2017-02-10. Checked 2026-10-16 and 2026-10-17. [^bocada]: [Bocada says automated BackupOps is coming](https://www.blocksandfiles.com/data-protection/2022/08/17/bocada-says-automated-backupops-is-coming/1606025) — Blocks & Files, 2022-08-17. Checked 2026-10-16. [^opsio]: [BackupOps Explained for Data Protection](https://opsiocloud.com/knowledge-base/what-is-backupops/) — Opsio, 2025-08-28 (updated 2025-10-02); evidence of label use only. Checked 2026-10-16. [^csf2]: [The NIST Cybersecurity Framework (CSF) 2.0](https://nvlpubs.nist.gov/nistpubs/CSWP/NIST.CSWP.29.pdf) — NIST CSWP 29, 2024-02-26. Checked 2026-10-16. [^nist53]: [NIST SP 800-53 Rev. 5 control catalog in OSCAL (release 5.2.0), control CP-9 "System Backup"](https://raw.githubusercontent.com/usnistgov/oscal-content/main/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json) — NIST, catalog last modified 2026-05-11. Checked 2026-10-16 and 2026-10-17. [^nist34]: [Contingency Planning Guide for Federal Information Systems (SP 800-34 Rev. 1)](https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-34r1.pdf) — NIST, 2010. Checked 2026-10-16. [^nist209]: [Security Guidelines for Storage Infrastructure (SP 800-209)](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-209.pdf) — NIST, October 2020. Checked 2026-10-16. [^ncsc2019]: [Offline backups in an online world](https://www.ncsc.gov.uk/blog-post/offline-backups-in-an-online-world) — NCSC (UK), 2019-08-13. Checked 2026-10-16. [^ncsccloud]: [Principles for ransomware-resistant cloud backups](https://www.ncsc.gov.uk/collection/ransomware-resistant-backups/principles-for-ransomware-resistant-cloud-backups) — NCSC (UK), version 1.0, 2024-11-22. Checked 2026-10-16. [^ncsconprem]: [Principles for ransomware-resistant on-premises backups](https://www.ncsc.gov.uk/collection/ransomware-resistant-backups/principles-for-ransomware-resistant-on-premises-backups) — NCSC (UK), version 1.0, 2024-11-22. Checked 2026-10-16. [^cisarw]: [#StopRansomware Guide](https://www.cisa.gov/stopransomware/ransomware-guide) — CISA, MS-ISAC, NSA and FBI, September 2023 edition. Checked 2026-10-16. [^kev]: [Known Exploited Vulnerabilities Catalog (JSON feed)](https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json) — CISA, catalog version 2026.10.16. Checked 2026-10-16. [^cisa2012]: [Data Backup Options](https://www.cisa.gov/sites/default/files/publications/data_backup_options.pdf) — Carnegie Mellon University for US-CERT, 2012. Checked 2026-10-16. [^dora]: [Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA), Article 12](https://eur-lex.europa.eu/eli/reg/2022/2554/oj/eng) — EU, 2022-12-14; applies from 2025-01-17. Checked 2026-10-16. [^hipaa]: [45 CFR 164.308 Administrative safeguards](https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-C/part-164/subpart-C/section-164.308) — eCFR (US), as of 2026-10-14. Checked 2026-10-16. [^hipaanprm]: [HIPAA Security Rule To Strengthen the Cybersecurity of Electronic Protected Health Information (proposed rule)](https://www.federalregister.gov/documents/2025/01/06/2024-30983/hipaa-security-rule-to-strengthen-the-cybersecurity-of-electronic-protected-health-information) — HHS, Federal Register, 2025-01-06. Checked 2026-10-16. [^paubox]: [HHS pushes final HIPAA security rule update to 2027](https://www.paubox.com/blog/hhs-pushes-final-hipaa-security-rule-update-to-2027) — Paubox (vendor blog), 2026-07-15. Checked 2026-10-16. [^ico]: [Right to erasure](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/individual-rights/individual-rights/right-to-erasure/) — Information Commissioner's Office (UK), undated. Checked 2026-10-16. [^sophos2024]: [The impact of compromised backups on ransomware outcomes](https://www.sophos.com/en-us/blog/the-impact-of-compromised-backups-on-ransomware-outcomes) — Sophos, 2024 (survey early 2024). Checked 2026-10-16. [^unisuper]: [A joint statement from UniSuper CEO Peter Chun, and Google Cloud CEO, Thomas Kurian](https://www.unisuper.com.au/about-us/media-centre/2024/a-joint-statement-from-unisuper-and-google-cloud) — UniSuper, 2024-05-08. Checked 2026-10-16. [^gcve2024]: [Sharing details on a recent incident impacting one of our customers](https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident) — Google Cloud Customer Support, 2024-05-25. Checked 2026-10-16. [^codespaces]: [Hosting service Code Spaces goes out of business due a cyber attack](https://securityaffairs.com/25912/cyber-crime/code-spaces-out-of-business.html) — Security Affairs, 2014-06-20 (quotes the company's statement). Checked 2026-10-16. [^ovh]: [OVHcloud must pay damages for lost backup data](https://www.blocksandfiles.com/data-protection/2023/03/23/ovhcloud-must-pay-damages-for-lost-backup-data/1594496) — Blocks & Files, 2023-03-23. Checked 2026-10-16. [^veeam321]: [What is the 3-2-1 Backup Rule?](https://www.veeam.com/blog/321-backup-rule.html) — Veeam, 2024-02-05 (updated 2026-07-03). Checked 2026-10-16. [^veeamsynth]: [Synthetic Full Backup](https://helpcenter.veeam.com/docs/vbr/userguide/synthetic_full_backup.html) — Veeam Backup & Replication 13 User Guide, page updated 2025-07-09. Checked 2026-10-16 and 2026-10-17. [^vaultlock]: [AWS Backup Vault Lock](https://docs.aws.amazon.com/aws-backup/latest/devguide/vault-lock.html) — AWS Backup Developer Guide. Checked 2026-10-16. [^putlock]: [PutBackupVaultLockConfiguration](https://docs.aws.amazon.com/aws-backup/latest/APIReference/API_PutBackupVaultLockConfiguration.html) — AWS Backup API Reference. Checked 2026-10-16. [^lagvault]: [Logically air-gapped vault](https://docs.aws.amazon.com/aws-backup/latest/devguide/logicallyairgappedvault.html) — AWS Backup Developer Guide. Checked 2026-10-16. [^restoretest]: [Restore testing](https://docs.aws.amazon.com/aws-backup/latest/devguide/restore-testing.html) — AWS Backup Developer Guide. Checked 2026-10-16. [^rtpapi]: [RestoreTestingPlanForCreate](https://docs.aws.amazon.com/aws-backup/latest/APIReference/API_RestoreTestingPlanForCreate.html) — AWS Backup API Reference. Checked 2026-10-16. [^valapi]: [PutRestoreValidationResult](https://docs.aws.amazon.com/aws-backup/latest/APIReference/API_PutRestoreValidationResult.html) — AWS Backup API Reference. Checked 2026-10-16. [^planapi]: [BackupRuleInput](https://docs.aws.amazon.com/aws-backup/latest/APIReference/API_BackupRuleInput.html) — AWS Backup API Reference. Checked 2026-10-16. [^lifecycle]: [Lifecycle](https://docs.aws.amazon.com/aws-backup/latest/APIReference/API_Lifecycle.html) — AWS Backup API Reference. Checked 2026-10-16. [^xacct]: [Creating backup copies across AWS accounts](https://docs.aws.amazon.com/aws-backup/latest/devguide/create-cross-account-backup.html) — AWS Backup Developer Guide. Checked 2026-10-16. [^awsenc]: [Encryption for backups in AWS Backup](https://docs.aws.amazon.com/aws-backup/latest/devguide/encryption.html) — AWS Backup Developer Guide. Checked 2026-10-16. [^legalhold]: [Legal holds and AWS Backup](https://docs.aws.amazon.com/aws-backup/latest/devguide/legalhold.html) — AWS Backup Developer Guide. Checked 2026-10-16 and 2026-10-17. [^awseks]: [AWS Backup adds logically air-gapped vault support for Amazon EKS](https://aws.amazon.com/about-aws/whats-new/2026/03/aws-backup-logically-air-gapped-vault-amazon-eks/) — AWS What's New, 2026-03-12. Checked 2026-10-16. [^awsassign]: [Assigning resources using JSON](https://docs.aws.amazon.com/aws-backup/latest/devguide/assigning-resources-json.html) — AWS Backup Developer Guide. Checked 2026-10-16. [^awsmalware]: [Malware protection in AWS Backup](https://docs.aws.amazon.com/aws-backup/latest/devguide/malware-protection.html) — AWS Backup Developer Guide. Checked 2026-10-16. [^s3lock]: [Locking objects with Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) — Amazon S3 User Guide. Checked 2026-10-16 and 2026-10-17. [^s3repl]: [What does Amazon S3 replicate?](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-what-is-isnot-replicated.html) — Amazon S3 User Guide. Checked 2026-10-16 and 2026-10-17. [^ebssnap]: [Amazon EBS snapshots](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html) — Amazon EBS User Guide. Checked 2026-10-16. [^awsresil]: [Shared Responsibility Model for Resiliency](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/shared-responsibility-model-for-resiliency.html) — AWS Well-Architected Framework, Reliability Pillar. Checked 2026-10-16. [^azimm]: [Concept of Immutable Vault for Azure Backup](https://learn.microsoft.com/en-us/azure/backup/backup-azure-immutable-vault-concept) — Microsoft Learn, 2026-09-09. Checked 2026-10-16. [^gcpvault]: [Backup vaults for immutable and indelible backups](https://docs.cloud.google.com/backup-disaster-recovery/docs/concepts/backup-vault) — Google Cloud documentation, updated 2026-10-15. Checked 2026-10-16. [^msrm]: [Shared responsibility in the cloud](https://learn.microsoft.com/en-us/azure/security/fundamentals/shared-responsibility) — Microsoft Learn, 2026-08-24. Checked 2026-10-16. [^m365]: [Overview of Microsoft 365 Backup](https://learn.microsoft.com/en-us/microsoft-365/backup/backup-overview) — Microsoft Learn, 2026-08-18. Checked 2026-10-16. [^velerohooks]: [Backup Hooks](https://velero.io/docs/v1.18/backup-hooks/) — Velero documentation, v1.18. Checked 2026-10-16. [^veleroschedule]: [Schedule API Type](https://velero.io/docs/v1.18/api-types/schedule/) — Velero documentation, v1.18. Checked 2026-10-16. [^velerofsb]: [File System Backup](https://velero.io/docs/v1.18/file-system-backup/) — Velero documentation, v1.18. Checked 2026-10-16. [^velerolimits]: [Backup Reference: "Cannot support backup data immutability"](https://velero.io/docs/v1.18/backup-reference/) — Velero documentation, v1.18. Checked 2026-10-16 and 2026-10-17. [^velero8724]: [Mandatory S3:DeleteObject Permission in Velero Forces Risky Maintenance Operations (#8724)](https://github.com/velero-io/velero/discussions/8724) — GitHub discussion, maintainers' replies, 2025-02-25. Checked 2026-10-16. [^velerodm]: [CSI Snapshot Data Movement](https://velero.io/docs/v1.18/csi-snapshot-data-movement/) — Velero documentation, v1.18. Checked 2026-10-16. [^velerobsl]: [BackupStorageLocation API type](https://velero.io/docs/v1.18/api-types/backupstoragelocation/) — Velero documentation, v1.18. Checked 2026-10-16. [^velerofreeze]: [examples/nginx-app/with-pv.yaml](https://github.com/velero-io/velero/blob/main/examples/nginx-app/with-pv.yaml) — Velero repository, main branch. Checked 2026-10-16. [^velerocncf]: [Velero Joins the CNCF Sandbox](https://velero.io/blog/velero-joins-cncf-sandbox/) — Velero blog, 2026-07-30. Checked 2026-10-16. [^restic-forget]: [Removing backup snapshots](https://restic.readthedocs.io/en/stable/060_forget.html) — restic documentation, 0.19.1. Checked 2026-10-16. [^restic-check]: [Working with repositories](https://restic.readthedocs.io/en/stable/045_working_with_repos.html) — restic documentation, 0.19.1. Checked 2026-10-16. [^gnutar]: [Using tar to Perform Incremental Dumps](https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html) — GNU tar manual. Checked 2026-10-16. [^pgdump]: [SQL Dump](https://www.postgresql.org/docs/current/backup-dump.html) — PostgreSQL 18 documentation, section 25.1. Checked 2026-10-16 and 2026-10-17. [^pgverify]: [pg_verifybackup](https://www.postgresql.org/docs/current/app-pgverifybackup.html) — PostgreSQL 18 documentation. Checked 2026-10-16. [^etcd]: [Operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/) — Kubernetes documentation. Checked 2026-10-16. [^pss]: [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) — Kubernetes documentation. Checked 2026-10-16. [^adfr]: [Active Directory forest recovery guide](https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/manage/forest-recovery-guide/ad-forest-recovery-guide) — Microsoft Learn, 2025-07-09. Checked 2026-10-16. [^vss]: [Volume Shadow Copy Service (VSS)](https://learn.microsoft.com/en-us/windows-server/storage/file-server/volume-shadow-copy-service) — Microsoft Learn, 2025-07-07. Checked 2026-10-16. [^ebsinit]: [Initialize Amazon EBS volumes](https://docs.aws.amazon.com/ebs/latest/userguide/initalize-volume.html) — Amazon EBS User Guide. Checked 2026-10-17. [^ebsfsr]: [Amazon EBS fast snapshot restore](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-snapshot-restore.html) — Amazon EBS User Guide. Checked 2026-10-17. [^s3delete]: [DeleteObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) — Amazon S3 API Reference. Checked 2026-10-17. [^veleroaws]: [Velero plugin for AWS, README (IAM policy)](https://github.com/velero-io/velero-plugin-for-aws/blob/main/README.md) — Velero project, main branch. Checked 2026-10-17. [^kopiafeat]: [Features](https://kopia.io/docs/features/) — Kopia documentation. Checked 2026-10-17. [^pgversions]: [Versioning Policy](https://www.postgresql.org/support/versioning/) — PostgreSQL Global Development Group. Checked 2026-10-17. **Further reading:** the whole SRE data-integrity chapter and PostgreSQL's [continuous archiving and point-in-time recovery](https://www.postgresql.org/docs/current/continuous-archiving.html). **Changelog.** 2026-10-16: first edition (research, ledger, labs 1 and 2 executed, lab 3 not executed). 2026-10-17: revised after independent review r1 (Velero limits and privileged hooks, backup-system hardening, legal holds, attribution fixes; sources opened 2026-10-16). 2026-10-17: revised after the final QA score (Velero immutability as documented, S3 Object Lock route, replication and legal-hold preconditions, restore-time sizing, runbook figure scan step, wrapped code, merged comparison table, labs re-run). 2026-10-17, post-score polish: EBS fast snapshot restore added to restore sizing, GitLab follow-ups labelled as listed work, NIST CP-9(7) limited to deletion with the extension marked as the Atlas's, synthetic-full trade-off worded as sourced, chain highlight drawn with thick arrows, long Velero sentences split.