BackupOps

BackupOps is an informal name for backup and recovery operations: deciding which data needs copies, making those copies independent of the systems they protect, keeping them safe from deletion, tampering and lost keys, and proving on a schedule that they can be restored recently enough and fast enough. Google’s SRE book passes on the saying: “No one really wants to make backups; what people really want are restores.”^[1] This page therefore treats BackupOps as restore engineering: a copy counts only when a test has restored it. It serves engineers, risk owners and auditors; beginners can stop after this part.

Backups fail quietly, and the failure is discovered at the worst moment. Four documented patterns recur.

• Silent failure without an owner. In January 2017 GitLab.com’s primary database was wiped by mistake. The daily backup script had run pg_dump 9.2 against PostgreSQL 9.6 and failed, and its failure e-mails were rejected for lack of DMARC signing; recovery came from a six-hour-old LVM snapshot taken for staging. Why was the procedure never tested? “Because there was no ownership, as a result nobody was responsible for testing this procedure.”^[7]
• Attackers target backups. CISA warns that “many ransomware variants attempt to find and subsequently delete or encrypt accessible backups”.^[8] In a Sophos-commissioned survey of 2,974 IT staff at organisations hit by ransomware (early 2024), 94% said attackers tried to compromise their backups and 57% of attempts succeeded (self-reported, so indicative only).^[9]
• Redundancy is not recoverability. In 2024 a provisioning error by Google operators deleted the Australian customer UniSuper’s Google Cloud VMware Engine private cloud, duplicate included; backups outside the deleted service limited the loss (case 2).^[10][11]
• Obligations. DORA requires most EU financial entities to document backup policies and test them periodically; HIPAA requires “retrievable exact copies” of electronic health data (methods).^[6][12]

With BackupOps, every data class has an owner, a policy, a copy beyond production credentials, and dated restore evidence.

A replica copies deletions as faithfully as good writes: “replication and redundancy are not recoverability”. Nor is an archive a backup: “backups can be loaded back into an application, while archives cannot”.^[1] A snapshot is a backup only if it is independent: Amazon EBS snapshots are stored in Amazon S3,^[13] so they outlive the volume, but the same account credentials can delete them, as Code Spaces found in 2014 (case 3).

Editorial distinction: BackupOps restores data; DisasterRecoveryOps restores services (failover, failback, the decision to invoke) and owns the RTO and RPO definitions used here. Nor is BackupOps a product: a backup suite gives you jobs, not evidence that restores work.

• The 3-2-1 rule. A 2012 paper for US-CERT attributes it to Peter Krogh’s The DAM Book (2nd edition, 2009): three copies, on two media types, one offsite.^[14]
• The ransomware era. In 2019 the NCSC added an “offline rule”, because 3-2-1 “does not require any backup location to be offline”;^[15] vendors extended the rule (3-2-1-1-0, methods), and the NCSC published ransomware-resistant backup principles in November 2024.^[16][5] Cloud services now offer root-proof retention locks and scheduled restore testing (tooling).

Mental model: four questions for every copy. (1) Independent? Does it share a site, account, credentials or keys with the source? (2) Intact? Is it immutable and checksummed? (3) Recent enough? Is it newer than the RPO allows? (4) Restorable in time? Has a test restored it within the time budget? A copy that passes all four is proven recoverable; the rest are hopes. (This framing is the Atlas’s own.)

flowchart TB
  S["StorageOps: snapshots, replication"] -->|"raw material"| B
  D["DBOps: dumps, WAL, PITR"] -->|"raw material"| B
  X["ResponseOps: ransomware incident"] -->|"asks for a clean point"| B
  B["BackupOps: independent copies, proven restores"]
  B -->|"restorable data"| R["DisasterRecoveryOps: service recovery"]
  B -->|"restore evidence"| C["ComplianceOps: audits"]
  B <-->|"erasure requests; re-applied after restores"| P["PrivacyOps: erasure"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class B core
  class S,D,X,R,C,P neutral

This charter is the Atlas’s proposal; no official BackupOps manifesto exists. Mission: every data asset that matters can be restored, recently enough and fast enough, by people who are not its usual operators, even when production and its credentials are compromised.

1. The restore is the product. A recovery point without a successful test is unproven.
2. Count independent failure domains, not copies. 3-2-1 is a floor. Ask what single event (a fire, a deleted account, a stolen password, a lost key) would destroy every copy.
3. One copy that production credentials cannot delete. Immutable, offline or held by a separate party; NIST SP 800-53 adds dual authorisation for deleting backups (CP-9(7)) and cryptographic protection (CP-9(8)).^[18]
4. Retain by time, not by count. The NCSC recommends storing backups “according to a fixed time period, rather than a fixed number of backups”.^[5]
5. Keys and access must survive the disaster, including the loss of all corporate identities.^[5]
6. Every asset has an owner, a policy and evidence, the cheapest of GitLab’s failures to prevent.^[7]
7. Proportionality. Tier by business impact; data that can be rebuilt need not be backed up.

Non-goals: high availability, service failover and long-term archiving, which other pages own.

R = responsible, A = accountable, C = consulted, I = informed. Adjust names to your organisation.

Day 0 is design (classify, set policy, build the isolated copy); Day 1 onboards each system with a first verified restore; Day 2 is the loop of jobs, copies, tests and expiry, which each recovery point follows.

stateDiagram-v2
  direction TB
  [*] --> Created: job completes
  Created --> Verified: checksum and catalogue OK
  Created --> Failed: job or check error
  Verified --> Isolated: copied to independent vault
  Isolated --> Locked: retention lock applies
  Locked --> Proven: restore test passes
  Locked --> Suspect: restore test fails
  Suspect --> Failed: cannot be repaired
  Proven --> Expired: retention ends
  Locked --> Expired: retention ends (untested)
  Expired --> [*]: deleted by policy
  Failed --> [*]: alert owner, rerun

Exceptions are recorded with an owner, a compensating control and an expiry date.

The central idea, on premises or in the cloud, is a trust boundary: the isolated copy lives where production identities cannot delete it, and a sandbox proves it works.

flowchart TB
  subgraph PROD["Production account or site"]
    A["Apps, databases, volumes, SaaS"] -->|"quiesce, copy"| E["Backup engine and hooks"]
    E --> P[("Primary repository")]
  end
  subgraph ISO["Isolated recovery account or site"]
    V[("Locked vault: immutable copy")] -->|"scheduled restore"| T["Restore-test sandbox"]
    K["Escrowed recovery keys"] -.->|"decrypt"| T
  end
  P -->|"copy job, write-only identity"| V
  E -->|"job status"| M["Monitoring and evidence"]
  T -->|"test results"| M
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A,E,P neutral
  class V,K,T core
  class M neutral

Design rules per component are in Building blocks. Alternatives: offline media isolate best but restore slowly; a second provider adds independence at the cost of egress and skills; small teams can run one locked vault in a separate account.

flowchart LR
  F["Sunday full"] ==> M["Mon incr."] ==> T["Tue incr."] ==> W["Wed incr."]
  F -.-> D["Wed differential"]
  F --> S["Thu synthetic full"]
  W --> S
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class F,M,T,W core
  class D,S neutral

Happy path. Schedule fires → pre-hook pauses writes → engine copies → post-hook resumes → checksum and catalogue → copy to the locked vault → success recorded → a scheduled test restores a sampled point, validates it and writes evidence → retention expires it.

Exception path. A job fails or never starts → retry → page the owner before the RPO is breached. A failed restore test is an incident at the asset’s tier: every point of that type is unproven until a new test passes. An unrequested deletion or retention change on the vault → a security alert for ResponseOps, because attacks on backups can precede the main attack.^[5]

Derive the schedule from the RPO, including copy delay. Worst-case loss for a copy is roughly the interval between backups plus the time until a backup reaches the copy you would restore from. Worked example: a nightly job starts at 01:00 and reaches the isolated vault at 05:00. If everything else is lost at 04:59, the newest vault copy holds data from 01:00 the previous day: about 28 hours of loss against a 24-hour RPO. Back up more often or copy faster.

Size the restore against its budget. Restore time is retrieval (cold tiers, offsite media) plus transfer plus recovery steps (log replay, application start) plus validation. Illustrative calculation: 4 TB at a measured 250 MB/s on one stream needs 16,000 seconds, about 4.4 hours, against a 2-hour budget. Four streams, if source, network and target sustain them, cut that to about 67 minutes; with 20 minutes of log replay and 15 of validation the restore fits, at 102 minutes. Cloud restores can hide time: a volume created from an EBS snapshot may be slow until its blocks arrive from S3, and even at the highest provisioned initialisation rate, 300 MiB/s per volume (as of October 2026), 4 TiB takes about 3.9 hours.^[20] Fast snapshot restore avoids the wait (the volume “is fully initialized at creation”), but it is enabled per snapshot and Availability Zone and billed for every minute it is on.^[20][21] Time each phase in tests at production size.

Make copies consistent. A crash-consistent copy looks like the moment after a power cut. On Windows, Volume Shadow Copy Service writers guarantee “a consistent data set to back up”.^[22] Velero runs hooks in pods, such as fsfreeze around a volume snapshot; by default a failing hook is an error and a hook times out after 30 seconds.^[23] Velero’s fsfreeze example runs a privileged sidecar,^[24] which Pod Security “baseline” and “restricted” namespaces reject,^[25] and if the post-hook never runs the volume stays frozen and writes hang. Prefer application-level quiescing or crash-consistent CSI snapshots where the application tolerates them. Velero’s file-system backup is “less consistent than the snapshot approaches”.^[26] For databases use the engine’s tools: a pg_dump “represents a snapshot of the database at the time pg_dump began running”.^[27]

Build the isolated copy. Use a retention-locked vault in a separate account, offline media or a second provider. The backup client should be able to create backups but not delete them, and should hold no credentials while idle.^[15] Cloud locks come in two modes: governance, which privileged users can still change, and compliance, which nobody can change: AWS Backup Vault Lock after a grace period of at least 72 hours, S3 Object Lock at once.^[28][29][30]

Design retention by time. The NCSC’s example keeps daily backups for a month and monthly backups for a year.^[5] Count-based rules can be gamed: restic’s documentation shows how an attacker who adds empty snapshots to an append-only repository can make a “keep three weekly” policy delete every legitimate one, and recommends the time-based --keep-within.^[31] Keep some points longer than an attacker’s plausible dwell time.

Keep keys recoverable. An attacker “doesn’t need to actually delete the data itself if they can simply delete or modify the encryption key”.^[5] Escrow a recovery key out of band (the NCSC suggests paper or a QR code in a safe), restrict key deletion, and decrypt in every sandbox test. etcd snapshots hold every Kubernetes object, so the Kubernetes documentation advises encrypting them.^[32]

Test restores on a ladder. Each rung finds failures the one below cannot:

1. Integrity checks: checksums; restic check --read-data, which reads every pack file;^[33] pg_verifybackup, which “does not and cannot include every check” a running server performs.^[34]
2. Sample restores: a file, a mailbox or a table, restored and compared.
3. Full sandbox restores with automated validation (Lab 2). AWS Backup restore testing, for example, restores the latest or a random recovery point, accepts your validation result and deletes the test resources.^[35][36]
4. Exercises: the team rebuilds a service from backups against the clock, with DisasterRecoveryOps.

The SRE book: “the only test that should let you sleep at night is a full end-to-end test”, automated and run continuously.^[1] NIST SP 800-53 likewise calls for integrity testing and sample restores (CP-9(1), CP-9(2)).^[18]

Protect SaaS data deliberately. In Microsoft’s shared-responsibility model, whatever the deployment type, “you always retain” responsibility for your data.^[37] Microsoft 365 Backup keeps backups “within the protected services’ data boundaries”,^[38] which buys fast restores, not independence from the provider or a compromised tenant administrator; decide per application whether to keep a copy outside it.

Back up Kubernetes at two levels. etcd holds cluster state (KubernetesOps). A tool such as Velero backs up namespaces and volume data, using CSI snapshots or Kopia-based file copies.^[32][26] Its schedules keep backups for 30 days unless ttl says otherwise.^[39] Velero’s v1.18 documentation says it does not support immutable storage. On Amazon S3 with Object Lock, backups work, but deleting one leaves the old object versions behind. Azure works only with version-level immutability. Google Cloud bucket locks do not work.^[40] Velero’s credential also needs delete rights, for expiry, deletion and Kopia maintenance.^[41] Artefact 3 compares three routes to a locked copy.

Back up the identity plane. Directories, identity-provider configuration and privileged-access vaults are tier 1 data: other restores usually need them first (the Atlas’s inference). Microsoft’s forest recovery guide covers the case where all domain controllers fail;^[42] procedures belong to IdentityOps.

The regulations bind specific sectors in specific places; confirm what applies to you with your compliance team (ComplianceOps).

Choose by the job, then by product, and re-check vendor documentation before relying on any row.

Selection criteria: source coverage, consistency, locks and who can delete what, keys, restore speed, test automation, total cost, and exit (can you read backups without the vendor?).

Automate the checkable: policy as code, tag-based selection that protects new resources by default, inventory diffs, machine-validated restore tests, and malware scanning (AWS Backup can have Amazon GuardDuty scan each point).^[48] NIST SP 800-53 asks for dual authorisation to delete or destroy backups (CP-9(7));^[18] the Atlas recommends two people for anything else that reduces protection: shorter retention, unlocking, or restoring over production. Language models can summarise job logs and draft runbooks; they cannot prove recoverability, and no AI agent should be able to delete or unlock backups.

Three artefacts make the policy executable; names, accounts and ARNs are placeholders.

Artefact 1: a protection policy kept as code, recording the data owner’s decisions.

Example status: YAML syntax-checked; a policy document, not a tool configuration.

# protection-policy.yaml: reviewed like code; the data owner approves changes
version: 1
tiers:
  tier1:                       # revenue- or safety-critical data
    rpo_hours: 1               # from the business impact analysis
    isolated_rpo_hours: 28     # accepted worst case if only the vault survives
    restore_budget_minutes: 120
    consistency: application   # engine dump, VSS writer or freeze hook
    copies:
      - {where: primary-repository, every: hour, keep_days: 2}
      - {where: primary-repository, every: day, keep_days: 7}
      - {where: isolated-vault, every: day, lock: compliance, keep_days: 35}
      - {where: isolated-vault, every: month, lock: compliance, keep_days: 400}
    restore_test: {every_days: 30, level: full-sandbox}
  tier2:
    rpo_hours: 24
    restore_budget_minutes: 480
    consistency: crash-acceptable
    copies:
      - {where: isolated-vault, lock: governance, keep_days: 35}
    restore_test: {every_days: 90, level: sample}
do_not_back_up:
  - {class: build-artifacts, reason: "rebuilt from source in under 30 minutes"}
assets:
  - {id: orders-db, tier: tier1, owner: orders-team, engine: postgresql}
  - {id: k8s-prod, tier: tier2, owner: platform-team, engine: velero}

Artefact 2: an AWS Backup plan for tier 1 (without the monthly 400-day rule): hourly points kept two days, and a daily point copied to a locked vault in a separate recovery account. Cross-account copies need both accounts in one AWS Organizations organisation, cross-account backup enabled from its management account, a destination vault access policy, and a customer managed KMS key for resource types AWS Backup does not fully manage, such as Amazon RDS.^[54][55] StartWindowMinutes must be at least 60, so an hourly job can start up to an hour late; count that in the RPO.^[56]

Example status: JSON syntax-checked; not executed (requires an AWS account). Use with aws backup create-backup-plan --backup-plan file://plan.json.

{
  "BackupPlanName": "tier1-orders",
  "Rules": [
    {
      "RuleName": "hourly-local",
      "TargetBackupVaultName": "prod-primary",
      "ScheduleExpression": "cron(0 * ? * * *)",
      "StartWindowMinutes": 60,
      "CompletionWindowMinutes": 120,
      "Lifecycle": {"DeleteAfterDays": 2},
      "RecoveryPointTags": {"tier": "1"}
    },
    {
      "RuleName": "daily-isolated-copy",
      "TargetBackupVaultName": "prod-primary",
      "ScheduleExpression": "cron(30 1 ? * * *)",
      "StartWindowMinutes": 60,
      "CompletionWindowMinutes": 240,
      "Lifecycle": {"DeleteAfterDays": 7},
      "CopyActions": [
        {
          "DestinationBackupVaultArn":
            "arn:aws:backup:eu-west-1:444455556666:backup-vault:recovery-locked",
          "Lifecycle": {"DeleteAfterDays": 35}
        }
      ],
      "RecoveryPointTags": {"tier": "1"}
    }
  ]
}

The recovery vault’s access policy names the production account, not the whole organisation, so another compromised account cannot fill the locked vault with junk that must then be kept.^[54]

Example status: JSON syntax-checked; not executed. Apply in the recovery account with aws backup put-backup-vault-access-policy --backup-vault-name recovery-locked --policy file://vault-policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CopyFromProductionOnly",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::111122223333:root"},
      "Action": "backup:CopyIntoBackupVault",
      "Resource": "*"
    }
  ]
}

Artefact 3: a Velero schedule with freeze hooks, snapshot data moved off the cluster, 35-day retention, and backups that survive deletion of the schedule.^[39][23] Its location recovery-bucket is a bucket in the recovery account. Three routes lead to a locked copy; none was tested for this page:

• Object Lock on recovery-bucket itself, which Velero’s documentation says works on S3.^[40] A delete without a version ID only adds a delete marker.^[57] A deleted backup must then be recovered from old versions by hand, and orphaned versions cost storage at least until their lock ends. Use compliance mode, because governance mode yields to anyone holding s3:BypassGovernanceRetention.^[30] A bucket’s default lock counts from when each version was written, but Kopia reuses stored data across backups.^[30][58] So the lock on data that newer backups still use can end before those backups do (the Atlas’s inference). Deleting a version requires s3:DeleteObjectVersion; keep it out of Velero’s role, as Velero’s documented AWS policy does.^[59][60]
• Replication into a locked bucket. S3 does not replicate deletions if the rule is a current, Filter-based one with delete-marker replication left off.^[57] Prove the copy by restoring from it with a second Velero whose location is ReadOnly.^[61]
• AWS Backup for Amazon EKS, which supports logically air-gapped vaults (as of October 2026).^[49]

If Velero stops between the hooks, unfreeze by hand with kubectl exec <pod> -c fsfreeze -- /sbin/fsfreeze --unfreeze /data.

Example status: YAML syntax-checked; not executed (requires a cluster with Velero and its node-agent, whose data-mover pods run as root,^[62] a CSI snapshot class and a storage location named recovery-bucket).

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: orders-nightly
  namespace: velero
spec:
  schedule: "0 1 * * *"
  useOwnerReferencesInBackup: false   # backups outlive the schedule
  template:
    includedNamespaces: ["orders"]
    snapshotVolumes: true
    snapshotMoveData: true            # needs node-agent; moves snapshot data
    storageLocation: recovery-bucket  # recovery account; see the text
    ttl: 840h0m0s                     # 35 days
    hooks:
      resources:
        - name: freeze-data-volume
          includedNamespaces: ["orders"]
          includedResources: ["pods"]
          labelSelector:
            matchLabels: {app: orders-api}
          pre:
            - exec:
                container: fsfreeze           # privileged sidecar in the app pod
                command: ["/sbin/fsfreeze", "--freeze", "/data"]
                onError: Fail
                timeout: 30s
          post:
            - exec:
                container: fsfreeze
                command: ["/sbin/fsfreeze", "--unfreeze", "/data"]
                onError: Fail                 # a failed unfreeze must surface
                timeout: 30s

Labs 1 and 2 were executed on 2026-10-16, and their scripts again on 2026-10-17 after line wrapping (Ubuntu 20.04, GNU tar 1.30, PostgreSQL 13.13, Python 3.8); Lab 3 was not executed.

Objective: see a backup chain work, verify a restore against checksums, and watch a restore test catch what jobs do not report. Prerequisites: Linux (or WSL) with GNU tar, sha256sum and diff. Cost and risk: none; everything stays in ./backup-lab. Time: 20 minutes.

Example status: syntax-checked with bash -n and executed (outputs below).

#!/usr/bin/env bash
# Lab 1: full + incremental backups with GNU tar, then a verified restore.
# Usage: ./backup-lab.sh init|full|change|incr|restore|corrupt|race
set -euo pipefail
LAB="${LAB:-$PWD/backup-lab}"
SRC="$LAB/src"; REPO="$LAB/repo"; RESTORE="$LAB/restore"

init() {     # sample data
  rm -rf "$LAB"; mkdir -p "$SRC/docs" "$REPO"
  printf 'invoice 1\n' > "$SRC/docs/inv-001.txt"
  printf 'invoice 2\n' > "$SRC/docs/inv-002.txt"
  head -c 1048576 /dev/urandom > "$SRC/blob.bin"
}
full() {     # level 0: everything; the snapshot file records what was seen
  tar --create --file="$REPO/full-0.tar" \
      --listed-incremental="$REPO/state.snar" -C "$SRC" .
}
change() {   # a day of work: modify, add, delete
  printf 'invoice 2 (corrected)\n' > "$SRC/docs/inv-002.txt"
  printf 'invoice 3\n' > "$SRC/docs/inv-003.txt"
  rm "$SRC/docs/inv-001.txt"
}
incr() {     # level 1: only changes since level 0, plus evidence files
  tar --create --file="$REPO/incr-1.tar" \
      --listed-incremental="$REPO/state.snar" -C "$SRC" .
  (cd "$SRC" && find . -type f -print0 | sort -z | xargs -0 sha256sum) \
    > "$REPO/source-1.sha256"
  (cd "$REPO" && sha256sum full-0.tar incr-1.tar) > "$REPO/archives.sha256"
  tar --list --file="$REPO/incr-1.tar"
}
restore() {  # verify archives, restore the chain oldest first, verify the result
  (cd "$REPO" && sha256sum --check archives.sha256)
  rm -rf "$RESTORE"; mkdir -p "$RESTORE"
  for a in full-0.tar incr-1.tar; do
    tar --extract --file="$REPO/$a" --listed-incremental=/dev/null -C "$RESTORE"
  done
  (cd "$RESTORE" && sha256sum --check --quiet "$REPO/source-1.sha256")
  diff -r "$SRC" "$RESTORE"
  n=$(find "$RESTORE" -type f | wc -l)
  echo "RESTORE VERIFIED: $n files match the source"
}
corrupt() {  # simulate silent corruption of the incremental archive
  printf 'X' | dd of="$REPO/incr-1.tar" bs=1 seek=1024 conv=notrunc status=none
}
race() {     # full, change and incremental within milliseconds of each other
  init; full; change; incr
}

case "${1:-}" in
  init|full|change|incr|restore|corrupt|race) "$1" ;;
  *) echo "usage: $0 init|full|change|incr|restore|corrupt|race" >&2; exit 2 ;;
esac

1. Save the script as backup-lab.sh, run chmod +x backup-lab.sh, then ./backup-lab.sh init and ./backup-lab.sh full.
2. Wait a few seconds, then run change and incr. The incremental lists only ./, ./docs/, ./docs/inv-002.txt and ./docs/inv-003.txt: the modified and new files; the deletion is recorded in directory metadata.^[63]
3. Run restore. Expected output: full-0.tar: OK, incr-1.tar: OK, RESTORE VERIFIED: 3 files match the source (inv-001.txt stays deleted because GNU tar replays deletions).^[63]
4. Break the chain: extract only incr-1.tar into an empty directory: two files, no blob.bin.
5. Corrupt the media: run corrupt, then restore. Expected: incr-1.tar: FAILED, exit status 1.
6. Race the clock: run race, then restore, several times. In our runs the incremental missed the edit to inv-002.txt in 7 to 20 of every 20 runs, depending on the machine; every tar exited 0, and only the restore check failed (./docs/inv-002.txt: FAILED). Our diagnosis: the kernel stamps file times from a clock that advances every few milliseconds, while tar records the full backup’s start precisely, so an edit made moments later can look older. The GNU tar manual warns that incremental dumps “depend crucially on time stamps” (its examples: altered time stamps, a clock set backwards);^[63] the same dependency explains this race.

Validation: steps 3, 5 and 6 behave as described. Troubleshooting: unrecognized option '--listed-incremental' means a non-GNU tar. Teardown: rm -rf backup-lab.

Objective: restore the stored copy into a scratch database, validate it against metadata written at backup time, and keep evidence. Prerequisites: a disposable, supported PostgreSQL (never production; we ran 13.13, unsupported since November 2025),^[64] client tools of the same or newer major version, Python 3. Cost and risk: none locally. Time: 45 minutes.

Example status: syntax-checked and executed against a throwaway PostgreSQL 13.13 cluster (socket-only, created with initdb).

# A throwaway, socket-only cluster (Debian/Ubuntu paths shown)
PGBIN=/usr/lib/postgresql/13/bin    # the major version we ran; use yours
SOCK="$HOME/pglab/sock"
mkdir -p "$SOCK"
"$PGBIN/initdb" -D "$HOME/pglab/data" -U labadmin --auth=trust > /dev/null
"$PGBIN/pg_ctl" -D "$HOME/pglab/data" -l "$HOME/pglab/pg.log" -w start -o \
  "-c listen_addresses='' -c unix_socket_permissions=0700 -k $SOCK -p 54329"
export PGHOST="$SOCK" PGPORT=54329 PGUSER=labadmin
createdb shop
psql -X -q -d shop -f setup-shop.sql

Example status: executed (PostgreSQL 13.13).

-- setup-shop.sql: fake data for the lab
CREATE TABLE customers (id int PRIMARY KEY, email text NOT NULL UNIQUE);
CREATE TABLE orders (
  id int PRIMARY KEY,
  customer_id int NOT NULL REFERENCES customers(id),
  amount numeric(10,2) NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);
INSERT INTO customers
  SELECT g, 'user' || g || '@example.com' FROM generate_series(1, 500) g;
INSERT INTO orders (id, customer_id, amount)
  SELECT g, 1 + (g % 500), round((g * 7 % 10000) / 100.0, 2)
  FROM generate_series(1, 20000) g;

The backup script refuses client tools older than the server (GitLab’s failure) and records a checksum and data fingerprint.

Example status: syntax-checked with bash -n and executed.

#!/usr/bin/env bash
# pg-backup.sh: back up one database and write metadata for later restore tests.
set -euo pipefail
SRC_DB="${SRC_DB:?set SRC_DB}"
REPO="${REPO:-$PWD/repo}"
CHECK="SELECT count(*) || ':' || md5(string_agg(
  id || ',' || customer_id || ',' || amount, ';' ORDER BY id)) FROM orders"
mkdir -p "$REPO"

# Guard: client tools must not be older than the server (the GitLab 2017 failure)
server_num=$(psql -X -At -d "$SRC_DB" -c 'SHOW server_version_num')
server_major=$(( server_num / 10000 ))
client_major=$(pg_dump --version | awk '{print $3}' | cut -d. -f1)
if [ "$client_major" -lt "$server_major" ]; then
  echo "FAIL: pg_dump $client_major is older than server $server_major" >&2
  exit 1
fi

stamp=$(date -u +%Y%m%dT%H%M%SZ)
dump="$REPO/$SRC_DB-$stamp.dump"
fingerprint=$(psql -X -At -d "$SRC_DB" -c "$CHECK")   # lab: database is quiet
pg_dump --format=custom --file="$dump" "$SRC_DB"
sha=$(sha256sum "$dump" | cut -d' ' -f1)
fmt='{"db":"%s","created_at":"%s","sha256":"%s",'
fmt+='"fingerprint":"%s","server_major":%d}\n'
printf "$fmt" "$SRC_DB" "$(date -u +%FT%TZ)" "$sha" "$fingerprint" \
  "$server_major" > "$dump.meta.json"
echo "$dump"

Example status: syntax-checked with bash -n and executed.

#!/usr/bin/env bash
# restore-test.sh: restore a stored dump into a scratch database, validate it
# against the metadata written at backup time, append an evidence record.
set -euo pipefail
DUMP="${1:?usage: $0 <dump file>}"
META="$DUMP.meta.json"
SCRATCH_DB="restoretest_$(date -u +%Y%m%d%H%M%S)"
CHECK="SELECT count(*) || ':' || md5(string_agg(
  id || ',' || customer_id || ',' || amount, ';' ORDER BY id)) FROM orders"
field() {  # one field from the metadata file
  python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))[sys.argv[2]])' \
    "$META" "$1"
}
q() { psql -X -At -d "$SCRATCH_DB" -c "$1"; }

status=PASSED; reason=ok
# 1. Integrity of the stored copy, before spending time on a restore
if [ "$(sha256sum "$DUMP" | cut -d' ' -f1)" != "$(field sha256)" ]; then
  status=FAILED; reason=checksum_mismatch
elif ! pg_restore --list "$DUMP" > /dev/null 2>&1; then
  status=FAILED; reason=unreadable_archive
fi

# 2. Restore into a scratch database (dropped on exit) and time it
t0=$(date +%s); t1=$t0
trap 'dropdb --if-exists "$SCRATCH_DB"' EXIT
if [ "$status" = PASSED ]; then
  createdb "$SCRATCH_DB"
  if pg_restore --exit-on-error --no-owner --dbname="$SCRATCH_DB" "$DUMP"; then
    t1=$(date +%s)
    # 3. Validate data and structure; run one real application query
    actual=$(q "$CHECK")
    fks=$(q "SELECT count(*) FROM pg_constraint WHERE contype = 'f'")
    top=$(q "SELECT customer_id FROM orders
             GROUP BY 1 ORDER BY sum(amount) DESC LIMIT 1")
    if [ "$actual" != "$(field fingerprint)" ]; then
      status=FAILED; reason=fingerprint_mismatch
    elif [ "$fks" -lt 1 ] || [ -z "$top" ]; then
      status=FAILED; reason=validation_query
    fi
  else
    status=FAILED; reason=restore_error
  fi
fi

# 4. Evidence: one JSON line per test, kept with the backup records
fmt='{"dump":"%s","tested_at":"%s","backup_created_at":"%s","restore_seconds":%d,'
fmt+='"status":"%s","reason":"%s"}\n'
printf "$fmt" "$(basename "$DUMP")" "$(date -u +%FT%TZ)" "$(field created_at)" \
  $((t1 - t0)) "$status" "$reason" | tee -a "$(dirname "$DUMP")/evidence.jsonl"
[ "$status" = PASSED ]

1. Run d=$(SRC_DB=shop ./pg-backup.sh), then ./restore-test.sh "$d".
2. Expected output: one JSON line ending "status":"PASSED","reason":"ok", exit status 0, and no restoretest_* database left.
3. Break it: truncate -s 60000 "$d" and rerun. Expected: "reason":"checksum_mismatch", exit status 1, and a harmless NOTICE from dropdb --if-exists.
4. Schedule the test; alert when the newest evidence line is older than the tier’s interval.

Validation: step 2 passes and step 3 fails as described. On a live database the pre-dump fingerprint can differ from the dump; validate invariants instead (row counts within a tolerance, newest created_at within the RPO, constraints present). Troubleshooting: role "…" does not exist: pg_dump omits roles; restore them from pg_dumpall --globals-only or keep --no-owner.^[27] Unix-domain socket path … is too long (maximum 107 bytes): use a shorter directory (we hit this). Teardown: "$PGBIN/pg_ctl" -D "$HOME/pglab/data" -m fast stop, then rm -rf "$HOME/pglab" repo.

Objective: a governance-mode locked vault, one tagged resource backed up, and a weekly restore test you validate. Prerequisites: a sandbox AWS account, AWS CLI v2, an IAM role AWS Backup can assume, one small EBS volume tagged backup-tier=lab. Cost and risk: recovery-point storage and a charge per restore test.^[35] Never use compliance mode in a lab: after its grace period nobody, including AWS, can delete the vault while it holds recovery points.^[29]

Example status: syntax-checked with bash -n; not executed (requires an AWS account). JSON files below are syntax-checked.

#!/usr/bin/env bash
set -euo pipefail
export AWS_REGION="eu-west-1"
VAULT="lab-locked-vault"

aws backup create-backup-vault --backup-vault-name "$VAULT"
# Governance mode: no --changeable-for-days, so authorised users can unlock
aws backup put-backup-vault-lock-configuration --backup-vault-name "$VAULT" \
  --min-retention-days 7 --max-retention-days 35
aws backup describe-backup-vault --backup-vault-name "$VAULT" \
  --query '{locked: Locked, min: MinRetentionDays, max: MaxRetentionDays}'

PLAN_ID=$(aws backup create-backup-plan --backup-plan file://lab-plan.json \
  --query BackupPlanId --output text)
aws backup create-backup-selection --backup-plan-id "$PLAN_ID" \
  --backup-selection file://lab-selection.json

aws backup create-restore-testing-plan \
  --restore-testing-plan file://lab-restore-plan.json
aws backup create-restore-testing-selection \
  --restore-testing-plan-name lab_weekly \
  --restore-testing-selection file://lab-restore-selection.json

lab-plan.json keeps 7 days, inside the lock’s limits (the vault rejects anything outside them):

{
  "BackupPlanName": "lab-daily",
  "Rules": [
    {
      "RuleName": "daily-to-locked-vault",
      "TargetBackupVaultName": "lab-locked-vault",
      "ScheduleExpression": "cron(0 5 ? * * *)",
      "StartWindowMinutes": 60,
      "CompletionWindowMinutes": 180,
      "Lifecycle": {"DeleteAfterDays": 7}
    }
  ]
}

lab-selection.json selects the tagged volume; tag conditions use the aws:ResourceTag/ key format:^[65]

{
  "SelectionName": "lab-tagged",
  "IamRoleArn": "arn:aws:iam::111122223333:role/LabBackupRestoreRole",
  "Resources": ["*"],
  "Conditions": {
    "StringEquals": [
      {"ConditionKey": "aws:ResourceTag/backup-tier", "ConditionValue": "lab"}
    ]
  }
}

lab-restore-plan.json picks a random recovery point from the last seven days:^[66]

{
  "RestoreTestingPlanName": "lab_weekly",
  "ScheduleExpression": "cron(0 6 ? * MON *)",
  "StartWindowHours": 4,
  "RecoveryPointSelection": {
    "Algorithm": "RANDOM_WITHIN_WINDOW",
    "IncludeVaults": ["*"],
    "RecoveryPointTypes": ["SNAPSHOT"],
    "SelectionWindowDays": 7
  }
}

lab-restore-selection.json:

{
  "RestoreTestingSelectionName": "lab_ebs",
  "ProtectedResourceType": "EBS",
  "IamRoleArn": "arn:aws:iam::111122223333:role/LabBackupRestoreRole",
  "ProtectedResourceArns": ["*"],
  "ProtectedResourceConditions": {
    "StringEquals": [{"Key": "backup-tier", "Value": "lab"}]
  },
  "ValidationWindowHours": 2
}

1. Run the script. Expected: the describe-backup-vault query prints "locked": true with "min": 7 and "max": 35.
2. After the first backup and the next Monday, list the test’s restore jobs: aws backup list-restore-jobs --by-restore-testing-plan-arn <plan ARN>.
3. Validate the restored volume (attach it to a test instance and check a known file), then report: aws backup put-restore-validation-result --restore-job-id <id> --validation-status SUCCESSFUL --validation-status-message "file hash matched".^[36]

Validation: your status appears on the restore job. Troubleshooting: a retention error means DeleteAfterDays is outside the lock’s limits; if recent points are skipped, make SelectionWindowDays exceed the backup interval plus StartWindowHours.^[35] Teardown: delete the restore-testing selection and plan and the backup selection and plan; delete-backup-vault-lock-configuration; delete-recovery-point for each point; delete-backup-vault.

The method for three estates is in the artefacts and labs: a PostgreSQL database (Lab 2), AWS accounts (Artefact 2, Lab 3) and a Kubernetes platform (Artefact 3). The four sourced cases show what happens without it.

Case 1 — silent failure, no owner (GitLab.com, 2017, sourced). What happened: see the problem; disk snapshots were also off for the database servers, and copying the six-hour-old LVM snapshot over throttled disks took about 18 hours. Follow-up work GitLab listed as under way: Prometheus backup monitoring, automated testing of PostgreSQL backup recovery, hourly LVM snapshots and an owner for data durability.^[7] Success measure: a passing automated restore in the last day. Pitfall: alerts that travel through the failing system.

Case 2 — the provider deletes the environment (UniSuper, 2024, sourced). What happened: Google operators left a provisioning parameter blank, which gave the private cloud a fixed term and automatic deletion.^[10] The joint statement says duplication “in two geographies” went too and credits backups with an additional service provider;^[11] Google’s account describes one region and two zones, and credits untouched backups in Google Cloud Storage plus third-party backup software. Lesson: copies outside the deleted service and subscription saved the data; the accounts differ on whether copies outside Google were decisive. CISA suggests multi-cloud for cloud-to-cloud backups “in case all accounts under the same vendor are impacted”.^[8] Success measure (the Atlas’s recommendation): a drill restoring only from copies outside the primary provider.

Case 3 — one console login destroys data and backups (Code Spaces, 2014, sourced). What happened: an attacker with control-panel access deleted EBS snapshots, S3 buckets and machine images; the company said “most of our data, backups, machine configurations and offsite backups were either partially or completely deleted” and ceased trading.^[67] Lesson: copies reachable with the same credentials are not independent. Success measure: no single identity can delete production and its last copy.

Case 4 — “offsite” in the same building (OVHcloud fire, Strasbourg, 2021, sourced). One customer’s contract called the backup “physically isolated from the infrastructure in which the VPS server is set up”, yet both were in the same building; a French court awarded damages in 2023 (OVH appealed; outcome not checked).^[68] Lesson: verify where each copy is; the site-loss story is in DisasterRecoveryOps.

These KPIs are the Atlas’s proposal, built on the four questions.

The script computes them from an inventory export (illustrative data).

Example status: JSON and Python syntax-checked; executed with Python 3.8 (output below).

{
  "as_of": "2026-10-16T06:00:00Z",
  "test_window_days": {"1": 30, "2": 90, "3": 365},
  "jobs_last_30d": {"succeeded": 1188, "failed": 12},
  "assets": [
    {"id": "orders-db", "tier": 1, "protected": true, "rpo_hours": 1,
     "last_good_point": "2026-10-16T05:15:00Z", "isolated_copy": true,
     "last_test": "2026-10-02", "test_passed": true,
     "restore_min": 52, "budget_min": 120},
    {"id": "payments-db", "tier": 1, "protected": true, "rpo_hours": 1,
     "last_good_point": "2026-10-16T03:30:00Z", "isolated_copy": true,
     "last_test": "2026-09-30", "test_passed": true,
     "restore_min": 95, "budget_min": 120},
    {"id": "crm-saas", "tier": 1, "protected": false, "rpo_hours": 24,
     "last_good_point": null, "isolated_copy": false,
     "last_test": null, "test_passed": false,
     "restore_min": null, "budget_min": 480},
    {"id": "file-share", "tier": 2, "protected": true, "rpo_hours": 24,
     "last_good_point": "2026-10-15T22:00:00Z", "isolated_copy": true,
     "last_test": "2026-06-01", "test_passed": true,
     "restore_min": 300, "budget_min": 480},
    {"id": "k8s-prod", "tier": 2, "protected": true, "rpo_hours": 24,
     "last_good_point": "2026-10-15T07:00:00Z", "isolated_copy": false,
     "last_test": "2026-09-20", "test_passed": true,
     "restore_min": 140, "budget_min": 240},
    {"id": "wiki", "tier": 3, "protected": true, "rpo_hours": 168,
     "last_good_point": "2026-10-12T01:00:00Z", "isolated_copy": false,
     "last_test": null, "test_passed": false,
     "restore_min": null, "budget_min": 1440}
  ]
}

"""Compute restore-centred backup KPIs from an asset inventory (JSON)."""
import json
import sys
from datetime import date, datetime, timedelta


def ts(s):
    return datetime.fromisoformat(s.replace("Z", "+00:00"))


def pct(n, d):
    return f"{n}/{d} = {100 * n / d:.0f}%" if d else "n/a"


def main(path):
    with open(path) as f:
        inv = json.load(f)
    now = ts(inv["as_of"])
    window = {int(k): v for k, v in inv["test_window_days"].items()}
    assets = inv["assets"]
    protected = [a for a in assets if a["protected"]]

    def within_rpo(a):
        return now - ts(a["last_good_point"]) <= timedelta(hours=a["rpo_hours"])

    rpo_ok = [a for a in protected if within_rpo(a)]
    tested = [a for a in protected if a["test_passed"] and a["last_test"]
              and (now.date() - date.fromisoformat(a["last_test"])).days
              <= window[a["tier"]]]
    critical = [a for a in assets if a["tier"] <= 2]
    isolated = [a for a in critical if a["protected"] and a["isolated_copy"]]
    in_budget = [a for a in tested if a["restore_min"] <= a["budget_min"]]
    proven = [a for a in in_budget
              if a in rpo_ok and (a["tier"] > 2 or a["isolated_copy"])]

    jobs = inv["jobs_last_30d"]
    all_jobs = jobs["succeeded"] + jobs["failed"]
    print("Job success rate (vanity):  ", pct(jobs["succeeded"], all_jobs))
    print("Coverage:                   ", pct(len(protected), len(assets)))
    print("RPO compliance (protected): ", pct(len(rpo_ok), len(protected)))
    print("Restore-test currency:      ", pct(len(tested), len(protected)))
    print("Isolated copy (tier 1-2):   ", pct(len(isolated), len(critical)))
    print("Proven recoverable (all):   ", pct(len(proven), len(assets)))
    for a in assets:
        if a not in proven:
            print("  not proven:", a["id"])
    return 0 if len(proven) == len(assets) else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "inventory.json"))

Job success rate (vanity):   1188/1200 = 99%
Coverage:                    5/6 = 83%
RPO compliance (protected):  4/5 = 80%
Restore-test currency:       3/5 = 60%
Isolated copy (tier 1-2):    3/5 = 60%
Proven recoverable (all):    1/6 = 17%
  not proven: payments-db
  not proven: crm-saas
  not proven: file-share
  not proven: k8s-prod
  not proven: wiki

Worked calculation. At 06:00 payments-db’s newest good point is 2.5 hours old against a 1-hour RPO. file-share was last tested 137 days ago, outside its 90-day window. k8s-prod passes RPO (23 of 24 hours) and its test but has no isolated copy. crm-saas is unprotected and wiki never tested. Only orders-db (45-minute-old point, tested 14 days ago, 52 of 120 minutes, isolated) is proven: 1 of 6, or 17%, while 99% of jobs succeeded. Set targets per tier (for example 100% of tier 1, exceptions named and dated) and expect the number to fall when you first measure honestly.

Backups hold sensitive data for years, and the backup system is itself a target: on 2026-10-16 CISA’s Known Exploited Vulnerabilities catalogue listed at least 13 flaws in backup and data-protection products (Veeam, Veritas, Commvault, NAKIVO, Acronis, Arcserve), seven known to be used by ransomware.^[53] The NCSC’s on-premises principles begin “Make it possible to isolate your backup solution” and “Update your backup solution”.^[16]

Privacy. After a valid erasure request the UK ICO expects you to “take steps to ensure erasure from backup systems as well as live systems”; where backups cannot be overwritten at once, “the key issue is to put the backup data ‘beyond use’”.^[69] Atlas recommendation: keep an erasure log and re-apply it after every restore, or a restore resurrects deleted personal data (PrivacyOps). Legal holds stop points from expiring until removed, but only the points or object versions they are placed on (AWS Backup, S3 Object Lock).^[70][30] Residency: a vault in another country may move personal data across borders.

Costs come from stored volume (data × change rate × retention × copies), restores and egress, test compute and people. Illustrative calculation: 10 TB protected, 2% daily change, forever-incremental, 35 days: 10 + 10 × 0.02 × 35 = 17 TB before compression and deduplication, and as much again in the vault: 3.4 times the protected data.

Three traps: backups in a locked vault “cannot be deleted until their lifecycle completes”, so retention mistakes become persistent costs;^[29] cold tiers have minimum periods (90 days in AWS Backup);^[71] and deleting an incremental snapshot “might not reduce your data storage costs”.^[13] Data rebuildable within the RTO need not be backed up at all, which saves money and energy.

Auditors need the policy, coverage, dated restore-test records, the exception register and access reviews. Leading indicators are inventory drift and ageing evidence; the lagging one is a failed real restore. Quarterly, an outsider restores a random asset from the runbook alone. Services can automate checks (AWS Backup Audit Manager evaluates backups against controls you define).^[29]

ResponseOps leads the incident; BackupOps supplies clean, validated data.

flowchart TB
  A["Incident declared; ResponseOps leads"] --> B["Legal holds on candidate points; freeze deletions"]
  B --> C["Confirm vault, catalogue and keys intact"]
  C --> D{"Start of compromise known?"}
  D -- "Yes" --> E["Candidate: last point before it"]
  D -- "No" --> E2["Candidate: newest point"]
  E --> F["Scan the candidate"]
  E2 --> F
  F --> G["Restore into isolated staging"]
  G --> H{"Clean and valid?"}
  H -- "No: next older point" --> F
  H -- "Yes" --> I["Promote by business priority"]
  I --> J["Re-apply erasures; reconcile lost data"]
  J --> K["Review: dwell time versus retention"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class A,B risk
  class C,E,E2,F,G,I,J,K core
  class D,H decision

CISA advises looking for precursor “dropper” malware, because a ransomware event may signal an earlier, unresolved compromise;^[8] restoring before it is found risks reinfection (the Atlas’s inference).

• “The job is green, so we’re covered.” It ran; that is all. Instead: report proven recoverable.
• “We replicate and snapshot, so we have backups.” Not if the copies share the source’s path, array, account or credentials. Instead: point-in-time copies held elsewhere.
• “Restore testing is too expensive.” Less than discovering a gap during an outage. Instead: small automated tests, often.
• “Immutable means safe.” Locked copies can still be unrestorable (lost keys, disabled images) or hold malware. Instead: test and scan them.
• “Everyone owns backups.” Then nobody does. Instead: one accountable owner per asset.

Anyone holding data they cannot recreate needs backup and recovery; the decisions are how much, for which data and who owns it.

• A dedicated backup function pays off with many platforms, regulatory duties or a history of failed restores; small teams make backups an explicit platform-team duty, with the same policy and evidence.
• Do not back up data rebuildable within the RTO (build artefacts, caches, derived tables); protect the source and test the rebuild.
• Do not apply long immutable retention to data with erasure duties until you know how to keep it beyond use.
• Hidden responsibilities: key custody, the erasure log, egress costs, the inventory and separate administrators for the vault.

flowchart TB
  A{"Rebuildable from a source of truth within the RTO?"} -- "Yes" --> N["No backup: protect the source and test the rebuild"]
  A -- "No" --> B{"Business impact tier?"}
  B -- "Tier 1" --> T1["Consistent copies, locked isolated vault, monthly full restore test"]
  B -- "Tier 2" --> T2["Daily copies, isolated vault, quarterly sample restore"]
  B -- "Tier 3" --> T3["Weekly copies, one offsite copy, yearly test"]
  T1 --> S{"Only one provider holds every copy?"}
  T2 --> S
  S -- "Yes" --> I["Add a copy outside it, or record the accepted risk"]
  S -- "No" --> P["Apply the policy; prove it with a first restore"]
  T3 --> P
  I --> P
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A,B,S decision
  class T1,T2,T3,P,I core
  class N neutral

For tool choice, weight the selection criteria before any demo (for example restore speed 25%, locks 20%, coverage 20%, test automation 15%, exit 10%, cost 10%), then test the top two on your own data.

This is the Atlas’s model, not an industry standard.

Quick start (first week)

• Every data asset listed with an owner and a tier
• Alerts on missing success, outside production channels
• One tier 1 asset restored by hand, timed and validated
• Known: who can delete backups, and who can delete the keys
• Backup servers patched (check the KEV catalogue); MFA on destructive actions; consoles reachable only from admin hosts

Preflight: onboarding a new system

• Tier and RPO agreed with the data owner; “rebuild instead” considered
• Consistency method chosen and tested
• Copies land in the isolated vault within the lock’s limits
• Restore runbook written; first restore passed; erasure and residency answered

Before locking a vault

• Retention minimum and maximum match the policy; no “keep forever” points
• Cost for the whole locked period approved; erasure handling documented
• Governance mode run first; grace period understood (at least 72 hours in AWS Backup Vault Lock)^[28]
• Account-closure consequences understood^[29]

Each restore test and each quarter

• Point chosen at random or by rotation; restored into a sandbox with the escrowed key
• Validated with data checks and an application query; duration and evidence recorded
• Quarterly: KPIs, delete and key permissions, exceptions and backup-server patch levels reviewed

Protection policy (one page per data class)

Data class:       ____________________   Owner (accountable): ____________
Tier:             1 / 2 / 3     RPO: ____ h     Restore budget: ____ min
                  (the RTO comes from the DR plan)
Rebuild instead?  no / yes: how, and tested on ______
Consistency:      engine dump / VSS writer / freeze hook / crash-consistent OK
Primary copy:     ______ (retain __ d)
Isolated copy:    ______ (lock: governance / compliance; retain __ d)
Keys:             KMS key ______ ; recovery key escrowed at ______
Restore test:     every __ days; integrity / sample / full sandbox / exercise
Personal data:    yes / no; erasure handling ______ ; residency ______
Approved by:      owner ______  backup lead ______  security ______
Date:             ______   Review by: ______

Restore test record

Asset / point:    ______ / ______ (created ______)
Tester / date:    ______ / ______
Restore target:   sandbox ______ (isolated: yes / no)   Key: escrowed / primary
Duration:         ____ min against a budget of ____ min
Validation:       checksums __ | row counts __ | fingerprint __ | app query __
Result:           PASSED / FAILED   Failure reason: ______   Ticket: ______
Evidence:         ______

Exception record

Asset: ______   Requirement not met: ______
Risk accepted by (data owner): ______   Compensating control: ______
Expiry date: ______   Plan to close: ______

Architecture decision record: vault lock mode

Context:       tier; threat (ransomware, insider, provider error); regulation;
               erasure duties
Options:       (a) governance mode + separate account
               (b) compliance mode after the grace period   (c) offline copy
Decision:      ______   Retention min/max: ______   Grace period: ______
Consequences:  cost locked in for ____ days; erasure handled by ______;
               revisit on ______

Is “BackupOps” a job title or a team? Rarely. Name an owner and write a policy before creating a team.

How often should we test restores? By tier; this page’s examples are monthly full restores for tier 1, quarterly samples for tier 2 and yearly tests for tier 3, plus automated integrity checks. DORA’s Article 12 says testing “shall be undertaken periodically” without fixing an interval; check any technical standards that apply to you.^[6]

Do we need to back up SaaS and cloud data? Yes, or record why not. On AWS you are “responsible for managing resiliency of your data including backup, versioning, and replication strategies”.^[72] Native backup covers accidental deletion; an independent copy covers provider failure and tenant-admin compromise.

If the vault is immutable, do we still need encryption? Yes: immutability protects integrity, encryption confidentiality. NIST lists cryptographic protection separately (CP-9(8)).^[18]

Is an etcd snapshot a Kubernetes backup? It covers cluster state, not volume data or external databases; back up and test both (KubernetesOps).^[32]

• Isolated recovery environments. NIST advises restoring cyber-attack recovery copies onto isolated staging,^[43] and DORA requires segregated systems for restores on an entity’s own systems;^[6] our expectation is that such environments become standing infrastructure.
• Recovery without the owning account. AWS logically air-gapped vaults can be shared, or recovered through multi-party approval when the owning account is inaccessible,^[47] which answers the NCSC’s out-of-band access point.^[5]

• Decide: tiers, RPO and restore budget per tier, an owner per asset.
• Build: consistent copies; one isolated, locked copy beyond production credentials; escrowed keys.
• Prove: scheduled sandbox restores, validated and recorded; a human exercise each quarter.
• Measure: coverage, RPO compliance, test currency, isolated copies and proven recoverable, not job success.
• Watch out for: silent failures, count-based retention, keys in the blast radius, unpatched backup servers, replicas taken for backups.
• Next action: restore one critical asset this week and time it.

Situation (illustrative): a payments company keeps tier 1 backups in a separate AWS account under a governance-mode lock. Security wants compliance mode with 35-day minimum retention; finance fears locked costs, the privacy lead erasure requests.

Assumptions: ransomware is the main threat; the recovery account has separate administrators; retention is 35 days. Decision process: the attractive move, compliance mode for everything on day one, is wrong while the retention maths is unproven: a wrong maximum or a keep-forever point cannot be undone. A defensible path is governance mode for 60 days while retention, costs and tests settle, then compliance mode for tier 1 only, with erasure handling documented. Uncertainty: how likely an attacker is to reach the recovery account, and how a regulator weighs erasure against integrity. Revisit after any drill or incident.

You should now be able to explain what BackupOps covers; design a policy from an RPO; build an isolated, immutable copy; operate restore tests with evidence; and evaluate recoverability with metrics job success cannot game.

1. “Our database replicates synchronously to a second region, so we don’t need backups.” What do you answer?

2. Hourly jobs have a 60-minute start window and take 20 minutes; copying to the vault takes 30 more. The RPO is one hour. Is it met if only the vault survives?

3. Your retention rule is “keep the last 14 recovery points”. Why might a security reviewer object?

4. Investigators find the attacker entered 40 days before encrypting systems; isolated copies are kept for 30 days. What now, and what changes?

5. You restore a customer database from three weeks ago; last week a valid erasure request was fulfilled. What must happen?

“BackupOps” is a new label on an old job, and it adds nothing the established practice lacks. What has changed is the threat: backups once guarded against failing disks and careless deletes; now attackers target them and the servers that run them, because a working restore lets a victim refuse to pay. The work moves from making copies to proving them: independent, locked, recent enough, and restored on a schedule by someone who records the result. GitLab and Code Spaces did not lack backup technology; they lacked a tested, independent copy and an owner whose job was to know it would work. UniSuper had copies outside the deleted service, and its story ended in recovery. Restore something important this week; it will teach you more than any number of green jobs.

Sources were opened on 2026-10-16, or on 2026-10-17 where marked; dates are the sources’ own. Re-check living product documentation before relying on it; the KEV count and the AWS, Velero and Microsoft rows change fastest (suggested review: quarterly).

Further reading: the whole SRE data-integrity chapter and PostgreSQL’s continuous archiving and point-in-time recovery.

Changelog. 2026-10-16: first edition (research, ledger, labs 1 and 2 executed, lab 3 not executed). 2026-10-17: revised after independent review r1 (Velero limits and privileged hooks, backup-system hardening, legal holds, attribution fixes; sources opened 2026-10-16). 2026-10-17: revised after the final QA score (Velero immutability as documented, S3 Object Lock route, replication and legal-hold preconditions, restore-time sizing, runbook figure scan step, wrapped code, merged comparison table, labs re-run). 2026-10-17, post-score polish: EBS fast snapshot restore added to restore sizing, GitLab follow-ups labelled as listed work, NIST CP-9(7) limited to deletion with the extension marked as the Atlas’s, synthetic-full trade-off worded as sourced, chain highlight drawn with thick arrows, long Velero sentences split.