StorageOps

StorageOps is an informal name for storage operations, usually called storage administration: choosing the right storage for each workload, provisioning it, keeping it fast, intact and large enough, and retiring it safely, for block volumes, shared file systems and object stores, on premises, in clouds and under Kubernetes. This page treats it as storage physics put to work: every workload has an access pattern, every medium and protocol has limits, and the job is to match them and prove it with measurements. Beginners can stop after this part.

Storage rarely fails loudly; it fails quietly, because a workload was never matched to its storage or a limit went unwatched.

• Slow without being broken. Driving more IOPS than a cloud volume has “can cause increased I/O latency”; a volume restored from a snapshot is slow until its blocks arrive.^[7][8]
• Full at the worst moment. A full LVM thin pool queues writes for up to 60 seconds and then fails them, which “can result in file system damage that requires repair”.^[9]
• Redundancy that runs out of room. In AWS’s 2011 EBS outage, mass re-mirroring exhausted free space (case 2).^[10]
• Silent damage. A study of 1.53 million drives found over 400,000 checksum mismatches and warned that RAID “may also be unable to detect these problems”.^[11]
• Costs hidden in rules. Colder tiers charge minimum durations and object sizes;^[12] Azure’s archive can take up to 15 hours to rehydrate.^[13]

With StorageOps, storage comes from priced classes with measured limits, growth is forecast against lead time, and redundancy is scrubbed and healthy.

Not backup. A replica copies deletions faithfully. AWS calls an EBS snapshot “an incremental backup”,^[14] yet any user whose IAM permissions allow it can delete an unlocked one;^[15] whether a snapshot is a backup depends on its independence, which BackupOps tests. Not disaster recovery: failover decisions and RTO/RPO targets live in DisasterRecoveryOps. Not protection from your own deletes: “eleven nines” (S3 lists 99.999999999% for most classes) is a design target against hardware loss.^[12]

Editorial distinction: StorageOps here owns the storage system and its mechanics, BackupOps the independent copies and restore evidence, DisasterRecoveryOps service recovery. No standard draws these lines.

The label has no history; the practice has turning points. A 1987 Berkeley report by Patterson, Gibson and Katz proposed “Redundant Arrays of Inexpensive Disks” and five RAID levels.^[16] Networked file sharing was already in use: when RFC 1094 documented Sun’s NFS in 1989, implementations existed “for a variety of machines”;^[17] it grew into today’s NFS 4.1/4.2, alongside SMB 2 and 3.^[18][19][20] Block storage moved onto dedicated networks: Fibre Channel SANs, then iSCSI over TCP (RFC 3720 in 2004, consolidated in RFC 7143 in 2014).^[21] Ceph and similar systems spread data across servers,^[22] cloud object stores became strongly consistent,^[23] and the Container Storage Interface let one storage plugin serve many container orchestrators.^[24]

Mental model: three numbers and a queue. Throughput = IOPS × I/O size. Little’s law says the average number of items in a queuing system equals their arrival rate times the time each spends there,^[29] so I/Os in flight = IOPS × latency: a device completing 3,000 I/Os per second with 30 in flight gives about 10 ms latency.

Four questions per workload (the Atlas’s checklist): shape (I/O size, randomness, read/write mix, concurrency), size (capacity and growth), safety (failures to survive, integrity checks) and spend (per TiB-month, per request, per retrieval, minimums included).

flowchart TB
  W["Workloads"] -->|"access pattern, growth"| S
  K["KubernetesOps"] -->|"volume claims"| S
  Y["SysOps"] <-->|"disks, mounts"| S
  C["CloudSecOps"] -.->|"guardrails"| S
  S["StorageOps: media, protection, protocols, performance, capacity"]
  S -->|"snapshots, replicas"| B["BackupOps: tested copies"]
  S -->|"replication behaviour"| R["DisasterRecoveryOps: service recovery"]
  S -->|"unit costs"| F["FinOps: spend decisions"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class S core
  class W,K,Y,C,B,R,F neutral

This charter is the Atlas’s proposal; no official StorageOps manifesto exists. Mission: every byte kept sits on storage chosen for its workload, with watched limits and a recorded retirement.

1. Shape before product. Know the access pattern before choosing medium, protocol or service.
2. Name every limit, and alert on approach, not arrival.
3. Count failure domains, not copies. Disk failures are correlated and rise with age;^[30] spread copies across hosts, racks or zones.
4. Verify integrity continuously with checksums and scrubs.^[11][31]
5. Headroom is capacity. A full pool cannot re-protect itself: Ceph refuses backfill (recovery that copies a placement group’s entire contents) to nearly full devices.^[32][33]
6. Destructive operations are slow, scoped and reversible, with floors and rate limits.^[34]
7. The price includes the rules: requests, retrievals, minimum durations and sizes.
8. Retire deliberately: sanitise owned media, and plan cryptographic erase for cloud storage, as the data’s sensitivity requires.^[35]

NIST SP 800-209 recommends at least the four ISO/IEC 27040 roles: security administrator; storage administrator, with “view” and “modify” rights over storage but none over security; security auditor; and storage auditor, with view-only rights over configuration and health logs.^[5] One stolen account then cannot both change storage and hide it. R = responsible, A = accountable, C = consulted, I = informed.

Day 0 designs the catalogue and telemetry, Day 1 provisions from it, and Day 2 runs everything after.

stateDiagram-v2
  direction TB
  [*] --> Requested: intake form
  Requested --> Provisioned: class chosen
  Provisioned --> InService: mounted or bound
  InService --> Changing: expand, retier, retune
  Changing --> InService: change verified
  InService --> Degraded: device or node lost
  Degraded --> Rebuilding: spare or new replica
  Rebuilding --> InService: redundancy restored
  Rebuilding --> Lost: failure beyond tolerance
  Lost --> Provisioned: restore from BackupOps
  InService --> Released: owner releases
  Released --> Retained: policy says keep
  Released --> Sanitised: sanitise or crypto-erase
  Retained --> Sanitised: retention ends
  Sanitised --> [*]: record kept

Exceptions are recorded with an owner, a reason and an expiry date.

The same stack exists on premises, in clouds and in clusters; only ownership changes. In a cloud the provider runs media and protection; you still own class, performance settings, capacity and lifecycle.

flowchart TB
  C["Control plane: APIs, CSI drivers, classes"]
  C -->|"provision, expand, retune"| P
  C -->|"policies"| D
  subgraph DP["Data path"]
    direction TB
    A["Consumers: VMs, pods, applications"] --> P["Presentation: block, file or object protocol"]
    P --> D["Data services: snapshots, replication, tiering"]
    D --> R["Protection: RAID, erasure coding, replicas"]
    R --> M["Media: SSD, HDD, tape"]
  end
  M -->|"health, errors"| T["Telemetry: latency, IOPS, capacity, scrubs"]
  P -->|"I/O statistics"| T
  T -.->|"alerts and forecasts feed back"| C
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  class A,P,D,R,M core
  class C decision
  class T neutral

Trust boundaries. One management call can delete a pool, so NIST SP 800-209 recommends separating data, management, replication and backup traffic and restricting which hosts see each device and each copy: “In many cases, a host granted access to a device should not be allowed to access a copy.”^[5]

Cloud types encode the same physics: st1 and sc1 are throughput-oriented HDD volumes;^[37] io2 Block Express targets average latency under 500 microseconds for 16 KiB I/O.^[38]

Parity costs writes. In the 1987 RAID report, a small write to a parity array “uses 2 disks to perform 4 accesses—2 reads and 2 writes”, because new parity = (old data XOR new data) XOR old parity.^[16] RAID 5 rotates the parity but keeps that cost.

Rebuilds are exposure. In a field study of about 100,000 drives, annual replacement rates of 2–4% were common, against datasheet figures implying at most 0.88%, and replacements were correlated in time.^[30] Linux md’s sync_speed_min trades application I/O for a shorter rebuild.^[39] Long rebuilds of large drives favour double parity or m ≥ 2 (the Atlas’s inference). Erasure coding has overhead (k+m)/k, 1.5 for 4+2 against 3.0 for three replicas; Ceph strongly discourages m=1 in production and wants at least k+m failure domains, preferably k+m+1.^[22]

Replication lag is exposure. Azure’s zone-redundant writes return “only after the data is written to all replicas across the three availability zones”; asynchronous geo-replication “might result in data loss”.^[40] S3 Replication Time Control copies “99.9 percent of those objects within 15 minutes”.^[41]

Snapshots have mechanics. A classic copy-on-write snapshot (device-mapper’s original target, used by lvcreate --snapshot -L <size>) copies old data into every snapshot’s COW area as the origin is overwritten, so each snapshot adds write work, and a full COW area disables its snapshot.^[42] Thin snapshots share blocks with their origin, and snapshots of snapshots no longer slow down with chain depth as the older chained tables did.^[43] Any block snapshot holds only what reached the volume; Ceph calls its own “merely crash-consistent” unless I/O is paused first (for example with fsfreeze).^[44] Application-consistent copies are BackupOps work. Volumes that must agree, such as a database’s data and log, need quiescing or a group snapshot, which CSI requires to be write-order consistent.^[24]

Integrity needs checksums. RAID cannot tell which of two disagreeing copies is right. An OpenZFS scrub “verifies each block’s checksum” and repairs redundant pools;^[31] Linux md’s check action counts mismatches in mismatch_cnt.^[39]

What the datasheet leaves out.

• Acknowledged is not durable: drives with volatile write caches report completion before data reaches non-volatile media, so fsync must flush them;^[45] Ceph expects drives with power-loss protection,^[46] and an NFS export with async can lose acknowledged writes in a server crash.^[47]
• Flash wears out: endurance is rated in drive writes per day (DWPD); Ceph finds 1 DWPD enough for most data drives, but not 0.3 DWPD under a block pool serving hundreds of VMs.^[46]
• Instance-store NVMe is ephemeral: its data “does not persist if the instance is stopped, hibernated, or terminated”.^[48]
• Data reduction is a forecast: plan on the ratio your data achieves (ZFS reports compressratio), not a quoted “effective” capacity; encrypted data barely reduces (the Atlas’s reasoning).^[49]

S3 scales to “at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix”, gradually, answering 503 “Slow Down” meanwhile,^[51] and is strongly consistent for PUT and DELETE.^[23]

• Zoning and masking: NIST asks for both, a default zone of “deny all”, and masking “as close to the data as possible”.^[5]
• Multipathing moves I/O to another path when a cable, switch or controller fails,^[26] and does it silently: alert on paths multipath -ll reports as faulty or shaky.^[52] If all paths fail, no_path_retry decides whether I/O fails at once (the upstream default; built-in array profiles may differ) or queues, and unlimited queueing “can lead to D state processes being hung and not killable”. On arrays with SCSI-3 ALUA (Asymmetric Logical Unit Access), path priorities come from the array.^[53]
• NAS exports and shares are file-side masking: list the clients, keep root squashed (the NFS default; NIST also asks for nosuid),^[5] and use sec=krb5p where the network is untrusted, because the default sys flavour has “no cryptographic security”.^[47] SMB 3 can encrypt traffic per share.^[54]

Example status: checked against exports(5); not executed (no NFS server on the research host; krb5p needs Kerberos).

# /etc/exports: one project share, read-write for one subnet, read-only for another.
# sync and root_squash are the defaults, spelled out so a reviewer sees them.
/srv/projects  10.20.0.0/24(rw,sync,root_squash,sec=krb5p) \
               10.20.1.0/24(ro,sync,root_squash,sec=krb5p)

Clients outside both subnets get no access. Apply with exportfs -ra.^[47]

Happy path.

1. The owner submits the intake form (templates).
2. The storage team maps it to a class or says why not.
3. Self-service provisioning applies encryption and tags.
4. A check confirms class limits and dashboards.
5. Alerts, forecasts, scrubs and lifecycle rules run.
6. Changes follow the class runbook.
7. On release: retention, then deletion, media sanitisation or key destruction, and a record.

Exception path. No class fits: measure the real need, then add a priced class or split the workload. A pool degrades: rebuild before any other change. Runway falls below lead time: escalate to the budget owner that day.

Rules of the road for cloud block storage (AWS EBS, as of October 2026; other clouds publish similar rules):

• IOPS count I/Os of up to 256 KiB (SSD) or 1,024 KiB (HDD); small sequential I/Os merge, random ones each count. Large I/O hits the throughput limit first: 1,000 IOPS of 256 KiB is already 250 MiB/s.^[7]
• Queue depth sets achievable IOPS. AWS advises an average queue of one per 1,000 provisioned IOPS (“for a volume provisioned with 3,000 IOPS, the queue depth average must be 3”);^[7] by Little’s law that is the queue for about 1 ms per I/O (the Atlas’s arithmetic: 3,000 × 0.001 = 3).
• gp3: 3,000 IOPS and 125 MiB/s baseline, up to 80,000 IOPS and 2,000 MiB/s at extra cost; gp2 bursts on credits, then slows.^[25] The instance’s bandwidth can also be the limit.^[7]

Read the right counters. /proc/diskstats counts I/Os, sectors (always 512 bytes), time spent and a weighted time that measures “both I/O completion time and the backlog”;^[55][56] iostat turns them into await (queue plus service time) and aqu-sz (queue length).^[57] Averages hide tails, which at scale “may come to dominate overall service performance”.^[58] On EBS, per-minute averages also hide micro-bursts, which VolumeIOPSExceededCheck and VolumeThroughputExceededCheck flag; on Nitro instances, detailed performance statistics add the latency histograms needed for a 99th percentile.^[7]

Benchmark honestly. Use fio with direct=1 and an asynchronous engine (iodepth above 1 does nothing on synchronous engines, and libaio needs direct=1 to be asynchronous).^[60] Google’s guide adds: fill the disk with non-zero data first, and measure latency below the limits, because a saturated disk “pushes back on incoming I/Os”.^[61] Never write-test a device holding data; fio refuses only if it appears mounted.^[60]

A pod claims storage with a PersistentVolumeClaim (PVC); a StorageClass tells a CSI driver how to create the PersistentVolume (PV) through calls such as CreateVolume, ControllerPublishVolume (attach) and NodePublishVolume (mount).^[24]

flowchart TB
  C["PVC created with a zonal class"] --> M{"volumeBindingMode?"}
  M -->|"Immediate"| I["Volume created at once, perhaps in zone b"]
  I --> X["Pod needs zone a: Pending, unschedulable"]
  M -->|"WaitForFirstConsumer"| W["PVC Pending: waiting for first consumer"]
  W --> P["Pod created; scheduler picks a node in zone a"]
  P --> V["CSI CreateVolume in zone a; PV bound to PVC"]
  V --> A["Attach: ControllerPublishVolume"]
  A --> N["Kubelet: NodeStageVolume, NodePublishVolume"]
  N --> R["Pod Running with the volume mounted"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class M decision
  class I,X risk
  class C,W,P,V,A,N,R core

Set these deliberately in every class (Kubernetes 1.37 is current in October 2026):^[62]

• Binding mode. Immediate, the default, provisions “without knowledge of the Pod’s scheduling requirements”; WaitForFirstConsumer waits for the pod.^[63]
• Reclaim policy. Dynamic volumes inherit the class policy, “which defaults to Delete”: deleting the claim deletes the disk.^[64]
• Access mode. ReadWriteOnce means one node; ReadWriteOncePod (RWOP, CSI only) means one pod; modes “do not enforce write protection once the storage has been mounted”.^[64]
• Expansion. Needs allowVolumeExpansion: true and XFS, ext3 or ext4; claims never shrink (FAQ).^[64]
• Snapshots. CRDs whose controller is “the responsibility of the Kubernetes distribution”; CSI drivers only.^[65]
• Mutable performance. A VolumeAttributesClass (GA since 1.34) moves a claim between settings such as IOPS and throughput when the driver implements ModifyVolume.^[66]
• StatefulSets. persistentVolumeClaimRetentionPolicy decides whether claims survive deletion or scale-down (retained by default).^[67]

None is mandatory in general; sector regulation may make some so.

Select by, in order: semantics and limits; failure behaviour; whether your team can run it at 03:00; unit cost; exit cost.

Automate provisioning from the catalogue (InfraOps covers infrastructure as code), expansion, lifecycle rules and waste reports. Automation that deletes, detaches, shrinks or removes capacity needs a scope limit, a rate limit, a floor and a dry run (case 1); keep humans on deleting pools or buckets, shortening retention, forcing detaches and touching last copies.

AI assistants can read logs and propose causes; the PyPI storageops package does this for S3-compatible storage, and its vendor notes that “Logs, errors and configs you give Pi go to the model provider you configure”.^[75] Scrub names, keys and customer identifiers first, and test suggestions against counters.

Assumptions: an organisation on AWS with one Kubernetes platform; five classes cover almost every request; FinOps sets prices.

Example status: syntax-checked (YAML parsed), illustrative; limits from AWS documentation as of October 2026.

# storage-catalogue.yaml: one entry per class; reviewed quarterly
classes:
  - name: block-fast
    for: "online transaction processing (OLTP) databases, latency-sensitive queues"
    backing: "EBS io2 Block Express"
    limits: {latency_target_ms: 1, iops_max_per_volume: 256000}
    protection: "provider durability 99.999%; snapshots hourly (not a backup)"
    k8s_storage_class: io2-fast
  - name: block-general
    for: "most VM and pod volumes"
    backing: "EBS gp3"
    limits: {iops_baseline: 3000, iops_max: 80000, mibps_baseline: 125, mibps_max: 2000}
    protection: "provider durability 99.8-99.9%; snapshots daily"
    k8s_storage_class: gp3-standard
  - name: block-throughput
    for: "large sequential scans, logs being processed"
    backing: "EBS st1"
    limits: {note: "throughput-oriented HDD; poor for random I/O"}
  - name: file-shared
    for: "home and project directories"
    backing: "NFS 4.1 service"
    rules: ["root_squash on every export", "no SMBv1, no NFSv2/3 on new shares"]
  - name: object-tiered
    for: "logs, media, data lake files"
    backing: "S3 with lifecycle rules logs-tier and logs-expire"
    rules:
      - "objects of 128 KiB or less stay in STANDARD and still expire"
      - "noncurrent versions expire after 30 days"
review:
  owner: storage-platform-team
  next_review: 2027-01-15

Object lifecycle rules. A rule’s filter scopes every action in the rule,^[76] so logs/ gets two rules. logs-tier moves objects larger than 128 KiB (131,072 bytes) to Standard-IA at 30 days and Glacier Flexible Retrieval at 120; its explicit size filter is in line with S3’s default since September 2024 and still holds if the configuration’s default is varies_by_storage_class, which lets small objects into Glacier.^[77][78] logs-expire, filtered by prefix only, expires objects at 400 days and old versions 30 days after they become noncurrent, and aborts uploads left incomplete for 7 days (their parts are billed until then).^[79] With the expiry actions in the size-filtered rule, objects of 128 KiB or less and their old versions would never expire.

Example status: JSON validated against botocore’s current S3 API model on 2026-10-18; not executed against AWS.

{
  "Rules": [
    {
      "ID": "logs-tier",
      "Status": "Enabled",
      "Filter": {"And": {"Prefix": "logs/", "ObjectSizeGreaterThan": 131072}},
      "Transitions": [
        {"Days": 30, "StorageClass": "STANDARD_IA"},
        {"Days": 120, "StorageClass": "GLACIER"}
      ]
    },
    {
      "ID": "logs-expire",
      "Status": "Enabled",
      "Filter": {"Prefix": "logs/"},
      "Expiration": {"Days": 400},
      "NoncurrentVersionExpiration": {"NoncurrentDays": 30},
      "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
    }
  ]
}

Example status: syntax-checked, not executed against AWS.

bucket=example-logs-bucket
# 1. Save the current rules, if any.
aws s3api get-bucket-lifecycle-configuration --bucket "$bucket" > lifecycle-before.json
# 2. Merge them into lifecycle.json; a second person reviews the merged file.
# 3. Apply, then read back.
aws s3api put-bucket-lifecycle-configuration --bucket "$bucket" \
  --lifecycle-configuration file://lifecycle.json
aws s3api get-bucket-lifecycle-configuration --bucket "$bucket"

Capacity runway. A straight-line fit compared with lead time, as Google’s SRE book requires: a forecast “which extends beyond the lead time required for acquiring capacity”.^[68]

Example status: verified by execution on 2026-10-18 (Python 3.8) with a generated 90-day illustrative series and with one-row, blank-line and same-date inputs.

#!/usr/bin/env python3
"""Capacity runway: fit daily usage, compare runway with lead time.

Input CSV lines: YYYY-MM-DD,used_tib (one per day, oldest first)
Usage: python3 runway.py usage.csv USABLE_TIB LEAD_TIME_DAYS [threshold]
"""
import csv
import sys
from datetime import date


def load(path):
    rows = []
    with open(path, newline="") as f:
        for rec in csv.reader(f):
            if not rec or not rec[0].strip():
                continue  # tolerate blank lines
            rows.append((date.fromisoformat(rec[0].strip()), float(rec[1])))
    if len({d for d, _ in rows}) < 2:
        sys.exit("need rows on at least two different dates to fit a trend")
    return rows


def fit(xs, ys):
    n = len(xs)
    mx, my = sum(xs) / n, sum(ys) / n
    sxx = sum((x - mx) ** 2 for x in xs)
    slope = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sxx
    return slope, my - slope * mx


def main():
    path, usable, lead = sys.argv[1], float(sys.argv[2]), int(sys.argv[3])
    threshold = float(sys.argv[4]) if len(sys.argv) > 4 else 0.80
    rows = load(path)
    xs = [(d - rows[0][0]).days for d, _ in rows]
    ys = [u for _, u in rows]
    slope, _ = fit(xs, ys)
    used, limit = ys[-1], usable * threshold
    print(f"used {used:.1f} TiB of {usable:.1f} TiB usable ({used / usable:.0%})")
    print(f"growth {slope:.3f} TiB/day over {len(rows)} days")
    if slope <= 0:
        print("no growth trend: review again next cycle")
        return
    runway = (limit - used) / slope
    print(f"runway to {threshold:.0%}: {runway:.0f} days; lead time {lead} days")
    if runway <= lead:
        print("ACT NOW: runway is shorter than lead time")
    elif runway <= 2 * lead:
        print("ORDER: start procurement or expansion this cycle")
    else:
        print("OK: re-forecast next cycle")


if __name__ == "__main__":
    main()

On 90 days growing from about 300 to 353 TiB on a 500 TiB pool with a 60-day lead time it printed runway to 80%: 78 days; lead time 60 days and ORDER: start procurement or expansion this cycle. Add known one-off demand by hand.^[68]

Growth touches four layers: device, partition, volume manager, filesystem. On a SAN the device step has three parts: the LUN on the array, each path, and the multipath map.^[52] SysOps covers the common EBS-plus-partition case; this is the full chain with LVM.

Example status: syntax-checked, not executed (needs root on a disposable VM). Names are placeholders.

# 0. Snapshot first (cloud) or confirm a tested backup exists (BackupOps).
# 1. Grow the device: cloud console/API, or the array; then confirm the kernel sees it.
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT /dev/nvme1n1
# 1b. SAN LUN on multipath: rescan every path, then resize the map, and use the
#     /dev/mapper device in place of the NVMe device and partition below.
for p in sdc sdd; do echo 1 | sudo tee "/sys/block/$p/device/rescan"; done
sudo multipathd resize map mpatha
sudo multipath -ll
# 2. Grow the partition if the PV sits on one (growpart needs some free space in /tmp).
sudo growpart /dev/nvme1n1 1
# 3. Tell LVM the physical volume is bigger, then grow the LV and its filesystem together.
sudo pvresize /dev/nvme1n1p1
sudo lvextend -r -L +200G /dev/vg_data/lv_pg
# 4. Verify at every layer.
sudo pvs; sudo vgs; sudo lvs vg_data; df -hT /var/lib/pgsql

lvextend -r grows the filesystem with the logical volume through fsadm;^[81] pvresize works on a physical volume that already holds active logical volumes.^[82] Without LVM, grow XFS with xfs_growfs <mount point> (mounted) and ext4 with resize2fs <device> (online growth supported).^[72][83][84] Sizes are one-way (FAQ). With active/passive multipath arrays, also filter the underlying path disks out of lvm.conf, then rebuild the initrd (update-initramfs -u -k all on Ubuntu).^[26]

Objective: compute IOPS, throughput, I/O size, latency, queue and utilisation from /proc/diskstats and check Little’s law. Prerequisites: any Linux host with Python 3.6+; no privileges; 10 minutes. Cost and risk: none; read-only.

Example status: verified by execution on 2026-10-18 (Ubuntu, kernel 5.15, Python 3.8, unprivileged).

#!/usr/bin/env python3
"""Sample /proc/diskstats twice and print the three numbers plus the queue.

Usage: python3 diskstats_sample.py [seconds] [device ...]
Read-only; needs no privileges. Field numbers follow the kernel's iostats.rst.
"""
import sys
import time


def read_stats():
    stats = {}
    with open("/proc/diskstats") as f:
        for line in f:
            p = line.split()
            stats[p[2]] = [int(x) for x in p[3:14]]  # fields 1..11
    return stats


def main():
    secs = float(sys.argv[1]) if len(sys.argv) > 1 else 5.0
    wanted = sys.argv[2:]
    a = read_stats()
    t0 = time.monotonic()
    time.sleep(secs)
    b = read_stats()
    ms = (time.monotonic() - t0) * 1000.0
    print(f"{'device':<12}{'r/s':>8}{'w/s':>8}{'MiB/s':>8}{'avgKiB':>8}"
          f"{'await':>8}{'aqu-sz':>8}{'util%':>7}{'L=XW':>7}")
    for dev in sorted(b):
        if (wanted and dev not in wanted) or dev.startswith(("loop", "ram")):
            continue
        d = [y - x for x, y in zip(a.get(dev, b[dev]), b[dev])]
        reads, sect_r, ms_r, writes, sect_w, ms_w = d[0], d[2], d[3], d[4], d[6], d[7]
        io_ms, weighted_ms = d[9], d[10]
        ios = reads + writes
        if ios == 0:
            continue
        iops = ios / (ms / 1000.0)
        mib_s = (sect_r + sect_w) * 512 / 1048576 / (ms / 1000.0)  # 512-byte units
        avg_kib = (sect_r + sect_w) * 512 / 1024 / ios
        await_ms = (ms_r + ms_w) / ios
        aqu = weighted_ms / ms               # average I/Os in flight
        util = 100.0 * io_ms / ms
        little = iops * await_ms / 1000      # Little's law: L = X * W
        print(f"{dev:<12}{reads / (ms / 1000):>8.1f}{writes / (ms / 1000):>8.1f}"
              f"{mib_s:>8.2f}{avg_kib:>8.1f}{await_ms:>8.2f}{aqu:>8.2f}"
              f"{util:>7.1f}{little:>7.2f}")


if __name__ == "__main__":
    main()

1. Save as diskstats_sample.py; run python3 diskstats_sample.py 5.
2. Repeat during busy work, or name a device: python3 diskstats_sample.py 10 nvme0n1.

Expected output (shape): one line per active device and partition; on the research host under light load nvme0n1 1.8 15.0 0.30 18.2 0.99 0.02 3.1 0.02. Validation: aqu-sz and L=XW agree to rounding; they come from different counters,^[55] so agreement shows the law at work. Troubleshooting: no line → no I/O completed; lengthen the window. Counters reset at boot or re-attach.^[55] util% near 100 with low latency is normal on parallel devices.^[57] Teardown: none.

Objective: 4 KiB random reads at queue depths 1, 4, 16 and 64: watch IOPS level off while latency rises. Prerequisites: a disposable Linux VM with 6 GiB free and fio; 20 minutes. Cost and risk: uses provisioned IOPS and may slow neighbours on shared storage; writes only its own file.

Example status: syntax-checked, not executed (fio was not available on the research host).

#!/usr/bin/env bash
# lab2.sh: random-read IOPS and latency versus queue depth on a test file.
set -euo pipefail
dir=${1:?usage: lab2.sh /path/on/test/filesystem}
file="$dir/fio-lab2.dat"
fio --name=fill --filename="$file" --size=4G --rw=write --bs=1M \
    --direct=1 --ioengine=libaio --iodepth=8 >/dev/null   # fill with data first
for qd in 1 4 16 64; do
  fio --name="qd$qd" --filename="$file" --size=4G --rw=randread --bs=4k \
      --direct=1 --ioengine=libaio --iodepth="$qd" --runtime=30 --time_based \
      --ramp_time=5 --group_reporting --output-format=json > "qd$qd.json"
  python3 - "qd$qd.json" "$qd" <<'PY'
import json, sys
job = json.load(open(sys.argv[1]))["jobs"][0]["read"]
lat_ms = job["clat_ns"]["mean"] / 1e6
p99_ms = job["clat_ns"]["percentile"]["99.000000"] / 1e6
print(f"qd={sys.argv[2]:>3} iops={job['iops']:>9.0f} mean_ms={lat_ms:6.2f} "
      f"p99_ms={p99_ms:6.2f} little_qd={job['iops'] * lat_ms / 1000:5.1f}")
PY
done
rm -f "$file"

Expected output (shape): IOPS rise, then flatten while mean and p99 latency climb; on a 3,000-IOPS gp3 volume IOPS cap near 3,000 beyond a depth of about 3. Validation: little_qd ≈ qd; the flattening point matches the documented limit.^[25] Troubleshooting: the same IOPS at every depth means a synchronous engine or buffered I/O; check --ioengine=libaio --direct=1.^[60] A missing percentile key means an older fio; read clat_ns by hand. Teardown: the script deletes its file; delete the JSON files and the VM.

Objective: a zonal gp3 class that waits for the first consumer; write data, snapshot, restore into a new claim, expand online. Prerequisites: a disposable EKS cluster (Kubernetes 1.35–1.37, not Auto Mode) with the EBS CSI driver ebs.csi.aws.com, snapshot CRDs and controller, and an IAM role for the driver’s controller (AWS suggests EKS Pod Identity; managed policy or a narrower one);^[65][85] 30 minutes. Cost and risk: two small volumes and a snapshot for an hour. Tip: the driver’s encrypted default is false; EBS encryption by default, set per Region, encrypts every new volume regardless.^[86][87]

Example status: YAML and bash syntax-checked, not executed (no cluster). Parameter names from the EBS CSI driver documentation (release v1.67.0).^[86]

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-lab
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "4000"
  throughput: "250"
  encrypted: "true"
reclaimPolicy: Delete           # lab only; production data classes often use Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-lab
driver: ebs.csi.aws.com
deletionPolicy: Delete
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: storage-lab
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: gp3-lab
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: writer
  namespace: storage-lab
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "date > /data/hello.txt && sleep 3600"]
      volumeMounts:
        - {name: data, mountPath: /data}
  volumes:
    - name: data
      persistentVolumeClaim: {claimName: data}

kubectl create namespace storage-lab
kubectl apply -f lab3.yaml
kubectl -n storage-lab get pvc data -w   # Pending, then Bound; Ctrl-C to stop watching
kubectl -n storage-lab exec writer -- cat /data/hello.txt
cat <<'YAML' | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata: {name: data-snap, namespace: storage-lab}
spec:
  volumeSnapshotClassName: ebs-lab
  source: {persistentVolumeClaimName: data}
YAML
kubectl -n storage-lab get volumesnapshot data-snap -w   # wait for READYTOUSE true
cat <<'YAML' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: data-restored, namespace: storage-lab}
spec:
  storageClassName: gp3-lab
  accessModes: ["ReadWriteOnce"]
  resources: {requests: {storage: 10Gi}}
  dataSource: {name: data-snap, kind: VolumeSnapshot, apiGroup: snapshot.storage.k8s.io}
---
apiVersion: v1
kind: Pod
metadata: {name: reader, namespace: storage-lab}
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "cat /data/hello.txt && sleep 3600"]
      volumeMounts: [{name: data, mountPath: /data}]
  volumes: [{name: data, persistentVolumeClaim: {claimName: data-restored}}]
YAML
kubectl -n storage-lab wait --for=condition=Ready pod/reader --timeout=300s
kubectl -n storage-lab logs reader   # the date the writer pod stored
kubectl -n storage-lab patch pvc data \
  -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
until [ "$(kubectl -n storage-lab get pvc data \
    -o jsonpath='{.status.capacity.storage}')" = "20Gi" ]; do sleep 10; done
kubectl -n storage-lab exec writer -- df -h /data

Expected output (shape): data is Pending (“waiting for first consumer to be created before binding”) until the pod exists, then Bound;^[88] cat prints a date; the snapshot becomes READYTOUSE true; the resize is asynchronous, so the loop waits until capacity reads 20Gi, and df then shows about 20G. Validation: reader prints the same date as writer (data-restored binds only once reader exists; EBS restores start slow).^[8] Troubleshooting: snapshot never ready → controller or CRDs missing (kubectl get crd | grep snapshot); ProvisioningFailed → driver permissions or parameters; no expansion → allowVolumeExpansion missing or unsupported filesystem.^[64] Teardown: kubectl delete namespace storage-lab, then the cluster-scoped classes: kubectl delete storageclass gp3-lab and kubectl delete volumesnapshotclass ebs-lab. Confirm with kubectl get pv and the console that volumes and snapshot are gone.

1. Capacity removal that went too far (sourced: AWS S3, 2017). An engineer “using an established playbook” mistyped an input and removed servers behind the index subsystem, which serves every GET, LIST, PUT and DELETE in the region; it needed “a full restart”, and PUT returned at 13:54 PST, more than four hours after the 09:37 removal. AWS made the tool remove capacity more slowly and refuse to take “any subsystem below its minimum required capacity level”.^[34] Success criterion: every destructive tool has a floor, a rate limit and a tested refusal.

2. Replication without headroom (sourced: Amazon EBS, 2011). Nodes that lost their replicas all searched for space to re-mirror, exhausted free capacity and “did not back off aggressively enough”; about 13% of the zone’s volumes stuck, and 0.07% “could not be restored for customers in a consistent state”.^[10] Trade-off: self-healing that suits one failure swamps a cluster in a thousand. Success criterion: a per-pool headroom target (re-protect the largest failure domain) and recovery traffic that backs off.

3. Shared scratch for supercomputing (sourced: NERSC Perlmutter). A 44 PB all-flash Lustre scratch can purge files unread for eight weeks once it passes a fullness threshold, stripes files over one storage target by default (most are small) and lets users stripe large files wider before writing;^[89][73] its tape archive wants small files bundled.^[36] Lesson: the policy is part of the design (ResearchDataOps covers what researchers keep).

4. A colder class that raised the bill (illustrative). An application wrote its 50 KB log objects straight to Standard-IA, where each is billed as 128 KB:^[12] billed bytes rose 2.56-fold. Bundling first fixed it (break-even).

Runway, headroom and subscription ratio lead; incidents from full or slow storage lag. Avoid vanity numbers: raw petabytes, average latency across all volumes, and %util on SSDs.^[57]

Worked calculation (the Atlas’s arithmetic). A Ceph cluster has 12 hosts × 10 drives × 20 TB = 2,400 TB ≈ 2,183 TiB raw. With erasure coding 4+2, overhead 1.5,^[22] usable ≈ 1,455 TiB (three replicas: ≈ 728 TiB). If a host fails, the 11 survivors must absorb its data, and by default Ceph refuses backfill to a device above 90% full and stops client I/O at 95%.^[32] So re-protection needs used ≤ 11/12 × 0.90 ≈ 82.5% before any imbalance between devices, and below about 78% if the 85% nearfull warning should stay quiet. An 80% threshold gives plannable ≈ 1,164 TiB with a thin margin. With 1,020 TiB used and 1.2 TiB/day growth, runway = (1,164 − 1,020) ÷ 1.2 = 120 days; with a 90-day lead time, order within 30 days.

NIST SP 800-209 is a detailed public checklist for storage security;^[5] the quarterly review applies its essentials, plus separate traffic networks (trust boundaries) and iSCSI authentication, which “does not provide encryption or integrity protection to the channel”. Encrypt at rest by default and leave key custody with security: a lost key is lost data. New S3 buckets block public access by default;^[90] exposure guardrails belong to CloudSecOps. Erasure requests reach storage as deletes, versions and snapshots; policy belongs to PrivacyOps.

Retiring storage. Owned media leaving service are cleared, purged or destroyed (NIST SP 800-88 Rev. 2), with each device’s method and result recorded. Cloud volumes and buckets differ: the data owner “has no direct access to the physical ISM” (information storage media), so for such logical storage “cryptographic erase may be the only viable purge sanitization technique option”.^[35] Decide before sensitive data arrives: encrypt from the first write under a customer-managed key whose every copy you can destroy; at retirement delete volumes, snapshots and object versions, destroy the key where the data’s sensitivity requires it. NIST warns against trusting cryptographic erase where data or keys were backed up or escrowed without firm knowledge of where the keys are.^[35]

Cost drivers: capacity after protection overhead, provisioned performance, requests, retrievals, transfer, and rules:

Break-even for small objects (the Atlas’s arithmetic). Let p < 1 be the colder class’s price per GB relative to the hot class. An object of s KB billed as 128 KB saves storage only if p × 128 < s: at p = 0.5 it must exceed 64 KB, before transition and retrieval charges. AWS’s stated reason for its default since September 2024 (no transitions under 128 KB) is the per-object transition charge: “the transition costs can outweigh the storage savings”.^[77] The billable minimum is a second.

Capacity. Derive alerts from runway: a 70% alert on a pool growing two points a week leaves about six weeks before an 82.5% re-protection ceiling, less than an eight-week lead time.

Sustainability. Storage draws power around the clock, so the cost levers also save energy: delete what nobody needs, tier cold data to denser media, prefer erasure coding to triple replication where performance allows; GreenOps measures.

Track catalogue conformance, forecast accuracy, drills (fail a disk or node in a test pool quarterly and time the rebuild; AWS Fault Injection Service can pause I/O or inject latency on EBS to test alerts),^[7] change records for destructive operations, and sanitisation records matched to inventory.

Trigger: an alert or a user report about one volume. Rule zero: delete, detach or rebuild nothing until you have named the layer at fault.

flowchart TB
  Q{"What is wrong?"} -->|"slow"| S["Measure the three numbers; compare with class limits"]
  Q -->|"full"| F["Find the full layer; grow bottom-up; thin pool first"]
  Q -->|"will not attach"| A1["Read pod events and VolumeAttachments"]
  A1 --> A2{"Old node state?"}
  A2 -->|"alive"| A3["Stop the old pod properly; wait for detach"]
  A2 -->|"confirmed off"| A4["Taint out-of-service; volume detaches"]
  A2 -->|"unknown"| A5["Escalate; never force a detach"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  class Q,A2 decision
  class A5 risk
  class S,F,A1,A3,A4 core

Slow. (1) Measure on the host (Lab 1 or iostat -x 1) and in provider metrics for the same minutes. (2) Compare with class limits: IOPS-bound or throughput-bound?^[7] (3) Neither: check instance bandwidth, snapshot initialisation,^[8] gp2 credits,^[25] rebuilds, and VolumeStalledIOCheck, which flags a volume that cannot complete I/O.^[7] (4) Fix the named limit and verify latency.

Full. (1) df -hT and df -i (SysOps covers inodes and deleted-but-open files). (2) For thin pools, lvs -o name,data_percent,metadata_percent; at 100%, writes queue for 60 seconds by default and then fail,^[9] so extend the pool first (lvextend -L +<size> <vg>/<pool>), then check filesystems. (3) Grow layers bottom-up (grow a volume end to end); EBS allows four modifications per volume in 24 hours.^[59]

Will not attach (Kubernetes). (1) kubectl describe pod: for a volume attached elsewhere, FailedAttachVolume starts “Multi-Attach error” up to Kubernetes 1.36 and “Waiting for detach” from 1.37;^[91][92] FailedMount points at the node side.^[93] (2) kubectl get volumeattachment names the holding node; check its CSI driver logs and attach limit (kubectl get csinode).^[94] (3) Old node healthy: drain or delete the old pod properly and wait. (4) Node confirmed shut down: verify first that it “is already in shutdown or power off state”,^[95] then add the out-of-service taint with the NoExecute effect (below); pods are force-deleted and volumes detached at once.^[96] Remove the taint once the node has recovered (a trailing - removes it).^[97] (5) Unknown: see the judgement call.

Example status: syntax-checked, not executed (no cluster).

node=NODE_NAME   # only a node confirmed powered off
kubectl taint nodes "$node" node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
# after the node has recovered and its pods run elsewhere:
kubectl taint nodes "$node" node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-

Afterwards: record the layer at fault and add the alert that would have caught it.

• One class for everything: some workloads slow, others overpaid. Build a small catalogue on access patterns.
• RAID 5 under write-heavy databases: four disk operations per small write.^[16] Use mirrors or SSD classes.
• Tiering by age alone: small or short-lived objects can cost more in colder classes.
• Destructive tools without floors: a typo should not remove most of a system’s capacity.^[34]
• Myth: “A million-hour MTTF means disks rarely fail.” Field rates ran several times the datasheet figure (rebuilds are exposure).^[30]

A named storage practice pays off with several storage systems; latency-sensitive stateful workloads; growth where lead time matters; self-run storage where you own rebuilds; Kubernetes platforms serving many teams; or regulated data that must be sanitised and audited.

When not to: a small team on managed services with modest data can keep storage as a checklist inside CloudOps or SysOps: the provider’s general class, encryption, lifecycle rules, two alerts. Do not self-run software-defined storage to save money unless you can staff its 03:00 failures. If the problem is lost data, start with BackupOps.

flowchart TB
  S["New workload"] --> Q4{"`Can the team run storage
  at 03:00, at a scale that pays?`"}
  Q4 -->|"no"| M["Managed services only"]
  Q4 -->|"yes"| SR["`Self-run allowed:
  Ceph, ZFS, arrays;
  you own failure domains,
  headroom and rebuilds`"]
  M --> Q1
  SR --> Q1{"Whole objects fetched by key?"}
  Q1 -->|"yes"| O["`Object storage: cold classes
  only for objects of 128 KB or more`"]
  Q1 -->|"no"| Q2{"Files shared by many hosts?"}
  Q2 -->|"yes"| F["File storage: NFS or SMB"]
  Q2 -->|"no"| Q3{"`Small random I/O,
  latency-critical?`"}
  Q3 -->|"yes"| B1["Block: provisioned-IOPS SSD"]
  Q3 -->|"no"| Q5{"`Only large
  sequential streams?`"}
  Q5 -->|"yes"| B2["Block: throughput HDD"]
  Q5 -->|"no"| B3["`Block: general-purpose SSD,
  the default`"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class Q1,Q2,Q3,Q4,Q5 decision
  class S,O,F,B1,B2,B3,M core
  class SR risk

AWS recommends its general-purpose SSD volumes “for most workloads”, and its HDD types “deliver optimal performance only when I/O operations are large and sequential”.^[25][7] Self-running pays only if its lower cost per usable TiB survives the staff, hardware cycles and lock-in.

This maturity model is the Atlas’s proposal; there is no official StorageOps maturity model.

Quick wins: delete unattached volumes and orphaned snapshots after sign-off, abort incomplete uploads, move steady gp2 volumes to gp3 (AWS cites prices up to 20% lower).^[25]

New class readiness

• Access patterns served and refused
• Limits from the provider or a benchmark on filled data
• Failure domains, re-protection time, headroom target
• Encryption, roles, audit logging
• Snapshot and replication policy linked to the backup policy
• Unit price and lifecycle rules agreed with FinOps
• Kubernetes: binding mode, reclaim policy, expansion, snapshot class

Before a destructive change

• Scope checked by a second person; dry run done
• Floor and rate limit set; independent copy confirmed

Monthly review

• Runway vs lead time; headroom and subscription ratios
• Scrub completion and errors; degraded hours
• Waste report; last month’s forecast error

Decommissioning

• Owner released the data; retention met
• Owned media: clear, purge or destroy recorded per device (NIST SP 800-88)
• Cloud: volumes, snapshots and versions deleted; key destroyed where required; recorded per volume or bucket

Quarterly security review

• Storage administrator and auditor roles held by different people; leavers removed
• Zoning, masking and export lists match the inventory; default zone denies all
• No SMB 1 or NFS 1 and 2; root squashed; management interfaces on an admin network
• Audit logs reach the central store; key custody and deletion rights reviewed

Storage request (intake form).

# storage-request.yaml
requester: team-payments
workload: "ledger database (PostgreSQL 17)"
interface: block            # block | file | object
access_pattern:
  io_size_kib: 8-16
  random_or_sequential: random
  read_write_ratio: "70:30"
  peak_iops: 9000
  peak_mibps: 150
  latency_target_ms_p99: 5
size:
  initial_gib: 800
  growth_gib_per_month: 60
safety:
  survive: ["disk", "node", "availability zone (via replica)"]
  snapshot: "hourly, keep 48"
  backup_policy_ref: "BKP-PAY-01"      # BackupOps owns this
sensitivity: confidential
retention_after_release_days: 30
cost_centre: CC-1234

Class card (one per catalogue entry): name; workloads served and refused; backing system; limits; protection and failure domains; snapshot and replication defaults; headroom target; alerts; unit price; owner; next review.

Capacity review record.

Pool: ceph-rgw-prod-1    Date: 2026-10-18    Reviewer: <name>
Usable TiB: 1,455    Used TiB: 1,020    Threshold: 80%    Re-protect ceiling: 82%
Growth (90 d): 1.2 TiB/day    Runway to threshold: 120 days    Lead time: 90 days
Known inorganic demand: archive migration, +80 TiB in December
Decision: order 2 hosts by 2026-11-15    Owner: <name>    Next review: 2026-11-18

Destructive change record: change ID; exact scope; command and dry-run output; floor checked; rate limit; second approver; recovery path; window; result.

Is StorageOps a job title? Not in our searches. The work sits with storage, infrastructure or platform engineers; NIST SP 800-209 names storage administrator and auditor roles.^[5]

Is a snapshot a backup? Only if independent of its source (credentials, account, system) and proven by a restore; BackupOps sets the test.

Why do I get fewer IOPS than provisioned? Too few I/Os in flight, I/O so large that throughput binds first, an instance limit, or a volume still initialising from a snapshot.^[7][8]

Can I shrink a volume? Generally not in place: EBS volumes cannot shrink, XFS shrinks only in a limited way, ext4 only offline, and Kubernetes does not shrink claims. Copy to a smaller volume.^{[59][72][83][64]}

Should databases use Kubernetes volumes? They can, with the right class (zonal binding, Retain, tested expansion and snapshots) and a team that understands detach on node failure; DBOps weighs managed databases.

• Denser drives: shingled (SMR) drives raise capacity but care about workload shape, which in Backblaze’s words makes qualification less “swap the drive” and more “plan the project”; HAMR raises density another way, and the two can be combined.^[27]
• Changed-block metadata: CSI defines group snapshots and an alpha service reporting the blocks changed between snapshots.^[24]

“Self-driving” storage claims are marketing until they publish failure behaviour.

• Prerequisites: Linux filesystems and mounts (SysOps); TCP/IP basics.
• One day: this page to the decision framework; Lab 1; your cloud’s volume-type page.
• One week, on a disposable VM: Lab 2; fill an LVM thin pool on a loop device; one class card.
• Thirty days: Lab 3; a draft catalogue; one pool’s forecast; one past incident replayed against the runbook.
• Specialise: Ceph and Rook, HPC filesystems, storage security or CSI drivers. Portfolio: benchmark three classes on filled disks.

Situation (illustrative). At 02:10 a node hosting a PostgreSQL primary stops reporting. A replica has been promoted, so the service is up with one replica fewer. The old pod is stuck Terminating; its replacement waits on FailedAttachVolume because the EBS volume is still attached to the old node, which the console shows as “running” but unreachable.

Assumptions: service already failed over, so minutes cost redundancy, not availability; cloud API access exists; no evidence the node is off.

Decision: B. A is fast, but its safety rests on an unchecked precondition, and “running but unreachable” is exactly when the old pod may still write; stopping the instance turns an unknown into a known. Also decide whether to keep the 6-minute forced detach (disable-force-detach-on-timeout turns it off, at the price of manual recovery).^[95] Uncertainty: had the service not failed over, faster action would be justified, still after confirming the node is off.

After this page you can explain what StorageOps covers and why the label is informal; calculate throughput, queue and runway; choose an interface and class; design a Kubernetes class; run the slow, full and attach runbook; decide when not to self-run storage.

1. A volume shows 1,000 IOPS, 250 MiB/s and rising latency; its limits are 3,000 IOPS and 250 MiB/s. Why will more IOPS not help?

2. An application keeps 32 reads in flight on a device that completes at most 8,000 per second. Expected latency? And at 16?

3. A team wants erasure coding 4+1 on five Ceph hosts. What do you say?

4. A pool is 70% full, grows 1.5 points a week, and expansion takes ten weeks. The alert is at 85%. Enough?

5. You must move 50 KB objects to a colder S3 class priced at 40% of Standard per GB. Do you save on storage?

StorageOps is an old job under a rarely used name, and its substance has not changed since the RAID report: disks are slow, they fail, and they fail together. What has changed is where the physics hides: in a cloud volume’s published limit, a cold tier’s minimum duration, a StorageClass field. The durable skill is to read those numbers as physics: measure the shape of the work, name the limit it meets, keep room to heal, and make every destructive action slow enough for a second thought.

Re-check quarterly: EBS limits and modification rules, S3 lifecycle defaults, Kubernetes event text and feature states, and every named version.

Changelog

• 2026-10-18: first version (Lab 1 and the runway script executed), condensed the same day to the length band.
• 2026-10-18 (review r1): Ceph re-protection ceiling corrected (about 82%, was 92%); general-purpose SSD default in the decision tree; lifecycle overwrite and expiry warning; SAN and NAS operations; cloud crypto-erase; minor fixes; 17 sources added.
• 2026-10-18 (score r2): lifecycle example split so small objects also expire; snapshot mechanics and consistency; NFS export example; ALUA, backfill and failure domain glossed; duplicates cut; re-check list.
• 2026-10-18, post-score polish: S3 expiry wording; initrd step; snapshot-lock source; Figure 6 self-run note; two lists.