SysOps

SysOps (systems operations) is the hands-on work of keeping servers healthy: supervising services, managing users and privileges, reading logs, scheduling jobs, installing packages, keeping filesystems and clocks in order, and finding out why a machine is slow or down. Most people who do this work are called system administrators; “SysOps” is an informal name for the same job, and was the name of an AWS certification until 2025.

It is for beginners meeting servers, on-call engineers who need a dependable diagnostic routine, and leaders deciding how much host work their platform still needs.

Every abstraction runs on a host: containers share a kernel, and a Kubernetes node is a Linux server with agents. When an abstraction leaks (a node turns NotReady, a VM stops answering, a batch job runs three times slower), someone must read the host; teams without that skill restart and hope.

Host work by hand is expensive too. Google’s SRE book says the traditional “sysadmin approach” gets costlier as a service grows, “because the size of the team necessarily scales with the load generated by the system”.^[1] The problem is not host knowledge but host work without method or automation.

• Not a defined discipline. No standards body, foundation or manifesto defines “SysOps”. The work goes by its established name: the US occupation “Network and Computer Systems Administrators” (O*NET 15-1244.00) includes reviewing “system and application logs” and verifying “completion of scheduled jobs, including system backups”,^[2] and LFCS^[3] and RHCSA^[4] certify system administrators.
• Not the other “sysop”. “Sysop” (system operator) also names a bulletin-board or online-community administrator,^[5] MediaWiki’s administrator group,^[6] and, for TechTarget, a role in which “a sysop monitors consoles” and usually reports to the sysadmin.^[7]
• Not CloudOps. AWS renamed SysOps Administrator – Associate to CloudOps Engineer – Associate “to reflect the evolving nature of cloud operations and a shift in industry terminology”; the last SOA-C02 exam was on 2025-09-29.^[8][9] The guest OS belongs here; the provider side to CloudOps.
• Not hosts as code. Terraform, golden images and drift detection belong to InfraOps; this page covers what an image must contain and how to diagnose its hosts.

System administration was a profession long before “-Ops” labels: USENIX’s LISA (Large Installation System Administration) conference ran for 35 years until, as Tom Limoncelli wrote in November 2022, “The Usenix LISA conference is no more”.^[10] His co-written textbook, The Practice of System and Network Administration, is in its third edition.^[11]

Three shifts changed the job without removing it.

1. SRE. Google’s SRE book (2016) defines its approach against the traditional sysadmin model.^[1]
2. Pets and cattle. Randy Bias, crediting a Bill Baker talk, contrasts pets, “indispensable or unique systems that can never be down”, with cattle, servers “built using automated tools” and “designed for failure”.^[12]
3. IaaS. In the cloud the customer still owns “the guest operating system (including updates and security patches)”.^[13]

The tools moved too: systemd’s journal and timers, pressure stall information (PSI, 2018),^[14] agent-based access,^[15] and OpenSSH installed by default from Windows Server 2025 (its sshd service must still be enabled).^[16]

Mental model. A host is a set of finite resources (CPUs, memory, storage capacity and I/O, network, and software limits such as file descriptors) shared by services under a supervisor. Diagnosis finds which resource is saturated or failing, and who is using it. Operation keeps baseline, headroom and evidence in place so diagnosis is rarely needed.

Hosts hand telemetry to ObservabilityOps, wider incidents to IncidentOps, and network and storage faults to NetOps and StorageOps.

Charter (this Atlas’s proposal, not a published standard). Every host is known, reachable, explainable and replaceable.

1. Measure before acting. A restart destroys evidence; take a snapshot first.
2. Errors and saturation before utilisation. Queues and failures are easier to interpret than percentages.^[20]
3. Named access and least privilege. No shared root, including shared agent accounts; every privileged action has a person and a log line.
4. One baseline per role. Drift from it is a defect, not a style.
5. Replace cattle, repair pets, learn from both. A replacement that fails the same way fixes nothing.
6. Every scheduled job has an owner, a log and an alert on failure.
7. Headroom is a feature. Disk, inodes, memory and file descriptors run out at night.
8. No change you cannot undo without a snapshot, a backup or a rehearsal.

R = responsible, A = accountable, C = consulted, I = informed.

In small teams one person holds every column; the RACI still says who approves a privilege grant or a deletion. Incident command belongs to IncidentOps, access approval workflows to AccessOps.

stateDiagram-v2
  state "Requested" as Req
  state "Provisioned" as Prov
  state "Baselined" as Base
  state "In service" as Live
  state "Changing" as Chg
  state "Degraded" as Deg
  state "Retired" as Ret
  [*] --> Req
  Req --> Prov: image and size chosen
  Prov --> Base: config, access, agents
  Base --> Live: readiness checks pass
  Live --> Chg: change
  Chg --> Live: verified
  Chg --> Deg: fails
  Live --> Deg: alert
  Deg --> Live: repaired or rolled back
  Deg --> Prov: replaced
  Live --> Ret: end of life
  Ret --> [*]

• Day 0 (design). Choose the OS release for its support window. Ubuntu LTS releases get “5 years of standard security maintenance”, 10 with ESM and up to 15 with the Legacy add-on through Ubuntu Pro (as of October 2026);^[23] Windows Server LTSC releases get “five years of mainstream support and five years of extended support”.^[24]
• Day 1 (build). Provision from an image, apply the baseline, enrol in monitoring, logging, backup and access control, and run the readiness checklist (Checklists).
• Day 2 (operate). Triage, capacity, scheduled jobs, reboots after updates, drift correction.
• Retire. Follow the decommissioning checklist (Checklists).

Host changes follow the change process (ITOps); make routine, low-risk ones (a sudo rule from an approved request) pre-approved standard changes so nobody skips it.

flowchart TB
  subgraph EXT["Outside the host"]
    OP["Operator: SSH, agent or console"]:::risk
    CM["Config tool: baseline"]:::neutral
    TS["Time sources"]:::neutral
  end
  subgraph HOST["The host"]
    SV["systemd: services, timers"]:::core
    A["Applications"]:::neutral
    AG["Log agent, backup timer"]:::neutral
    KR["Kernel: cgroups, firewall, MAC"]:::core
  end
  subgraph OFF["Off-host stores"]
    TEL["Logs and metrics"]:::neutral
    BK["Backups"]:::neutral
  end
  OP -->|"named login, sudo"| SV
  CM -->|"apply"| HOST
  TS -->|"chrony"| KR
  SV -->|"runs"| A
  SV -->|"runs"| AG
  SV --> KR
  AG -->|"ship"| TEL
  AG -->|"copy"| BK
  KR --> HW["Hardware or hypervisor"]:::neutral
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

flowchart TB
  A["Alert or complaint"]:::neutral --> R{"Reachable?"}:::decision
  R -->|no| OOB["Console or status checks"]:::risk
  OOB --> PF["Replace on platform fault; else fix boot, network, sshd"]:::neutral
  R -->|yes| SN["Snapshot: load, logs, units"]:::core
  SN --> ER{"Errors?"}:::decision
  ER -->|yes| FE["Follow the error"]:::core
  ER -->|no| SA{"Saturation?"}:::decision
  SA -->|yes| NM["Name resource, consumer"]:::core
  SA -->|no| UT["Utilisation, then app"]:::neutral
  FE --> MI["Mitigate, verify, record"]:::core
  NM --> MI
  UT --> MI
  PF --> MI
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

• Trigger and actors: an alert, report or failed job; the on-call engineer, the service owner for business decisions, and an incident commander once more than one host is affected (IncidentOps).
• Exception path: on AWS, a failed system status check needs AWS (stopping and starting an EBS-backed instance “in most cases results in the instance being migrated to a new host”); a failed instance check is yours.^[25]
• Output: a triage note (Templates) naming resource, consumer, evidence and fix.

Five rules:

• Prefer timers to cron for new jobs. Under cron, SHELL is /bin/sh, an unescaped % in a command is “changed into newline characters”, and output is mailed to MAILTO or the crontab owner,^[29] often nowhere. A timer logs to the journal, Persistent=true catches up missed runs, and RandomizedDelaySec= spreads jobs “to reduce workload spikes”;^[30] OnFailure= starts another unit when the job’s unit enters the “failed” state:^[19] point it at your alerting (Lab 2).
• Cap and keep logs on purpose. With Storage=auto, journald keeps logs on disk only if /var/log/journal exists; auto was the default in systemd 245, but current releases set it at build time,^[31] so set it explicitly. Default caps: 10% of the filesystem used, 15% kept free, each at most 4 GiB. logrotate’s copytruncate leaves “a very small time slice” in which “some logging data might be lost”.^[32]
• Change SSH safely. For most keywords sshd uses “the first obtained value”, reading Include files in lexical order, but AllowGroups, AllowUsers, DenyGroups and DenyUsers append.^[33] Test with sshd -t, check sshd -T,^[28] and keep a second session open.
• Never step a clock by hand. chrony slews time; makestep steps it only above a threshold, in the first updates after start.^[34]
• Own accounts and files on purpose. Run each service as a system account (useradd --system --shell /usr/sbin/nologin) that alone can write its data directory (install -d -o myapp -g myapp -m 0750 /var/lib/myapp). For a leaver, usermod -L disables only the password: lock the account by also expiring it (usermod -L -e 1 <user>)^[35] and remove their SSH keys. File modes, umask and ACLs are prerequisites (LFCS tests ACLs^[3]).

Patch programmes belong to PatchOps; on one host, three questions matter.

• What changed? dnf history undo <id> reverses a transaction unless the RPM database state prevents it.^[36] apt has no undo: /var/log/dpkg.log records changes, and the way back is a named older version (apt install pkg=version)^[37] while a repository still has it.
• Does anything still run old code? After a library update, running services keep the old version until they restart.^[38] On the RHEL family, dnf needs-restarting -r reports “whether a reboot is required (exit code 1) or not (exit code 0)” and -s lists the services.^[39] On Ubuntu, /var/run/reboot-required marks a pending reboot, and from 24.04 LTS needrestart restarts affected services automatically.^[38]
• Will it boot? “Errors in the /etc/fstab file can render a system unbootable.” Mount data volumes by UUID with nofail, and test with mount -a before any reboot;^[40] findmnt --verify checks the whole file.^[41]

Unattended updates are a baseline decision. Ubuntu “will apply security updates automatically”, but unattended-upgrades’ Automatic-Reboot defaults to false,^[38] so reboots pile up unless scheduled. Choose per role (automatic security updates plus a reboot window, or only the patch programme) and alert on reboots pending beyond policy.

Example status: Debian/Ubuntu lines and findmnt --verify executed read-only on Ubuntu 20.04 (2026-10-17); dnf lines and mount -a not executed (no RHEL host; mount -a changes mounts).

# Debian/Ubuntu: pending reboot, recent package changes, nightly runs
[ -f /var/run/reboot-required ] && echo "reboot pending"
grep -E ' (install|upgrade) ' /var/log/dpkg.log | tail -n 5
ls /var/log/unattended-upgrades/
# RHEL family (dnf-plugins-core): exit 1 = reboot needed
sudo dnf needs-restarting -r; echo "exit $?"
sudo dnf needs-restarting -s                  # services still on old files
sudo dnf history list | head -n 5
# Any family, after editing /etc/fstab and before rebooting
sudo findmnt --verify && sudo mount -a

SecOps chooses the benchmark; applying it to servers is host work. Three controls need the most care.

• Host firewall: default deny inbound. Allow the management path and the service’s ports, and change rules remotely only with a way back that runs unless cancelled. In firewalld, --reload restores the permanent configuration, dropping untested runtime changes, and --runtime-to-permanent keeps them;^[42] systemd-run --on-active= can schedule the reload.^[43]
• Mandatory access control stays enforcing. SELinux (RHEL family) and AppArmor (loaded by default on Ubuntu^[44]) confine services beyond file permissions. Permissive SELinux permits accesses “that would be denied by policy” and only audits them,^[45] so putting the whole host in permissive mode for one application unconfines every service. Read the denial (audit2why for SELinux,^[46] apparmor="DENIED" lines for AppArmor^[44]), then fix the label (restorecon), a boolean (setsebool -P) or the profile. If a stop-gap is unavoidable, relax one SELinux domain (semanage permissive -a <domain>)^[47] or one AppArmor profile (aa-complain),^[44] recorded with SecOps as an expiring exception.
• SSH keys have a lifecycle. Inventory every key file and command sshd uses, remove leavers’ keys, and list revoked keys in RevokedKeys, replaced atomically: if sshd cannot read it, it refuses every public-key login.^[33] At scale, short-lived OpenSSH user certificates replace key lists (Deep comparisons); central identity belongs to IdentityOps.

Rescan (for example with CIS-CAT^[48]) after each baseline change; fix each failure or record an exception with owner and expiry.

Example status: syntax-checked; not executed (needs root and firewalld). The transient-timer mechanism (fire, and cancel by stopping the timer) was tested with systemd-run --user (systemd 245), and the key loop run read-only on Ubuntu 20.04 (2026-10-17).

# Firewall change with an automatic way back (test host first)
sudo systemd-run --unit=fw-undo --on-active=5min firewall-cmd --reload
svc=cockpit                                   # example: a service you do not offer
sudo firewall-cmd --remove-service="$svc"     # runtime only
# Open a NEW SSH session. If it works, keep the change and cancel the undo:
sudo firewall-cmd --runtime-to-permanent && sudo systemctl stop fw-undo.timer
# Mandatory access control: mode, and why something was denied
getenforce 2>/dev/null || sudo aa-status | head -n 3
sudo audit2why -a 2>/dev/null | tail -n 20
# Keys: where sshd reads them. `systemctl cat ssh` shows -o options that
# sshd -T misses (Ubuntu's ec2-instance-connect adds an AuthorizedKeysCommand).
sudo sshd -T | grep -E '^authorizedkeys(file|command) '
akf=$(sudo sshd -T | awk '$1 == "authorizedkeysfile" { $1 = ""; print }')
# Key entries (not comments or blank lines) per file, for every account's home
getent passwd | while IFS=: read -r user _ uid _ _ home _; do
  for f in $akf; do
    f=${f//"%h"/$home}; f=${f//"%u"/$user}; f=${f//"%U"/$uid}
    [ "${f#/}" = "$f" ] && f="$home/$f"          # relative paths start at home
    sudo test -f "$f" && echo "$user $f $(sudo grep -cvE '^[[:space:]]*(#|$)' "$f")"
  done
done

Brendan Gregg’s USE method (industry practice, not a standard): “For every resource, check utilization, saturation, and errors.” Utilisation is “the average time that the resource was busy servicing work”; saturation is “the degree to which the resource has extra work which it can’t service, often queued”.^[49] Netflix’s performance team published ten commands for a host’s first minute (uptime, dmesg | tail, vmstat 1, mpstat -P ALL 1, pidstat 1, iostat -xz 1, free -m, sar -n DEV 1, sar -n TCP,ETCP 1, top; several need sysstat) and the reading order: “Look for errors and saturation metrics, as they are both easy to interpret, and then resource utilization.”^[20]

Five traps:

1. Load average is not CPU load. It counts runnable tasks and “tasks in the uninterruptible sleep state”, which “includes tasks blocked on disk I/O and some locks”.^[17] High load with idle CPUs points at storage, NFS or locks.
2. “Free” memory is the wrong number. MemAvailable is “An estimate of how much memory is available for starting new applications, without swapping”.^[52] Modern free computes used as total minus available;^[53] a small free column is usually page cache.
3. %util lies about SSDs and arrays. For “devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits”;^[54] read queue size and latency.
4. A VM can be slow for reasons outside it. vmstat column st is “Time stolen from a virtual machine”;^[55] on EC2, CloudWatch CPUUtilization includes hypervisor time.^[56] Burstable T instances run on CPU credits. Out of credits, Standard mode drops to a baseline; Unlimited mode keeps bursting and bills you if the 24-hour average exceeds the baseline.^[57] As of October 2026, T2 launches in Standard mode and T3, T3a, T4g and T8i in Unlimited (T3 on a Dedicated Host: Standard only), unless the account default was changed.^[58]
5. Averages hide bursts. Gregg describes CPU saturation while monitoring, using five-minute averages, “showed CPU utilization was never higher than 80%”.^[49] PSI averages over 10, 60 and 300 seconds: “some” is the share of time at least some tasks stalled, “full” the share in which all non-idle tasks stalled at once.^[14]

Windows Server. Same method, different tools: Get-Counter for processor, memory and disk counters,^[59] Get-WinEvent -FilterHashtable for errors (faster than Where-Object; Level=2 selects errors)^[22] and Get-Service for services.^[21]

Idempotent tools, whose repeated runs leave the same result as one run,^[62] make a baseline safe to re-apply every hour.

Automate toil first: work that is manual, repetitive and “scales linearly as a service grows”,^[63] such as baseline application, housekeeping and a triage snapshot on alert. Keep humans for decisions that destroy data or evidence. AI assistants explain output and draft commands but invent flags, and pasted logs may hold secrets or personal data: review anything run as root. Cross-host event correlation belongs to AIOps.

Assumptions: a Debian/Ubuntu or RHEL-family server with systemd, OpenSSH and chrony; administrators in group ops-admins; an application myapp logging to /var/log/myapp. Deliver the files through configuration management (InfraOps).

Example status: directives checked against upstream man pages on 2026-10-17; not executed as a set. Validate on a test host first.

# /etc/ssh/sshd_config.d/00-baseline.conf
# "00-" sorts first. For most keywords sshd keeps the FIRST value it reads,
# but AllowGroups/AllowUsers/DenyGroups/DenyUsers lines anywhere add up.
# The main sshd_config must Include this directory before its own settings.
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
AllowGroups ssh-users ops-admins

# /etc/sudoers.d/ops-admins   (no dot in the file name: sudo skips such files)
# Install: sudo install -m 0440 -o root -g root ops-admins /etc/sudoers.d/
# Members of ops-admins may run commands as root after entering their own password.
%ops-admins ALL=(ALL:ALL) ALL

sudo reads /etc/sudoers.d only through an @includedir (or older #includedir) line, and skips file names that “contain a ‘.’ character”, so ops-admins.conf would be ignored silently; the default mode is 0440, and the file “must not be world-writable”.^[64] visudo -cf <file> checks only syntax; visudo -c also checks ownership and permissions.^[27] Each administrator needs a password: issue it from central identity (IdentityOps) so it is revoked with the account.

# /etc/systemd/journald.conf.d/10-retention.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=4G

# /etc/chrony/chrony.conf (Debian/Ubuntu) or /etc/chrony.conf (RHEL family): excerpt
# Use either leap-smearing sources only or non-smearing sources only.
# On EC2, replace the pool line with the local Amazon Time Sync Service:
#   server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4
pool time.example.com iburst      # your organisation's (non-smearing) servers
makestep 1.0 3
leapsecmode slew                  # default "system": the kernel steps back 1 s
rtcsync

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    su myapp myapp
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    sharedscripts
    postrotate
        systemctl reload myapp.service >/dev/null 2>&1 || true
    endscript
}

su makes logrotate act as the owner, as recommended for directories “in control of non-privileged users”; sharedscripts runs the reload once, not once per file.^[32]

Example status: syntax-checked with bash -n; guard tested with stubs; not executed. Requires root on a test host.

# Validate before reloading anything; keep a second SSH session open.
sudo grep -iE '^(Include|@includedir|#includedir)' /etc/ssh/sshd_config /etc/sudoers
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|allowgroups) '
sudo visudo -c                                # whole tree, ownership and modes
sudo systemctl restart systemd-journald && journalctl --disk-usage
chronyc tracking | grep -E '^(Reference ID|System time|Leap status)'
sudo logrotate --debug /etc/logrotate.d/myapp   # dry run: no changes are made
# Reload sshd only if its config is valid AND you are in an allowed group.
in_group() { id -nG "$USER" | tr ' ' '\n' | grep -qxE 'ssh-users|ops-admins'; }
if sudo sshd -t && in_group; then
  # The unit is "ssh" on Debian/Ubuntu and "sshd" on the RHEL family.
  sudo systemctl reload ssh 2>/dev/null || sudo systemctl reload sshd
else
  echo "STOP: sshd config invalid, or you are in neither group (lock-out)"
fi

On EC2, AWS recommends its local Time Sync Service, which smears leap seconds, and advises against mixing smeared and non-smeared sources during one.^[65][66]

Use a disposable VM for anything needing root.

Objective: collect the errors and saturation signals of the diagnostic method in one read-only pass. Prerequisites: a Linux host with systemd, procps and sysstat; 20 minutes. Cost and risk: the script changes nothing; step 3 loads the CPUs for a minute.

1. Save the script as triage.sh; run bash triage.sh | tee triage-$(date +%F-%H%M).txt.
2. Run sudo bash triage.sh and compare.
3. On a disposable VM, start twice as many busy loops as CPUs (for i in $(seq $((2 * $(nproc)))); do yes > /dev/null & done), wait a minute, rerun the script, then pkill yes. On a shared host, confine it: two taskset -c 0 nice -n 19 yes > /dev/null & loops, then taskset -c 0 bash triage.sh.
4. For each CHECK line, name its USE cell (resource; utilisation, saturation or error).

Example status: verified by execution as an unprivileged user (Ubuntu 20.04, kernel 5.15, systemd 245, 2026-10-17), including the confined step 3; output not reproduced (it describes a production host).

#!/usr/bin/env bash
# triage.sh: read-only first-minutes snapshot of one Linux host (USE-style).
# Changes nothing. Run with sudo to see all open files and the kernel log.
set -uo pipefail
warn() { printf '  CHECK: %s\n' "$*"; }
section() { printf '\n== %s ==\n' "$*"; }
fsx=(-x tmpfs -x devtmpfs -x squashfs -x overlay)   # skip pseudo filesystems

ncpu=$(nproc)
section "Load and CPU (cpus=${ncpu})"
read -r l1 l5 l15 _ < /proc/loadavg
echo "  load 1/5/15 min: ${l1} ${l5} ${l15}"
if awk -v l="$l1" -v n="$ncpu" 'BEGIN { exit !(l > n) }'; then
  warn "1-min load above CPU count (load also counts tasks in D state)"
fi
vmstat 1 5 | tail -n 4 | awk -v n="$ncpu" '
  { r += $1; si += $7; so += $8; wa += $16; st += $17; c++ }
  END {
    printf "  vmstat avg: r=%.1f si=%.0f so=%.0f wa=%.0f%% st=%.0f%%\n",
      r/c, si/c, so/c, wa/c, st/c
    if (r/c > n) print "  CHECK: run queue above CPU count (CPU saturation)"
    if (si/c + so/c > 0) print "  CHECK: swapping in or out (memory saturation)"
    if (st/c > 10) print "  CHECK: steal above 10% (hypervisor contention)"
  }'

section "Pressure stall information"
if head -n 1 /proc/pressure/cpu > /dev/null 2>&1; then   # a read, not just -r
  for r in cpu memory io; do
    printf '  %-6s %s\n' "$r" "$(head -n 1 "/proc/pressure/$r")"
  done
else
  echo "  PSI not readable: not built into this kernel, or disabled at boot"
fi

section "Memory"
awk '/^MemTotal:/ {t=$2} /^MemAvailable:/ {a=$2} /^SwapTotal:/ {st=$2}
     /^SwapFree:/ {sf=$2}
  END {
    printf "  MemAvailable %.0f%% of %.1f GiB; swap used %.0f MiB\n",
      100*a/t, t/1048576, (st-sf)/1024
    if (a/t < 0.10) print "  CHECK: MemAvailable below 10%"
  }' /proc/meminfo
journalctl -k -b --no-pager -q 2>/dev/null | grep -i -E 'out of memory|oom-kill' |
  tail -n 3 | sed 's/^/  /'

section "Filesystems (blocks and inodes)"
df -P "${fsx[@]}" | awk 'NR > 1 { u = $5; sub("%", "", u)
  if (u+0 >= 85) printf "  CHECK: %s at %s%% blocks\n", $6, u }'
df -P -i "${fsx[@]}" | awk 'NR > 1 && $5 != "-" { u = $5; sub("%", "", u)
  if (u+0 >= 85) printf "  CHECK: %s at %s%% inodes\n", $6, u }'
df -h "${fsx[@]}" | sed 's/^/  /'
deleted=$(find /proc/[0-9]*/fd -lname '*(deleted)' 2>/dev/null | wc -l)
echo "  open-but-deleted file descriptors visible to this user: ${deleted}"

section "Services and recent errors"
failed_units() {
  systemctl list-units --state=failed --no-legend --plain 2>/dev/null
}
failed=$(failed_units | wc -l)
echo "  failed units: ${failed}"
failed_units | sed 's/^/  /'
journalctl -p err -b --since "-1h" --no-pager -q -n 10 2>/dev/null | sed 's/^/  /'

section "Top processes"
ps -eo pid,user,pcpu,pmem,stat,comm --sort=-pcpu | head -n 6 | sed 's/^/  /'
dstate=$(ps -eo stat= | grep -c '^D')
[ "$dstate" -gt 0 ] && warn "${dstate} task(s) in D state: usually waiting on I/O"

section "Network and time"
ss -s | head -n 2 | sed 's/^/  /'
timedatectl show -p NTPSynchronized --value 2>/dev/null |
  sed 's/^/  NTP synchronised: /'
exit 0

Expected output (shape): seven sections, from == Load and CPU (cpus=4) == to == Network and time ==, with CHECK: lines where a threshold is crossed; after step 3, CHECK: run queue above CPU count (CPU saturation), usually the load check, and a much higher cpu some avg10. Validation: echo $? prints 0; every section header appears; step 3 produces the run-queue check. Troubleshooting: vmstat: command not found → install procps; “PSI not readable” → no PSI in the kernel, or disabled at boot (some RHEL-family kernels): add psi=1 to the kernel command line;^[67] no journal lines → join systemd-journal or adm, or use sudo; odd wa/st values → check column positions in vmstat 1 1. Teardown: pkill yes; delete saved output that names hosts.

Objective: move a nightly job to a systemd timer that logs, catches up, spreads load, alerts on failure and runs with least privilege; then prove a restore. Prerequisites: root on a disposable VM with systemd 245 or later; 45 minutes. Cost and risk: low; the job writes only to /var/backups/etc. The archive is not a backup until copied off-host (BackupOps).

Example status: script run 17 times against scratch directories (14 archives kept) and the restore run there without sudo; the three units pass systemd-analyze verify (systemd 245); not installed or started (2026-10-17).

#!/usr/bin/env bash
# /usr/local/sbin/backup-etc.sh: keep 14 dated archives of /etc on this host.
set -euo pipefail
dest=/var/backups/etc
keep=14
stamp=$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$dest"
tar --create --gzip --file "$dest/etc-$stamp.tar.gz" --directory / etc
# Read the archive back: fails loudly if it is unreadable.
tar --list --gzip --file "$dest/etc-$stamp.tar.gz" > /dev/null
ls -1t "$dest"/etc-*.tar.gz | tail -n +$((keep + 1)) | xargs -r rm --
count=$(ls -1 "$dest"/etc-*.tar.gz | wc -l)
echo "backup-etc: wrote $dest/etc-$stamp.tar.gz; $count archives kept"

# /etc/systemd/system/backup-etc.service
[Unit]
Description=Archive /etc to /var/backups/etc
OnFailure=notify-failure@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup-etc.sh
Nice=10
IOSchedulingClass=idle
MemoryMax=512M
ProtectSystem=strict
ReadWritePaths=/var/backups/etc
PrivateTmp=yes
NoNewPrivileges=yes
ProtectHome=read-only

# /etc/systemd/system/notify-failure@.service
# Logs an error-priority journal line; point ExecStart at your pager instead.
[Unit]
Description=Report the failure of %i

[Service]
Type=oneshot
ExecStart=/usr/bin/logger -p user.err "unit %i failed"

# /etc/systemd/system/backup-etc.timer
[Unit]
Description=Nightly /etc archive

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=30min
Persistent=true

[Install]
WantedBy=timers.target

1. Install: sudo install -m 0755 backup-etc.sh /usr/local/sbin/, then sudo install -m 0644 backup-etc.service notify-failure@.service backup-etc.timer /etc/systemd/system/, then the root-only directory: sudo install -d -m 0700 /var/backups/etc.
2. Check: sudo systemd-analyze verify /etc/systemd/system/backup-etc.{service,timer}; ignore notes about other units.
3. Enable: sudo systemctl daemon-reload && sudo systemctl enable --now backup-etc.timer.
4. Run once: sudo systemctl start backup-etc.service; read journalctl -u backup-etc.service -n 5 --no-pager.
5. Restore one file and compare it (block below).
6. Score the sandbox: systemd-analyze security backup-etc.service reports an exposure level from 0.0 to 10.0 (lower is tighter).^[68] Remove ProtectSystem=strict, reload, and compare.
7. Comment out the old crontab line and record the change.

latest=$(sudo sh -c 'ls -1t /var/backups/etc/etc-*.tar.gz | head -n 1')
scratch=$(mktemp -d)
sudo tar -xzf "$latest" -C "$scratch" etc/hostname
sudo diff "$scratch/etc/hostname" /etc/hostname && echo "restore OK: $latest"
sudo rm -r -- "$scratch"

What the settings do: ProtectSystem=strict makes the filesystem read-only except /dev, /proc and /sys, and ReadWritePaths= re-opens only the backup directory, which must exist (step 1); NoNewPrivileges= blocks privilege gain through setuid binaries; PrivateTmp= gives a private /tmp.^[69] Above MemoryMax=, the OOM killer runs inside the unit.^[18] The job still runs as root and reads private keys in /etc, hence the 0700 directory. Expected output: systemctl list-timers backup-etc.timer shows a NEXT time between 02:30 and 03:00; the journal shows backup-etc: wrote /var/backups/etc/etc-<timestamp>.tar.gz; 1 archives kept; step 5 prints restore OK. Validation: systemctl show backup-etc.service -p Result prints Result=success. To test the alert, add exit 1 as the script’s second line, start the service, and find unit backup-etc.service failed in journalctl -p err -n 5. Troubleshooting: status=226/NAMESPACE → /var/backups/etc missing; Read-only file system → path not in ReadWritePaths=; status=203/EXEC → script not executable or wrong path; timer never fires → systemctl is-enabled backup-etc.timer. Teardown: sudo systemctl disable --now backup-etc.timer, remove the three unit files and the script, run sudo systemctl daemon-reload, and delete /var/backups/etc if unneeded.

Objective: see why deleting a log that a process holds open frees nothing, and recover the space without a restart. Prerequisites: a Linux host with GNU coreutils; 200 MiB free in $TMPDIR; 15 minutes. Cost and risk: none for the main part (own temporary directory).

Example status: main part verified by execution as an unprivileged user (2026-10-17); optional part syntax-checked, not executed.

#!/usr/bin/env bash
# lab3.sh: deleting a file that a process still holds open frees no space.
set -euo pipefail
avail() { df -m --output=avail . | tail -n 1; }
work=$(mktemp -d "${TMPDIR:-/tmp}/sysops-lab3.XXXXXX")
cd "$work"
head -c 200M /dev/zero > app.log            # a 200 MiB "log"
tail -f app.log > /dev/null 2>&1 &           # a process that keeps the file open
holder=$!
sleep 1
echo "before rm:  $(du -sm . | cut -f1) MiB in dir; fs avail $(avail) MiB"
rm app.log
echo "after rm:   $(du -sm . | cut -f1) MiB in dir; fs avail $(avail) MiB"
fd=$(find "/proc/$holder/fd" -lname '*app.log (deleted)' | head -n 1)
echo "still open: $fd -> $(readlink "$fd")"
: > "$fd"                                    # truncate via the descriptor
echo "truncated:  fs avail $(avail) MiB"
kill "$holder"
cd / && rmdir "$work"

Expected output (shape): before rm shows about 201 MiB in the directory; after rm about 1 MiB but the same free space; still open prints /proc/<pid>/fd/<n> -> …/app.log (deleted); truncated shows about 200 MiB more free space. Validation: the three free-space figures behave as described. On real hosts, lsof +L1 selects “open files that have been unlinked”.^[70] Troubleshooting: df: unrecognized option '--output=avail' → non-GNU or old coreutils; read df -m . instead.

Optional (root, disposable VM only): inode exhaustion. mkfs.ext4 -N sets the number of inodes.

truncate -s 64M /root/tiny.img
mkfs.ext4 -q -N 2048 /root/tiny.img
mkdir -p /mnt/tiny && mount -o loop /root/tiny.img /mnt/tiny
mkdir /mnt/tiny/many
for i in $(seq 1 3000); do
  : > "/mnt/tiny/many/f$i" 2>/dev/null || { echo "write failed at file $i"; break; }
done
df -h /mnt/tiny; df -i /mnt/tiny      # blocks mostly free, IUse% 100%
umount /mnt/tiny && rm /root/tiny.img

Teardown: the main script removes its own directory; the optional part ends with umount and rm.

1. Cloudflare and the 2016 leap second (sourced). At midnight UTC on 1 January 2017 the clock went back one second, and Cloudflare’s DNS software computed a negative duration and panicked. At peak “approximately 0.2% of DNS queries” were affected and “less than 1% of all HTTP requests” saw errors; the root cause was “the belief that time cannot go backwards”.^[71] SysOps lesson: by default chrony lets the kernel step the clock back; leapsecmode slew corrects it over about 12 seconds,^[34] so the kit sets it. Leap seconds are to be abandoned “by or before 2035”,^[65] but makestep and manual changes still step clocks; monotonic clocks for durations are the application’s fix. Success metric: no clock step outside boot.

2. Recovering hosts that cannot boot: CrowdStrike, July 2024 (sourced). Microsoft estimated that the faulty CrowdStrike update “affected 8.5 million Windows devices”.^[72] Its first server guidance was manual: boot Hyper-V VMs from a Windows Server ISO, or reach physical servers through the vendor’s remote management interface (iLO, iDRAC, CIMC), then delete files matching C-00000291*.sys. It later “released a USB tool to help automate this manual repair process”,^[73] which still boots each machine from recovery media.^[74] Client guidance warned “You may be asked to enter your BitLocker recovery key”.^[75] SysOps lesson: consoles, recovery media and recovery keys must work before the outage; no agent fixes a host that cannot boot. The EC2 serial console, for example, is off by default and on Linux needs a password-based OS user, which AWS AMIs lack (Deep comparisons).^[76] Keep a break-glass account with a vaulted, rotated password that cannot log in over SSH.

3. “The new release made the API slow” (illustrative scenario). Latency doubles a week after a release and the team starts a rollback. Triage shows CPU pinned near the baseline: the instances are T2s in Standard credit mode, and CPUCreditBalance is zero because rising traffic used up the credits.^[57] Method: the USE table, then credit balance and specification. Trade-off: Unlimited mode or a non-burstable instance costs more; the rollback fixes nothing. Success metric: page to named resource in under 15 minutes.

Worked calculation (illustrative numbers). A 500 GiB ext4 data volume holds 410 GiB and grew 63 GiB in 7 days: 9 GiB a day. The default 5% reserve (25 GiB) is closed to non-root processes,^[77] so applications see a full disk at about 475 GiB: runway is (475 − 410) ÷ 9 ≈ 7.2 days, not the 10 that size minus used suggests.

• Least privilege. Named access on every path in (Deep comparisons), and know who can reach credentials: in Lab 2, only root and the backup job can read archives holding every secret in /etc.
• Baselines and standards. Apply the benchmark SecOps chose (Applying a hardening baseline). CIS Benchmarks are consensus guides^[48] and NIST SP 800-123 (2008) lists server-security activities;^[78] neither is mandatory in itself, but contracts, regulators or customers can make one so (ComplianceOps).
• Logs as evidence and personal data. Plan retention with NIST SP 800-92 (2006; Rev. 1 is a 2023 draft);^[79][80] logs hold user names and IP addresses, so agree retention with PrivacyOps. OpenSSH warns that “Logging with a DEBUG level violates the privacy of users”.^[33]
• Jurisdiction. Retention and audit duties vary by country, sector and contract; this is not legal advice.

Host costs hide in idle capacity and people’s time: “just in case” sizing, ownerless hosts, pets only one person can maintain. Keep headroom for a burst plus the time to add capacity, no more. Rightsizing belongs to FinOps, carbon to GreenOps.

Repairing a Linux host that does not boot. At the console, interrupt the boot loader (on EC2, set GRUB up for the serial console beforehand^[83]) and append systemd.unit=rescue.target (all filesystems, basic services) or systemd.unit=emergency.target (a shell and nothing else) to the kernel line; a failed check of a required filesystem lands in emergency mode by itself. There / may still be read-only: remount it (mount -o remount,rw /).^[84] Fix or comment out the bad /etc/fstab line, run findmnt --verify, and read why the previous boot failed with journalctl -b -1 (persistent journal only).^[85] Both shells ask for the root password through sulogin, which skips it for a locked root account only when forced,^[86] so decide whether break-glass includes a vaulted root password, and rehearse this path in the console drill.

Protecting a process with oom_score_adj=-1000 is “equivalent to disabling OOM-killing entirely for that task”,^[87] so the kernel kills something else; prefer memory limits on the noisy service.

flowchart TB
  A["Filesystem above 90%"]:::risk --> B["df -h and df -i"]:::core
  B --> C{"Blocks or inodes?"}:::decision
  C -->|inodes| D["Remove small-file sprawl"]:::core
  C -->|blocks| E{"du close to df?"}:::decision
  E -->|no| F["lsof +L1"]:::core
  F --> G["Truncate logs or restart holder"]:::core
  E -->|yes| H["du -x: largest paths"]:::core
  H --> I{"Safe to remove?"}:::decision
  I -->|yes| J["Remove, rotate or move"]:::core
  I -->|no| K["Extend volume with owner"]:::neutral
  D ----> L
  G --> L["Verify, fix cause, record"]:::core
  J --> L
  K --> L
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Example status: commands syntax-checked; read-only except the marked step. Replace /var with the alerting mount.

M=/var                                        # the mount from the alert
df -h "$M"; df -i "$M"                        # 1. blocks or inodes?
sudo du -xh --max-depth=2 "$M" 2>/dev/null | sort -rh | head -n 15   # 2. where?
sudo lsof +L1 2>/dev/null | awk 'NR==1 || /deleted/'   # 3. deleted but still open?
journalctl --disk-usage                       # 4. is the journal the culprit?
# 5. CHANGE (ticket first): reclaim journal space above 1 GiB
sudo journalctl --vacuum-size=1G
df -h "$M"                                    # 6. verify headroom returned

1. Stabilise. At 100% with services failing, free a little space first: vacuum the journal (step 5), or remove rotated logs already shipped off-host.
2. Find the cause. Steps 1–4. Typical causes: a log without rotation, a core dump, a runaway cache, a stuck temporary directory.
3. Fix. Truncate a deleted-but-open file through /proc/<pid>/fd/<n> (Lab 3) only if it is a log or cache you would delete anyway; truncating a database, queue or spool file destroys live data. Otherwise restart the holder in a quiet period.
4. Roll back. Deleting is irreversible; when unsure, move files to another volume.
5. Prevent. Add the missing rotation or cap, an inode alert and disk runway in the weekly report; record the cause.

Example status (branch “Extend volume”): syntax-checked; not executed (needs root); steps from AWS’s guide.^[88]

# After the owner has grown the volume (snapshot first); same M as above.
lsblk                                         # partition smaller than its disk?
sudo growpart /dev/nvme0n1 1                  # if so, grow it (note the space)
df -hT "$M"                                   # file system type; run ONE of:
sudo xfs_growfs -d "$M"                       # XFS: takes the mount point
sudo resize2fs "$(findmnt -no SOURCE "$M")"   # ext4: takes the device
df -hT "$M"                                   # size now matches the volume
# growpart needs free space for a temporary directory (stabilise first).
# With LVM, extend the logical volume with LVM tools instead.

• “Restart first.” It destroys evidence. Take the Lab 1 snapshot; it takes a minute.
• Unrecorded hand edits. Each makes the next host different and the next incident slower.
• Disabling a protection to make something work (a firewall rule, SELinux or AppArmor, a sandbox setting). Read the denial and fix the policy.
• Myths: high load means busy CPUs; low “free” memory is a problem (traps 1 and 2); the cloud needs no SysOps, when on IaaS the guest OS and its patches are yours.^[13]

Invest in host skills and a baseline when you run guest operating systems (VMs, bare metal, IaaS, self-managed Kubernetes nodes), when regulators or customers want host-level evidence, when you own long-lived pets (licensed software, appliances, legacy systems), or when servers sit at remote sites (EdgeOps).

Invest less, or differently, when the provider runs the OS (SaaS, managed databases, serverless): the work moves to CloudOps and ServerlessOps, and NoOps explains what remains. With many identical hosts, invest in images and configuration code (InfraOps) rather than repair. A separate “SysOps team” beside a platform team can become a hand-off queue; consider giving the platform team host ownership.

Hidden responsibilities: OS end-of-life migrations, reboots, access reviews, restore tests, console access, and someone on call who can read a kernel log.

flowchart TB
  Q1{"You run the guest OS?"}:::decision
  Q1 -->|no| N1["Provider runs the host"]:::neutral
  Q1 -->|yes| Q2{"Identical to many?"}:::decision
  Q2 -->|"yes: cattle"| Q3{"Failure understood?"}:::decision
  Q3 -->|yes| R1["Fix image, replace"]:::core
  Q3 -->|no| R2["Capture evidence, replace"]:::core
  Q2 -->|"no: pet"| Q4{"Rebuild within RTO?"}:::decision
  Q4 -->|yes| R3["Replace, codify gaps"]:::core
  Q4 -->|no| R4["Snapshot, repair, codify"]:::risk
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

The tree assumes evidence (triage snapshot, off-host logs, disk snapshot) is cheap; where it is not, agree in advance how much to collect. Recovery time objectives belong to DisasterRecoveryOps.

This is this Atlas’s model, not an industry standard.

New host readiness

• In inventory, with an owner and an OS end-of-life date
• Baseline passing; firewall default-deny; SELinux or AppArmor enforcing; exceptions recorded
• Named access only; no root login; sudo through groups; agent sessions mapped to named accounts
• Metrics, logs and alerts arriving; one test alert received
• Time synchronised; journal persistent and capped
• Update and reboot policy set; reboot-pending alert in place
• Off-host backup running; first restore tested
• Console access, the boot menu and a rescue-mode boot tested with break-glass credentials

Before a risky host change

• Change record approved; rollback written down
• Snapshot or backup taken and verified
• Second session open; automatic undo for SSH or firewall changes
• Config validated (sshd -t, visudo -c, systemd-analyze verify; findmnt --verify before a reboot)

Decommissioning

• Owner confirms; data retention decided
• Access, keys, certificates and service accounts revoked
• Removed from monitoring, backup and patch schedules
• Disks wiped or destroyed; inventory updated

HOST RUNBOOK: <service> on <host role>
Owner / escalation:      <team, on-call rota, service owner>
What runs here:          <units and timers, ports, data paths>
Health checks:           <commands and expected output>
Common alerts:           <alert> -> <first command> -> <known fix>
Safe restarts:           <unit>, <impact>, <quiet window>
Data and backups:        <paths>, <backup job>, <last restore test>
Access:                  <who has sudo, how it is granted, console path>
Last reviewed:           <YYYY-MM-DD> by <name>

TRIAGE NOTE (fill in while working)
Time / host / alert:     <UTC time> / <host> / <alert>
Reachable:               yes | no (console used: <which>)
Errors seen:             <kernel log, failed units, OOM>
Saturation seen:         <run queue, PSI, swap, await, drops>
Constrained resource:    <CPU | memory | disk I/O | capacity | network | limit>
Consumer:                <process or unit>
Action and verification: <what was done, what proved it worked>
Follow-up:               <ticket for prevention>

Ways onto a host, and who reaches root (AWS examples, as of October 2026).

On EC2 that IAM action is ec2-instance-connect:SendSerialConsoleSSHPublicKey.^[76]

Is “SysOps” the same as system administration? In practice, yes: the label is informal, and the US occupational classification (O*NET) and the certifications call the job system administration.^[2][3]

My monitoring shows 40% CPU, yet users see slowness. How? Bursts hidden by averages, steal time, or saturation elsewhere: check run queue, PSI and I/O queues.^[49][55][14]

Is agent-based access safer than SSH? It closes inbound ports and removes keys, but by default every permitted IAM principal shares one passwordless-sudo account; it is safer only with Run As, scoped IAM and session logging (Deep comparisons).

Is a snapshot a backup? Not on its own: a snapshot on the same system fails with it (BackupOps).

When does a host problem become an incident? When several hosts or a user-facing service are affected, or the fix needs decisions beyond the host owner (IncidentOps).

• Pressure-driven memory management. systemd-oomd “uses cgroups-v2 and pressure stall information (PSI) to monitor and take corrective action before an OOM occurs in the kernel space”;^[95] distributions differ on enabling it.
• Tracing. perf and eBPF-based tools answer “why” after USE answers “where” (see further reading).
• Short-lived access. Per-request SSH certificates valid for hours make the CA the critical system (AccessOps).

Skills maps: the LFCS domains^[3] and RHCSA objectives^[4] (packages, recovery, firewalls, SELinux).

• Decide: who owns hosts; pet or cattle; which baseline; the update and reboot policy.
• Do: named access on every path, one baseline, alerting timers, capped journals, synchronised time, off-host backups, tested consoles.
• Diagnose: errors, then saturation, then utilisation; load includes D state; read MemAvailable; distrust %util on SSDs.
• Measure: conformance, job success, disk runway, reboot debt, OS end-of-life exposure.
• Watch out for: deleted-but-open files, inodes, clock steps, silent jobs, shared agent accounts, untested fstab edits.

Situation (illustrative). At 14:00 the only server running a licensed reporting system is swapping and nearly unresponsive; month-end reports are due at 18:00. A rebuild from documentation is estimated at six hours and has never been tested.

Assumptions: a disk snapshot takes minutes; triage has named the cause, a runaway export job not needed today; the licence allows a second instance later.

Decision: snapshot, repair in place (stop the export job or cap it with MemoryHigh=), deliver the reports, and schedule the rebuild as a rehearsed project. Option B looks principled, but an untested rebuild under a deadline turns degradation into an outage. Uncertainty: if triage cannot name the cause within 30 minutes, the repair estimate is unknown and the balance changes; escalate to the service owner.

After this page you can explain what SysOps covers and why the label is informal; diagnose a slow or full host with the USE method; configure timers, journald, sudo, SSH and a host firewall safely; decide whether to repair or replace a host.

1. A host shows load average 24 on 8 CPUs, mpstat shows 70% idle, and ps lists many tasks in state D. Where do you look next?

2. You put PasswordAuthentication no in /etc/ssh/sshd_config.d/99-hardening.conf, but password logins still work. Name two causes.

3. A sudoers rule in /etc/sudoers.d/admins.conf has no effect. Why?

4. A nightly cron job has not produced output for two weeks and nobody noticed. Name two changes that would have caught it.

5. A cattle host in an auto-scaling group fails in a way nobody has seen before. Replace it immediately?

SysOps is an old job under a fading name. The label will keep moving (AWS has moved it to CloudOps), but the host under every platform stays: a kernel that queues work, disks that fill, clocks that drift, jobs that fail quietly, doors that lead to root. The durable skill is a habit, not a command: measure before acting, read errors and saturation before percentages, and turn every fix into a baseline, timer, alert or image, so the same problem never needs the same person twice.

Claim-by-claim evidence: research/sysops-ledger.md. Re-check the AWS rows, OS support windows and systemd or chrony defaults quarterly, and the versions on your own hosts.

Further reading: Linux Performance (Brendan Gregg’s tools map); chronyc(1) (tracking output).

Changelog. 2026-10-16, r0: first version. 2026-10-17, r1: revised after reviews/sysops-r1.md (access paths, console, packages, firewall, MAC, time, labs; Figure 1 removed). 2026-10-17, r2: revised after reviews/sysops-score.md (descriptor limits, boot recovery, per-domain permissive mode, accounts and file modes, Figure 4 routing; all sources re-checked). 2026-10-17, post-score polish: guarded sshd reload, key-file inventory, RevokedKeys and GRUB preconditions, filesystem growth, T3 and chrony details.