---
title: "SysOps"
slug: sysops
category: cloud
term_status: overlapping-synonym
evidence_status: verified
summary: "SysOps (systems operations) is the hands-on work of keeping servers healthy: services, users and privileges, logs, scheduled jobs, packages, filesystems, time and performance triage. The label is informal; the established practice is system administration."
audience: [Beginners, System administrators, On-call engineers, Architects, Engineering leaders]
last_verified: 2026-10-17
key_takeaways:
  - "SysOps is an informal name for system administration. No body defines it, AWS dropped it from its certification in 2025, and 'sysop' also names BBS and wiki administrators. The practice behind it is long established."
  - "This page's emphasis is one Linux host in depth: a command-line method that finds what a server is short of before anyone guesses or restarts."
  - "Triage with the USE method (utilisation, saturation, errors for every resource), and read errors and saturation first: kernel log, run queue, pressure stall information, swapping, I/O queues."
  - "Load average also counts tasks blocked in uninterruptible sleep, and MemAvailable, not 'free', tells you how much memory is left. Misreading either sends the fix to the wrong resource."
  - "Every way onto a host is a way to root: SSH keys, agent sessions and consoles each need named identities, scoped permissions and logs, and the console must work before the outage."
  - "Many host outages are mundane: a full disk, exhausted inodes, a deleted log still held open, a silent job, a clock step, a reboot that does not come back. Each has a cheap check and a cheap prevention."
related: [cloudops, infraops, patchops, storageops, netops, containerops, endpointops, secops]
---

## What SysOps is

### In 30 seconds

**SysOps** (systems operations) is the hands-on work of keeping servers healthy: supervising services, managing users and
privileges, reading logs, scheduling jobs, installing packages, keeping filesystems and clocks in order, and finding out why a machine
is slow or down. Most people who do this work are called system administrators; "SysOps" is an informal name for the same job, and
was the name of an AWS certification until 2025.

It is for beginners meeting servers, on-call engineers who need a dependable diagnostic routine, and leaders deciding how much
host work their platform still needs.

### At a glance

| Question | Answer |
|---|---|
| Goal | Hosts that stay available, observable, recoverable and on an agreed baseline |
| Users | System administrators, on-call engineers, anyone with a shell on a server |
| Scope on this page | Linux servers in depth, Windows Server equivalents in brief: services, users and sudo, remote access, logs, scheduling, packages and reboots, filesystems, time, host firewall and mandatory access control, host-level backups, performance triage |
| Out of scope | Provider control plane → [CloudOps](../cloudops/); hosts as code → [InfraOps](../infraops/); patch programmes → [PatchOps](../patchops/); storage arrays → [StorageOps](../storageops/); user devices → [EndpointOps](../endpointops/); choosing baselines → [SecOps](../secops/) |
| Prerequisites | None; a Linux shell helps |
| Key outputs | A host baseline, runbooks, triage notes naming the constrained resource, healthy jobs, evidence of who did what |
| Term status | Overlapping synonym: an informal label for system administration (see [What SysOps is not](#what-sysops-is-not)) |

### The problem it solves

Every abstraction runs on a host: containers share a kernel, and a Kubernetes node is a Linux server with agents. When an
abstraction leaks (a node turns NotReady, a VM stops answering, a batch job runs three times slower), someone must read the host;
teams without that skill restart and hope.

Host work by hand is expensive too. Google's SRE book says the traditional "sysadmin approach" gets costlier as a service grows,
"because the size of the team necessarily scales with the load generated by the system".[^sre-intro] The problem is not host
knowledge but host work without method or automation.

| Without a host discipline | With one |
|---|---|
| "The server is slow" ends in a restart | Triage names the saturated resource and its consumer |
| Hand-configured, slightly different servers | One baseline, applied by tooling and verified |
| Cron jobs fail silently for weeks | Jobs log; failures alert |
| A disk fills at 03:00 | A headroom alert fired days earlier |
| Everyone shares root | Named accounts, sudo, and logs of who ran what |

### What SysOps is not

- **Not a defined discipline.** No standards body, foundation or manifesto defines "SysOps". The work goes by its established
  name: the US occupation "Network and Computer Systems Administrators" (O*NET 15-1244.00) includes reviewing "system and
  application logs" and verifying "completion of scheduled jobs, including system backups",[^onet] and LFCS[^lfcs] and RHCSA[^rhcsa]
  certify system administrators.
- **Not the other "sysop".** "Sysop" (system operator) also names a bulletin-board or online-community administrator,[^wp-sysop]
  MediaWiki's administrator group,[^mw-rights] and, for TechTarget, a role in which "a sysop monitors consoles" and usually reports
  to the sysadmin.[^tt-sysop]
- **Not CloudOps.** AWS renamed SysOps Administrator – Associate to CloudOps Engineer – Associate "to reflect the evolving nature of
  cloud operations and a shift in industry terminology"; the last SOA-C02 exam was on 2025-09-29.[^aws-cert][^aws-dates] The guest
  OS belongs here; the provider side to [CloudOps](../cloudops/).
- **Not hosts as code.** Terraform, golden images and drift detection belong to [InfraOps](../infraops/); this page covers what an image
  must contain and how to diagnose its hosts.

::: note Term status and editorial distinction
This Atlas classes SysOps as an **overlapping synonym** for system administration and teaches the practice under that name. The
Atlas split (host → SysOps; image and runtime → [ContainerOps](../containerops/); orchestrator → [KubernetesOps](../kubernetesops/);
baselines chosen in [SecOps](../secops/), applied here) is an **editorial distinction**, not a standard.
:::

## Where SysOps comes from and where it sits

### Origins and evolution

System administration was a profession long before "-Ops" labels: USENIX's LISA (Large Installation System Administration)
conference ran for 35 years until, as Tom Limoncelli wrote in November 2022, "The Usenix LISA conference is no more".[^lisa] His
co-written textbook, *The Practice of System and Network Administration*, is in its third edition.[^tposna]

Three shifts changed the job without removing it.

1. **SRE.** Google's SRE book (2016) defines its approach against the traditional sysadmin model.[^sre-intro]
2. **Pets and cattle.** Randy Bias, crediting a Bill Baker talk, contrasts pets, "indispensable or unique systems that can never be
   down", with cattle, servers "built using automated tools" and "designed for failure".[^pets]
3. **IaaS.** In the cloud the customer still owns "the guest operating system (including updates and security patches)".[^aws-srm]

The tools moved too: systemd's journal and timers, pressure stall information (PSI, 2018),[^psi] agent-based access,[^ssm-session]
and OpenSSH installed by default from Windows Server 2025 (its sshd service must still be enabled).[^win-ssh]

### Glossary and mental model

| Term | Meaning |
|---|---|
| Host | One operating-system instance: bare metal, VM or cloud |
| Load average | Exponentially damped moving sums of runnable plus uninterruptible tasks; 1, 5 and 15 minutes are formula constants, not exact windows[^loadavg] |
| Run queue | Runnable tasks, running or waiting for a CPU (`r` in `vmstat`); above the CPU count means CPU saturation |
| PSI | Pressure stall information: the share of time tasks stall on CPU, memory or I/O |
| MemAvailable | Kernel estimate of memory available for new work without swapping: the memory number to alert on |
| D state | Uninterruptible sleep, usually waiting for I/O; raises load without using CPU |
| Inode | Per-file metadata record; their number is limited, so a disk can be "full" with free blocks |
| cgroup | Control group: a kernel grouping of processes for resource management, behind systemd's per-service limits[^sd-resource] |
| Unit, drop-in, timer | A systemd unit file describes a service, timer (a schedule that starts a service) or mount; a drop-in (`<unit>.d/*.conf`) changes settings "without having to modify unit files"[^sd-unit] |
| Baseline | The agreed configuration every host of a role must match |

**Mental model.** A host is a set of finite resources (CPUs, memory, storage capacity and I/O, network, and software limits such as
file descriptors) shared by services under a supervisor. **Diagnosis** finds which resource is saturated or failing, and who is using
it. **Operation** keeps baseline, headroom and evidence in place so diagnosis is rarely needed.

### How SysOps relates to other Ops

| Neighbour | Shared scope | This page | The neighbour |
|---|---|---|---|
| [CloudOps](../cloudops/) | Cloud VMs | Inside the guest OS | Accounts, quotas, Regions, provider events |
| [InfraOps](../infraops/) | Host configuration | Baseline content; diagnosing hosts | IaC, images, drift detection |
| [PatchOps](../patchops/) | Updates | Package mechanics and reboots on one host | Rings, windows, compliance across the estate |
| [SecOps](../secops/) | Hardening | Applying baselines on servers | Choosing baselines and approving exceptions |
| [IdentityOps](../identityops/) | Accounts | Local accounts, sudo, SSH keys on the host | Central identity and its lifecycle |
| [EndpointOps](../endpointops/) | OS management | Servers | Laptops and phones |
| [ContainerOps](../containerops/), [KubernetesOps](../kubernetesops/) | Kernel features | The host and its kernel | Images, runtimes, clusters |
| [SRE](../sre/) | Reliability work | Host method and toil reduction | SLOs and error budgets |
| [ITOps](../itops/) | Operations process | Technical host work | Change enablement, CMDB, service desk |

Hosts hand telemetry to [ObservabilityOps](../observabilityops/), wider incidents to [IncidentOps](../incidentops/), and network
and storage faults to [NetOps](../netops/) and [StorageOps](../storageops/).

## Principles and operating model

### Charter and principles

**Charter (this Atlas's proposal, not a published standard).** Every host is known, reachable, explainable and replaceable.

1. **Measure before acting.** A restart destroys evidence; take a snapshot first.
2. **Errors and saturation before utilisation.** Queues and failures are easier to interpret than percentages.[^netflix60]
3. **Named access and least privilege.** No shared root, including shared agent accounts; every privileged action has a person
   and a log line.
4. **One baseline per role.** Drift from it is a defect, not a style.
5. **Replace cattle, repair pets, learn from both.** A replacement that fails the same way fixes nothing.
6. **Every scheduled job has an owner, a log and an alert on failure.**
7. **Headroom is a feature.** Disk, inodes, memory and file descriptors run out at night.
8. **No change you cannot undo** without a snapshot, a backup or a rehearsal.

### Fundamentals: the building blocks of a host

| Building block | Linux | Windows Server | Typical failure |
|---|---|---|---|
| Service supervision | systemd units, `systemctl` | Service Control Manager, `Get-Service`[^get-service] | Crash loops, failed units nobody notices |
| Logs | journald, rsyslog, `/var/log` | Event logs, `Get-WinEvent`[^winevent] | Logs fill the disk, or vanish on reboot |
| Scheduling | systemd timers, cron | Task Scheduler | Silent failure; jobs that never ran |
| Packages | apt or dnf and their history | Windows Update, installers | Pending reboots; failed boots |
| Users and privilege | Users, groups, sudo | Local groups, domain accounts | Shared administrator accounts |
| Remote access | OpenSSH, or an agent such as Session Manager | OpenSSH, PowerShell remoting | Key sprawl, open ports, shared accounts |
| Filesystems | ext4, XFS, LVM, mounts | NTFS, ReFS, volumes | Full blocks or inodes |
| Time | chrony or systemd-timesyncd | Windows Time service | Clock skew breaks TLS and authentication |

### Roles, ownership and RACI

R = responsible, A = accountable, C = consulted, I = informed.

| Activity | SysOps engineer | Service owner | Security (SecOps) | Platform or InfraOps team |
|---|---|---|---|---|
| Define the baseline | R (operational settings) | C | A (security settings) | C |
| Apply and verify the baseline | R/A | I | C | C (images) |
| Diagnose host problems | R/A | C | I | C |
| Grant privileged access | R (implements) | A (approves need) | C | I |
| Host-level backups | R | A (decides scope) | I | C |
| Capacity and headroom | R | A | I | C |
| Decommission a host | R | A | C (wipe, revoke) | C |

In small teams one person holds every column; the RACI still says who **approves** a privilege grant or a deletion. Incident
command belongs to [IncidentOps](../incidentops/), access approval workflows to [AccessOps](../accessops/).

### Lifecycle of a host

::: figure The life of a host from request to retirement
```mermaid
stateDiagram-v2
  state "Requested" as Req
  state "Provisioned" as Prov
  state "Baselined" as Base
  state "In service" as Live
  state "Changing" as Chg
  state "Degraded" as Deg
  state "Retired" as Ret
  [*] --> Req
  Req --> Prov: image and size chosen
  Prov --> Base: config, access, agents
  Base --> Live: readiness checks pass
  Live --> Chg: change
  Chg --> Live: verified
  Chg --> Deg: fails
  Live --> Deg: alert
  Deg --> Live: repaired or rolled back
  Deg --> Prov: replaced
  Live --> Ret: end of life
  Ret --> [*]
```
Text equivalent: A host is requested, provisioned from a chosen image and size, baselined (configuration, access, agents) and put
in service when readiness checks pass. A verified change returns it to service; a failed change or an alert leaves it degraded. A
degraded host is repaired or rolled back in place, or replaced by a new host from the image. At end of life it is retired.
:::

- **Day 0 (design).** Choose the OS release for its support window. Ubuntu LTS releases get "5 years of standard security
  maintenance", 10 with ESM and up to 15 with the Legacy add-on through Ubuntu Pro (as of October 2026);[^ubuntu] Windows Server LTSC
  releases get "five years of mainstream support and five years of extended support".[^win-servicing]
- **Day 1 (build).** Provision from an image, apply the baseline, enrol in monitoring, logging, backup and access control, and run
  the readiness checklist ([Checklists](#checklists)).
- **Day 2 (operate).** Triage, capacity, scheduled jobs, reboots after updates, drift correction.
- **Retire.** Follow the decommissioning checklist ([Checklists](#checklists)).

### Operating model and governance

| Cadence | Activity | Output |
|---|---|---|
| Daily | Failed units and jobs, pending reboots, host alerts | Tickets, or "none" |
| Weekly | Headroom and drift report | Hosts at risk, with days-to-full |
| Monthly | Sample a restore; review privileged access, including IAM roles that reach hosts; patch status to PatchOps | Audit evidence |
| Quarterly | Baseline review with SecOps; console drill; OS end-of-life forecast; toil measure | Baseline version, drill record, EOL plan |

Host changes follow the change process ([ITOps](../itops/)); make routine, low-risk ones (a sudo rule from an approved request)
pre-approved standard changes so nobody skips it.

## Architecture and workflow

### Reference architecture: one well-run host

::: figure Reference architecture of one well-run host, with its trust boundaries
```mermaid
flowchart TB
  subgraph EXT["Outside the host"]
    OP["Operator: SSH, agent or console"]:::risk
    CM["Config tool: baseline"]:::neutral
    TS["Time sources"]:::neutral
  end
  subgraph HOST["The host"]
    SV["systemd: services, timers"]:::core
    A["Applications"]:::neutral
    AG["Log agent, backup timer"]:::neutral
    KR["Kernel: cgroups, firewall, MAC"]:::core
  end
  subgraph OFF["Off-host stores"]
    TEL["Logs and metrics"]:::neutral
    BK["Backups"]:::neutral
  end
  OP -->|"named login, sudo"| SV
  CM -->|"apply"| HOST
  TS -->|"chrony"| KR
  SV -->|"runs"| A
  SV -->|"runs"| AG
  SV --> KR
  AG -->|"ship"| TEL
  AG -->|"copy"| BK
  KR --> HW["Hardware or hypervisor"]:::neutral
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: Three zones. Outside: the operator (SSH, agent or console, then sudo), marked as the risk because every way in
reaches root; the configuration tool applying the baseline; time sources. Inside: systemd runs the applications, a log agent and a
backup timer, which ship logs and metrics and copy backups off-host, on a kernel with cgroups, firewall and MAC whose clock chrony
disciplines. Below: hardware or a hypervisor.
:::

| Component | Trust boundary and trade-off |
|---|---|
| Access path | Every path reaches root: SSH keys, agent sessions and consoles ([Deep comparisons](#deep-comparisons)) |
| systemd | Unit files are configuration: version them |
| Kernel | cgroup limits protect neighbours but cause throttling and OOM kills |
| Telemetry | Local-only logs die with the host ([ObservabilityOps](../observabilityops/)) |
| Backups | A copy on the same disk is not a backup ([BackupOps](../backupops/)) |

### End-to-end workflow: from alert to explanation

::: figure Host triage workflow, including the unreachable-host path
```mermaid
flowchart TB
  A["Alert or complaint"]:::neutral --> R{"Reachable?"}:::decision
  R -->|no| OOB["Console or status checks"]:::risk
  OOB --> PF["Replace on platform fault; else fix boot, network, sshd"]:::neutral
  R -->|yes| SN["Snapshot: load, logs, units"]:::core
  SN --> ER{"Errors?"}:::decision
  ER -->|yes| FE["Follow the error"]:::core
  ER -->|no| SA{"Saturation?"}:::decision
  SA -->|yes| NM["Name resource, consumer"]:::core
  SA -->|no| UT["Utilisation, then app"]:::neutral
  FE --> MI["Mitigate, verify, record"]:::core
  NM --> MI
  UT --> MI
  PF --> MI
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: An alert arrives. If the host is unreachable, use the console or provider status checks: replace the host on a
platform fault, otherwise fix boot, network or sshd from the console. If reachable, take a snapshot (load, kernel log, failed units).
Follow any error; otherwise name the saturated resource and its consumer; failing that, check utilisation, then the application.
Every branch ends with mitigate, verify and record.
:::

- **Trigger and actors:** an alert, report or failed job; the on-call engineer, the service owner for business decisions, and an
  incident commander once more than one host is affected ([IncidentOps](../incidentops/)).
- **Exception path:** on AWS, a failed *system* status check needs AWS (stopping and starting an EBS-backed instance "in most cases
  results in the instance being migrated to a new host"); a failed *instance* check is yours.[^ec2-status]
- **Output:** a triage note ([Templates](#templates)) naming resource, consumer, evidence and fix.

## Practices, methods and tools

### Core practices and playbooks

| Practice | Output | Verification |
|---|---|---|
| Supervise services | Units with `Restart=on-failure`[^sd-service] | `systemctl list-units --state=failed` is empty |
| Schedule jobs | Timers with `OnFailure=` | `systemctl list-timers` shows next and last runs |
| Manage logs | journald and logrotate settings | `journalctl --disk-usage` under the cap |
| Control access | sudoers drop-ins, key-only SSH | `visudo -c`[^visudo] and `sshd -t`[^sshd] pass |
| Keep time | chrony configuration | `chronyc tracking` synchronised, small offset |
| Manage filesystems | Alerts on blocks and inodes | `df -h` and `df -i` below thresholds |
| Back up host state | Archive job, off-host copy | A sampled restore succeeds |
| Apply baselines | Scan report, documented exceptions | Pass rate tracked |

Five rules:

- **Prefer timers to cron for new jobs.** Under cron, SHELL is `/bin/sh`, an unescaped `%` in a command is "changed into newline
  characters", and output is mailed to `MAILTO` or the crontab owner,[^crontab] often nowhere. A timer logs to the journal,
  `Persistent=true` catches up missed runs, and `RandomizedDelaySec=` spreads jobs "to reduce workload spikes";[^sd-timer]
  `OnFailure=` starts another unit when the job's unit enters the "failed" state:[^sd-unit] point it at your alerting (Lab 2).
- **Cap and keep logs on purpose.** With `Storage=auto`, journald keeps logs on disk only if `/var/log/journal` exists; `auto` was
  the default in systemd 245, but current releases set it at build time,[^journald] so set it explicitly. Default caps: 10% of the
  filesystem used, 15% kept free, each at most 4 GiB. logrotate's `copytruncate` leaves "a very small time slice"
  in which "some logging data might be lost".[^logrotate]
- **Change SSH safely.** For most keywords sshd uses "the first obtained value", reading `Include` files in lexical order, but
  `AllowGroups`, `AllowUsers`, `DenyGroups` and `DenyUsers` append.[^sshd-config] Test with `sshd -t`, check `sshd -T`,[^sshd] and
  keep a second session open.
- **Never step a clock by hand.** chrony slews time; `makestep` steps it only above a threshold, in the first updates after
  start.[^chrony]
- **Own accounts and files on purpose.** Run each service as a system account (`useradd --system --shell /usr/sbin/nologin`) that
  alone can write its data directory (`install -d -o myapp -g myapp -m 0750 /var/lib/myapp`). For a leaver, `usermod -L` disables
  only the password: lock the account by also expiring it (`usermod -L -e 1 <user>`)[^usermod] and remove their SSH keys. File
  modes, umask and ACLs are prerequisites (LFCS tests ACLs[^lfcs]).

### Packages, reboots and the next boot

Patch programmes belong to [PatchOps](../patchops/); on one host, three questions matter.

- **What changed?** `dnf history undo <id>` reverses a transaction unless the RPM database state prevents it.[^dnf-history] apt
  has no undo: `/var/log/dpkg.log` records changes, and the way back is a named older version (`apt install pkg=version`)[^apt]
  while a repository still has it.
- **Does anything still run old code?** After a library update, running services keep the old version until they
  restart.[^ubuntu-auto] On the RHEL family, `dnf needs-restarting -r` reports "whether a reboot is required (exit code 1) or not
  (exit code 0)" and `-s` lists the services.[^dnf-nr] On Ubuntu, `/var/run/reboot-required` marks a pending reboot, and from 24.04
  LTS needrestart restarts affected services automatically.[^ubuntu-auto]
- **Will it boot?** "Errors in the /etc/fstab file can render a system unbootable." Mount data volumes by UUID with `nofail`, and test
  with `mount -a` before any reboot;[^ebs-fstab] `findmnt --verify` checks the whole file.[^findmnt]

Unattended updates are a baseline decision. Ubuntu "will apply security updates automatically", but unattended-upgrades'
`Automatic-Reboot` defaults to false,[^ubuntu-auto] so reboots pile up unless scheduled. Choose per role (automatic security updates
plus a reboot window, or only the patch programme) and alert on reboots pending beyond policy.

_Example status: Debian/Ubuntu lines and `findmnt --verify` executed read-only on Ubuntu 20.04 (2026-10-17); dnf lines and `mount -a`
not executed (no RHEL host; `mount -a` changes mounts)._

```bash
# Debian/Ubuntu: pending reboot, recent package changes, nightly runs
[ -f /var/run/reboot-required ] && echo "reboot pending"
grep -E ' (install|upgrade) ' /var/log/dpkg.log | tail -n 5
ls /var/log/unattended-upgrades/
# RHEL family (dnf-plugins-core): exit 1 = reboot needed
sudo dnf needs-restarting -r; echo "exit $?"
sudo dnf needs-restarting -s                  # services still on old files
sudo dnf history list | head -n 5
# Any family, after editing /etc/fstab and before rebooting
sudo findmnt --verify && sudo mount -a
```

### Applying a hardening baseline on one host

[SecOps](../secops/) chooses the benchmark; applying it to servers is host work. Three controls need the most care.

- **Host firewall: default deny inbound.** Allow the management path and the service's ports, and change rules remotely only with
  a way back that runs unless cancelled. In firewalld, `--reload` restores the permanent configuration, dropping untested runtime
  changes, and `--runtime-to-permanent` keeps them;[^firewalld] `systemd-run --on-active=` can schedule the reload.[^sd-run]
- **Mandatory access control stays enforcing.** SELinux (RHEL family) and AppArmor (loaded by default on Ubuntu[^apparmor]) confine
  services beyond file permissions. Permissive SELinux permits accesses "that would be denied by policy" and only audits
  them,[^selinux] so putting the whole host in permissive mode for one application unconfines every service. Read the denial
  (`audit2why` for SELinux,[^audit2why] `apparmor="DENIED"` lines for AppArmor[^apparmor]), then fix the label (`restorecon`), a
  boolean (`setsebool -P`) or the profile. If a stop-gap is unavoidable, relax one SELinux domain
  (`semanage permissive -a <domain>`)[^semanage-perm] or one AppArmor profile (`aa-complain`),[^apparmor] recorded with SecOps as an
  expiring exception.
- **SSH keys have a lifecycle.** Inventory every key file and command sshd uses, remove leavers' keys, and list revoked keys in
  `RevokedKeys`, replaced atomically: if sshd cannot read it, it refuses every public-key login.[^sshd-config] At scale,
  short-lived OpenSSH user certificates replace key lists ([Deep comparisons](#deep-comparisons)); central identity belongs to
  [IdentityOps](../identityops/).

Rescan (for example with CIS-CAT[^cis]) after each baseline change; fix each failure or record an exception with owner and expiry.

_Example status: syntax-checked; not executed (needs root and firewalld). The transient-timer mechanism (fire, and cancel by stopping
the timer) was tested with `systemd-run --user` (systemd 245), and the key loop run read-only on Ubuntu 20.04 (2026-10-17)._

```bash
# Firewall change with an automatic way back (test host first)
sudo systemd-run --unit=fw-undo --on-active=5min firewall-cmd --reload
svc=cockpit                                   # example: a service you do not offer
sudo firewall-cmd --remove-service="$svc"     # runtime only
# Open a NEW SSH session. If it works, keep the change and cancel the undo:
sudo firewall-cmd --runtime-to-permanent && sudo systemctl stop fw-undo.timer
# Mandatory access control: mode, and why something was denied
getenforce 2>/dev/null || sudo aa-status | head -n 3
sudo audit2why -a 2>/dev/null | tail -n 20
# Keys: where sshd reads them. `systemctl cat ssh` shows -o options that
# sshd -T misses (Ubuntu's ec2-instance-connect adds an AuthorizedKeysCommand).
sudo sshd -T | grep -E '^authorizedkeys(file|command) '
akf=$(sudo sshd -T | awk '$1 == "authorizedkeysfile" { $1 = ""; print }')
# Key entries (not comments or blank lines) per file, for every account's home
getent passwd | while IFS=: read -r user _ uid _ _ home _; do
  for f in $akf; do
    f=${f//"%h"/$home}; f=${f//"%u"/$user}; f=${f//"%U"/$uid}
    [ "${f#/}" = "$f" ] && f="$home/$f"          # relative paths start at home
    sudo test -f "$f" && echo "$user $f $(sudo grep -cvE '^[[:space:]]*(#|$)' "$f")"
  done
done
```

### The diagnostic method: USE and the first 60 seconds

Brendan Gregg's USE method (industry practice, not a standard): "For every resource, check utilization, saturation, and errors."
Utilisation is "the average time that the resource was busy servicing work"; saturation is "the degree to which the resource has
extra work which it can't service, often queued".[^use] Netflix's performance team published ten commands for a host's first minute (`uptime`, `dmesg | tail`, `vmstat 1`,
`mpstat -P ALL 1`, `pidstat 1`, `iostat -xz 1`, `free -m`, `sar -n DEV 1`, `sar -n TCP,ETCP 1`, `top`; several need sysstat) and
the reading order: "Look for errors and saturation metrics, as they are both easy to interpret, and then resource utilization."[^netflix60]

| Resource | Utilisation | Saturation | Errors |
|---|---|---|---|
| CPU | `mpstat -P ALL 1` (100 minus idle) | `vmstat 1` column `r` above the CPU count;[^use-linux] `/proc/pressure/cpu` | Rare: machine-check messages |
| Memory | `free -m`, column `available` | `vmstat 1` `si`/`so` (swapping); `/proc/pressure/memory` | OOM kills: `journalctl -k \| grep -i oom` |
| Disk I/O | `iostat -xz 1` `%util` (see trap 3) | `aqu-sz`, `await`; `/proc/pressure/io`; tasks in D state | I/O errors in the kernel log |
| Disk capacity | `df -h`, `df -i` | Not meaningful: when full, writes fail | `ENOSPC` ("No space left on device") |
| Network | `sar -n DEV 1` against link speed | Drops, overruns, TCP retransmits (`sar -n TCP,ETCP 1`) | `ip -s link` errors |
| Software limits | Open files, processes, cgroup limits | `nr_throttled` in the service cgroup's `cpu.stat` (cpu controller on)[^cgroup2] | `EMFILE` ("Too many open files") |

Five traps:

1. **Load average is not CPU load.** It counts runnable tasks and "tasks in the uninterruptible sleep state", which "includes tasks
   blocked on disk I/O and some locks".[^loadavg] High load with idle CPUs points at storage, NFS or locks.
2. **"Free" memory is the wrong number.** `MemAvailable` is "An estimate of how much memory is available for starting new
   applications, without swapping".[^proc] Modern `free` computes `used` as total minus available;[^free] a small `free` column is
   usually page cache.
3. **`%util` lies about SSDs and arrays.** For "devices serving requests in parallel, such as RAID arrays and modern SSDs, this number
   does not reflect their performance limits";[^iostat] read queue size and latency.
4. **A VM can be slow for reasons outside it.** `vmstat` column `st` is "Time stolen from a virtual machine";[^vmstat] on EC2,
   CloudWatch `CPUUtilization` includes hypervisor time.[^ec2-metrics] Burstable T instances run on CPU credits. Out of credits,
   Standard mode drops to a baseline; Unlimited mode keeps bursting and bills you if the 24-hour average exceeds the
   baseline.[^ec2-credits] As of October 2026, T2 launches in Standard mode and T3, T3a, T4g and T8i in Unlimited (T3 on a
   Dedicated Host: Standard only), unless the account default was changed.[^ec2-credit-cfg]
5. **Averages hide bursts.** Gregg describes CPU saturation while monitoring, using five-minute averages, "showed CPU utilization
   was never higher than 80%".[^use] PSI averages over 10, 60 and 300 seconds: "some" is the share of time at least some tasks
   stalled, "full" the share in which all non-idle tasks stalled at once.[^psi]

**Windows Server.** Same method, different tools: `Get-Counter` for processor, memory and disk counters,[^get-counter]
`Get-WinEvent -FilterHashtable` for errors (faster than `Where-Object`; `Level=2` selects errors)[^winevent] and `Get-Service` for
services.[^get-service]

### Tooling landscape

| Job | Representative tools | Selection notes |
|---|---|---|
| Supervision and logs | systemd and journald; Windows Service Control Manager and event logs | Built in; learn them before agents |
| Live triage | procps (`ps`, `vmstat`, `free`), sysstat (`iostat`, `mpstat`, `pidstat`, `sar`), `ss`, `lsof` | Install sysstat in advance; enable `sar` collection for history |
| Host metrics history | Metrics agents; on EC2 the CloudWatch agent provides `mem_used_percent` and `disk_used_percent`[^cw-agent] | The default EC2 metric list has no memory metric[^ec2-metrics] |
| Remote access | OpenSSH; AWS Systems Manager Session Manager[^ssm-session] | An agent removes open ports and keys but makes provider IAM your root control ([Deep comparisons](#deep-comparisons)) |
| Fleet commands and configuration | Ansible; SSM Run Command[^ssm-walk] | Run Command is root-equivalent; governance in [AutomationOps](../automationops/) |
| Baseline assessment | CIS-CAT Pro Assessor[^cis] and similar scanners | Scan results need an exception process |

Idempotent tools, whose repeated runs leave the same result as one run,[^ansible] make a baseline safe to re-apply every hour.

### Automation and AI

Automate toil first: work that is manual, repetitive and "scales linearly as a service grows",[^sre-toil] such as baseline
application, housekeeping and a triage snapshot on alert. Keep humans for decisions that destroy data or evidence. AI assistants
explain output and draft commands but invent flags, and pasted logs may hold secrets or personal data: review anything run as root.
Cross-host event correlation belongs to [AIOps](../aiops/).

## SysOps in practice

### Reference implementation: a baseline host kit

**Assumptions:** a Debian/Ubuntu or RHEL-family server with systemd, OpenSSH and chrony; administrators in group `ops-admins`; an
application `myapp` logging to `/var/log/myapp`. Deliver the files through configuration management ([InfraOps](../infraops/)).

_Example status: directives checked against upstream man pages on 2026-10-17; not executed as a set. Validate on a test host first._

```text
# /etc/ssh/sshd_config.d/00-baseline.conf
# "00-" sorts first. For most keywords sshd keeps the FIRST value it reads,
# but AllowGroups/AllowUsers/DenyGroups/DenyUsers lines anywhere add up.
# The main sshd_config must Include this directory before its own settings.
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
AllowGroups ssh-users ops-admins
```

```text
# /etc/sudoers.d/ops-admins   (no dot in the file name: sudo skips such files)
# Install: sudo install -m 0440 -o root -g root ops-admins /etc/sudoers.d/
# Members of ops-admins may run commands as root after entering their own password.
%ops-admins ALL=(ALL:ALL) ALL
```

sudo reads `/etc/sudoers.d` only through an `@includedir` (or older `#includedir`) line, and skips file names that "contain a '.'
character", so `ops-admins.conf` would be ignored silently; the default mode is 0440, and the file "must not be
world-writable".[^sudoers] `visudo -cf <file>` checks only syntax; `visudo -c` also checks ownership and permissions.[^visudo] Each
administrator needs a password: issue it from central identity ([IdentityOps](../identityops/)) so it is revoked with the account.

```ini
# /etc/systemd/journald.conf.d/10-retention.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=4G
```

```text
# /etc/chrony/chrony.conf (Debian/Ubuntu) or /etc/chrony.conf (RHEL family): excerpt
# Use either leap-smearing sources only or non-smearing sources only.
# On EC2, replace the pool line with the local Amazon Time Sync Service:
#   server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4
pool time.example.com iburst      # your organisation's (non-smearing) servers
makestep 1.0 3
leapsecmode slew                  # default "system": the kernel steps back 1 s
rtcsync
```

```text
# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    su myapp myapp
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    sharedscripts
    postrotate
        systemctl reload myapp.service >/dev/null 2>&1 || true
    endscript
}
```

`su` makes logrotate act as the owner, as recommended for directories "in control of non-privileged users"; `sharedscripts` runs
the reload once, not once per file.[^logrotate]

_Example status: syntax-checked with `bash -n`; guard tested with stubs; not executed. Requires root on a test host._

```bash
# Validate before reloading anything; keep a second SSH session open.
sudo grep -iE '^(Include|@includedir|#includedir)' /etc/ssh/sshd_config /etc/sudoers
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|allowgroups) '
sudo visudo -c                                # whole tree, ownership and modes
sudo systemctl restart systemd-journald && journalctl --disk-usage
chronyc tracking | grep -E '^(Reference ID|System time|Leap status)'
sudo logrotate --debug /etc/logrotate.d/myapp   # dry run: no changes are made
# Reload sshd only if its config is valid AND you are in an allowed group.
in_group() { id -nG "$USER" | tr ' ' '\n' | grep -qxE 'ssh-users|ops-admins'; }
if sudo sshd -t && in_group; then
  # The unit is "ssh" on Debian/Ubuntu and "sshd" on the RHEL family.
  sudo systemctl reload ssh 2>/dev/null || sudo systemctl reload sshd
else
  echo "STOP: sshd config invalid, or you are in neither group (lock-out)"
fi
```

On EC2, AWS recommends its local Time Sync Service, which smears leap seconds, and advises against mixing
smeared and non-smeared sources during one.[^aws-time][^aws-ntp]

### Hands-on labs

Use a disposable VM for anything needing root.

#### Lab 1 (beginner): a first-minutes triage snapshot

**Objective:** collect the errors and saturation signals of [the diagnostic method](#the-diagnostic-method-use-and-the-first-60-seconds) in one read-only pass.
**Prerequisites:** a Linux host with systemd, procps and sysstat; 20 minutes.
**Cost and risk:** the script changes nothing; step 3 loads the CPUs for a minute.

1. Save the script as `triage.sh`; run `bash triage.sh | tee triage-$(date +%F-%H%M).txt`.
2. Run `sudo bash triage.sh` and compare.
3. On a disposable VM, start twice as many busy loops as CPUs
   (`for i in $(seq $((2 * $(nproc)))); do yes > /dev/null & done`), wait a minute, rerun the script, then `pkill yes`. On a
   shared host, confine it: two `taskset -c 0 nice -n 19 yes > /dev/null &` loops, then `taskset -c 0 bash triage.sh`.
4. For each `CHECK` line, name its USE cell (resource; utilisation, saturation or error).

_Example status: verified by execution as an unprivileged user (Ubuntu 20.04, kernel 5.15, systemd 245, 2026-10-17), including the
confined step 3; output not reproduced (it describes a production host)._

```bash
#!/usr/bin/env bash
# triage.sh: read-only first-minutes snapshot of one Linux host (USE-style).
# Changes nothing. Run with sudo to see all open files and the kernel log.
set -uo pipefail
warn() { printf '  CHECK: %s\n' "$*"; }
section() { printf '\n== %s ==\n' "$*"; }
fsx=(-x tmpfs -x devtmpfs -x squashfs -x overlay)   # skip pseudo filesystems

ncpu=$(nproc)
section "Load and CPU (cpus=${ncpu})"
read -r l1 l5 l15 _ < /proc/loadavg
echo "  load 1/5/15 min: ${l1} ${l5} ${l15}"
if awk -v l="$l1" -v n="$ncpu" 'BEGIN { exit !(l > n) }'; then
  warn "1-min load above CPU count (load also counts tasks in D state)"
fi
vmstat 1 5 | tail -n 4 | awk -v n="$ncpu" '
  { r += $1; si += $7; so += $8; wa += $16; st += $17; c++ }
  END {
    printf "  vmstat avg: r=%.1f si=%.0f so=%.0f wa=%.0f%% st=%.0f%%\n",
      r/c, si/c, so/c, wa/c, st/c
    if (r/c > n) print "  CHECK: run queue above CPU count (CPU saturation)"
    if (si/c + so/c > 0) print "  CHECK: swapping in or out (memory saturation)"
    if (st/c > 10) print "  CHECK: steal above 10% (hypervisor contention)"
  }'

section "Pressure stall information"
if head -n 1 /proc/pressure/cpu > /dev/null 2>&1; then   # a read, not just -r
  for r in cpu memory io; do
    printf '  %-6s %s\n' "$r" "$(head -n 1 "/proc/pressure/$r")"
  done
else
  echo "  PSI not readable: not built into this kernel, or disabled at boot"
fi

section "Memory"
awk '/^MemTotal:/ {t=$2} /^MemAvailable:/ {a=$2} /^SwapTotal:/ {st=$2}
     /^SwapFree:/ {sf=$2}
  END {
    printf "  MemAvailable %.0f%% of %.1f GiB; swap used %.0f MiB\n",
      100*a/t, t/1048576, (st-sf)/1024
    if (a/t < 0.10) print "  CHECK: MemAvailable below 10%"
  }' /proc/meminfo
journalctl -k -b --no-pager -q 2>/dev/null | grep -i -E 'out of memory|oom-kill' |
  tail -n 3 | sed 's/^/  /'

section "Filesystems (blocks and inodes)"
df -P "${fsx[@]}" | awk 'NR > 1 { u = $5; sub("%", "", u)
  if (u+0 >= 85) printf "  CHECK: %s at %s%% blocks\n", $6, u }'
df -P -i "${fsx[@]}" | awk 'NR > 1 && $5 != "-" { u = $5; sub("%", "", u)
  if (u+0 >= 85) printf "  CHECK: %s at %s%% inodes\n", $6, u }'
df -h "${fsx[@]}" | sed 's/^/  /'
deleted=$(find /proc/[0-9]*/fd -lname '*(deleted)' 2>/dev/null | wc -l)
echo "  open-but-deleted file descriptors visible to this user: ${deleted}"

section "Services and recent errors"
failed_units() {
  systemctl list-units --state=failed --no-legend --plain 2>/dev/null
}
failed=$(failed_units | wc -l)
echo "  failed units: ${failed}"
failed_units | sed 's/^/  /'
journalctl -p err -b --since "-1h" --no-pager -q -n 10 2>/dev/null | sed 's/^/  /'

section "Top processes"
ps -eo pid,user,pcpu,pmem,stat,comm --sort=-pcpu | head -n 6 | sed 's/^/  /'
dstate=$(ps -eo stat= | grep -c '^D')
[ "$dstate" -gt 0 ] && warn "${dstate} task(s) in D state: usually waiting on I/O"

section "Network and time"
ss -s | head -n 2 | sed 's/^/  /'
timedatectl show -p NTPSynchronized --value 2>/dev/null |
  sed 's/^/  NTP synchronised: /'
exit 0
```

**Expected output (shape):** seven sections, from `== Load and CPU (cpus=4) ==` to `== Network and time ==`, with `CHECK:` lines
where a threshold is crossed; after step 3, `CHECK: run queue above CPU count (CPU saturation)`, usually the load check, and a much
higher `cpu some` avg10.
**Validation:** `echo $?` prints `0`; every section header appears; step 3 produces the run-queue check.
**Troubleshooting:** `vmstat: command not found` → install procps; "PSI not readable" → no PSI in the kernel, or disabled at boot
(some RHEL-family kernels): add `psi=1` to the kernel command line;[^psi-boot] no journal lines → join `systemd-journal` or `adm`, or
use sudo; odd `wa`/`st` values → check column positions in `vmstat 1 1`.
**Teardown:** `pkill yes`; delete saved output that names hosts.

#### Lab 2 (intermediate): replace a cron job with a sandboxed timer

**Objective:** move a nightly job to a systemd timer that logs, catches up, spreads load, alerts on failure and runs with least
privilege; then prove a restore.
**Prerequisites:** root on a disposable VM with systemd 245 or later; 45 minutes.
**Cost and risk:** low; the job writes only to `/var/backups/etc`. The archive is not a backup until copied off-host
([BackupOps](../backupops/)).

_Example status: script run 17 times against scratch directories (14 archives kept) and the restore run there without `sudo`; the
three units pass `systemd-analyze verify` (systemd 245); not installed or started (2026-10-17)._

```bash
#!/usr/bin/env bash
# /usr/local/sbin/backup-etc.sh: keep 14 dated archives of /etc on this host.
set -euo pipefail
dest=/var/backups/etc
keep=14
stamp=$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$dest"
tar --create --gzip --file "$dest/etc-$stamp.tar.gz" --directory / etc
# Read the archive back: fails loudly if it is unreadable.
tar --list --gzip --file "$dest/etc-$stamp.tar.gz" > /dev/null
ls -1t "$dest"/etc-*.tar.gz | tail -n +$((keep + 1)) | xargs -r rm --
count=$(ls -1 "$dest"/etc-*.tar.gz | wc -l)
echo "backup-etc: wrote $dest/etc-$stamp.tar.gz; $count archives kept"
```

```ini
# /etc/systemd/system/backup-etc.service
[Unit]
Description=Archive /etc to /var/backups/etc
OnFailure=notify-failure@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup-etc.sh
Nice=10
IOSchedulingClass=idle
MemoryMax=512M
ProtectSystem=strict
ReadWritePaths=/var/backups/etc
PrivateTmp=yes
NoNewPrivileges=yes
ProtectHome=read-only
```

```ini
# /etc/systemd/system/notify-failure@.service
# Logs an error-priority journal line; point ExecStart at your pager instead.
[Unit]
Description=Report the failure of %i

[Service]
Type=oneshot
ExecStart=/usr/bin/logger -p user.err "unit %i failed"
```

```ini
# /etc/systemd/system/backup-etc.timer
[Unit]
Description=Nightly /etc archive

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=30min
Persistent=true

[Install]
WantedBy=timers.target
```

1. Install: `sudo install -m 0755 backup-etc.sh /usr/local/sbin/`, then
   `sudo install -m 0644 backup-etc.service notify-failure@.service backup-etc.timer /etc/systemd/system/`, then the root-only
   directory: `sudo install -d -m 0700 /var/backups/etc`.
2. Check: `sudo systemd-analyze verify /etc/systemd/system/backup-etc.{service,timer}`; ignore notes about other units.
3. Enable: `sudo systemctl daemon-reload && sudo systemctl enable --now backup-etc.timer`.
4. Run once: `sudo systemctl start backup-etc.service`; read `journalctl -u backup-etc.service -n 5 --no-pager`.
5. Restore one file and compare it (block below).
6. Score the sandbox: `systemd-analyze security backup-etc.service` reports an exposure level from 0.0 to 10.0 (lower is
   tighter).[^sd-analyze] Remove `ProtectSystem=strict`, reload, and compare.
7. Comment out the old crontab line and record the change.

```bash
latest=$(sudo sh -c 'ls -1t /var/backups/etc/etc-*.tar.gz | head -n 1')
scratch=$(mktemp -d)
sudo tar -xzf "$latest" -C "$scratch" etc/hostname
sudo diff "$scratch/etc/hostname" /etc/hostname && echo "restore OK: $latest"
sudo rm -r -- "$scratch"
```

**What the settings do:** `ProtectSystem=strict` makes the filesystem read-only except `/dev`, `/proc` and `/sys`, and
`ReadWritePaths=` re-opens only the backup directory, which must exist (step 1); `NoNewPrivileges=` blocks privilege gain through
setuid binaries; `PrivateTmp=` gives a private `/tmp`.[^sd-exec] Above `MemoryMax=`, the OOM killer runs inside the
unit.[^sd-resource] The job still runs as root and reads private keys in `/etc`, hence the 0700 directory.
**Expected output:** `systemctl list-timers backup-etc.timer` shows a NEXT time between 02:30 and 03:00; the journal shows
`backup-etc: wrote /var/backups/etc/etc-<timestamp>.tar.gz; 1 archives kept`; step 5 prints `restore OK`.
**Validation:** `systemctl show backup-etc.service -p Result` prints `Result=success`. To test the alert, add `exit 1` as the
script's second line, start the service, and find `unit backup-etc.service failed` in `journalctl -p err -n 5`.
**Troubleshooting:** `status=226/NAMESPACE` → `/var/backups/etc` missing; `Read-only file system` → path not in `ReadWritePaths=`;
`status=203/EXEC` → script not executable or wrong path; timer never fires → `systemctl is-enabled backup-etc.timer`.
**Teardown:** `sudo systemctl disable --now backup-etc.timer`, remove the three unit files and the script, run
`sudo systemctl daemon-reload`, and delete `/var/backups/etc` if unneeded.

#### Lab 3 (advanced): the space that `rm` did not free

**Objective:** see why deleting a log that a process holds open frees nothing, and recover the space without a restart.
**Prerequisites:** a Linux host with GNU coreutils; 200 MiB free in `$TMPDIR`; 15 minutes.
**Cost and risk:** none for the main part (own temporary directory).

_Example status: main part verified by execution as an unprivileged user (2026-10-17); optional part syntax-checked, not executed._

```bash
#!/usr/bin/env bash
# lab3.sh: deleting a file that a process still holds open frees no space.
set -euo pipefail
avail() { df -m --output=avail . | tail -n 1; }
work=$(mktemp -d "${TMPDIR:-/tmp}/sysops-lab3.XXXXXX")
cd "$work"
head -c 200M /dev/zero > app.log            # a 200 MiB "log"
tail -f app.log > /dev/null 2>&1 &           # a process that keeps the file open
holder=$!
sleep 1
echo "before rm:  $(du -sm . | cut -f1) MiB in dir; fs avail $(avail) MiB"
rm app.log
echo "after rm:   $(du -sm . | cut -f1) MiB in dir; fs avail $(avail) MiB"
fd=$(find "/proc/$holder/fd" -lname '*app.log (deleted)' | head -n 1)
echo "still open: $fd -> $(readlink "$fd")"
: > "$fd"                                    # truncate via the descriptor
echo "truncated:  fs avail $(avail) MiB"
kill "$holder"
cd / && rmdir "$work"
```

**Expected output (shape):** `before rm` shows about 201 MiB in the directory; `after rm` about 1 MiB but **the same** free space;
`still open` prints `/proc/<pid>/fd/<n> -> …/app.log (deleted)`; `truncated` shows about 200 MiB more free space.
**Validation:** the three free-space figures behave as described. On real hosts, `lsof +L1` selects "open files that have been
unlinked".[^lsof]
**Troubleshooting:** `df: unrecognized option '--output=avail'` → non-GNU or old coreutils; read `df -m .` instead.

**Optional (root, disposable VM only): inode exhaustion.** `mkfs.ext4 -N` sets the number of inodes.

```bash
truncate -s 64M /root/tiny.img
mkfs.ext4 -q -N 2048 /root/tiny.img
mkdir -p /mnt/tiny && mount -o loop /root/tiny.img /mnt/tiny
mkdir /mnt/tiny/many
for i in $(seq 1 3000); do
  : > "/mnt/tiny/many/f$i" 2>/dev/null || { echo "write failed at file $i"; break; }
done
df -h /mnt/tiny; df -i /mnt/tiny      # blocks mostly free, IUse% 100%
umount /mnt/tiny && rm /root/tiny.img
```

**Teardown:** the main script removes its own directory; the optional part ends with `umount` and `rm`.

### Use cases and scenarios

**1. Cloudflare and the 2016 leap second (sourced).** At midnight UTC on 1 January 2017 the clock went back one second, and
Cloudflare's DNS software computed a negative duration and panicked. At peak "approximately 0.2% of DNS queries" were affected and
"less than 1% of all HTTP requests" saw errors; the root cause was "the belief that time cannot go backwards".[^cf-leap] *SysOps
lesson:* by default chrony lets the kernel step the clock back; `leapsecmode slew` corrects it over about 12 seconds,[^chrony] so
the kit sets it. Leap seconds are to be abandoned "by or before 2035",[^aws-time] but `makestep` and manual
changes still step clocks; monotonic clocks for durations are the application's fix. *Success metric:* no clock step outside boot.

**2. Recovering hosts that cannot boot: CrowdStrike, July 2024 (sourced).** Microsoft estimated that the faulty CrowdStrike update
"affected 8.5 million Windows devices".[^ms-crowd] Its first server guidance was manual: boot Hyper-V VMs from a Windows Server ISO,
or reach physical servers through the vendor's remote management interface (iLO, iDRAC, CIMC), then delete files matching
`C-00000291*.sys`. It later "released a USB tool to help automate this manual repair process",[^kb-server] which still boots each machine from recovery media.[^kb-tool] Client guidance warned "You may be asked to enter
your BitLocker recovery key".[^kb-client] *SysOps lesson:* consoles, recovery media and recovery keys must work **before** the outage;
no agent fixes a host that cannot boot. The EC2 serial console, for example, is off by default and on Linux needs a password-based
OS user, which AWS AMIs lack ([Deep comparisons](#deep-comparisons)).[^ec2-serial] Keep a break-glass account with a vaulted, rotated
password that cannot log in over SSH.

**3. "The new release made the API slow" (illustrative scenario).** Latency doubles a week after a release and the team starts a
rollback. Triage shows CPU pinned near the baseline: the instances are T2s in Standard credit mode, and `CPUCreditBalance` is zero
because rising traffic used up the credits.[^ec2-credits] *Method:* the USE table, then credit balance and specification.
*Trade-off:* Unlimited mode or a non-burstable instance costs more; the rollback fixes nothing. *Success metric:* page to named
resource in under 15 minutes.

## Measure, govern, pay

### Metrics and KPIs

| KPI | Formula | Data source | Pitfall |
|---|---|---|---|
| Baseline conformance | Hosts passing every check ÷ hosts in inventory | Scanner or config-management report | An incomplete inventory inflates it; reconcile with the platform's list |
| Monitoring coverage | Hosts sending metrics, logs and alerts ÷ hosts | Agent heartbeats | "Agent installed" is not "alert tested" |
| Scheduled-job success | Successful runs ÷ expected runs | Timer and journal data | Jobs that never ran raise no failure: compare each timer's LAST time with its schedule, or use a heartbeat |
| Disk runway | Space applications can still use ÷ average daily growth | `df` history | Growth is lumpy; use a 7- or 14-day window |
| Reboot debt | Hosts with a reboot pending longer than policy ÷ hosts | Reboot-required checks | A pending reboot means old code still runs |
| Time to named resource | Median minutes from page to a note naming the resource | Incident timeline | A skill indicator, not a target |
| Toil share | Manual, repetitive host work ÷ operations hours | Time tracking or ticket tags | Google SRE's 50% ceiling is its own goal[^sre-toil] |
| OS end-of-life exposure | Hosts past standard support ÷ hosts | Inventory plus vendor calendars[^ubuntu][^win-servicing] | Paid extended support still needs a migration date |

**Worked calculation (illustrative numbers).** A 500 GiB ext4 data volume holds 410 GiB and grew 63 GiB in 7 days: 9 GiB a day. The
default 5% reserve (25 GiB) is closed to non-root processes,[^tune2fs] so applications see a full disk at about 475 GiB: runway is
(475 − 410) ÷ 9 ≈ 7.2 days, not the 10 that size minus used suggests.

### Quality, security and compliance

- **Least privilege.** Named access on every path in ([Deep comparisons](#deep-comparisons)), and know who can reach credentials:
  in Lab 2, only root and the backup job can read archives holding every secret in `/etc`.
- **Baselines and standards.** Apply the benchmark [SecOps](../secops/) chose ([Applying a hardening baseline](#applying-a-hardening-baseline-on-one-host)).
  CIS Benchmarks are consensus guides[^cis] and NIST SP 800-123 (2008) lists server-security activities;[^nist-123] neither is
  mandatory in itself, but contracts, regulators or customers can make one so ([ComplianceOps](../complianceops/)).
- **Logs as evidence and personal data.** Plan retention with NIST SP 800-92 (2006; Rev. 1 is a 2023 draft);[^nist-92][^nist-92r1]
  logs hold user names and IP addresses, so agree retention with [PrivacyOps](../privacyops/). OpenSSH warns that "Logging with a
  DEBUG level violates the privacy of users".[^sshd-config]
- **Jurisdiction.** Retention and audit duties vary by country, sector and contract; this is not legal advice.

### Cost, capacity and sustainability

Host costs hide in idle capacity and people's time: "just in case" sizing, ownerless hosts, pets only one person can maintain.
Keep headroom for a burst plus the time to add capacity, no more. Rightsizing belongs to [FinOps](../finops/), carbon to
[GreenOps](../greenops/).

### Assurance of the operating model

| Risk indicator | Healthy signal | Check |
|---|---|---|
| Alerts that never fire | Each host alert tested quarterly | Synthetic test alert |
| Stale runbooks | Runbooks updated after each incident that used them | Post-incident review |
| Standing privilege | Sudoers, SSH keys, and IAM principals that can start sessions, send commands or open consoles | Access review ([AccessOps](../accessops/)) |
| Backups never restored | A sampled restore each month | Restore test log |
| Console access untested | Break-glass login works on sample hosts | Quarterly drill |

## When things go wrong

### Common failures and troubleshooting

| Symptom | Likely cause | Diagnose | Remediate and prevent |
|---|---|---|---|
| Disk full, but `du` finds little | Deleted files still open | `lsof +L1`[^lsof] | Truncate via `/proc/<pid>/fd/<n>` (logs only) or restart the holder; fix rotation |
| "No space left on device" with free blocks | Inodes exhausted | `df -i`[^df] | Remove small-file sprawl; alert on inode use |
| Root can write, applications cannot | ext4 reserved blocks (normally 5%) in use[^tune2fs] | `df` shows 100% | Free space; do not lower the reserve as a fix |
| High load, idle CPU | Tasks in D state waiting for disk or NFS[^loadavg] | `ps -eo stat,comm \| grep '^D'`, `iostat -xz 1` | Fix storage or mounts; set timeouts |
| A process disappears | OOM killer | `journalctl -k \| grep -i oom` | Add memory or set `MemoryHigh=`/`MemoryMax=`;[^sd-resource] alert on MemAvailable or PSI |
| "Too many open files" (`EMFILE`) | Descriptor leak, or systemd's default soft limit of 1024 (hard limit 524288)[^sd-system-conf] | `ls /proc/<pid>/fd \| wc -l` against Max open files in `/proc/<pid>/limits` | Leak: fix the code. Low limit: systemd marks `LimitNOFILE=` "Do not use." and expects programs to raise their own soft limit; for one that does not, a drop-in with `LimitNOFILE=65536:524288` raises only the soft limit, if the program never uses `select(2)`, which fails above descriptor 1023[^sd-exec] |
| Service keeps restarting | Crash loop | `systemctl status`, `journalctl -u <unit>` | Fix the cause; restart limits stop the loop |
| Nightly job "ran" but did nothing | cron environment, `%`, host was off | Journal, `systemctl list-timers` | Timer with `Persistent=true` and `OnFailure=`;[^sd-timer] job-success KPI |
| TLS or Kerberos errors on one host | Clock offset | `chronyc tracking` | Fix time sources; alert on offset |
| Cannot log in over SSH | sshd config error, full disk, firewall | Console or provider status checks[^ec2-status] | `sshd -t` before every reload; firewall changes with an automatic undo |
| Host does not come back after a reboot | `/etc/fstab` error, missing volume, kernel or boot change | Serial or out-of-band console | Fix from the console (below); `nofail` on data volumes; `findmnt --verify` before reboots[^ebs-fstab] |

**Repairing a Linux host that does not boot.** At the console, interrupt the boot loader (on EC2, set GRUB up for the serial
console beforehand[^ec2-grub]) and append `systemd.unit=rescue.target`
(all filesystems, basic services) or `systemd.unit=emergency.target` (a shell and nothing else) to the kernel line; a failed check
of a required filesystem lands in emergency mode by itself. There `/` may still be read-only: remount it
(`mount -o remount,rw /`).[^sd-special] Fix or comment out the bad `/etc/fstab` line, run `findmnt --verify`, and read why the
previous boot failed with `journalctl -b -1` (persistent journal only).[^journalctl] Both shells ask for the root password through
sulogin, which skips it for a locked root account only when forced,[^sulogin] so decide whether break-glass includes a vaulted
root password, and rehearse this path in the console drill.

Protecting a process with `oom_score_adj=-1000` is "equivalent to disabling OOM-killing entirely for that task",[^oom] so the kernel
kills something else; prefer memory limits on the noisy service.

### Worked runbook: a filesystem is filling

::: figure Runbook for a filling filesystem
```mermaid
flowchart TB
  A["Filesystem above 90%"]:::risk --> B["df -h and df -i"]:::core
  B --> C{"Blocks or inodes?"}:::decision
  C -->|inodes| D["Remove small-file sprawl"]:::core
  C -->|blocks| E{"du close to df?"}:::decision
  E -->|no| F["lsof +L1"]:::core
  F --> G["Truncate logs or restart holder"]:::core
  E -->|yes| H["du -x: largest paths"]:::core
  H --> I{"Safe to remove?"}:::decision
  I -->|yes| J["Remove, rotate or move"]:::core
  I -->|no| K["Extend volume with owner"]:::neutral
  D ----> L
  G --> L["Verify, fix cause, record"]:::core
  J --> L
  K --> L
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: A filesystem passes 90%. Inodes exhausted: remove small-file sprawl. Blocks full and du much smaller than df:
find deleted-but-open files with lsof +L1, then truncate logs or restart the holder. Blocks full and du agrees: find the largest paths
with du -x; remove, rotate or move them if safe, else extend the volume with the owner. Then verify, fix the cause, record.
:::

_Example status: commands syntax-checked; read-only except the marked step. Replace `/var` with the alerting mount._

```bash
M=/var                                        # the mount from the alert
df -h "$M"; df -i "$M"                        # 1. blocks or inodes?
sudo du -xh --max-depth=2 "$M" 2>/dev/null | sort -rh | head -n 15   # 2. where?
sudo lsof +L1 2>/dev/null | awk 'NR==1 || /deleted/'   # 3. deleted but still open?
journalctl --disk-usage                       # 4. is the journal the culprit?
# 5. CHANGE (ticket first): reclaim journal space above 1 GiB
sudo journalctl --vacuum-size=1G
df -h "$M"                                    # 6. verify headroom returned
```

1. **Stabilise.** At 100% with services failing, free a little space first: vacuum the journal (step 5), or remove rotated logs
   already shipped off-host.
2. **Find the cause.** Steps 1–4. Typical causes: a log without rotation, a core dump, a runaway cache, a stuck temporary directory.
3. **Fix.** Truncate a deleted-but-open file through `/proc/<pid>/fd/<n>` (Lab 3) only if it is a log or cache you would delete
   anyway; truncating a database, queue or spool file destroys live data. Otherwise restart the holder in a quiet period.
4. **Roll back.** Deleting is irreversible; when unsure, move files to another volume.
5. **Prevent.** Add the missing rotation or cap, an inode alert and disk runway in the weekly report; record the cause.

_Example status (branch "Extend volume"): syntax-checked; not executed (needs root); steps from AWS's guide.[^ebs-extend]_

```bash
# After the owner has grown the volume (snapshot first); same M as above.
lsblk                                         # partition smaller than its disk?
sudo growpart /dev/nvme0n1 1                  # if so, grow it (note the space)
df -hT "$M"                                   # file system type; run ONE of:
sudo xfs_growfs -d "$M"                       # XFS: takes the mount point
sudo resize2fs "$(findmnt -no SOURCE "$M")"   # ext4: takes the device
df -hT "$M"                                   # size now matches the volume
# growpart needs free space for a temporary directory (stabilise first).
# With LVM, extend the logical volume with LVM tools instead.
```

### Anti-patterns and myths

- **"Restart first."** It destroys evidence. Take the Lab 1 snapshot; it takes a minute.
- **Unrecorded hand edits.** Each makes the next host different and the next incident slower.
- **Disabling a protection to make something work** (a firewall rule, SELinux or AppArmor, a sandbox setting). Read the denial and
  fix the policy.
- **Myths:** high load means busy CPUs; low "free" memory is a problem (traps 1 and 2); the cloud needs no SysOps, when on IaaS
  the guest OS and its patches are yours.[^aws-srm]

## Deciding and adopting

### When to use SysOps and when not to

**Invest in host skills and a baseline when** you run guest operating systems (VMs, bare metal, IaaS, self-managed Kubernetes
nodes), when regulators or customers want host-level evidence, when you own long-lived pets (licensed software, appliances, legacy
systems), or when servers sit at remote sites ([EdgeOps](../edgeops/)).

**Invest less, or differently, when** the provider runs the OS (SaaS, managed databases, serverless): the work moves to
[CloudOps](../cloudops/) and [ServerlessOps](../serverlessops/), and [NoOps](../noops/) explains what remains. With many identical
hosts, invest in images and configuration code ([InfraOps](../infraops/)) rather than repair. A separate "SysOps team" beside a
platform team can become a hand-off queue; consider giving the platform team host ownership.

**Hidden responsibilities:** OS end-of-life migrations, reboots, access reviews, restore tests, console access, and someone on
call who can read a kernel log.

### Decision framework

::: figure Repair, replace or hand the host to a provider
```mermaid
flowchart TB
  Q1{"You run the guest OS?"}:::decision
  Q1 -->|no| N1["Provider runs the host"]:::neutral
  Q1 -->|yes| Q2{"Identical to many?"}:::decision
  Q2 -->|"yes: cattle"| Q3{"Failure understood?"}:::decision
  Q3 -->|yes| R1["Fix image, replace"]:::core
  Q3 -->|no| R2["Capture evidence, replace"]:::core
  Q2 -->|"no: pet"| Q4{"Rebuild within RTO?"}:::decision
  Q4 -->|yes| R3["Replace, codify gaps"]:::core
  Q4 -->|no| R4["Snapshot, repair, codify"]:::risk
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
```
Text equivalent: If you do not operate the guest OS, the provider runs the host. For cattle: if the failure is understood, fix the
image and replace; if not, capture evidence, then replace. For a pet: if it can be rebuilt within the
recovery time objective, replace it and codify the gaps; if not, snapshot, repair in place (the riskiest path) and codify later.
:::

The tree assumes evidence (triage snapshot, off-host logs, disk snapshot) is cheap; where it is not, agree in advance how much to
collect. Recovery time objectives belong to [DisasterRecoveryOps](../disasterrecoveryops/).

### Maturity model

This is **this Atlas's model**, not an industry standard.

| Level | Observable behaviour | Evidence | Next step |
|---|---|---|---|
| 0. Ad hoc | Hand-built hosts; shared root; fixes by restart | No inventory | Inventory and named accounts |
| 1. Documented | Inventory, runbooks, named access, basic alerts | Runbooks used in incidents | Write the baseline |
| 2. Baselined | One baseline per role, applied by tooling, exceptions recorded | Conformance report | Measure jobs, headroom, reboots, toil |
| 3. Measured | KPIs reviewed monthly; jobs and disks alert before failure | KPI trend | Replace cattle instead of repairing |
| 4. Replaceable | Most hosts rebuilt from images; console and restore drills | Rebuild and drill records | Pets on a migration plan |

### Adoption roadmap: 30/60/90 days

| Window | Actions | Exit criteria |
|---|---|---|
| Days 1–30 | Build the inventory; remove shared root and agent accounts; install sysstat and the triage script; alert on disk, inodes, memory, failed units | Inventory reconciled; every host alerts on the four signals |
| Days 31–60 | Write the baseline with SecOps and apply it with configuration management; move cron jobs to timers; cap journals; set the update and reboot policy | Conformance measured; no job without log and alert |
| Days 61–90 | Restore sample; console drill; EOL calendar; triage training with Lab 1 and the runbook | First KPI review; drill passed or gaps ticketed |

## Toolkit

### Checklists

**New host readiness**
- [ ] In inventory, with an owner and an OS end-of-life date
- [ ] Baseline passing; firewall default-deny; SELinux or AppArmor enforcing; exceptions recorded
- [ ] Named access only; no root login; sudo through groups; agent sessions mapped to named accounts
- [ ] Metrics, logs and alerts arriving; one test alert received
- [ ] Time synchronised; journal persistent and capped
- [ ] Update and reboot policy set; reboot-pending alert in place
- [ ] Off-host backup running; first restore tested
- [ ] Console access, the boot menu and a rescue-mode boot tested with break-glass credentials

**Before a risky host change**
- [ ] Change record approved; rollback written down
- [ ] Snapshot or backup taken and verified
- [ ] Second session open; automatic undo for SSH or firewall changes
- [ ] Config validated (`sshd -t`, `visudo -c`, `systemd-analyze verify`; `findmnt --verify` before a reboot)

**Decommissioning**
- [ ] Owner confirms; data retention decided
- [ ] Access, keys, certificates and service accounts revoked
- [ ] Removed from monitoring, backup and patch schedules
- [ ] Disks wiped or destroyed; inventory updated

### Templates

```text
HOST RUNBOOK: <service> on <host role>
Owner / escalation:      <team, on-call rota, service owner>
What runs here:          <units and timers, ports, data paths>
Health checks:           <commands and expected output>
Common alerts:           <alert> -> <first command> -> <known fix>
Safe restarts:           <unit>, <impact>, <quiet window>
Data and backups:        <paths>, <backup job>, <last restore test>
Access:                  <who has sudo, how it is granted, console path>
Last reviewed:           <YYYY-MM-DD> by <name>
```

```text
TRIAGE NOTE (fill in while working)
Time / host / alert:     <UTC time> / <host> / <alert>
Reachable:               yes | no (console used: <which>)
Errors seen:             <kernel log, failed units, OOM>
Saturation seen:         <run queue, PSI, swap, await, drops>
Constrained resource:    <CPU | memory | disk I/O | capacity | network | limit>
Consumer:                <process or unit>
Action and verification: <what was done, what proved it worked>
Follow-up:               <ticket for prevention>
```

### Deep comparisons

**Ways onto a host, and who reaches root** (AWS examples, as of October 2026).

| Path | Who reaches root by default | Make it safe | Evidence |
|---|---|---|---|
| SSH with keys | Anyone with a key in `authorized_keys` whom `AllowGroups` admits, via sudo | Key inventory; removal on leaving; `RevokedKeys`, kept readable[^sshd-config] | sshd and sudo log lines |
| SSH with user certificates | Holders of a certificate from a CA in `TrustedUserCAKeys` | Short lifetimes (`-V`; the default never expires); a protected CA key[^ssh-keygen] | Certificate key ID in sshd logs |
| Session Manager | Every IAM principal allowed `ssm:StartSession` on the node,[^ssm-iam] as a shared `ssm-user` with passwordless sudo[^ssm-user] | Run As: an `SSMSessionRunAs` tag per principal names the OS account;[^ssm-runas] or remove `ssm-user`'s sudo rule;[^ssm-user] scope IAM to instances or tags | CloudTrail records `StartSession`;[^ssm-ct] session content only once logging is configured, and never for port forwarding or SSH[^ssm-logging] |
| Run Command | Any principal allowed to run `AWS-` documents: "administrator access to the node"[^ssm-walk] | Allow `ssm:SendCommand` only for specific documents and instances | CloudTrail records `SendCommand`[^ssm-ct] |
| Serial or out-of-band console | Principals with console permission who know a local password | EC2: account access per Region, the serial-console IAM action (below) scoped to instances, a password-based break-glass user;[^ec2-serial] else hypervisor console or BMC | Local login records |

On EC2 that IAM action is `ec2-instance-connect:SendSerialConsoleSSHPublicKey`.[^ec2-serial]

### FAQ

**Is "SysOps" the same as system administration?** In practice, yes: the label is informal, and the US occupational
classification (O*NET) and the certifications call the job system administration.[^onet][^lfcs]

**My monitoring shows 40% CPU, yet users see slowness. How?** Bursts hidden by averages, steal time, or saturation elsewhere: check
run queue, PSI and I/O queues.[^use][^vmstat][^psi]

**Is agent-based access safer than SSH?** It closes inbound ports and removes keys, but by default every permitted IAM principal
shares one passwordless-sudo account; it is safer only with Run As, scoped IAM and session logging ([Deep comparisons](#deep-comparisons)).

**Is a snapshot a backup?** Not on its own: a snapshot on the same system fails with it ([BackupOps](../backupops/)).

**When does a host problem become an incident?** When several hosts or a user-facing service are affected, or the fix needs
decisions beyond the host owner ([IncidentOps](../incidentops/)).

## Going further

### Advanced topics and emerging direction

- **Pressure-driven memory management.** systemd-oomd "uses cgroups-v2 and pressure stall information (PSI) to monitor and take
  corrective action before an OOM occurs in the kernel space";[^oomd] distributions differ on enabling it.
- **Tracing.** `perf` and eBPF-based tools answer "why" after USE answers "where" (see further reading).
- **Short-lived access.** Per-request SSH certificates valid for hours make the CA the critical system ([AccessOps](../accessops/)).

### Learning path

| Stage | Do this | Proof |
|---|---|---|
| Prerequisites | Shell, files and permissions, an editor | Navigate and edit on a VM |
| 1 day | Lab 1 on your own VM; the USE method[^use] | Explain each `CHECK` line |
| 1 week | Labs 2 and 3; validated sudoers and sshd drop-ins; a firewall change with an automatic undo | A VM that passes the readiness checklist |
| 30 days | A baseline applied by configuration management; cron jobs moved to timers; one SELinux or AppArmor denial fixed without disabling it | Conformance report and runbook |
| Advanced | `perf` and eBPF; kernel tunables; Windows Server | A written diagnosis of a real slowdown |

Skills maps: the LFCS domains[^lfcs] and RHCSA objectives[^rhcsa] (packages, recovery, firewalls, SELinux).

## Wrap-up

### Executive recap and cheat sheet

- **Decide:** who owns hosts; pet or cattle; which baseline; the update and reboot policy.
- **Do:** named access on every path, one baseline, alerting timers, capped journals, synchronised time, off-host backups, tested
  consoles.
- **Diagnose:** errors, then saturation, then utilisation; load includes D state; read MemAvailable; distrust `%util` on SSDs.
- **Measure:** conformance, job success, disk runway, reboot debt, OS end-of-life exposure.
- **Watch out for:** deleted-but-open files, inodes, clock steps, silent jobs, shared agent accounts, untested fstab edits.

### A judgement call: repair the pet or rebuild it?

**Situation (illustrative).** At 14:00 the only server running a licensed reporting system is swapping and nearly unresponsive;
month-end reports are due at 18:00. A rebuild from documentation is estimated at six hours and has never been tested.

| Option | For | Against |
|---|---|---|
| A. Repair in place | Fastest likely route; keeps the evidence | Fixes the symptom; the next failure will be the same |
| B. Rebuild now | Ends the pet | Six untested hours against a four-hour deadline |

**Assumptions:** a disk snapshot takes minutes; triage has named the cause, a runaway export job not needed today; the licence allows
a second instance later.

**Decision:** snapshot, repair in place (stop the export job or cap it with `MemoryHigh=`), deliver the reports, and schedule the
rebuild as a rehearsed project. Option B looks principled, but an untested rebuild under a deadline turns degradation into an
outage. **Uncertainty:** if triage cannot name the cause within 30 minutes, the repair estimate is unknown and the balance changes;
escalate to the service owner.

### Learning objectives and mastery check

After this page you can **explain** what SysOps covers and why the label is informal; **diagnose** a slow or full host with the USE
method; **configure** timers, journald, sudo, SSH and a host firewall safely; **decide** whether to repair or replace a host.

1. A host shows load average 24 on 8 CPUs, `mpstat` shows 70% idle, and `ps` lists many tasks in state D. Where do you look next?

::: answer Show answer for question 1
Storage or network filesystems: load includes tasks in uninterruptible sleep, typically blocked on I/O.[^loadavg] Check `iostat -xz 1`
(queue size, await), `/proc/pressure/io`, NFS mounts and kernel-log I/O errors. More CPUs would not help.
:::

2. You put `PasswordAuthentication no` in `/etc/ssh/sshd_config.d/99-hardening.conf`, but password logins still work. Name two causes.

::: answer Show answer for question 2
First, sshd keeps the first value it reads for most keywords,[^sshd-config] so an earlier drop-in or a main-file line above the
`Include` can set `PasswordAuthentication yes`. Second, with `UsePAM yes`, "PAM keyboard-interactive authentication usually serves
an equivalent role to password authentication".[^deb-sshd] Check `sshd -T`, set `KbdInteractiveAuthentication no`, and rename the
drop-in to sort first.[^sshd]
:::

3. A sudoers rule in `/etc/sudoers.d/admins.conf` has no effect. Why?

::: answer Show answer for question 3
sudo skips included files whose names contain a dot.[^sudoers] Rename it `admins`, install it with mode 0440 owned by root, and run
`visudo -c`.[^visudo]
:::

4. A nightly cron job has not produced output for two weeks and nobody noticed. Name two changes that would have caught it.

::: answer Show answer for question 4
A systemd timer with `OnFailure=` pointing at your alerting, and `Persistent=true` to catch up missed runs;[^sd-timer][^sd-unit] and a
job-success KPI that compares each timer's last run with its schedule, so a job that never started is noticed too.
:::

5. A cattle host in an auto-scaling group fails in a way nobody has seen before. Replace it immediately?

::: answer Show answer for question 5
Replace it, but capture evidence first (triage snapshot, off-host logs, optionally a disk snapshot) and study the copy; otherwise the
replacement may fail the same way ([decision framework](#decision-framework)).
:::

### Conclusion

SysOps is an old job under a fading name. The label will keep moving (AWS has moved it to CloudOps), but the host under every
platform stays: a kernel that queues work, disks that fill, clocks that drift, jobs that fail quietly, doors that lead to root. The durable skill is a habit, not a command: measure before acting, read errors and saturation before percentages, and turn
every fix into a baseline, timer, alert or image, so the same problem never needs the same person twice.

## Sources and further reading

Claim-by-claim evidence: `research/sysops-ledger.md`. Re-check the AWS rows, OS support windows and systemd or chrony defaults
quarterly, and the versions on your own hosts.

[^aws-cert]: [AWS Certified CloudOps Engineer – Associate](https://aws.amazon.com/certification/certified-cloudops-engineer-associate/) — AWS. Checked 2026-10-17.
[^aws-dates]: [Coming soon to AWS Certification](https://aws.amazon.com/certification/coming-soon/) — AWS, 2025. Checked 2026-10-17.
[^tt-sysop]: [What is a systems operator (sysop)?](https://www.techtarget.com/searchitoperations/definition/system-operator-sysop) — Paul Kirvan, TechTarget, 2023-04-24. Checked 2026-10-17.
[^wp-sysop]: [Sysop](https://en.wikipedia.org/wiki/Sysop) — Wikipedia. Checked 2026-10-17.
[^mw-rights]: [Manual:User rights](https://www.mediawiki.org/wiki/Manual:User_rights) — MediaWiki. Checked 2026-10-17.
[^onet]: [Network and Computer Systems Administrators, 15-1244.00](https://www.onetonline.org/link/summary/15-1244.00) — O*NET OnLine, US Department of Labor, updated 2026. Checked 2026-10-17.
[^lfcs]: [Linux Foundation Certified System Administrator (LFCS)](https://training.linuxfoundation.org/certification/linux-foundation-certified-sysadmin-lfcs/) — The Linux Foundation. Checked 2026-10-17.
[^rhcsa]: [Red Hat Certified System Administrator](https://www.redhat.com/en/services/certification/rhcsa) — Red Hat. Checked 2026-10-17.
[^tposna]: [The Practice of System and Network Administration, 3rd ed.](https://the-sysadmin-book.com/) — Limoncelli, Hogan, Chalup. Checked 2026-10-17.
[^lisa]: [Usenix LISA is no more. Here's my retrospective.](https://everythingsysadmin.com/2022/11/reflections-on-usenix-lisa.html) — Tom Limoncelli, 2022-11-12. Checked 2026-10-17.
[^sre-intro]: [Introduction](https://sre.google/sre-book/introduction/) — Benjamin Treynor Sloss, *Site Reliability Engineering*, Google, 2016. Checked 2026-10-17.
[^sre-toil]: [Eliminating Toil](https://sre.google/sre-book/eliminating-toil/) — Vivek Rau, *Site Reliability Engineering*, Google, 2016. Checked 2026-10-17.
[^pets]: [The History of Pets vs Cattle and How to Use the Analogy Properly](https://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/) — Randy Bias, Cloudscaling, 2016-09-29. Checked 2026-10-17.
[^use]: [The USE Method](https://www.brendangregg.com/usemethod.html) — Brendan Gregg, 2012. Checked 2026-10-17.
[^use-linux]: [USE Method: Linux Performance Checklist](https://www.brendangregg.com/USEmethod/use-linux.html) — Brendan Gregg, updated 2013-09-29. Checked 2026-10-17.
[^netflix60]: [Linux Performance Analysis in 60,000 Milliseconds](https://www.brendangregg.com/Articles/Netflix_Linux_Perf_Analysis_60s.pdf) — Netflix Technology Blog, 2015-11-30 (copy hosted by co-author Brendan Gregg). Checked 2026-10-17.
[^loadavg]: [Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html) — Brendan Gregg, 2017-08-08. Checked 2026-10-17.
[^psi]: [PSI – Pressure Stall Information](https://docs.kernel.org/accounting/psi.html) — Johannes Weiner, kernel.org, 2018. Checked 2026-10-17.
[^psi-boot]: [CONFIG_PSI_DEFAULT_DISABLED](https://cateee.net/lkddb/web-lkddb/PSI_DEFAULT_DISABLED.html) — Linux Kernel Driver DataBase (Kconfig help). Checked 2026-10-17.
[^proc]: [The /proc Filesystem](https://docs.kernel.org/filesystems/proc.html) — kernel.org. Checked 2026-10-17.
[^free]: [free(1)](https://man7.org/linux/man-pages/man1/free.1.html) — procps-ng. Checked 2026-10-17.
[^iostat]: [iostat(1)](https://man7.org/linux/man-pages/man1/iostat.1.html) — sysstat. Checked 2026-10-17.
[^vmstat]: [vmstat(8)](https://man7.org/linux/man-pages/man8/vmstat.8.html) — procps-ng. Checked 2026-10-17.
[^oom]: [proc_pid_oom_score_adj(5)](https://man7.org/linux/man-pages/man5/proc_pid_oom_score_adj.5.html) — Linux man-pages. Checked 2026-10-17.
[^oomd]: [systemd-oomd.service(8)](https://man7.org/linux/man-pages/man8/systemd-oomd.service.8.html) — systemd 262. Checked 2026-10-17.
[^cgroup2]: [Control Group v2](https://docs.kernel.org/admin-guide/cgroup-v2.html) — kernel.org. Checked 2026-10-17.
[^df]: [df(1)](https://man7.org/linux/man-pages/man1/df.1.html) — GNU coreutils. Checked 2026-10-17.
[^lsof]: [lsof(8)](https://man7.org/linux/man-pages/man8/lsof.8.html) — man7.org. Checked 2026-10-17.
[^tune2fs]: [tune2fs(8)](https://man7.org/linux/man-pages/man8/tune2fs.8.html) — e2fsprogs. Checked 2026-10-17.
[^sd-unit]: [systemd.unit(5)](https://man7.org/linux/man-pages/man5/systemd.unit.5.html) — systemd 262. Checked 2026-10-17.
[^sd-service]: [systemd.service(5)](https://man7.org/linux/man-pages/man5/systemd.service.5.html) — systemd 262. Checked 2026-10-17.
[^sd-exec]: [systemd.exec(5)](https://man7.org/linux/man-pages/man5/systemd.exec.5.html) — systemd 262. Checked 2026-10-17.
[^sd-analyze]: [systemd-analyze(1)](https://man7.org/linux/man-pages/man1/systemd-analyze.1.html) — systemd 262. Checked 2026-10-17.
[^sd-resource]: [systemd.resource-control(5)](https://man7.org/linux/man-pages/man5/systemd.resource-control.5.html) — systemd 262. Checked 2026-10-17.
[^sd-timer]: [systemd.timer(5)](https://man7.org/linux/man-pages/man5/systemd.timer.5.html) — systemd 262. Checked 2026-10-17.
[^sd-run]: [systemd-run(1)](https://man7.org/linux/man-pages/man1/systemd-run.1.html) — systemd 262. Checked 2026-10-17.
[^sd-system-conf]: [systemd-system.conf(5)](https://man7.org/linux/man-pages/man5/systemd-system.conf.5.html) — systemd 262. Checked 2026-10-17.
[^sd-special]: [systemd.special(7)](https://man7.org/linux/man-pages/man7/systemd.special.7.html) — systemd 262. Checked 2026-10-17.
[^journalctl]: [journalctl(1)](https://man7.org/linux/man-pages/man1/journalctl.1.html) — systemd 262. Checked 2026-10-17.
[^sulogin]: [sulogin(8)](https://man7.org/linux/man-pages/man8/sulogin.8.html) — util-linux. Checked 2026-10-17.
[^journald]: [journald.conf(5)](https://man7.org/linux/man-pages/man5/journald.conf.5.html) — systemd 262; systemd 245 defaults to `auto`. Checked 2026-10-17.
[^crontab]: [crontab(5)](https://man7.org/linux/man-pages/man5/crontab.5.html) — cronie. Checked 2026-10-17.
[^logrotate]: [logrotate(8)](https://man7.org/linux/man-pages/man8/logrotate.8.html) — man7.org. Checked 2026-10-17.
[^dnf-history]: [DNF Command Reference: history](https://dnf.readthedocs.io/en/latest/command_ref.html) — DNF project. Checked 2026-10-17.
[^dnf-nr]: [DNF needs-restarting Plugin](https://dnf-plugins-core.readthedocs.io/en/latest/needs_restarting.html) — dnf-plugins-core 4.4.2. Checked 2026-10-17.
[^apt]: [apt(8)](https://manpages.debian.org/bookworm/apt/apt.8.en.html) — Debian bookworm. Checked 2026-10-17.
[^ubuntu-auto]: [Automatic updates](https://ubuntu.com/server/docs/how-to/software/automatic-updates/) — Canonical. Checked 2026-10-17.
[^ebs-fstab]: [Make an Amazon EBS volume available for use](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-using-volumes.html) — AWS. Checked 2026-10-17.
[^ebs-extend]: [Extend the file system after resizing an Amazon EBS volume](https://docs.aws.amazon.com/ebs/latest/userguide/recognize-expanded-volume-linux.html) — AWS. Checked 2026-10-17.
[^findmnt]: [findmnt(8)](https://man7.org/linux/man-pages/man8/findmnt.8.html) — util-linux. Checked 2026-10-17.
[^firewalld]: [firewall-cmd](https://firewalld.org/documentation/man-pages/firewall-cmd.html) — firewalld project. Checked 2026-10-17.
[^selinux]: [selinux(8)](https://man7.org/linux/man-pages/man8/selinux.8.html) — SELinux userspace. Checked 2026-10-17.
[^semanage-perm]: [semanage-permissive(8)](https://man7.org/linux/man-pages/man8/semanage-permissive.8.html) — SELinux userspace. Checked 2026-10-17.
[^audit2why]: [audit2allow(1) / audit2why](https://man7.org/linux/man-pages/man1/audit2why.1.html) — SELinux userspace. Checked 2026-10-17.
[^apparmor]: [AppArmor](https://ubuntu.com/server/docs/how-to/security/apparmor/) — Canonical. Checked 2026-10-17.
[^sshd-config]: [sshd_config(5)](https://man.openbsd.org/sshd_config) — OpenBSD. Checked 2026-10-17.
[^deb-sshd]: [sshd_config(5)](https://manpages.debian.org/bookworm/openssh-server/sshd_config.5.en.html) — Debian bookworm openssh-server. Checked 2026-10-17.
[^sshd]: [sshd(8)](https://man.openbsd.org/sshd) — OpenBSD. Checked 2026-10-17.
[^ssh-keygen]: [ssh-keygen(1)](https://man.openbsd.org/ssh-keygen) — OpenBSD. Checked 2026-10-17.
[^usermod]: [usermod(8)](https://man7.org/linux/man-pages/man8/usermod.8.html) — shadow-utils. Checked 2026-10-17.
[^sudoers]: [Sudoers Manual](https://www.sudo.ws/docs/man/sudoers.man/) — sudo project. Checked 2026-10-17.
[^visudo]: [visudo Manual](https://www.sudo.ws/docs/man/visudo.man/) — sudo project. Checked 2026-10-17.
[^chrony]: [chrony.conf(5)](https://chrony-project.org/doc/4.9/chrony.conf.html) — chrony 4.9 (released 2026-08-27). Checked 2026-10-17.
[^aws-time]: [Precision clock and time synchronization on your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html) — AWS. Checked 2026-10-17.
[^aws-ntp]: [Set the time reference on your EC2 instance to use the local Amazon Time Sync Service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configure-ec2-ntp.html) — AWS. Checked 2026-10-17.
[^cf-leap]: [How and why the leap second affected Cloudflare DNS](https://blog.cloudflare.com/how-and-why-the-leap-second-affected-cloudflare-dns/) — John Graham-Cumming, Cloudflare, 2017-01-01. Checked 2026-10-17.
[^nist-123]: [SP 800-123, Guide to General Server Security](https://csrc.nist.gov/pubs/sp/800/123/final) — NIST, 2008. Checked 2026-10-17.
[^cis]: [CIS Benchmarks](https://www.cisecurity.org/cis-benchmarks) — Center for Internet Security. Checked 2026-10-17.
[^nist-92]: [SP 800-92, Guide to Computer Security Log Management](https://csrc.nist.gov/pubs/sp/800/92/final) — NIST, 2006. Checked 2026-10-17.
[^nist-92r1]: [SP 800-92 Rev. 1 (initial public draft), Cybersecurity Log Management Planning Guide](https://csrc.nist.gov/pubs/sp/800/92/r1/ipd) — NIST, 2023-10-11. Checked 2026-10-17.
[^aws-srm]: [Shared Responsibility Model](https://aws.amazon.com/compliance/shared-responsibility-model/) — AWS. Checked 2026-10-17.
[^ssm-session]: [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) — AWS. Checked 2026-10-17.
[^ssm-user]: [Turn on or turn off ssm-user account administrative permissions](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-ssm-user-permissions.html) — AWS. Checked 2026-10-17.
[^ssm-runas]: [Turn on Run As support for Linux and macOS managed nodes](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-preferences-run-as.html) — AWS. Checked 2026-10-17.
[^ssm-iam]: [Restrict access to managed nodes](https://docs.aws.amazon.com/systems-manager/latest/userguide/getting-started-restrict-access-quickstart.html) — AWS. Checked 2026-10-17.
[^ssm-logging]: [Enabling and disabling session logging](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-logging.html) — AWS. Checked 2026-10-17.
[^ssm-ct]: [Logging AWS Systems Manager API calls with AWS CloudTrail](https://docs.aws.amazon.com/systems-manager/latest/userguide/monitoring-cloudtrail-logs.html) — AWS. Checked 2026-10-17.
[^ssm-walk]: [Walkthrough: Use the AWS CLI with Run Command](https://docs.aws.amazon.com/systems-manager/latest/userguide/walkthrough-cli.html) — AWS. Checked 2026-10-17.
[^ec2-status]: [Status checks for Amazon EC2 instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html) — AWS. Checked 2026-10-17.
[^ec2-serial]: [Configure access to the EC2 Serial Console](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configure-access-to-serial-console.html) — AWS. Checked 2026-10-17.
[^ec2-grub]: [Prerequisites for the EC2 Serial Console](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console-prerequisites.html) — AWS. Checked 2026-10-17.
[^ec2-metrics]: [List the available CloudWatch metrics for your instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html) — AWS. Checked 2026-10-17.
[^cw-agent]: [Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html) — AWS. Checked 2026-10-17.
[^ec2-credits]: [Key concepts for burstable performance instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html) — AWS. Checked 2026-10-17.
[^ec2-credit-cfg]: [Configure burstable performance instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances-how-to.html) — AWS. Checked 2026-10-17.
[^win-servicing]: [Comparison of Windows Server servicing channels](https://learn.microsoft.com/en-us/windows-server/get-started/servicing-channels-comparison) — Microsoft Learn. Checked 2026-10-17.
[^win-ssh]: [Get started with OpenSSH for Windows](https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse) — Microsoft Learn. Checked 2026-10-17.
[^winevent]: [Creating Get-WinEvent queries with FilterHashtable](https://learn.microsoft.com/en-us/powershell/scripting/samples/creating-get-winevent-queries-with-filterhashtable) — Microsoft Learn. Checked 2026-10-17.
[^get-counter]: [Get-Counter](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.diagnostics/get-counter) — Microsoft Learn. Checked 2026-10-17.
[^get-service]: [Get-Service](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-service) — Microsoft Learn. Checked 2026-10-17.
[^ubuntu]: [The Ubuntu lifecycle and release cadence](https://ubuntu.com/about/release-cycle) — Canonical. Checked 2026-10-17.
[^ms-crowd]: [Helping our customers through the CrowdStrike outage](https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/) — David Weston, Microsoft, 2024-07-20. Checked 2026-10-17.
[^kb-server]: [KB5042426: CrowdStrike issue impacting Windows servers](https://support.microsoft.com/en-us/help/5042426) — Microsoft Support, 2024. Checked 2026-10-17.
[^kb-client]: [KB5042421: CrowdStrike issue impacting Windows endpoints](https://support.microsoft.com/en-us/help/5042421) — Microsoft Support, 2024. Checked 2026-10-17.
[^kb-tool]: [KB5042429: New recovery tool to help with CrowdStrike issue impacting Windows devices](https://support.microsoft.com/en-us/help/5042429) — Microsoft Support, 2024. Checked 2026-10-17.
[^ansible]: [Glossary](https://docs.ansible.com/ansible/latest/reference_appendices/glossary.html) — Ansible documentation. Checked 2026-10-17.

**Further reading:** [Linux Performance](https://www.brendangregg.com/linuxperf.html) (Brendan Gregg's tools map);
[chronyc(1)](https://chrony-project.org/doc/4.9/chronyc.html) (`tracking` output).

**Changelog.** 2026-10-16, r0: first version. 2026-10-17, r1: revised after `reviews/sysops-r1.md` (access paths, console, packages,
firewall, MAC, time, labs; Figure 1 removed). 2026-10-17, r2: revised after `reviews/sysops-score.md` (descriptor limits, boot
recovery, per-domain permissive mode, accounts and file modes, Figure 4 routing; all sources re-checked). 2026-10-17, post-score
polish: guarded sshd reload, key-file inventory, `RevokedKeys` and GRUB preconditions, filesystem growth, T3 and chrony details.