NetOps

NetOps (network operations) is the day-to-day work of keeping a network available and performing: watching signals from devices and paths, detecting and isolating faults, escalating to carriers and vendors, running change windows, and planning capacity before demand arrives. It is usually done by a network operations centre (NOC) or an on-call network team.

The label is shorthand, and it has a second meaning: vendors also use “NetOps” for NetOps 2.0 or NetDevOps, the automation of network change. This page covers the running network; NetworkOps covers engineering change.

It is for NOC engineers, network administrators, on-call engineers who must tell “the network” from everything else, and managers deciding how to staff network operations.

A network fails in partial ways. A link can be up and still drop 2% of packets; a path can work in one direction only; a firewall can pass pings and silently break large transfers. Users see “the application is slow”, and every team suspects another.

• Not an industry standard. No standards body or professional association defines “NetOps”. The practice behind it is old and documented under other names: ITU-T M.3400 (2000) lists the management functions of telecommunication networks, with fault management meaning functions that enable “the detection, isolation and correction of abnormal operation of the telecommunication network and its environment”.^[1] The US occupational classification describes network support specialists who “Analyze, test, troubleshoot, and evaluate existing network systems”.^[2] The only formal definition found is military: a 2008 US Department of Defense instruction defined NetOps as its capabilities for operating and defending its Global Information Grid.^[3]
• Two meanings in trade use. TechTarget notes that historically “NetOps has simply been shorthand for network operations, or the running of the network”, and that the term also means NetOps 2.0, NetDevOps or network as code.^[4] Red Hat writes that NetOps 2.0 was shortened to NetOps and that the term now “generally refers to the agile methodology”;^[5] Kentik describes both uses.^[6] Vendor definitions are usage evidence, not standards.
• Not a security operations centre. MITRE’s SOC handbook says “the focus of a NOC is typically on availability and service level agreements, while the focus of the SOC is on data protection and cyberspace defense”.^[7] Attack detection, attribution and the security case belong to SOC Ops and SecOps. Keeping links usable during a volumetric DDoS (distributed denial-of-service) attack is network work, done with the SOC (Use cases).

Network operations is older than the Internet. AT&T opened a Network Control Center in New York in 1962, fed in real time from its most important toll switches, and replaced it with a Network Operations Center in Bedminster, New Jersey, in 1977.^[8]

• Management standards. ITU-T M.3400 (first edition 1992, revised 2000) organised telecom management into the five functional areas of the OSI management framework (ITU-T X.700, 1992), known as FCAPS.^[1][9] The IETF still runs an Operations and Management Area that is “logically divided into two separate functions: Network Management and Operations”.^[10]
• Telemetry generations. SNMP polling, then 64-bit interface counters (IF-MIB, RFC 1573, 1994; current text RFC 2863);^[11][12] flow export with NetFlow v9 (2004), sFlow v5 (2004) and IPFIX (RFC 5101, 2008; revised as RFC 7011);^{[13][14][15][16]} then streaming telemetry, which RFC 9232 (2022) contrasts with SNMP, CLI and syslog monitoring.^[17]
• The label. After the 2008 DoD definition, TechTarget credits Gartner analyst Andrew Lerner with coining “NetOps 2.0” in 2017, which moved the trade meaning towards automation.^[4]

Mental model. A network is a set of paths, each with four measurable properties (reachable, delay, loss, capacity) and some control-plane state (routes, sessions, neighbours). Symptoms appear at the edges; causes hide in the middle. NetOps runs a loop.

flowchart TB
  O["Observe: counters, flows, logs, probes"]:::core
  D{"Deviation from baseline or objective?"}:::decision
  O --> D
  D -->|"no"| O
  D -->|"yes"| I["Isolate the failing layer and segment"]:::core
  I --> R["Restore: reroute, repair or escalate"]:::core
  R --> V{"Verified from the user's side?"}:::decision
  V -->|"no"| I
  V -->|"yes"| P["Prevent: fix the cause, tune the alert, plan capacity"]:::core
  P --> O
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029

Hosts belong to SysOps, event correlation to AIOps, service-level objectives to SRE.

Charter (this Atlas’s proposal, not a published standard). Know the state of every path that matters, find faults before users do, restore service with evidence in hand, and change the network without surprises.

1. Measure paths, not only boxes. Device counters say an interface is up; probes from the user’s side say whether the service works. Use both.
2. Alert on symptoms, route on causes. Google’s SRE book separates what is broken (the symptom) from why (the cause) and warns that frequent pages make engineers “second-guess, skim, or even ignore incoming alerts”.^[22]
3. Evidence before escalation. A carrier acts on timestamps, loss figures and path data, not on “it’s slow”.
4. Management access must not depend on the network it manages (Rogers and Facebook: Use cases).
5. Few changes per window, each verifiable. Every change has pre-checks, post-checks and a rollback trigger.
6. Plan capacity for the failure case. A redundant pair is only redundant if one link can carry both loads.
7. Every alert has an owner, a runbook and a review date.

Cloud caveats (as of October 2026): AWS flow logs “do not capture real-time log streams for your network interfaces”^[31] and skip traffic such as Amazon DNS queries and DHCP;^[32] Google Cloud samples packets.^[33]

R = responsible, A = accountable, C = consulted, I = informed. Tiers are common practice, not a standard; small teams merge them.

Change approval and service levels belong to ITOps.

• Day 0 (design). Site-and-role naming, inventory, out-of-band access, a monitoring profile, clocks synchronised (NTP) so logs and probes line up; for every carrier link the circuit ID, contract and escalation path; for unstaffed sites, spares and a remote-hands contract.
• Day 1 (onboard). Monitoring added, baseline recorded, one test alert fired, configuration backups confirmed.
• Day 2 (operate). Alerts, incidents, hardware replacements, change windows, capacity reviews; appliance image upgrades follow the rings on PatchOps, one peer of a pair at a time.
• Retire. Remove from monitoring, cancel the circuit, revoke access.

MANRS, a routing-security initiative, asks operators to “Maintain globally accessible, up-to-date contact information in common routing databases”.^[34] A NOC that cannot reach a neighbouring network’s NOC at 03:00 has an incomplete runbook.

flowchart TB
  subgraph NET["Managed network"]
    DEV["Routers, switches, firewalls"]:::neutral
    LNK["Carrier and cloud links"]:::neutral
  end
  subgraph COL["Collectors, one set per region"]
    POLL["SNMP or telemetry"]:::core
    FLOW["Flow collector"]:::core
    LOGS["Syslog and traps"]:::core
    PROBE["Path probes"]:::core
  end
  STORE["Metric, flow and log stores"]:::core
  ALERT["Rules, grouping, inhibition"]:::decision
  NOC["NOC console, tickets, runbooks"]:::core
  OOB["Out-of-band access"]:::risk
  DEV --> POLL & FLOW & LOGS
  LNK --> PROBE
  POLL & FLOW & LOGS & PROBE --> STORE
  STORE --> ALERT --> NOC
  NOC -.->|"when in-band fails"| OOB
  OOB -.-> DEV
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

The monitoring path is part of the design. In AWS’s December 2021 event, congestion “immediately impacted the availability of real-time monitoring data for our internal operations teams”.^[36]

flowchart TB
  A["Alert or user report"]:::neutral --> M{"Planned work?"}:::decision
  M -->|"yes"| S{"As expected?"}:::decision
  M -->|"no"| T["Acknowledge; scope who, where, since when"]:::core
  T --> I["Isolate layer by layer"]:::core
  I --> W{"Where is the fault?"}:::decision
  W -->|"our network"| F["Mitigate: reroute, disable, roll back"]:::core
  W -->|"carrier"| C["Carrier ticket, escalate"]:::risk
  W -->|"another team"| H["Hand off with evidence"]:::neutral
  F --> V["Verify from the user's side"]:::core
  C --> V
  H --> V
  S -->|"no"| B["Roll back"]:::core
  B --> V
  V --> R["Close; record cause; review if major"]:::core
  S -->|"yes"| R
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

• Actors: NOC Tier 1 scopes, a network engineer isolates and mitigates, and an incident commander takes over when several teams or customers are involved (IncidentOps).
• Exception path: while a carrier works on its contract clock, keep your own mitigation (rerouting) going.
• Output: a triage note naming the failing layer and segment, the evidence and the action.

Carrier escalation. Give the circuit ID, impact, start time in UTC and evidence from both ends (counters, probe loss and delay, a trace each way). Allow intrusive tests (a loopback) only in a window you can afford. Track the contract clock yourself, escalate on its schedule, and ask for a reason for outage (RFO) before closing. Read the contract’s metric definitions before arguing a number: MEF 10.4, for example, defines one-way frame delay, inter-frame delay variation, frame loss ratio and availability.^[37] Carrier contracts are underpinning contracts in ITOps terms.

Hardware faults. Tell the vendor what you have already swapped. Keep spares where the contract’s replacement time (RMA, return merchandise authorisation) exceeds the outage you can accept. At unstaffed sites, give remote hands a written method of procedure: which slot or cable, what to check, how to confirm.

Changes in a window. Run the same checks before and after (interfaces and errors, routing neighbours and prefix counts, probes across the blast radius), then observe: some faults appear only when load returns.

stateDiagram-v2
  state "Approved change" as A
  state "Pre-checks captured" as P
  state "Change applied" as C
  state "Post-checks compared" as V
  state "Observation period" as O
  state "Rolled back" as R
  state "Closed" as D
  [*] --> A
  A --> P: snapshot state
  P --> C: inside the window
  C --> V: same checks again
  V --> O: results match
  V --> R: differences or alarms
  O --> D: quiet period ends
  O --> R: delayed fault
  R --> D: verified and recorded
  D --> [*]

Structured troubleshooting has several named methods: top-down from the application, bottom-up from the physical layer, divide-and-conquer from the network layer with ping and trace, follow-the-path, compare-configurations and swap-components. A Cisco Press chapter on them advises: “The best bottom-up approach is to first reduce the scope of the problem using a different strategy and then switch to the bottom-up approach for clearly bounded parts of the network topology”.^[38] So first scope the fault: one user, one site or everyone? Everyone points at a shared dependency (an uplink, a carrier, DNS) before any single device. Then follow one path bottom-up, as the ping manual orders it: the local host first, then “hosts and gateways further and further away”.^[39] Stop at the first step that fails; if a ping to the destination already works, start at step 4.

flowchart TB
  S["Symptom: slow, lossy or unreachable"]:::risk
  SC{"One user, one site or everyone?"}:::decision
  S --> SC
  SC -->|"everyone"| SH["Check shared dependencies first: uplink, carrier, DNS"]:::neutral
  SC -->|"one user or site"| L1["Step 1 Link (OSI 1-2): errors, CRC, optics"]:::core
  SH --> L1
  L1 --> L2["Step 2 Neighbour (OSI 2): ARP or ND, VLAN"]:::core
  L2 --> L3["Step 3 Route (OSI 3): lookup, gateway reachable"]:::core
  L3 --> P["Step 4 Path (OSI 3, hop by hop): trace both ways"]:::core
  P --> T4["Step 5 Transport (OSI 4): retransmits, MTU, ports"]:::core
  T4 --> AP["Step 6 Application (OSI 5-7): DNS, TLS, first byte"]:::core
  AP --> OUT["Name the failing layer and segment"]:::decision
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Five traps catch most beginners.

1. A spike or loss at one hop is usually cosmetic. Steenbergen explains that routers generate ICMP replies in software as “one of the lowest priority functions of the router”, “typically rate-limited to very low values”: a delayed reply shows as a spike, a dropped one as loss. “If you see a latency spike on one Traceroute hop, and it goes away in the next hop, you can easily deduce that the spike was actually caused by a cosmetic delay in the ICMP generation process”.^[44] mtr’s manual is more alarmist (a jump is “often an indication of a bad (or simply overloaded) link”);^[45] believe it only when the jump persists to the destination.
2. Traceroute shows the forward path; its times include the return path. Steenbergen: “you must also examine the reverse path Traceroute before you can be certain that a particular link is responsible”.^[44] Get a trace from the far end, a looking glass, or a measurement network such as RIPE Atlas.^[46]
3. Silence is not failure. Many networks drop UDP probes; traceroute -T uses TCP SYNs, a method “intended to bypass firewalls”.^[47] Lab 1’s run shows it.
4. Ping works, transfers hang: suspect the path MTU. In a PMTUD black hole, ICMP “fragmentation needed” messages are filtered; RFC 2923 says the failure “is especially difficult to debug, as pings and some interactive TCP connections to the destination host work”.^[21] Test with ping -M do -s 1472 <host> (1472 bytes plus 28 bytes of headers is 1500) and tracepath.^[39][48] IPv6 routers never fragment,^[49] so filtered ICMPv6 Packet Too Big messages cause the same black hole;^[50] test with ping -6 -M do -s 1452 <host> (plus 48 header bytes).
5. Averages hide bursts. A five-minute utilisation of 40% can contain seconds at line rate; look at discard and drop counters, not only utilisation.

None is mandatory in itself; a contract, regulator or customer can make one binding.

Representative tools (as of October 2026), vendor-neutral categories first:

Lock-in sits in the data: keep inventories, alert rules and runbooks in formats you can move.

Automate read-only work first: attach counters and probe results to the ticket when an alert fires; check every device against its monitoring profile. Pushing configuration is change engineering (NetworkOps, AutomationOps).

Automated remediation needs limits. In October 2025, flapping health checks made AWS’s Network Load Balancer fail over between Availability Zones and remove capacity; AWS is “adding a velocity control mechanism to limit the capacity a single NLB can remove when health check failures cause AZ failover”.^[67] The SRE book reports avoiding “magic” systems that learn thresholds or detect causality;^[22] correlation is AIOps. Language models explain output well but invent CLI syntax: review anything that touches a device.

Assumptions: Prometheus 3.x, snmp_exporter v0.30.1 (2026-01-07) with its shipped snmp.yml and if_mib module, blackbox_exporter v0.29.0, Alertmanager; devices accept SNMPv3 only from the collector. Names are placeholders.

The generator documents that auth_protocol “Defaults to MD5” and priv_protocol “Defaults to DES”,^[68] so set both; community strings “are sent unencrypted in SNMP v1 and v2c”.^[23] SHA-256 for SNMPv3 is RFC 7860: check device support.^[69] A sample on the project’s main branch loads extra auths as a second --config.file, with secrets from the environment.^[70]

Example status: YAML parsed with js-yaml; fields checked against the projects’ documentation (2026-10-18); not run against devices; PromQL not executed (operator precedence and label matching checked against the Prometheus documentation). Before use, run promtool check config, promtool check rules and amtool check-config.

# snmp-auth.yml: start the exporter with
#   snmp_exporter --config.file=snmp.yml --config.file=snmp-auth.yml \
#     --config.expand-environment-variables \
#     --web.listen-address=192.0.2.10:9116 --web.config.file=web-config.yml
# Credentials come from the environment, never from this file.
auths:
  noc_v3:
    version: 3
    username: ${SNMP_USER}
    security_level: authPriv
    password: ${SNMP_AUTH_PASS}
    auth_protocol: SHA256       # default is MD5
    priv_protocol: AES          # default is DES
    priv_password: ${SNMP_PRIV_PASS}

# blackbox.yml: ICMP module for path probes
modules:
  icmp_v4:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: ip4   # the exporter's default is ip6

# web-config.yml: TLS and basic auth for both exporters (exporter-toolkit format).
# Start blackbox_exporter with --web.listen-address=192.0.2.10:9115 and this file too.
# 192.0.2.10 is the collector behind both exporter names in prometheus.yml; the
# certificate must list those names.
tls_server_config:
  cert_file: /etc/exporters/tls.crt
  key_file: /etc/exporters/tls.key
basic_auth_users:
  prometheus: "<bcrypt-hash>"   # from: htpasswd -nBC 10 "" | tr -d ':\n'

# prometheus.yml (excerpt). Targets carry a site label for inhibition.
rule_files: [rules/netops.yml]
scrape_configs:
  - job_name: snmp
    scrape_interval: 60s
    scrape_timeout: 50s
    metrics_path: /snmp
    params: {auth: [noc_v3], module: [if_mib]}
    scheme: https
    basic_auth: {username: prometheus, password_file: /etc/prometheus/exporter.pass}
    tls_config: {ca_file: /etc/prometheus/noc-ca.crt}
    file_sd_configs: [{files: [targets/devices.yml]}]
    relabel_configs:
      - {source_labels: [__address__], target_label: __param_target}
      - {source_labels: [__param_target], target_label: instance}
      - {target_label: __address__, replacement: "snmp-exporter.example.net:9116"}
  - job_name: probe_icmp
    scrape_interval: 15s
    metrics_path: /probe
    params: {module: [icmp_v4]}
    scheme: https
    basic_auth: {username: prometheus, password_file: /etc/prometheus/exporter.pass}
    tls_config: {ca_file: /etc/prometheus/noc-ca.crt}
    file_sd_configs: [{files: [targets/probes.yml]}]
    relabel_configs:
      - {source_labels: [__address__], target_label: __param_target}
      - {source_labels: [__param_target], target_label: instance}
      - {target_label: __address__, replacement: "blackbox.example.net:9115"}
  - job_name: probe_icmp_from_branch07   # a blackbox at the site: the users' path
    scrape_interval: 15s
    metrics_path: /probe
    params: {module: [icmp_v4]}
    scheme: https
    basic_auth: {username: prometheus, password_file: /etc/prometheus/exporter.pass}
    tls_config: {ca_file: /etc/prometheus/noc-ca.crt}
    static_configs:
      - targets: [app.example.net]       # a service address users depend on
        labels: {site: branch07, source_site: branch07}
    relabel_configs:
      - {source_labels: [__address__], target_label: __param_target}
      - {source_labels: [__param_target], target_label: instance}
      - {target_label: __address__, replacement: "blackbox.branch07.example.net:9115"}

# targets/devices.yml and targets/probes.yml (file-based discovery)
- targets: [rtr1.branch07.example.net, sw1.branch07.example.net]
  labels: {site: branch07}

# rules/netops.yml: symptoms page; causes open tickets, except lost redundancy or
# lost visibility. Every rule names an owner and a runbook.
groups:
  - name: netops
    rules:
      - alert: SiteProbeLoss
        expr: avg_over_time(probe_success{job=~"probe_icmp.*"}[10m]) < 0.9
        for: 5m
        labels: {severity: page, owner: noc}
        annotations:
          summary: "More than 10% probe loss to {{ $labels.instance }}"
          runbook: "https://wiki.example.net/noc/site-probe-loss"
      - alert: UplinkDown            # uplink or carrier circuit: redundancy is gone
        expr: >
          ifOperStatus{ifAlias=~"(?i)(uplink|cid).*"} == 2
          and on (instance, ifIndex) ifAdminStatus == 1
        for: 2m
        labels: {severity: page, owner: noc}
        annotations:
          summary: "{{ $labels.instance }} uplink {{ $labels.ifAlias }} down"
          runbook: "https://wiki.example.net/noc/uplink-down"
      - alert: InterfaceDown         # any other admin-up port without a link
        expr: >
          ifOperStatus{ifAlias!~"(?i)(uplink|cid).*"} == 2
          and on (instance, ifIndex) ifAdminStatus == 1
        for: 15m
        labels: {severity: ticket, owner: network-eng}
        annotations:
          runbook: "https://wiki.example.net/noc/interface-down"
      - alert: InterfaceHighUtilisation   # either direction
        expr: >
          (rate(ifHCInOctets[5m]) * 8 / (ifHighSpeed * 1e6) > 0.8
            or rate(ifHCOutOctets[5m]) * 8 / (ifHighSpeed * 1e6) > 0.8)
          and on (instance, ifIndex) ifHighSpeed > 0
        for: 15m
        labels: {severity: ticket, owner: network-eng}
        annotations:
          runbook: "https://wiki.example.net/noc/high-utilisation"
      - alert: InterfaceErrors
        expr: increase(ifInErrors[15m]) > 50
        labels: {severity: ticket, owner: network-eng}
        annotations:
          runbook: "https://wiki.example.net/noc/interface-errors"
      - alert: SnmpTargetUnreachable   # blind to a device: paged like lost redundancy
        expr: up{job="snmp"} == 0
        for: 5m
        labels: {severity: page, owner: noc}
        annotations:
          runbook: "https://wiki.example.net/noc/snmp-unreachable"

# alertmanager.yml (excerpt): one page per site outage; tickets never page
route:
  receiver: noc-pager
  group_by: [site, alertname]
  routes:
    - receiver: noc-tickets
      matchers: ['severity="ticket"']
receivers:
  - name: noc-pager
  - name: noc-tickets
inhibit_rules:
  - source_matchers: ['alertname="SiteProbeLoss"']
    target_matchers: ['alertname=~"UplinkDown|InterfaceDown|SnmpTargetUnreachable"']
    equal: [site]

Design notes. rate() handles counter resets (“Breaks in monotonicity … are automatically adjusted for”),^[71] and for: keeps an alert pending until its condition has held for that long.^[72] up == 0 treats missing data as a signal: the device is down or unmanageable. The ICMP probe needs CAP_NET_RAW, root or a group inside net.ipv4.ping_group_range on Linux.^[64]

What pages. Symptoms (probe loss) page, and so do two definite, imminent causes: a lost uplink (redundancy is gone) and a device the collector cannot see; other causes open tickets. The SRE book’s rule for causes: “only worry about very definite, very imminent causes”.^[22] UplinkDown relies on a convention: an ifAlias that starts with uplink or cid, because PromQL regex matches are fully anchored.^[73] RFC 2863 defines ifAlias as a name the network manager sets, such as a carrier’s circuit identifier;^[12] Junos exposes the interface description there.^[74] InterfaceDown fires for every admin-up port without a link, which reports down(2),^[12] so shut unused access ports.

Who can reach the credentials. Both exporters are proxies that listen on all addresses over plain HTTP unless configured otherwise (the blackbox README’s log shows “TLS is disabled.”).^[64] Whoever reaches /snmp inherits the exporter’s SNMP credentials for any device; whoever reaches /probe can aim probes anywhere. Bind each exporter to the collector’s address, admit only the Prometheus servers, and enable TLS and basic authentication (--web.config.file, which covers /snmp too).^[23][75] On devices, give the SNMPv3 user a read-only view, accepted only from the collector. The ${…} variables keep secrets out of committed files, not off the host: root and the exporter’s user can read /proc/<pid>/environ,^[76] so use a dedicated user on a host only the NOC administers.

Where the probes sit. The central probe_icmp job pings management addresses from the data centre: the device answers, but are branch users served? The second job probes a service address from a blackbox exporter at the branch (source_site); add one per site, or use RIPE Atlas or perfSONAR.

Labs 1 to 3 are read-only and run as an ordinary user. The optional part of Lab 2 needs root on a disposable VM.

Objective: walk the layers from your host to a service and name the first one that fails. Prerequisites: Linux with iproute2, iputils (ping, tracepath) and curl; 15 minutes. Cost and risk: about 40 small packets to the target; nothing on the host changes.

Example status: verified by execution as an unprivileged user on Ubuntu 20.04 (iputils s20190709, iproute2 5.5, curl 7.68), target example.com:443, 2026-10-18; output summarised, not reproduced (it names a production host’s addresses).

#!/usr/bin/env bash
# netpath.sh: read-only, bottom-up path check from this host to HOST:PORT.
# Usage: bash netpath.sh HOST [PORT]      (changes nothing; no root needed)
set -uo pipefail
target=${1:?usage: netpath.sh HOST [PORT]}
port=${2:-443}
say() { printf '\n== %s ==\n' "$*"; }
flag() { printf '  CHECK: %s\n' "$*"; }
field() { awk -v k="$1" '{for (i = 1; i < NF; i++) if ($i == k) {print $(i+1); exit}}'; }

say "1. Name resolution"
addr=$(getent ahostsv4 "$target" | awk 'NR == 1 {print $1}')
[ -n "$addr" ] || { flag "cannot resolve $target: test DNS first"; exit 1; }
echo "  $target -> $addr"

say "2. Route and egress interface"
route=$(ip route get "$addr" | head -n 1)
echo "  $route"
dev=$(field dev <<<"$route"); gw=$(field via <<<"$route")

say "3. Link state and error counters on $dev"
ip -s link show dev "$dev" | sed 's/^/  /'
for c in rx_errors tx_errors rx_dropped tx_dropped rx_crc_errors; do
  v=$(cat "/sys/class/net/$dev/statistics/$c" 2>/dev/null || echo 0)
  [ "$v" -gt 0 ] && flag "$c=$v since boot: read again, is it rising?"
done

say "4. Next hop ${gw:-$addr} in the neighbour table"
ip neigh show "${gw:-$addr}" dev "$dev" | sed 's/^/  /'

say "5. Loss and RTT (gateway first, then target)"
[ -n "$gw" ] && ping -n -c 5 -i 0.5 -W 1 "$gw" | tail -n 2 | sed 's/^/  gw  /'
ping -n -c 10 -i 0.5 -W 1 "$addr" | tail -n 2 | sed 's/^/  dst /'

say "6. Forward path and path MTU (RTTs include the unseen return path)"
tracepath -n -m 20 "$addr" | sed 's/^/  /'

say "7. TCP connect to port $port"
if timeout 5 bash -c ": > /dev/tcp/$addr/$port" 2>/dev/null; then
  echo "  TCP $port open"
else
  flag "TCP $port refused, filtered or timed out"
fi
if [ "$port" = 443 ]; then
  fmt='  dns=%{time_namelookup}s tcp=%{time_connect}s'
  fmt+=' tls=%{time_appconnect}s first_byte=%{time_starttransfer}s\n'
  curl -s -o /dev/null --connect-timeout 5 -m 15 -w "$fmt" "https://$target/" ||
    flag "HTTPS request failed"
fi
exit 0

1. Run bash netpath.sh example.com 443, then against a service you support; name the layer each section tests.
2. Run bash netpath.sh no-such-host.invalid and bash netpath.sh example.com 23; note where each stops or flags.

Expected output (shape): seven == sections. The 2026-10-18 cloud-VM run: no CHECK lines; gateway RTT 0.08 ms; target 1 ms, 0% loss; tracepath showed [LOCALHOST] pmtu 9001 (the interface MTU), then pmtu 1500; hop 3 read 31 ms between hops of about 1.3 ms (trap 1); no reply from hop 5 to 20, yet TCP 443 connected (trap 3). The failure runs printed CHECK: cannot resolve (exit 1) and CHECK: TCP 23 refused, filtered or timed out. Validation: exit code 0 for the first run; every section mapped to a layer. Troubleshooting: ping: socket: Operation not permitted → your distribution restricts ICMP sockets (ping_group_range); use sudo or a VM. Empty tracepath output → UDP blocked locally; try traceroute -T -p 443 with sudo. Teardown: none; delete saved output that names hosts.

Objective: turn raw ping output into the KPIs a carrier contract or SLO uses, and see why one number is not enough. Prerequisites: Python 3.8 or later; iputils ping with -D and -O; 20 minutes. Cost and risk: 300 small packets at 5 per second. The iputils manual warns that “it is unwise to use ping during normal operations or from automated scripts” because of the load;^[39] keep the rate low and aim repeated runs at hosts you own or may test.

Example status: verified by execution on 2026-10-18 (Python 3.8.10): against a synthetic capture with known answers, against example.com and against 127.0.0.1. The optional netem part is syntax-checked only (needs root and kernel modules).

#!/usr/bin/env python3
"""pingkpi.py: loss, RTT and delay-variation KPIs from `ping -D -O` output.

Usage: ping -n -D -O -i 0.2 -c 300 HOST | python3 pingkpi.py --p95-ms 60 --loss-pct 1
All delay figures are ROUND-TRIP times, not the one-way metrics of RFC 7679.
"""
import argparse
import math
import re
import sys

REPLY = re.compile(r"icmp_seq=(\d+) .*time=([\d.]+) ms")
SUMMARY = re.compile(r"(\d+) packets transmitted, (\d+) received")


def pct(values, p):
    """Nearest-rank percentile of a non-empty list."""
    s = sorted(values)
    return s[max(0, math.ceil(p / 100 * len(s)) - 1)]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--p95-ms", type=float, default=100.0, help="SLO: p95 RTT")
    ap.add_argument("--loss-pct", type=float, default=1.0, help="SLO: loss")
    ap.add_argument("--pdv-ms", type=float, default=30.0, help="SLO: p99 PDV")
    a = ap.parse_args()

    rtt, sent = {}, None
    for line in sys.stdin:
        m = REPLY.search(line)
        if m and "DUP!" not in line:
            rtt.setdefault(int(m.group(1)), float(m.group(2)))
        s = SUMMARY.search(line)
        if s:
            sent = int(s.group(1))
    if not sent or not rtt:
        sys.exit("no ping summary or no replies: did ping run to completion?")

    seqs = range(1, sent + 1)  # iputils numbers echo requests from 1
    lost = [q for q in seqs if q not in rtt]
    burst = run = 0
    for q in seqs:
        run = run + 1 if q not in rtt else 0
        burst = max(burst, run)
    ordered = [rtt[q] for q in seqs if q in rtt]
    base = min(ordered)
    pdv = [r - base for r in ordered]  # RFC 5481 PDV: reference = minimum
    ipdv = [ordered[i] - ordered[i - 1] for i in range(1, len(ordered))]
    jit = 0.0
    for d in ipdv:  # RFC 3550 estimator, applied here to RTTs
        jit += (abs(d) - jit) / 16

    loss = 100.0 * len(lost) / sent
    p95 = pct(ordered, 95)
    pdv99 = pct(pdv, 99)
    print(f"sent={sent} received={len(ordered)} loss={loss:.2f}% "
          f"longest_loss_burst={burst}")
    print(f"rtt_ms min={base:.2f} p50={pct(ordered, 50):.2f} p95={p95:.2f} "
          f"max={max(ordered):.2f}")
    mean_abs = sum(map(abs, ipdv)) / max(1, len(ipdv))
    print(f"pdv_p99_ms={pdv99:.2f} ipdv_mean_abs_ms={mean_abs:.2f} "
          f"rfc3550_style_jitter_ms={jit:.2f}")
    breaches = []
    if loss > a.loss_pct:
        breaches.append(f"loss {loss:.2f}% > {a.loss_pct}%")
    if p95 > a.p95_ms:
        breaches.append(f"p95 {p95:.2f} ms > {a.p95_ms} ms")
    if pdv99 > a.pdv_ms:
        breaches.append(f"PDV p99 {pdv99:.2f} ms > {a.pdv_ms} ms")
    print("SLO: " + ("BREACH: " + "; ".join(breaches) if breaches else "met"))
    return 1 if breaches else 0


if __name__ == "__main__":
    sys.exit(main())

1. Run ping -n -D -O -i 0.2 -c 300 <host> | python3 pingkpi.py --p95-ms 60 --loss-pct 1 --pdv-ms 30.
2. Repeat at a busy and a quiet hour; compare p50, p95 and PDV, and decide which objective users feel.

PDV is measured against the minimum delay, which RFC 5481 prefers for service levels (“The one-sided PDV distribution can be constrained with a single statistic, such as an upper percentile, so it is preferred”);^[20] the smoothed figure is RFC 3550’s estimator, J(i) = J(i-1) + (|D(i-1,i)| - J(i-1))/16.^[77] Both are applied to round-trip times.

Expected output: three lines of figures and an SLO: line; exit 0 when met, 1 on a breach. The synthetic test (8 sent, 2 lost in a row, one duplicate) printed loss=25.00% longest_loss_burst=2, p95=60.00, pdv_p99_ms=20.00 and rfc3550_style_jitter_ms=3.12, matching a hand calculation; example.com gave 0% loss, p95 1.11 ms. Validation: loss and min agree with ping’s summary (ping’s min also counts duplicates). Troubleshooting: no ping summary → ping was interrupted, or its output is localised; run with LC_ALL=C. Teardown: none for the ping run; the optional netem block ends by deleting its namespace.

Optional (root, disposable VM only): impair a path you own. netem adds delay and loss on egress.^[78] The namespace uses the documentation range 192.0.2.0/24.

sudo ip netns add lab
sudo ip link add v0 type veth peer name v1
sudo ip link set v1 netns lab
sudo ip addr add 192.0.2.1/24 dev v0 && sudo ip link set v0 up
sudo ip netns exec lab ip addr add 192.0.2.2/24 dev v1
sudo ip netns exec lab ip link set v1 up
sudo tc qdisc add dev v0 root netem delay 40ms 8ms loss 2%
ping -n -D -O -i 0.2 -c 300 192.0.2.2 | python3 pingkpi.py --p95-ms 60 --loss-pct 1
sudo ip netns del lab        # teardown: deleting v1 also removes its peer v0

Expect a median near 40 ms and loss near 2%, varying between runs: 300 packets estimate 2% coarsely.

Objective: compute utilisation as an NMS does, handle counter wraps and resets, and see a 32-bit counter under-report a busy link. Prerequisites: Python 3.8 or later on Linux; 10 minutes. Cost and risk: none; it reads /sys/class/net/<if>/statistics, which holds the same 64-bit counters that ip -s link shows.^[40]

Example status: verified by execution on 2026-10-18 (Python 3.8.10): --selftest, and live on interface ens5 at a 2-second interval.

#!/usr/bin/env python3
"""ifrate.py: interface utilisation from byte counters, as an NMS computes it.

Live:     python3 ifrate.py IFACE --speed-mbps 1000 [--interval 5] [--samples 6]
Selftest: python3 ifrate.py --selftest   (shows why 32-bit counters mislead)
Reads /sys/class/net/IFACE/statistics: 64-bit counters, like IF-MIB ifHCInOctets.
"""
import argparse
import math
import sys
import time


def delta(prev, cur, bits, seconds, speed_bps):
    """Bytes between two readings, or None for a discontinuity (reset)."""
    if cur >= prev:
        d = cur - prev
    elif bits == 32:
        d = cur + 2**32 - prev  # assume exactly one wrap: the best guess
    else:
        return None  # a 64-bit counter does not wrap in practice: a reset
    if d * 8 / seconds > speed_bps:
        return None  # faster than the line: the reading cannot be real
    return d


def p95(values):
    s = sorted(values)
    return s[max(0, math.ceil(0.95 * len(s)) - 1)]


def selftest():
    speed, true_bps = 1e9, 600e6  # 1 Gb/s link carrying 600 Mb/s
    for bits, poll in ((64, 300), (32, 300), (32, 20)):
        counter, prev, got = 0, 0, []
        for _ in range(4):
            counter += int(true_bps / 8 * poll)
            reading = counter % 2**bits
            d = delta(prev, reading, bits, poll, speed)
            got.append("reset?" if d is None else f"{d * 8 / poll / 1e6:.0f}")
            prev = reading
        print(f"{bits}-bit counter, poll {poll:>3}s, true 600 Mb/s -> "
              f"measured Mb/s: {', '.join(got)}")
    d = delta(5_000_000, 1_000, 64, 300, speed)
    print(f"64-bit counter falls from 5000000 to 1000 -> {d} (discard)")


def live(a):
    base = f"/sys/class/net/{a.iface}/statistics/"
    names = ("rx_bytes", "tx_bytes", "rx_errors", "rx_dropped")

    def read():
        return {n: int(open(base + n).read()) for n in names}

    speed = a.speed_mbps * 1e6
    util = {"rx": [], "tx": []}
    prev, t0 = read(), time.monotonic()
    for _ in range(a.samples):
        time.sleep(a.interval)
        cur, t1 = read(), time.monotonic()
        sec = t1 - t0
        out = []
        for k in ("rx", "tx"):
            d = delta(prev[k + "_bytes"], cur[k + "_bytes"], a.bits, sec, speed)
            if d is None:
                out.append(f"{k}=discontinuity")
                continue
            util[k].append(100 * d * 8 / sec / speed)
            out.append(f"{k}={d * 8 / sec / 1e6:8.3f} Mb/s ({util[k][-1]:.3f}%)")
        errs = cur["rx_errors"] - prev["rx_errors"]
        drops = cur["rx_dropped"] - prev["rx_dropped"]
        print(" ".join(out) + f" rx_err+{errs} rx_drop+{drops}")
        prev, t0 = cur, t1
    for k, v in util.items():
        if v:
            print(f"{k}: p95 of {len(v)} samples = {p95(v):.3f}% of "
                  f"{a.speed_mbps:g} Mb/s")


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("iface", nargs="?")
    ap.add_argument("--speed-mbps", type=float)
    ap.add_argument("--interval", type=float, default=5.0)
    ap.add_argument("--samples", type=int, default=6)
    ap.add_argument("--bits", type=int, choices=(32, 64), default=64)
    ap.add_argument("--selftest", action="store_true")
    args = ap.parse_args()
    if args.selftest:
        selftest()
    elif args.iface and args.speed_mbps:
        live(args)
    else:
        sys.exit("give IFACE and --speed-mbps, or --selftest")

1. Run python3 ifrate.py --selftest.
2. Run it live: python3 ifrate.py <if> --speed-mbps <rate>. Virtual NICs often report no speed (cat /sys/class/net/<if>/speed fails), so pass the rate you are entitled to, such as a cloud instance’s documented bandwidth.
3. Explain the selftest’s second line using RFC 2863’s arithmetic in Metrics and KPIs.

Expected output: the selftest prints 600 four times for the 64-bit counter at 300 s polls, 27 four times for the 32-bit counter at 300 s, 600 four times for the 32-bit counter at 20 s, and None (discard) for the reset; live mode prints one line per interval and a p95 per direction. Validation: 27 Mb/s is 22.5 GB per 300 s modulo 2^32: plausible, wrong and silent. Troubleshooting: FileNotFoundError → wrong interface name (ls /sys/class/net); discontinuity lines after an interface reset are expected and discarded. Teardown: none.

1. Rogers, Canada, July 2022 (sourced). A configuration clean-up removed a routing filter; the core routers were flooded and crashed within minutes, and more than 12 million customers lost service for about a day (change review: NetworkOps). For the NOC, the independent assessment published by the regulator, the CRTC, is blunt: Rogers “had a management network that relied on the Rogers IP core network”; staff “could not pinpoint the root cause for about 14 hours”; and “Rogers had completed multiple configuration changes during the maintenance window”, which made the rollback choice hard. Its lessons include separating management “physically and logically from the data network”, backup NOC connectivity from third-party operators, fewer changes per window and alarm prioritisation.^[79] Success metric: every core device reachable by a path that does not use the core.

2. Facebook, October 2021 (sourced). During routine maintenance, “a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network”; a bug in the audit tool failed to stop it. DNS servers then withdrew their BGP advertisements, and “Our primary and out-of-band network access was down”, so engineers went on site, slowed by deliberate security; “storm” drills helped the restart.^[80] Lessons: out-of-band access must not share fate with the backbone; rehearse physical access; restore load gradually.

3. Volumetric DDoS at the edge (illustrative scenario). A 10 Gbit/s transit link sits at line rate and every site behind it loses packets; flows show most traffic going to one public address from thousands of sources. Method: flows name the target in minutes; the NOC runs the runbook agreed with the SOC, which owns attribution. Options: (a) blackhole the /32 at the upstream with a BGP community such as RFC 7999’s BLACKHOLE (“drop any traffic being sent towards this prefix”), agreed with each upstream beforehand: the network recovers, but the target goes dark;^[81] (b) a BGP Flowspec rule that drops only the attack’s pattern (protocol, port, packet length): the service stays up, but a broad rule drops legitimate traffic, and RFC 8955 warns that badly programmed controllers “may announce unintended Flow Specifications”;^[82] (c) a scrubbing provider under contract, slower to start and paid for. Success metric: mitigated within the runbook’s target time; utilisation and site loss back to baseline.

4. Branch voice quality (illustrative scenario). Calls at a branch are choppy every afternoon, yet the 100 Mbit/s uplink peaks at 55% (five-minute average). Lab 2’s probe shows PDV p99 of 45 ms from 14:00 to 16:00 while uplink discards rise: file-sync bursts fill the queue for seconds. Method: probes and discard counters, then flows to name the traffic. Options: a voice priority queue, shaping, or a larger uplink. Success metric: PDV p99 under 30 ms in business hours.

Time to detect, acknowledge and restore are defined on IncidentOps.

Worked calculation 1: counter wraps. A 32-bit counter holds 2^32 bytes, about 4.29 GB. At 1 Gbit/s that is 2^32 × 8 ÷ 10^9 ≈ 34.4 s, matching RFC 2863’s “at 1Gbs, the minimum is 34 seconds”.^[12] Polled every 300 s, the counter can wrap several times between readings and the poller cannot tell (Lab 3 reports 27 Mbit/s for a true 600). Use the 64-bit ifHC counters; RFC 2863 requires them for interfaces faster than 20 Mbit/s.^[12]

Worked calculation 2: what loss costs one TCP flow. Mathis and colleagues modelled Reno-style congestion avoidance as throughput ≈ (MSS ÷ RTT) × (C ÷ √p), with C ≈ 1.22.^[83] With MSS 1460 bytes, RTT 40 ms and p = 1%: (11,680 bits ÷ 0.04 s) × (1.22 ÷ 0.1) ≈ 3.6 Mbit/s per flow; at p = 0.01%, about 36 Mbit/s. CUBIC, the default in the Linux, Windows and Apple stacks,^[84] behaves differently, so treat this as an order of magnitude: small loss on a long path throttles every flow.

Worked calculation 3: runway for a redundant pair (illustrative numbers). Links A and B are 1 Gbit/s each, with p95 loads of 380 and 350 Mbit/s. If one fails, the survivor carries 730 Mbit/s, 73% of its capacity. Combined growth is 12 Mbit/s a week; with a planning ceiling of 85% (850 Mbit/s, this example’s choice), runway is (850 − 730) ÷ 12 = 10 weeks. If the carrier quotes 12 weeks to upgrade, the order is already late, although each link alone looks half empty.

Choosing targets. ITU-T Y.1541 Class 0, for real-time interactive traffic such as voice, sets upper bounds of 100 ms mean transfer delay, 50 ms delay variation and a 1 × 10^-3 loss ratio for public IP networks, with “An evaluation interval of 1 minute” suggested.^[53] For planning, G.114 recommends “to not exceed a one-way delay of 400 ms”, and below 150 ms “most applications, both speech and non-speech, will experience essentially transparent interactivity”.^[54] Set internal objectives from what users feel, then check that the carrier contract can deliver them.

These targets are not the statistics Lab 2 prints. Y.1541 bounds the mean one-way delay between the user-network interfaces, and its delay variation is the 1 − 10^-3 quantile of one-way delay minus the minimum: a p99.9 PDV, needing at least 1,000 packets per sample. G.114’s figures are one-way and end to end (“mouth-to-ear” for speech).^[53][54] Lab 2 prints round-trip p95 and p99 PDV. A round trip is never shorter than either leg, so a round-trip mean within a one-way bound meets it; a p99 PDV within 50 ms does not prove a p99.9 within it.

• Management-plane security. SNMPv3 with privacy; management protocols only from collector and jump-host addresses; the quarterly audit in Checklists.
• Separation from security monitoring. MITRE advises keeping network and security operations as distinct peers, which “will help avoid sidelining security in the name of network availability”.^[7] Suspected attacks go to SOC Ops; the NOC mitigates link saturation under a runbook agreed with the SOC.
• Privacy. Flow records, logs and captures contain IP addresses that can identify people; set retention with PrivacyOps and limit who may capture packets.
• Regulation. Telecom operators may answer to regulators: the CRTC published the independent Rogers assessment.^[79] Duties vary by country, sector and contract; this page is not legal advice.

• Utilisation targets depend on design. Google’s B4 paper notes that “WAN links are typically provisioned to 30-40% average utilization”, while B4’s traffic engineering lets “all links to average 70% utilization over long time periods”, using control of the applications at the edge that most networks do not have.^[85]
• Cloud bandwidth is an allowance. Smaller EC2 instances have a baseline and burst “up to” a figure for limited periods;^[86] the ENA driver counts packets “queued or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance” (bw_in_allowance_exceeded in ethtool -S).^[87]
• People. A 24×7 seat is 168 × 52 = 8,736 hours a year. At 1,736 productive hours per person (2,080 contract hours minus about 43 days of leave, holidays, sickness and training, as on SOC Ops), a seat needs about 5.0 people. MITRE agrees: “Each 24x7 seat requires roughly five full time equivalents (FTEs), including fill-in for vacation and sick leave”.^[7] Google’s SRE book derives a higher minimum, eight for a single-site 24/7 rotation, because it assumes two people on call at once and at most 25% of anyone’s time on call.^[88]
• Waste. Unused circuits keep billing; review the inventory with FinOps and GreenOps.

Situation. A branch’s probe loss alert fires three times in an afternoon, each for a few minutes. The branch has one carrier circuit and a cellular backup.

1. Stabilise. If voice or payments suffer, move those classes to the backup; record the time.
2. Collect evidence from your end with the block below, during an episode if possible.
3. Rule out your side. Interface errors at both ends of your link; utilisation and discards during the episodes; any change in the last 48 hours.
4. Open the carrier ticket from the template, with traces in both directions and what you ruled out. Ask for an intrusive test only in an agreed window.
5. Escalate on the contract clock, not when you feel like it.
6. Verify and close. Probe loss below objective for an agreed period after the repair; request the reason for outage; tune the alert if it fired late.

Example status: syntax-checked; run unprivileged against a public host on Ubuntu 20.04 (2026-10-18) in 222 s (mtr works without root there because mtr-packet has CAP_NET_RAW). Its mtr report showed 24% loss at hop 3 and 0% at hops 4 and 5, the destination: trap 1 in real data. The tcpdump line was not run (root; personal data).

# Evidence pack for a carrier ticket: read-only except the optional capture.
T=203.0.113.10                       # far end of the circuit (placeholder)
out=evidence-$(date -u +%Y%m%dT%H%MZ).txt
{
  date -u
  ip -s link show dev eth0           # your end: errors and drops
  ping -n -D -c 200 -i 0.2 "$T" | tail -n 2
  tracepath -n "$T"
  mtr -rn -c 100 "$T"                # per-hop loss: read it with the traps
  ss -tio dst "$T"                   # RTT and retransmissions on live sessions
} > "$out" 2>&1
echo "saved $out"
# Optional, root, change ticket first: 60 s of headers plus the first payload bytes.
# Send the carrier summaries, not the pcap: payload bytes can hold personal data.
# sudo timeout 60 tcpdump -ni eth0 -s 96 -w "${out%.txt}.pcap" host "$T"

Roll back: nothing changes except the optional capture file; delete captures when the ticket closes.

• “It’s the network” or “never the network”. Both are guesses; the isolation table is the answer.
• Ping as the service test. Routers answer ICMP on a slow, rate-limited path, and many networks filter it; probe the service’s protocol and port.
• Treating an emergency mitigation as too urgent to check. At Cloudflare in July 2020, while relieving backbone congestion, “a one line change started leaking all BGP routes into the backbone”; the outage lasted 27 minutes. The correct change “would have been to deactivate the term instead of the prefix-list”, and the fixes include a maximum-prefix limit on backbone BGP sessions.^[89] The post does not describe Cloudflare’s review process; the lesson is this Atlas’s inference: a mitigation is a change, so read what it will announce and keep a limit that catches what you miss.
• Myth: a redundant pair at 60% each is safe. One failure leaves 120% of a link.

Build a NetOps function when you operate your own campus, WAN, data-centre or service-provider network; buy carrier circuits for many sites; owe customers network service levels; or face a regulator. Invest less when a provider runs the network (SaaS office, managed SD-WAN, cloud-only): the work becomes managing the provider (service levels, evidence, escalation) and the rest moves to CloudOps. Hidden responsibilities: contracts and contacts, out-of-band access, inventory and spares, a DDoS runbook agreed with each upstream, a 24×7 rota.

flowchart TB
  Q0{"Do you operate the devices and circuits?"}:::decision
  Q0 -->|"no"| A0["Manage the provider: SLAs, evidence, escalation"]:::neutral
  Q0 -->|"yes"| Q1{"Does an outage out of hours stop the business?"}:::decision
  Q1 -->|"no"| A1["Business-hours team plus on-call"]:::neutral
  Q1 -->|"yes"| Q2{"Can you fund about five people per 24x7 seat?"}:::decision
  Q2 -->|"no"| A2["Hybrid: managed Tier 1 watch, in-house Tier 2 and 3"]:::core
  Q2 -->|"yes"| Q3{"Staff in several time zones?"}:::decision
  Q3 -->|"yes"| A3["Follow-the-sun NOC"]:::core
  Q3 -->|"no"| A4["In-house 24x7 shifts"]:::core
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

The tree assumes the staffing arithmetic in Cost, capacity and sustainability and a managed watch that escalates with evidence; one that only forwards alerts adds a hop, not judgement.

This is this Atlas’s model, not an industry standard.

Onboarding a device or circuit

• In the inventory with site, role, owner; circuits with ID, contract and escalation path
• SNMPv3 (or telemetry) polling and syslog working; one test alert received
• A probe covers the path users take; baseline recorded
• Out-of-band access tested; configuration backup confirmed

Before and after a change window

• Pre-checks captured; rollback written and timed; silences scoped and expiring
• Few changes, each independently reversible
• Post-checks match pre-checks; observation period completed before closing

Every shift

• Handover note read and signed; open carrier tickets checked against contract clocks
• No silence outlives its change

Quarterly security and governance audit

• No SNMPv1/v2c; exporters behind TLS and basic auth; out-of-band accounts named, logged and reviewed
• Every alert rule has an owner, a runbook and a review date; flow, log and capture retention as agreed

CARRIER TROUBLE TICKET
Circuit ID / A-end / B-end:  <id> / <site, address> / <site, address>
Impact:                      <down | loss | delay | errors>, <users or services>
Start (UTC) / still active:  <YYYY-MM-DD hh:mm> / <yes | intermittent: times>
Evidence:                    loss <x%> RTT p95 <y ms> over <n> probes; traces both ways
Ruled out on our side:       <interface errors, utilisation, changes in last 48 h>
Requested:                   <investigation | intrusive test window | RFO>
Our contact (24x7):          <NOC phone, ticket address>

SHIFT HANDOVER (NOC)
Shift / engineer:            <date, shift> / <name>
Open incidents:              <id, state, next action, owner>
Carrier tickets:             <carrier ref, circuit, next escalation time>
Changes in flight:           <change id, window end, post-checks done?>
Active silences:             <matcher, expiry, reason>
Watch items:                 <flapping links, rising errors, capacity alerts>

Is NetOps the same as NetworkOps or NetDevOps? In industry usage, largely yes.^[5] This Atlas splits them editorially: the running network here, the engineering of change (NetDevOps included) on NetworkOps.

Do I still need SNMP with streaming telemetry? Often: switches, routers and UPS units are usually SNMP-enabled.^[58] Use SNMPv3 and 64-bit counters.^[56][12]

Can cloud flow logs replace probes? No. They are delayed, incomplete by design and, in some clouds, sampled.^[31][33]

Should the NOC also be the SOC? Small organisations often share a watch floor; MITRE recommends distinct people, tools and funding so availability pressure does not crowd out security.^[7]

When does a network fault become an incident? When customers or several teams are affected, or the fix needs decisions beyond the network team: hand command to IncidentOps.

• Streaming telemetry at scale. Sub-minute counters expose bursts and flaps but multiply data volume.^[17]
• Active measurement between devices. STAMP measures one-way and round-trip delay, delay variation and loss between network endpoints,^[18] and BFD detects path failures within the network quickly.^[90]

• Decide: who runs the network out of hours; which paths matter; what objectives users need.
• Measure: probes per site (loss, delay p95, PDV), 64-bit interface counters, discards, alert actionability, capacity runway.
• Do: scope, then isolate bottom-up along the path; evidence before escalation; few verified changes per window; out-of-band drills.
• Watch out for: cosmetic traceroute spikes, PMTU black holes, 32-bit counters, monitoring that shares the failed path, exporters anyone can query, redundant pairs above 50%.

Situation (illustrative). At 10:00 a branch’s 1 Gbit/s primary circuit starts losing about 2% of packets in bursts; the carrier ticket is open with no estimate. The backup is a 200 Mbit/s circuit; the branch’s p95 load is 260 Mbit/s, of which voice and payments are 30 Mbit/s.

Decision: C if the routers already support class-based routing to the backup: 30 Mbit/s fits easily and bulk traffic keeps the primary’s capacity. Without it, choose B and escalate hard: A would push 260 Mbit/s into 200 all afternoon, while the primary loses packets only in bursts. Prepare C before the next fault. Uncertainty: the loss model describes Reno-style TCP; CUBIC flows and real-time media degrade differently, and the bursts may worsen.

After this page you can explain what NetOps covers and how the label is used; isolate a fault layer by layer; compute utilisation, loss and delay KPIs; escalate to a carrier with evidence; decide a NOC model.

1. Traceroute shows 80 ms at hop 6 and 12 ms at hops 7 to 10. Is hop 6 congested?

2. Users can ping a remote server and log in, but file transfers stall. What do you test first?

3. A 1 Gbit/s interface polled every 5 minutes through ifInOctets shows 30 Mbit/s at peak. Why distrust it?

4. Two 10 Gbit/s links each run at 6 Gbit/s p95. Is the pair redundant?

5. Name two design changes that the Rogers assessment recommends for any operator’s NOC.

6. A colleague exposes snmp_exporter on all addresses without authentication, “because it only reads”. What is the risk?

NetOps is an old job under a contested label. The job stays: a network fails partially and silently, and somebody must measure the paths users take, read the evidence honestly, and keep a way in when the network itself is down.

Claim-by-claim evidence: research/netops-ledger.md. Re-check tool versions, cloud flow-log behaviour and AWS bandwidth rules every six months.

Changelog. 2026-10-18, r0: first version (research, ledger, labs executed where safe). 2026-10-18, r1: revised after the independent review (exporter security, paging rules, DDoS, hardware faults, OSI mapping, targets, staffing, dates). 2026-10-18, post-score polish: IPv6 path MTU, ifAlias anchoring, sampling error, FCAPS sources, BGP row, audit checklist, Figure 1 retry branch; repeats trimmed.