--- title: "NetOps" slug: netops category: cloud term_status: overlapping-synonym evidence_status: verified summary: "NetOps (network operations) is the day-to-day work of running a network: watching its signals, isolating faults, escalating to carriers, running change windows and planning capacity. The label is shorthand; vendors also use it for NetDevOps." audience: [Beginners, NOC engineers, Network administrators, On-call and SRE engineers, Operations managers] last_verified: 2026-10-18 key_takeaways: - "NetOps is shorthand for network operations, and vendors also use it for NetOps 2.0 or NetDevOps. This Atlas teaches the running network here and network automation on NetworkOps; the split is editorial." - "Measure paths as well as boxes: device counters say a link is up, probes from the user's side say whether the service works." - "Isolate systematically: scope the fault first, then walk the layers bottom-up along the path. A traceroute spike or loss that does not continue to later hops is usually the router de-prioritising or rate-limiting the ICMP replies it generates, not a forwarding fault." - "Small loss is expensive: under Reno-style congestion control, 1% loss at 40 ms round-trip time caps one TCP flow near 3.6 Mbit/s, whatever the link speed." - "Management access must not depend on the network it manages: Rogers (2022) and Facebook (2021) both lost remote access during their outages." - "Change windows need pre- and post-checks and few changes each; capacity plans must hold when one link of a pair fails." related: [networkops, itops, incidentops, observabilityops, cloudops, kubernetesops, edgeops, sysops, soc-ops] --- ## What NetOps is ### In 30 seconds **NetOps** (network operations) is the day-to-day work of keeping a network available and performing: watching signals from devices and paths, detecting and isolating faults, escalating to carriers and vendors, running change windows, and planning capacity before demand arrives. It is usually done by a network operations centre (NOC) or an on-call network team. The label is shorthand, and it has a second meaning: vendors also use "NetOps" for NetOps 2.0 or NetDevOps, the automation of network change. This page covers the running network; [NetworkOps](../networkops/) covers engineering change. It is for NOC engineers, network administrators, on-call engineers who must tell "the network" from everything else, and managers deciding how to staff network operations. ### At a glance | Question | Answer | |---|---| | Goal | Networks that are reachable, fast enough and repairable, with faults found before users report them | | Users | NOC engineers, network administrators, on-call engineers, operations managers | | Scope on this page | Signal sources (SNMP, flows, logs, probes), fault isolation, carrier and hardware escalation, DDoS mitigation on links, change windows, capacity, NOC staffing | | Out of scope | Network automation, YANG and source of truth → [NetworkOps](../networkops/); virtual private cloud (VPC) design → [CloudOps](../cloudops/#network-foundations); pod networking (the Container Network Interface, CNI) → [KubernetesOps](../kubernetesops/#reference-architecture); attack detection and attribution → [SOC Ops](../soc-ops/); Wi-Fi radio design (not covered) | | Prerequisites | IP addressing, routing and TCP basics; a Linux shell helps | | Key outputs | Alerts with runbooks, triage notes naming the failing segment, carrier tickets with evidence, verified changes, capacity forecasts | | Term status | Overlapping synonym for network operations (see [What NetOps is not](#what-netops-is-not)) | ### The problem it solves A network fails in partial ways. A link can be up and still drop 2% of packets; a path can work in one direction only; a firewall can pass pings and silently break large transfers. Users see "the application is slow", and every team suspects another. | Without a NetOps practice | With one | |---|---| | "It's the network" is argued, not measured | Probes alert first; triage names the failing segment | | Carrier tickets stall for lack of evidence | Tickets carry timestamps, loss and path data from both ends | | Several changes per window, rollback guesswork | Few changes, pre- and post-checks, a rollback trigger | | Remote access dies with the network | Out-of-band access tested every quarter | ### What NetOps is not - **Not an industry standard.** No standards body or professional association defines "NetOps". The practice behind it is old and documented under other names: ITU-T M.3400 (2000) lists the management functions of telecommunication networks, with fault management meaning functions that enable "the detection, isolation and correction of abnormal operation of the telecommunication network and its environment".[^m3400] The US occupational classification describes network support specialists who "Analyze, test, troubleshoot, and evaluate existing network systems".[^onet] The only formal definition found is military: a 2008 US Department of Defense instruction defined NetOps as its capabilities for operating and defending its Global Information Grid.[^dodi] - **Two meanings in trade use.** TechTarget notes that historically "NetOps has simply been shorthand for network operations, or the running of the network", and that the term also means NetOps 2.0, NetDevOps or network as code.[^tt-netops2] Red Hat writes that NetOps 2.0 was shortened to NetOps and that the term now "generally refers to the agile methodology";[^rh-netops] Kentik describes both uses.[^kentik] Vendor definitions are usage evidence, not standards. - **Not a security operations centre.** MITRE's SOC handbook says "the focus of a NOC is typically on availability and service level agreements, while the focus of the SOC is on data protection and cyberspace defense".[^mitre] Attack detection, attribution and the security case belong to [SOC Ops](../soc-ops/) and [SecOps](../secops/). Keeping links usable during a volumetric DDoS (distributed denial-of-service) attack is network work, done with the SOC ([Use cases](#use-cases-and-scenarios)). ::: note Term status and editorial distinction This Atlas classes NetOps as an **overlapping synonym**: shorthand for network operations, a long-established practice, and a near synonym of [NetworkOps](../networkops/). NetOps *consumes* telemetry and runs the network; NetworkOps *engineers* change (models, source of truth, pre-change verification, CI/CD). That split is an **editorial distinction** that no standard defines. It also runs against current vendor usage, which leans towards the automation meaning: if you came for NetOps 2.0, read NetworkOps next. ::: ## Where NetOps comes from and where it sits ### Origins and evolution Network operations is older than the Internet. AT&T opened a Network Control Center in New York in 1962, fed in real time from its most important toll switches, and replaced it with a Network Operations Center in Bedminster, New Jersey, in 1977.[^att] - **Management standards.** ITU-T M.3400 (first edition 1992, revised 2000) organised telecom management into the five functional areas of the OSI management framework (ITU-T X.700, 1992), known as FCAPS.[^m3400][^rfc6632] The IETF still runs an Operations and Management Area that is "logically divided into two separate functions: Network Management and Operations".[^ietf-ops] - **Telemetry generations.** SNMP polling, then 64-bit interface counters (IF-MIB, RFC 1573, 1994; current text RFC 2863);[^rfc1573][^rfc2863] flow export with NetFlow v9 (2004), sFlow v5 (2004) and IPFIX (RFC 5101, 2008; revised as RFC 7011);[^rfc3954][^sflow][^rfc5101][^rfc7011] then streaming telemetry, which RFC 9232 (2022) contrasts with SNMP, CLI and syslog monitoring.[^rfc9232] - **The label.** After the 2008 DoD definition, TechTarget credits Gartner analyst Andrew Lerner with coining "NetOps 2.0" in 2017, which moved the trade meaning towards automation.[^tt-netops2] ### Glossary and mental model | Term | Meaning | |---|---| | NOC | Network operations centre: the team (often also a room) that watches and repairs the network | | FCAPS | Fault, configuration, accounting, performance, security: the M.3400 management areas[^m3400] | | `ifHCInOctets` | IF-MIB 64-bit count of bytes received on an interface; utilisation is computed from its rate of change[^rfc2863] | | Trap, syslog | Messages a device pushes when something happens (link down, neighbour lost) | | Flow record | A summary of packets sharing properties (addresses, ports, protocol) seen at one point during an interval[^rfc7011] | | Active probe | A test packet stream you send to measure reachability, delay and loss (ping, TWAMP, STAMP, HTTP checks)[^rfc8762] | | RTT, one-way delay | Round-trip time (what ping shows) versus delay in one direction, which needs synchronised clocks[^rfc7679] | | PDV, IPDV, jitter | Delay variation against the minimum delay (PDV) or against the previous packet (IPDV); "jitter" is the loose name[^rfc5481] | | PMTU | Path MTU: the largest packet that crosses the whole path unfragmented[^rfc2923] | | Out-of-band (OOB) | Management access that avoids the production network: console servers, a separate circuit, cellular | | Acronyms | SNMP: Simple Network Management Protocol; ICMP: the protocol of ping and traceroute replies; BGP: Border Gateway Protocol, routing between networks; VLAN: virtual LAN; CRC: frame checksum; MSS: TCP maximum segment size; NMS: network management system; SLO: service-level objective | **Mental model.** A network is a set of paths, each with four measurable properties (reachable, delay, loss, capacity) and some control-plane state (routes, sessions, neighbours). Symptoms appear at the edges; causes hide in the middle. NetOps runs a loop. ::: figure The NetOps operating loop ```mermaid flowchart TB O["Observe: counters, flows, logs, probes"]:::core D{"Deviation from baseline or objective?"}:::decision O --> D D -->|"no"| O D -->|"yes"| I["Isolate the failing layer and segment"]:::core I --> R["Restore: reroute, repair or escalate"]:::core R --> V{"Verified from the user's side?"}:::decision V -->|"no"| I V -->|"yes"| P["Prevent: fix the cause, tune the alert, plan capacity"]:::core P --> O classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 ``` Text equivalent: NetOps observes counters, flows, logs and probes and asks whether anything deviates from its baseline or objective. If not, it keeps observing. If so, it isolates the failing layer and segment, restores service (reroute, repair or escalate) and verifies from the user's side, isolating again until service is restored. Then it prevents recurrence (fix the cause, tune the alert, plan capacity) and observes again. ::: ### How NetOps relates to other Ops | Neighbour | Shared scope | NetOps (this page) | The neighbour | |---|---|---|---| | [NetworkOps](../networkops/) | Network change | Runs windows, verifies health after changes | Network as code, source of truth, pre-change tests | | [CloudOps](../cloudops/) | Cloud connectivity | Monitors cloud links, reads flow logs | VPC, transit and private-access design | | [KubernetesOps](../kubernetesops/) | Packet paths | Physical and cloud network under the nodes | CNI plugin, pod and service networking | | [ITOps](../itops/) | Process | Network runbooks and carrier escalation | Incident, change and SLA processes, CMDB | | [IncidentOps](../incidentops/) | Outages | Diagnosis and repair of the network | Incident command and communication | | [ObservabilityOps](../observabilityops/) | Telemetry | Network signals and their meaning | Telemetry pipeline and platform | | [SOC Ops](../soc-ops/) | Watch floors | Availability and service levels | Detecting and responding to attacks | | [EdgeOps](../edgeops/) | Remote sites | WAN links and branch connectivity | Edge compute fleets | Hosts belong to [SysOps](../sysops/), event correlation to [AIOps](../aiops/), service-level objectives to [SRE](../sre/). ## Principles and operating model ### Charter and principles **Charter (this Atlas's proposal, not a published standard).** Know the state of every path that matters, find faults before users do, restore service with evidence in hand, and change the network without surprises. 1. **Measure paths, not only boxes.** Device counters say an interface is up; probes from the user's side say whether the service works. Use both. 2. **Alert on symptoms, route on causes.** Google's SRE book separates what is broken (the symptom) from why (the cause) and warns that frequent pages make engineers "second-guess, skim, or even ignore incoming alerts".[^sre-mon] 3. **Evidence before escalation.** A carrier acts on timestamps, loss figures and path data, not on "it's slow". 4. **Management access must not depend on the network it manages** (Rogers and Facebook: [Use cases](#use-cases-and-scenarios)). 5. **Few changes per window, each verifiable.** Every change has pre-checks, post-checks and a rollback trigger. 6. **Plan capacity for the failure case.** A redundant pair is only redundant if one link can carry both loads. 7. **Every alert has an owner, a runbook and a review date.** ### Fundamentals: the signal sources | Source | What it tells you | Mechanism | Blind spots | |---|---|---|---| | SNMP polling | Interface state, byte, error and discard counters, device health | Collector reads MIB objects every 1–5 minutes | Averages hide bursts; v1/v2c send the community string in clear, with no cryptographic authentication[^snmpexp][^rfc3410] | | SNMP traps, syslog | Events: link down, neighbour lost, configuration saved | Device pushes messages, mostly over UDP | Lost exactly when the network is congested;[^rfc5426] acknowledged SNMP INFORMs[^rfc3416] and syslog over TLS[^rfc5425] reduce the loss | | Flow export | Who talks to whom, how much, on which ports | NetFlow v9, IPFIX or sampled sFlow records[^rfc3954][^rfc7011][^sflow] | 1-in-N sampling: counts are scaled up by N, and fewer samples mean larger errors (small flows, short intervals);[^sflow-basics] UDP export loss; no payload | | Streaming telemetry | High-rate counters and state changes | Subscriptions, for example gNMI `SAMPLE` for counters and `ON_CHANGE` for state[^gnmi] | Model and subscription design belong to [NetworkOps](../networkops/) | | Routing monitoring | BGP sessions, prefix counts, route changes | BGP Monitoring Protocol (BMP)[^rfc7854], session state | Shows control plane, not forwarding | | Active probes | Reachability, delay, loss and delay variation along a path | ICMP, TCP or HTTP tests; TWAMP or STAMP between endpoints[^rfc8762] | Probes take the probe's path, not necessarily the user's | | Cloud flow logs | Accepted and rejected flows on cloud interfaces | Provider-generated records | Not real time and not complete (below) | | Packet capture | Ground truth for one segment | `tcpdump`, Wireshark | Needs privilege, holds personal data, does not scale | Cloud caveats (as of October 2026): AWS flow logs "do not capture real-time log streams for your network interfaces"[^aws-flow-basics] and skip traffic such as Amazon DNS queries and DHCP;[^aws-flow-lim] Google Cloud samples packets.[^gcp-flow] ### Roles, ownership and RACI R = responsible, A = accountable, C = consulted, I = informed. Tiers are common practice, not a standard; small teams merge them. | Activity | NOC Tier 1 | Network engineer (Tier 2/3) | NOC manager | Service owner | Carrier or vendor | |---|---|---|---|---|---| | Watch, acknowledge, scope alerts | R | I | A | I | – | | Isolate the failing layer and segment | R (runbook steps) | R/A | I | C | C | | Escalate to a carrier under contract | R | C | A | I | R (repair) | | Emergency mitigation (reroute, disable) | C | R/A | I | C | – | | Execute a planned change | I | R | A | C | C | | Verify after a change | R (checks) | R/A | I | I | – | | Capacity forecast and upgrade request | C | R | A | C (budget) | C (lead time) | | Alert tuning and runbook upkeep | R | R | A | I | – | Change approval and service levels belong to [ITOps](../itops/#change-enablement-in-depth). ### Lifecycle of a network element - **Day 0 (design).** Site-and-role naming, inventory, out-of-band access, a monitoring profile, clocks synchronised (NTP) so logs and probes line up; for every carrier link the circuit ID, contract and escalation path; for unstaffed sites, spares and a remote-hands contract. - **Day 1 (onboard).** Monitoring added, baseline recorded, one test alert fired, configuration backups confirmed. - **Day 2 (operate).** Alerts, incidents, hardware replacements, change windows, capacity reviews; appliance image upgrades follow the rings on [PatchOps](../patchops/#core-practices-and-playbooks), one peer of a pair at a time. - **Retire.** Remove from monitoring, cancel the circuit, revoke access. ### Operating model and governance | Cadence | Activity | Output | |---|---|---| | Every shift | Handover: open incidents, carrier tickets, changes in flight, silenced alerts | Signed handover note ([Templates](#templates)) | | Daily | Review open carrier tickets against contract times | Escalations sent | | Weekly | Capacity and error trends; noisiest alerts | Upgrade requests; alerts fixed or deleted | | Monthly | KPI review; change outcomes | Scorecard; process fixes | | Quarterly | Out-of-band drill; contact lists checked with carriers and peers | Drill record; updated contacts | MANRS, a routing-security initiative, asks operators to "Maintain globally accessible, up-to-date contact information in common routing databases".[^manrs] A NOC that cannot reach a neighbouring network's NOC at 03:00 has an incomplete runbook. ## Architecture and workflow ### Reference architecture: a NOC monitoring stack ::: figure Reference architecture of a NOC monitoring stack with its out-of-band path ```mermaid flowchart TB subgraph NET["Managed network"] DEV["Routers, switches, firewalls"]:::neutral LNK["Carrier and cloud links"]:::neutral end subgraph COL["Collectors, one set per region"] POLL["SNMP or telemetry"]:::core FLOW["Flow collector"]:::core LOGS["Syslog and traps"]:::core PROBE["Path probes"]:::core end STORE["Metric, flow and log stores"]:::core ALERT["Rules, grouping, inhibition"]:::decision NOC["NOC console, tickets, runbooks"]:::core OOB["Out-of-band access"]:::risk DEV --> POLL & FLOW & LOGS LNK --> PROBE POLL & FLOW & LOGS & PROBE --> STORE STORE --> ALERT --> NOC NOC -.->|"when in-band fails"| OOB OOB -.-> DEV classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: Devices feed three collectors (SNMP or streaming telemetry, flows, syslog and traps); carrier and cloud links are measured by path probes. All four write to metric, flow and log stores. Alert rules with grouping and inhibition feed the NOC console, tickets and runbooks. A dashed out-of-band path, marked as a risk, lets the NOC reach devices when in-band access fails. ::: | Component | Responsibility | Trust boundary and trade-off | |---|---|---| | Collectors | Poll close to the devices | Regional collectors survive a WAN cut; monitor them too | | Path probes | Measure from where users sit | At sites and in clouds, not only the data centre | | Stores | Keep metrics, flows, logs | IP addresses in flows and logs make retention a privacy decision | | Alerting | Few, routed alerts | Alertmanager inhibition is "suppressing notifications for certain alerts if certain other alerts are already firing"[^am] | | Out-of-band | Console access over a separate path | Also an attack path: named accounts, logs, drills | The monitoring path is part of the design. In AWS's December 2021 event, congestion "immediately impacted the availability of real-time monitoring data for our internal operations teams".[^aws-2021] ### End-to-end workflow: from alert to closure ::: figure From alert to closure, including the carrier and hand-off paths ```mermaid flowchart TB A["Alert or user report"]:::neutral --> M{"Planned work?"}:::decision M -->|"yes"| S{"As expected?"}:::decision M -->|"no"| T["Acknowledge; scope who, where, since when"]:::core T --> I["Isolate layer by layer"]:::core I --> W{"Where is the fault?"}:::decision W -->|"our network"| F["Mitigate: reroute, disable, roll back"]:::core W -->|"carrier"| C["Carrier ticket, escalate"]:::risk W -->|"another team"| H["Hand off with evidence"]:::neutral F --> V["Verify from the user's side"]:::core C --> V H --> V S -->|"no"| B["Roll back"]:::core B --> V V --> R["Close; record cause; review if major"]:::core S -->|"yes"| R classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: An alert or user report arrives. Planned work with the expected effect closes; an unexpected effect is rolled back, then verified. Otherwise acknowledge and scope it (who, where, since when) and isolate layer by layer. Mitigate a fault in your network (reroute, disable, roll back), open and escalate a carrier ticket, or hand off to another team with evidence. Every fault path is verified from the user's side, then closed with a recorded cause and, if major, a review. ::: - **Actors:** NOC Tier 1 scopes, a network engineer isolates and mitigates, and an incident commander takes over when several teams or customers are involved ([IncidentOps](../incidentops/)). - **Exception path:** while a carrier works on its contract clock, keep your own mitigation (rerouting) going. - **Output:** a triage note naming the failing layer and segment, the evidence and the action. ## Practices, methods and tools ### Core practices and playbooks | Practice | Trigger | Output | Verification | |---|---|---|---| | Alert triage | Alert or report | Scope: users, sites, start time | Every alert acknowledged within target | | Fault isolation | Confirmed symptom | Failing layer and segment named | A test that failed now passes | | Carrier escalation | Fault outside your network | Ticket with circuit ID and evidence | Carrier reference; contract clock tracked | | Hardware replacement | Dead optic, power supply or card | Vendor case with serial number and logs; spare or RMA | Part replaced under a written procedure; post-checks pass | | Change window execution | Approved change | Pre- and post-check records | Post-checks match pre-checks | | Maintenance suppression | Planned work | Silences scoped to the devices and time | Silences expire; none left open | | Capacity review | Weekly or monthly | Forecast per link and pair | Upgrades ordered before lead time runs out | | Out-of-band drill | Quarterly | Console reached on sample devices | Drill record | **Carrier escalation.** Give the circuit ID, impact, start time in UTC and evidence from both ends (counters, probe loss and delay, a trace each way). Allow intrusive tests (a loopback) only in a window you can afford. Track the contract clock yourself, escalate on its schedule, and ask for a reason for outage (RFO) before closing. Read the contract's metric definitions before arguing a number: MEF 10.4, for example, defines one-way frame delay, inter-frame delay variation, frame loss ratio and availability.[^mef] Carrier contracts are underpinning contracts in [ITOps](../itops/#service-levels-slas-olas-and-underpinning-contracts) terms. **Hardware faults.** Tell the vendor what you have already swapped. Keep spares where the contract's replacement time (RMA, return merchandise authorisation) exceeds the outage you can accept. At unstaffed sites, give remote hands a written method of procedure: which slot or cable, what to check, how to confirm. **Changes in a window.** Run the same checks before and after (interfaces and errors, routing neighbours and prefix counts, probes across the blast radius), then observe: some faults appear only when load returns. ::: figure Change window with pre-checks, post-checks and a rollback trigger ```mermaid stateDiagram-v2 state "Approved change" as A state "Pre-checks captured" as P state "Change applied" as C state "Post-checks compared" as V state "Observation period" as O state "Rolled back" as R state "Closed" as D [*] --> A A --> P: snapshot state P --> C: inside the window C --> V: same checks again V --> O: results match V --> R: differences or alarms O --> D: quiet period ends O --> R: delayed fault R --> D: verified and recorded D --> [*] ``` Text equivalent: An approved change starts with pre-checks that snapshot the state. The change is applied inside the window and the same checks are run again. If the results match, an observation period follows and the change closes when it ends quietly. If the post-checks differ or alarms fire, or a fault appears during observation, the change is rolled back; the rollback is verified and recorded before closing. ::: ### The fault-isolation method Structured troubleshooting has several named methods: top-down from the application, bottom-up from the physical layer, divide-and-conquer from the network layer with ping and trace, follow-the-path, compare-configurations and swap-components. A Cisco Press chapter on them advises: "The best bottom-up approach is to first reduce the scope of the problem using a different strategy and then switch to the bottom-up approach for clearly bounded parts of the network topology".[^ranjbar] So first scope the fault: one user, one site or everyone? Everyone points at a shared dependency (an uplink, a carrier, DNS) before any single device. Then follow one path bottom-up, as the ping manual orders it: the local host first, then "hosts and gateways further and further away".[^ping] Stop at the first step that fails; if a ping to the destination already works, start at step 4. ::: figure Layer-by-layer isolation along the path ```mermaid flowchart TB S["Symptom: slow, lossy or unreachable"]:::risk SC{"One user, one site or everyone?"}:::decision S --> SC SC -->|"everyone"| SH["Check shared dependencies first: uplink, carrier, DNS"]:::neutral SC -->|"one user or site"| L1["Step 1 Link (OSI 1-2): errors, CRC, optics"]:::core SH --> L1 L1 --> L2["Step 2 Neighbour (OSI 2): ARP or ND, VLAN"]:::core L2 --> L3["Step 3 Route (OSI 3): lookup, gateway reachable"]:::core L3 --> P["Step 4 Path (OSI 3, hop by hop): trace both ways"]:::core P --> T4["Step 5 Transport (OSI 4): retransmits, MTU, ports"]:::core T4 --> AP["Step 6 Application (OSI 5-7): DNS, TLS, first byte"]:::core AP --> OUT["Name the failing layer and segment"]:::decision classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: Scope the symptom; if everyone is affected, check shared dependencies (uplink, carrier, DNS) first. Then walk six steps and stop at the first that fails: link, OSI layers 1 and 2 (error and CRC frame-checksum counters, optics); neighbour, layer 2 (ARP or IPv6 neighbour discovery, VLAN); route, layer 3; path, layer 3 hop by hop (trace both ways); transport, layer 4 (retransmissions, MTU, ports); application, layers 5 to 7 (DNS, TLS, first byte). The output names the failing layer and segment. ::: | Step (OSI layer) | Linux commands (read-only) | What a failure looks like | |---|---|---| | 1 Link (1–2) | `ip -s link`, `ethtool -S ` | Rising `rx_crc_errors`, drops; the kernel counts buffer-exhaustion drops as `rx_missed_errors`[^kstats] | | 2 Neighbour (2) | `ip neigh show` | `FAILED` or `INCOMPLETE` entry for the gateway | | 3 Route (3) | `ip route get ` | Wrong egress interface or gateway; the command prints the route "exactly as the kernel sees it"[^ip-route] | | 4 Path (3, per hop) | `tracepath -n`, `mtr -rn -c 100`, `traceroute -T -p 443` | Loss or delay that starts at one hop and continues to the destination | | 5 Transport (4) | `ss -tio dst ` | Retransmission count rising (`timer` field); `rtt` far above the ping RTT[^ss] | | 6 Application (5–7) | `dig`, `curl -w` timing variables | Slow `time_namelookup`, `time_connect` or `time_starttransfer`[^curl] | Five traps catch most beginners. 1. **A spike or loss at one hop is usually cosmetic.** Steenbergen explains that routers generate ICMP replies in software as "one of the lowest priority functions of the router", "typically rate-limited to very low values": a delayed reply shows as a spike, a dropped one as loss. "If you see a latency spike on one Traceroute hop, and it goes away in the next hop, you can easily deduce that the spike was actually caused by a cosmetic delay in the ICMP generation process".[^ras] mtr's manual is more alarmist (a jump is "often an indication of a bad (or simply overloaded) link");[^mtr] believe it only when the jump persists to the destination. 2. **Traceroute shows the forward path; its times include the return path.** Steenbergen: "you must also examine the reverse path Traceroute before you can be certain that a particular link is responsible".[^ras] Get a trace from the far end, a looking glass, or a measurement network such as RIPE Atlas.[^ripe-atlas] 3. **Silence is not failure.** Many networks drop UDP probes; `traceroute -T` uses TCP SYNs, a method "intended to bypass firewalls".[^traceroute] Lab 1's run shows it. 4. **Ping works, transfers hang: suspect the path MTU.** In a PMTUD black hole, ICMP "fragmentation needed" messages are filtered; RFC 2923 says the failure "is especially difficult to debug, as pings and some interactive TCP connections to the destination host work".[^rfc2923] Test with `ping -M do -s 1472 ` (1472 bytes plus 28 bytes of headers is 1500) and `tracepath`.[^ping][^tracepath] IPv6 routers never fragment,[^rfc8200] so filtered ICMPv6 Packet Too Big messages cause the same black hole;[^rfc8201] test with `ping -6 -M do -s 1452 ` (plus 48 header bytes). 5. **Averages hide bursts.** A five-minute utilisation of 40% can contain seconds at line rate; look at discard and drop counters, not only utilisation. ### Methods, standards and frameworks | Source | What it gives NetOps | Status | |---|---|---| | ITU-T M.3400 (2000) | Fault, performance and other management function sets[^m3400] | Recommendation; a vocabulary, not an obligation | | IETF IPPM: RFC 7679, 7680, 3393, 5481 | Definitions of one-way delay, loss, delay variation[^rfc7679][^rfc7680][^rfc3393][^rfc5481] | IETF RFCs (RFC 5481 informational) | | ITU-T Y.1541 (2011), G.114 (2003) | Performance objectives, delay planning limits[^y1541][^g114] | Recommendations you may adopt | | MEF 10.4 (2018) | Carrier Ethernet service metrics[^mef] | Industry standard | | RFC 7454 (BCP 194) | BGP operations, for example "It is RECOMMENDED to configure a limit on the number of routes to be accepted from a peer"[^rfc7454] | Best current practice | | MANRS | Filtering, anti-spoofing and contact-information norms[^manrs] | Voluntary programme | | SNMP security considerations | "deployment of SNMP versions prior to SNMPv3 is NOT RECOMMENDED"[^rfc4293] | IETF recommendation | None is mandatory in itself; a contract, regulator or customer can make one binding. ### Tooling landscape Representative tools (as of October 2026), vendor-neutral categories first: | Job | Representative tools | Selection notes | |---|---|---| | Device and interface monitoring | LibreNMS (auto-discovering, SNMP-based);[^librenms] Zabbix SNMP items;[^zabbix] Prometheus with `snmp_exporter`[^snmpexp] | SNMPv3, discovery, scale per collector | | Layer 2/3 inventory | Netdisco: IP and MAC data from SNMP, CLI or device APIs[^netdisco] | Finds the switch port a host is on | | Flow collection | nfdump (NetFlow v1 to v9, IPFIX, sFlow);[^nfdump] pmacct[^pmacct] | Storage cost, sampling, enrichment with BGP data | | Path measurement | SmokePing;[^smokeping] perfSONAR;[^perfsonar] `blackbox_exporter`;[^blackbox] RIPE Atlas[^ripe-atlas] | Where probes sit matters more than the tool | | Packet analysis | `tcpdump`; Wireshark's TCP analysis flags such as `tcp.analysis.retransmission`[^tcpdump][^wireshark] | Privilege, storage and privacy controls | | Alert routing | Prometheus Alertmanager: grouping, inhibition, silences[^am] | Maintenance silences must expire | | Commercial platforms | Kentik, for example, says it unifies flows, cloud flow logs, SNMP, telemetry and synthetic tests[^kentik] | Vendor claims: test with your data | Lock-in sits in the data: keep inventories, alert rules and runbooks in formats you can move. ### Automation and AI Automate read-only work first: attach counters and probe results to the ticket when an alert fires; check every device against its monitoring profile. Pushing configuration is change engineering ([NetworkOps](../networkops/), [AutomationOps](../automationops/)). Automated remediation needs limits. In October 2025, flapping health checks made AWS's Network Load Balancer fail over between Availability Zones and remove capacity; AWS is "adding a velocity control mechanism to limit the capacity a single NLB can remove when health check failures cause AZ failover".[^aws-2025] The SRE book reports avoiding "magic" systems that learn thresholds or detect causality;[^sre-mon] correlation is [AIOps](../aiops/). Language models explain output well but invent CLI syntax: review anything that touches a device. ## NetOps in practice ### Reference implementation: an SNMPv3 and probe monitoring baseline **Assumptions:** Prometheus 3.x, `snmp_exporter` v0.30.1 (2026-01-07) with its shipped `snmp.yml` and `if_mib` module, `blackbox_exporter` v0.29.0, Alertmanager; devices accept SNMPv3 only from the collector. Names are placeholders. The generator documents that `auth_protocol` "Defaults to MD5" and `priv_protocol` "Defaults to DES",[^snmpexp-gen] so set both; community strings "are sent unencrypted in SNMP v1 and v2c".[^snmpexp] SHA-256 for SNMPv3 is RFC 7860: check device support.[^rfc7860] A sample on the project's main branch loads extra auths as a second `--config.file`, with secrets from the environment.[^snmpexp-auth] _Example status: YAML parsed with js-yaml; fields checked against the projects' documentation (2026-10-18); not run against devices; PromQL not executed (operator precedence and label matching checked against the Prometheus documentation). Before use, run `promtool check config`, `promtool check rules` and `amtool check-config`._ ```yaml # snmp-auth.yml: start the exporter with # snmp_exporter --config.file=snmp.yml --config.file=snmp-auth.yml \ # --config.expand-environment-variables \ # --web.listen-address=192.0.2.10:9116 --web.config.file=web-config.yml # Credentials come from the environment, never from this file. auths: noc_v3: version: 3 username: ${SNMP_USER} security_level: authPriv password: ${SNMP_AUTH_PASS} auth_protocol: SHA256 # default is MD5 priv_protocol: AES # default is DES priv_password: ${SNMP_PRIV_PASS} ``` ```yaml # blackbox.yml: ICMP module for path probes modules: icmp_v4: prober: icmp timeout: 5s icmp: preferred_ip_protocol: ip4 # the exporter's default is ip6 ``` ```yaml # web-config.yml: TLS and basic auth for both exporters (exporter-toolkit format). # Start blackbox_exporter with --web.listen-address=192.0.2.10:9115 and this file too. # 192.0.2.10 is the collector behind both exporter names in prometheus.yml; the # certificate must list those names. tls_server_config: cert_file: /etc/exporters/tls.crt key_file: /etc/exporters/tls.key basic_auth_users: prometheus: "" # from: htpasswd -nBC 10 "" | tr -d ':\n' ``` ```yaml # prometheus.yml (excerpt). Targets carry a site label for inhibition. rule_files: [rules/netops.yml] scrape_configs: - job_name: snmp scrape_interval: 60s scrape_timeout: 50s metrics_path: /snmp params: {auth: [noc_v3], module: [if_mib]} scheme: https basic_auth: {username: prometheus, password_file: /etc/prometheus/exporter.pass} tls_config: {ca_file: /etc/prometheus/noc-ca.crt} file_sd_configs: [{files: [targets/devices.yml]}] relabel_configs: - {source_labels: [__address__], target_label: __param_target} - {source_labels: [__param_target], target_label: instance} - {target_label: __address__, replacement: "snmp-exporter.example.net:9116"} - job_name: probe_icmp scrape_interval: 15s metrics_path: /probe params: {module: [icmp_v4]} scheme: https basic_auth: {username: prometheus, password_file: /etc/prometheus/exporter.pass} tls_config: {ca_file: /etc/prometheus/noc-ca.crt} file_sd_configs: [{files: [targets/probes.yml]}] relabel_configs: - {source_labels: [__address__], target_label: __param_target} - {source_labels: [__param_target], target_label: instance} - {target_label: __address__, replacement: "blackbox.example.net:9115"} - job_name: probe_icmp_from_branch07 # a blackbox at the site: the users' path scrape_interval: 15s metrics_path: /probe params: {module: [icmp_v4]} scheme: https basic_auth: {username: prometheus, password_file: /etc/prometheus/exporter.pass} tls_config: {ca_file: /etc/prometheus/noc-ca.crt} static_configs: - targets: [app.example.net] # a service address users depend on labels: {site: branch07, source_site: branch07} relabel_configs: - {source_labels: [__address__], target_label: __param_target} - {source_labels: [__param_target], target_label: instance} - {target_label: __address__, replacement: "blackbox.branch07.example.net:9115"} ``` ```yaml # targets/devices.yml and targets/probes.yml (file-based discovery) - targets: [rtr1.branch07.example.net, sw1.branch07.example.net] labels: {site: branch07} ``` ```yaml # rules/netops.yml: symptoms page; causes open tickets, except lost redundancy or # lost visibility. Every rule names an owner and a runbook. groups: - name: netops rules: - alert: SiteProbeLoss expr: avg_over_time(probe_success{job=~"probe_icmp.*"}[10m]) < 0.9 for: 5m labels: {severity: page, owner: noc} annotations: summary: "More than 10% probe loss to {{ $labels.instance }}" runbook: "https://wiki.example.net/noc/site-probe-loss" - alert: UplinkDown # uplink or carrier circuit: redundancy is gone expr: > ifOperStatus{ifAlias=~"(?i)(uplink|cid).*"} == 2 and on (instance, ifIndex) ifAdminStatus == 1 for: 2m labels: {severity: page, owner: noc} annotations: summary: "{{ $labels.instance }} uplink {{ $labels.ifAlias }} down" runbook: "https://wiki.example.net/noc/uplink-down" - alert: InterfaceDown # any other admin-up port without a link expr: > ifOperStatus{ifAlias!~"(?i)(uplink|cid).*"} == 2 and on (instance, ifIndex) ifAdminStatus == 1 for: 15m labels: {severity: ticket, owner: network-eng} annotations: runbook: "https://wiki.example.net/noc/interface-down" - alert: InterfaceHighUtilisation # either direction expr: > (rate(ifHCInOctets[5m]) * 8 / (ifHighSpeed * 1e6) > 0.8 or rate(ifHCOutOctets[5m]) * 8 / (ifHighSpeed * 1e6) > 0.8) and on (instance, ifIndex) ifHighSpeed > 0 for: 15m labels: {severity: ticket, owner: network-eng} annotations: runbook: "https://wiki.example.net/noc/high-utilisation" - alert: InterfaceErrors expr: increase(ifInErrors[15m]) > 50 labels: {severity: ticket, owner: network-eng} annotations: runbook: "https://wiki.example.net/noc/interface-errors" - alert: SnmpTargetUnreachable # blind to a device: paged like lost redundancy expr: up{job="snmp"} == 0 for: 5m labels: {severity: page, owner: noc} annotations: runbook: "https://wiki.example.net/noc/snmp-unreachable" ``` ```yaml # alertmanager.yml (excerpt): one page per site outage; tickets never page route: receiver: noc-pager group_by: [site, alertname] routes: - receiver: noc-tickets matchers: ['severity="ticket"'] receivers: - name: noc-pager - name: noc-tickets inhibit_rules: - source_matchers: ['alertname="SiteProbeLoss"'] target_matchers: ['alertname=~"UplinkDown|InterfaceDown|SnmpTargetUnreachable"'] equal: [site] ``` Design notes. `rate()` handles counter resets ("Breaks in monotonicity … are automatically adjusted for"),[^prom-rate] and `for:` keeps an alert pending until its condition has held for that long.[^prom-rules] `up == 0` treats missing data as a signal: the device is down or unmanageable. The ICMP probe needs `CAP_NET_RAW`, root or a group inside `net.ipv4.ping_group_range` on Linux.[^blackbox] **What pages.** Symptoms (probe loss) page, and so do two definite, imminent causes: a lost uplink (redundancy is gone) and a device the collector cannot see; other causes open tickets. The SRE book's rule for causes: "only worry about very definite, very imminent causes".[^sre-mon] `UplinkDown` relies on a convention: an `ifAlias` that *starts with* `uplink` or `cid`, because PromQL regex matches are fully anchored.[^prom-basics] RFC 2863 defines `ifAlias` as a name the network manager sets, such as a carrier's circuit identifier;[^rfc2863] Junos exposes the interface description there.[^junos-desc] `InterfaceDown` fires for every admin-up port without a link, which reports `down(2)`,[^rfc2863] so shut unused access ports. **Who can reach the credentials.** Both exporters are proxies that listen on all addresses over plain HTTP unless configured otherwise (the blackbox README's log shows "TLS is disabled.").[^blackbox] Whoever reaches `/snmp` inherits the exporter's SNMP credentials for any device; whoever reaches `/probe` can aim probes anywhere. Bind each exporter to the collector's address, admit only the Prometheus servers, and enable TLS and basic authentication (`--web.config.file`, which covers `/snmp` too).[^snmpexp][^webcfg] On devices, give the SNMPv3 user a read-only view, accepted only from the collector. The `${…}` variables keep secrets out of committed files, not off the host: root and the exporter's user can read `/proc//environ`,[^proc-environ] so use a dedicated user on a host only the NOC administers. **Where the probes sit.** The central `probe_icmp` job pings management addresses from the data centre: the device answers, but are branch users served? The second job probes a service address from a blackbox exporter at the branch (`source_site`); add one per site, or use RIPE Atlas or perfSONAR. ### Hands-on labs Labs 1 to 3 are read-only and run as an ordinary user. The optional part of Lab 2 needs root on a disposable VM. #### Lab 1 (beginner): a bottom-up path check **Objective:** walk the layers from your host to a service and name the first one that fails. **Prerequisites:** Linux with iproute2, iputils (`ping`, `tracepath`) and curl; 15 minutes. **Cost and risk:** about 40 small packets to the target; nothing on the host changes. _Example status: verified by execution as an unprivileged user on Ubuntu 20.04 (iputils s20190709, iproute2 5.5, curl 7.68), target example.com:443, 2026-10-18; output summarised, not reproduced (it names a production host's addresses)._ ```bash #!/usr/bin/env bash # netpath.sh: read-only, bottom-up path check from this host to HOST:PORT. # Usage: bash netpath.sh HOST [PORT] (changes nothing; no root needed) set -uo pipefail target=${1:?usage: netpath.sh HOST [PORT]} port=${2:-443} say() { printf '\n== %s ==\n' "$*"; } flag() { printf ' CHECK: %s\n' "$*"; } field() { awk -v k="$1" '{for (i = 1; i < NF; i++) if ($i == k) {print $(i+1); exit}}'; } say "1. Name resolution" addr=$(getent ahostsv4 "$target" | awk 'NR == 1 {print $1}') [ -n "$addr" ] || { flag "cannot resolve $target: test DNS first"; exit 1; } echo " $target -> $addr" say "2. Route and egress interface" route=$(ip route get "$addr" | head -n 1) echo " $route" dev=$(field dev <<<"$route"); gw=$(field via <<<"$route") say "3. Link state and error counters on $dev" ip -s link show dev "$dev" | sed 's/^/ /' for c in rx_errors tx_errors rx_dropped tx_dropped rx_crc_errors; do v=$(cat "/sys/class/net/$dev/statistics/$c" 2>/dev/null || echo 0) [ "$v" -gt 0 ] && flag "$c=$v since boot: read again, is it rising?" done say "4. Next hop ${gw:-$addr} in the neighbour table" ip neigh show "${gw:-$addr}" dev "$dev" | sed 's/^/ /' say "5. Loss and RTT (gateway first, then target)" [ -n "$gw" ] && ping -n -c 5 -i 0.5 -W 1 "$gw" | tail -n 2 | sed 's/^/ gw /' ping -n -c 10 -i 0.5 -W 1 "$addr" | tail -n 2 | sed 's/^/ dst /' say "6. Forward path and path MTU (RTTs include the unseen return path)" tracepath -n -m 20 "$addr" | sed 's/^/ /' say "7. TCP connect to port $port" if timeout 5 bash -c ": > /dev/tcp/$addr/$port" 2>/dev/null; then echo " TCP $port open" else flag "TCP $port refused, filtered or timed out" fi if [ "$port" = 443 ]; then fmt=' dns=%{time_namelookup}s tcp=%{time_connect}s' fmt+=' tls=%{time_appconnect}s first_byte=%{time_starttransfer}s\n' curl -s -o /dev/null --connect-timeout 5 -m 15 -w "$fmt" "https://$target/" || flag "HTTPS request failed" fi exit 0 ``` 1. Run `bash netpath.sh example.com 443`, then against a service you support; name the layer each section tests. 2. Run `bash netpath.sh no-such-host.invalid` and `bash netpath.sh example.com 23`; note where each stops or flags. **Expected output (shape):** seven `==` sections. The 2026-10-18 cloud-VM run: no `CHECK` lines; gateway RTT 0.08 ms; target 1 ms, 0% loss; `tracepath` showed `[LOCALHOST] pmtu 9001` (the interface MTU), then `pmtu 1500`; hop 3 read 31 ms between hops of about 1.3 ms (trap 1); `no reply` from hop 5 to 20, yet TCP 443 connected (trap 3). The failure runs printed `CHECK: cannot resolve` (exit 1) and `CHECK: TCP 23 refused, filtered or timed out`. **Validation:** exit code `0` for the first run; every section mapped to a layer. **Troubleshooting:** `ping: socket: Operation not permitted` → your distribution restricts ICMP sockets (`ping_group_range`); use `sudo` or a VM. Empty `tracepath` output → UDP blocked locally; try `traceroute -T -p 443` with sudo. **Teardown:** none; delete saved output that names hosts. #### Lab 2 (intermediate): loss, delay and delay variation against an objective **Objective:** turn raw `ping` output into the KPIs a carrier contract or SLO uses, and see why one number is not enough. **Prerequisites:** Python 3.8 or later; iputils `ping` with `-D` and `-O`; 20 minutes. **Cost and risk:** 300 small packets at 5 per second. The iputils manual warns that "it is unwise to use ping during normal operations or from automated scripts" because of the load;[^ping] keep the rate low and aim repeated runs at hosts you own or may test. _Example status: verified by execution on 2026-10-18 (Python 3.8.10): against a synthetic capture with known answers, against example.com and against 127.0.0.1. The optional `netem` part is syntax-checked only (needs root and kernel modules)._ ```python #!/usr/bin/env python3 """pingkpi.py: loss, RTT and delay-variation KPIs from `ping -D -O` output. Usage: ping -n -D -O -i 0.2 -c 300 HOST | python3 pingkpi.py --p95-ms 60 --loss-pct 1 All delay figures are ROUND-TRIP times, not the one-way metrics of RFC 7679. """ import argparse import math import re import sys REPLY = re.compile(r"icmp_seq=(\d+) .*time=([\d.]+) ms") SUMMARY = re.compile(r"(\d+) packets transmitted, (\d+) received") def pct(values, p): """Nearest-rank percentile of a non-empty list.""" s = sorted(values) return s[max(0, math.ceil(p / 100 * len(s)) - 1)] def main(): ap = argparse.ArgumentParser() ap.add_argument("--p95-ms", type=float, default=100.0, help="SLO: p95 RTT") ap.add_argument("--loss-pct", type=float, default=1.0, help="SLO: loss") ap.add_argument("--pdv-ms", type=float, default=30.0, help="SLO: p99 PDV") a = ap.parse_args() rtt, sent = {}, None for line in sys.stdin: m = REPLY.search(line) if m and "DUP!" not in line: rtt.setdefault(int(m.group(1)), float(m.group(2))) s = SUMMARY.search(line) if s: sent = int(s.group(1)) if not sent or not rtt: sys.exit("no ping summary or no replies: did ping run to completion?") seqs = range(1, sent + 1) # iputils numbers echo requests from 1 lost = [q for q in seqs if q not in rtt] burst = run = 0 for q in seqs: run = run + 1 if q not in rtt else 0 burst = max(burst, run) ordered = [rtt[q] for q in seqs if q in rtt] base = min(ordered) pdv = [r - base for r in ordered] # RFC 5481 PDV: reference = minimum ipdv = [ordered[i] - ordered[i - 1] for i in range(1, len(ordered))] jit = 0.0 for d in ipdv: # RFC 3550 estimator, applied here to RTTs jit += (abs(d) - jit) / 16 loss = 100.0 * len(lost) / sent p95 = pct(ordered, 95) pdv99 = pct(pdv, 99) print(f"sent={sent} received={len(ordered)} loss={loss:.2f}% " f"longest_loss_burst={burst}") print(f"rtt_ms min={base:.2f} p50={pct(ordered, 50):.2f} p95={p95:.2f} " f"max={max(ordered):.2f}") mean_abs = sum(map(abs, ipdv)) / max(1, len(ipdv)) print(f"pdv_p99_ms={pdv99:.2f} ipdv_mean_abs_ms={mean_abs:.2f} " f"rfc3550_style_jitter_ms={jit:.2f}") breaches = [] if loss > a.loss_pct: breaches.append(f"loss {loss:.2f}% > {a.loss_pct}%") if p95 > a.p95_ms: breaches.append(f"p95 {p95:.2f} ms > {a.p95_ms} ms") if pdv99 > a.pdv_ms: breaches.append(f"PDV p99 {pdv99:.2f} ms > {a.pdv_ms} ms") print("SLO: " + ("BREACH: " + "; ".join(breaches) if breaches else "met")) return 1 if breaches else 0 if __name__ == "__main__": sys.exit(main()) ``` 1. Run `ping -n -D -O -i 0.2 -c 300 | python3 pingkpi.py --p95-ms 60 --loss-pct 1 --pdv-ms 30`. 2. Repeat at a busy and a quiet hour; compare p50, p95 and PDV, and decide which objective users feel. PDV is measured against the minimum delay, which RFC 5481 prefers for service levels ("The one-sided PDV distribution can be constrained with a single statistic, such as an upper percentile, so it is preferred");[^rfc5481] the smoothed figure is RFC 3550's estimator, `J(i) = J(i-1) + (|D(i-1,i)| - J(i-1))/16`.[^rfc3550] Both are applied to round-trip times. **Expected output:** three lines of figures and an `SLO:` line; exit `0` when met, `1` on a breach. The synthetic test (8 sent, 2 lost in a row, one duplicate) printed `loss=25.00% longest_loss_burst=2`, `p95=60.00`, `pdv_p99_ms=20.00` and `rfc3550_style_jitter_ms=3.12`, matching a hand calculation; example.com gave 0% loss, p95 1.11 ms. **Validation:** `loss` and `min` agree with ping's summary (ping's `min` also counts duplicates). **Troubleshooting:** `no ping summary` → ping was interrupted, or its output is localised; run with `LC_ALL=C`. **Teardown:** none for the ping run; the optional `netem` block ends by deleting its namespace. **Optional (root, disposable VM only): impair a path you own.** `netem` adds delay and loss on egress.[^netem] The namespace uses the documentation range 192.0.2.0/24. ```bash sudo ip netns add lab sudo ip link add v0 type veth peer name v1 sudo ip link set v1 netns lab sudo ip addr add 192.0.2.1/24 dev v0 && sudo ip link set v0 up sudo ip netns exec lab ip addr add 192.0.2.2/24 dev v1 sudo ip netns exec lab ip link set v1 up sudo tc qdisc add dev v0 root netem delay 40ms 8ms loss 2% ping -n -D -O -i 0.2 -c 300 192.0.2.2 | python3 pingkpi.py --p95-ms 60 --loss-pct 1 sudo ip netns del lab # teardown: deleting v1 also removes its peer v0 ``` Expect a median near 40 ms and loss near 2%, varying between runs: 300 packets estimate 2% coarsely. #### Lab 3 (advanced): from counters to utilisation, and why 32-bit counters lie **Objective:** compute utilisation as an NMS does, handle counter wraps and resets, and see a 32-bit counter under-report a busy link. **Prerequisites:** Python 3.8 or later on Linux; 10 minutes. **Cost and risk:** none; it reads `/sys/class/net//statistics`, which holds the same 64-bit counters that `ip -s link` shows.[^kstats] _Example status: verified by execution on 2026-10-18 (Python 3.8.10): `--selftest`, and live on interface `ens5` at a 2-second interval._ ```python #!/usr/bin/env python3 """ifrate.py: interface utilisation from byte counters, as an NMS computes it. Live: python3 ifrate.py IFACE --speed-mbps 1000 [--interval 5] [--samples 6] Selftest: python3 ifrate.py --selftest (shows why 32-bit counters mislead) Reads /sys/class/net/IFACE/statistics: 64-bit counters, like IF-MIB ifHCInOctets. """ import argparse import math import sys import time def delta(prev, cur, bits, seconds, speed_bps): """Bytes between two readings, or None for a discontinuity (reset).""" if cur >= prev: d = cur - prev elif bits == 32: d = cur + 2**32 - prev # assume exactly one wrap: the best guess else: return None # a 64-bit counter does not wrap in practice: a reset if d * 8 / seconds > speed_bps: return None # faster than the line: the reading cannot be real return d def p95(values): s = sorted(values) return s[max(0, math.ceil(0.95 * len(s)) - 1)] def selftest(): speed, true_bps = 1e9, 600e6 # 1 Gb/s link carrying 600 Mb/s for bits, poll in ((64, 300), (32, 300), (32, 20)): counter, prev, got = 0, 0, [] for _ in range(4): counter += int(true_bps / 8 * poll) reading = counter % 2**bits d = delta(prev, reading, bits, poll, speed) got.append("reset?" if d is None else f"{d * 8 / poll / 1e6:.0f}") prev = reading print(f"{bits}-bit counter, poll {poll:>3}s, true 600 Mb/s -> " f"measured Mb/s: {', '.join(got)}") d = delta(5_000_000, 1_000, 64, 300, speed) print(f"64-bit counter falls from 5000000 to 1000 -> {d} (discard)") def live(a): base = f"/sys/class/net/{a.iface}/statistics/" names = ("rx_bytes", "tx_bytes", "rx_errors", "rx_dropped") def read(): return {n: int(open(base + n).read()) for n in names} speed = a.speed_mbps * 1e6 util = {"rx": [], "tx": []} prev, t0 = read(), time.monotonic() for _ in range(a.samples): time.sleep(a.interval) cur, t1 = read(), time.monotonic() sec = t1 - t0 out = [] for k in ("rx", "tx"): d = delta(prev[k + "_bytes"], cur[k + "_bytes"], a.bits, sec, speed) if d is None: out.append(f"{k}=discontinuity") continue util[k].append(100 * d * 8 / sec / speed) out.append(f"{k}={d * 8 / sec / 1e6:8.3f} Mb/s ({util[k][-1]:.3f}%)") errs = cur["rx_errors"] - prev["rx_errors"] drops = cur["rx_dropped"] - prev["rx_dropped"] print(" ".join(out) + f" rx_err+{errs} rx_drop+{drops}") prev, t0 = cur, t1 for k, v in util.items(): if v: print(f"{k}: p95 of {len(v)} samples = {p95(v):.3f}% of " f"{a.speed_mbps:g} Mb/s") if __name__ == "__main__": ap = argparse.ArgumentParser() ap.add_argument("iface", nargs="?") ap.add_argument("--speed-mbps", type=float) ap.add_argument("--interval", type=float, default=5.0) ap.add_argument("--samples", type=int, default=6) ap.add_argument("--bits", type=int, choices=(32, 64), default=64) ap.add_argument("--selftest", action="store_true") args = ap.parse_args() if args.selftest: selftest() elif args.iface and args.speed_mbps: live(args) else: sys.exit("give IFACE and --speed-mbps, or --selftest") ``` 1. Run `python3 ifrate.py --selftest`. 2. Run it live: `python3 ifrate.py --speed-mbps `. Virtual NICs often report no speed (`cat /sys/class/net//speed` fails), so pass the rate you are entitled to, such as a cloud instance's documented bandwidth. 3. Explain the selftest's second line using RFC 2863's arithmetic in [Metrics and KPIs](#metrics-and-kpis). **Expected output:** the selftest prints `600` four times for the 64-bit counter at 300 s polls, **`27`** four times for the 32-bit counter at 300 s, `600` four times for the 32-bit counter at 20 s, and `None (discard)` for the reset; live mode prints one line per interval and a p95 per direction. **Validation:** 27 Mb/s is 22.5 GB per 300 s modulo 2^32: plausible, wrong and silent. **Troubleshooting:** `FileNotFoundError` → wrong interface name (`ls /sys/class/net`); `discontinuity` lines after an interface reset are expected and discarded. **Teardown:** none. ### Use cases and scenarios **1. Rogers, Canada, July 2022 (sourced).** A configuration clean-up removed a routing filter; the core routers were flooded and crashed within minutes, and more than 12 million customers lost service for about a day (change review: [NetworkOps](../networkops/)). For the NOC, the independent assessment published by the regulator, the CRTC, is blunt: Rogers "had a management network that relied on the Rogers IP core network"; staff "could not pinpoint the root cause for about 14 hours"; and "Rogers had completed multiple configuration changes during the maintenance window", which made the rollback choice hard. Its lessons include separating management "physically and logically from the data network", backup NOC connectivity from third-party operators, fewer changes per window and alarm prioritisation.[^crtc] *Success metric:* every core device reachable by a path that does not use the core. **2. Facebook, October 2021 (sourced).** During routine maintenance, "a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network"; a bug in the audit tool failed to stop it. DNS servers then withdrew their BGP advertisements, and "Our primary and out-of-band network access was down", so engineers went on site, slowed by deliberate security; "storm" drills helped the restart.[^fb] *Lessons:* out-of-band access must not share fate with the backbone; rehearse physical access; restore load gradually. **3. Volumetric DDoS at the edge (illustrative scenario).** A 10 Gbit/s transit link sits at line rate and every site behind it loses packets; flows show most traffic going to one public address from thousands of sources. *Method:* flows name the target in minutes; the NOC runs the runbook agreed with the SOC, which owns attribution. *Options:* (a) blackhole the /32 at the upstream with a BGP community such as RFC 7999's BLACKHOLE ("drop any traffic being sent towards this prefix"), agreed with each upstream beforehand: the network recovers, but the target goes dark;[^rfc7999] (b) a BGP Flowspec rule that drops only the attack's pattern (protocol, port, packet length): the service stays up, but a broad rule drops legitimate traffic, and RFC 8955 warns that badly programmed controllers "may announce unintended Flow Specifications";[^rfc8955] (c) a scrubbing provider under contract, slower to start and paid for. *Success metric:* mitigated within the runbook's target time; utilisation and site loss back to baseline. **4. Branch voice quality (illustrative scenario).** Calls at a branch are choppy every afternoon, yet the 100 Mbit/s uplink peaks at 55% (five-minute average). Lab 2's probe shows PDV p99 of 45 ms from 14:00 to 16:00 while uplink discards rise: file-sync bursts fill the queue for seconds. *Method:* probes and discard counters, then flows to name the traffic. *Options:* a voice priority queue, shaping, or a larger uplink. *Success metric:* PDV p99 under 30 ms in business hours. ## Measure, govern, pay ### Metrics and KPIs | KPI | Formula | Data source | Pitfall | |---|---|---|---| | Path availability | Minutes the path met its objective ÷ minutes in the period | Probes, one-minute intervals | Device "up" is not path "available" | | Loss ratio | Lost probes ÷ sent probes | Probes | Short runs are coarse; report bursts too | | Delay p95 | 95th percentile of RTT (or one-way delay with synchronised clocks) | Probes | Averages hide queues | | Delay variation | PDV p99 (delay minus the sample minimum)[^rfc5481] | Probes | Several "jitter" formulas exist: name yours | | Utilisation p95 | 95th percentile of 5-minute rate ÷ link or contracted rate | `ifHCInOctets`, `ifHCOutOctets` | 32-bit counters wrap; averages hide bursts | | Errors and discards | Increase per interval and interface | IF-MIB counters | Rising discards with modest utilisation mean bursts | | Alert actionability | Alerts that led to action ÷ alerts | Alert and ticket records | Silenced noise is still noise | | Carrier time to restore | Ticket open to verified restoration, per carrier | NOC tickets | Measure from your clock, not theirs | | Capacity runway | (Planning ceiling − current p95) ÷ weekly growth | Utilisation trend | Compare with the upgrade lead time | Time to detect, acknowledge and restore are defined on [IncidentOps](../incidentops/#metrics-and-kpis). **Worked calculation 1: counter wraps.** A 32-bit counter holds 2^32 bytes, about 4.29 GB. At 1 Gbit/s that is 2^32 × 8 ÷ 10^9 ≈ 34.4 s, matching RFC 2863's "at 1Gbs, the minimum is 34 seconds".[^rfc2863] Polled every 300 s, the counter can wrap several times between readings and the poller cannot tell (Lab 3 reports 27 Mbit/s for a true 600). Use the 64-bit `ifHC` counters; RFC 2863 requires them for interfaces faster than 20 Mbit/s.[^rfc2863] **Worked calculation 2: what loss costs one TCP flow.** Mathis and colleagues modelled Reno-style congestion avoidance as throughput ≈ (MSS ÷ RTT) × (C ÷ √p), with C ≈ 1.22.[^mathis] With MSS 1460 bytes, RTT 40 ms and p = 1%: (11,680 bits ÷ 0.04 s) × (1.22 ÷ 0.1) ≈ 3.6 Mbit/s per flow; at p = 0.01%, about 36 Mbit/s. CUBIC, the default in the Linux, Windows and Apple stacks,[^cubic] behaves differently, so treat this as an order of magnitude: small loss on a long path throttles every flow. **Worked calculation 3: runway for a redundant pair (illustrative numbers).** Links A and B are 1 Gbit/s each, with p95 loads of 380 and 350 Mbit/s. If one fails, the survivor carries 730 Mbit/s, 73% of its capacity. Combined growth is 12 Mbit/s a week; with a planning ceiling of 85% (850 Mbit/s, this example's choice), runway is (850 − 730) ÷ 12 = 10 weeks. If the carrier quotes 12 weeks to upgrade, the order is already late, although each link alone looks half empty. **Choosing targets.** ITU-T Y.1541 Class 0, for real-time interactive traffic such as voice, sets upper bounds of 100 ms mean transfer delay, 50 ms delay variation and a 1 × 10^-3 loss ratio for public IP networks, with "An evaluation interval of 1 minute" suggested.[^y1541] For planning, G.114 recommends "to not exceed a one-way delay of 400 ms", and below 150 ms "most applications, both speech and non-speech, will experience essentially transparent interactivity".[^g114] Set internal objectives from what users feel, then check that the carrier contract can deliver them. These targets are not the statistics Lab 2 prints. Y.1541 bounds the *mean* one-way delay between the user-network interfaces, and its delay variation is the 1 − 10^-3 quantile of one-way delay minus the minimum: a p99.9 PDV, needing at least 1,000 packets per sample. G.114's figures are one-way and end to end ("mouth-to-ear" for speech).[^y1541][^g114] Lab 2 prints round-trip p95 and p99 PDV. A round trip is never shorter than either leg, so a round-trip mean within a one-way bound meets it; a p99 PDV within 50 ms does not prove a p99.9 within it. ### Quality, security and compliance - **Management-plane security.** SNMPv3 with privacy; management protocols only from collector and jump-host addresses; the quarterly audit in [Checklists](#checklists). - **Separation from security monitoring.** MITRE advises keeping network and security operations as distinct peers, which "will help avoid sidelining security in the name of network availability".[^mitre] Suspected attacks go to [SOC Ops](../soc-ops/); the NOC mitigates link saturation under a runbook agreed with the SOC. - **Privacy.** Flow records, logs and captures contain IP addresses that can identify people; set retention with [PrivacyOps](../privacyops/) and limit who may capture packets. - **Regulation.** Telecom operators may answer to regulators: the CRTC published the independent Rogers assessment.[^crtc] Duties vary by country, sector and contract; this page is not legal advice. ### Cost, capacity and sustainability - **Utilisation targets depend on design.** Google's B4 paper notes that "WAN links are typically provisioned to 30-40% average utilization", while B4's traffic engineering lets "all links to average 70% utilization over long time periods", using control of the applications at the edge that most networks do not have.[^b4] - **Cloud bandwidth is an allowance.** Smaller EC2 instances have a baseline and burst "up to" a figure for limited periods;[^aws-bw] the ENA driver counts packets "queued or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance" (`bw_in_allowance_exceeded` in `ethtool -S`).[^aws-ena] - **People.** A 24×7 seat is 168 × 52 = 8,736 hours a year. At 1,736 productive hours per person (2,080 contract hours minus about 43 days of leave, holidays, sickness and training, as on [SOC Ops](../soc-ops/#lab-2-intermediate-how-many-analysts-does-the-queue-need)), a seat needs about 5.0 people. MITRE agrees: "Each 24x7 seat requires roughly five full time equivalents (FTEs), including fill-in for vacation and sick leave".[^mitre] Google's SRE book derives a higher minimum, eight for a single-site 24/7 rotation, because it assumes two people on call at once and at most 25% of anyone's time on call.[^sre-oncall] - **Waste.** Unused circuits keep billing; review the inventory with [FinOps](../finops/) and [GreenOps](../greenops/). ### Assurance of the operating model | Risk indicator | Healthy signal | Check | |---|---|---| | Alerts nobody acts on | Actionability above an agreed floor | Monthly alert review | | Stale carrier contacts | Every circuit has a current escalation path | Quarterly contact test | | Unmonitored devices | Inventory and monitoring lists match | Monthly reconciliation | ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnose | Remediate and prevent | |---|---|---|---| | Loss or delay at one traceroute hop only | That router de-prioritises or rate-limits the ICMP replies it generates | Later hops are clean[^ras] | None; teach the rule | | Pings work, large transfers hang | PMTUD black hole[^rfc2923] | `ping -M do -s 1472` (IPv6: `ping -6 -M do -s 1452`), `tracepath` | Allow ICMP "fragmentation needed" and ICMPv6 Packet Too Big; fix MTU or clamp MSS | | Modest utilisation, rising discards | Microbursts filling queues | Discard counters, short-interval probes | Queueing policy, shaping, capacity | | Rising CRC errors | Optics, cable or port | `ethtool -S`, IF-MIB errors | Clean or replace; check the far end | | Graph spikes or falls to zero | Counter wrap or reset | Counter width; device reboot time | 64-bit counters; `rate()` handles resets[^prom-rate] | | Slow in one direction only | Asymmetric routing | Trace from the far end or RIPE Atlas[^ripe-atlas] | Fix routing policy, or accept and document | | BGP sessions flap or prefix counts jump | Unstable link or peer; route leak | Session logs; BMP peer-down reasons and route counts[^rfc7854] | Fix the link; per-peer prefix limits and filters[^rfc7454][^manrs] via [NetworkOps](../networkops/) | | Fifty alerts from one site at once | Upstream link or device down | Topology, probe to the site | Inhibition by site label ([reference implementation](#reference-implementation-an-snmpv3-and-probe-monitoring-baseline)) | | Monitoring goes dark in an outage | Monitoring shares the failed path (AWS, December 2021) | Is the collector reachable? | Regional collectors; out-of-band alerting[^aws-2021] | | Edge link saturated, loss at every site behind it | Volumetric DDoS | Flows: top destinations | Blackhole, Flowspec[^rfc7999][^rfc8955] or scrubbing per the runbook; tell the SOC | | Gaps in flow data | UDP export loss or sampling | IPFIX sequence numbers | Provision the export path; "UDP MUST NOT be used unless the application can tolerate some loss"[^rfc7011] | | Cloud VM slow, interface error-free | Instance bandwidth or packet allowance | `ethtool -S` allowance counters[^aws-ena] | Larger instance or spread traffic ([CloudOps](../cloudops/)) | ### Worked runbook: intermittent loss on a carrier circuit **Situation.** A branch's probe loss alert fires three times in an afternoon, each for a few minutes. The branch has one carrier circuit and a cellular backup. 1. **Stabilise.** If voice or payments suffer, move those classes to the backup; record the time. 2. **Collect evidence from your end** with the block below, during an episode if possible. 3. **Rule out your side.** Interface errors at both ends of your link; utilisation and discards during the episodes; any change in the last 48 hours. 4. **Open the carrier ticket** from the [template](#templates), with traces in both directions and what you ruled out. Ask for an intrusive test only in an agreed window. 5. **Escalate on the contract clock**, not when you feel like it. 6. **Verify and close.** Probe loss below objective for an agreed period after the repair; request the reason for outage; tune the alert if it fired late. _Example status: syntax-checked; run unprivileged against a public host on Ubuntu 20.04 (2026-10-18) in 222 s (`mtr` works without root there because `mtr-packet` has `CAP_NET_RAW`). Its `mtr` report showed 24% loss at hop 3 and 0% at hops 4 and 5, the destination: trap 1 in real data. The `tcpdump` line was not run (root; personal data)._ ```bash # Evidence pack for a carrier ticket: read-only except the optional capture. T=203.0.113.10 # far end of the circuit (placeholder) out=evidence-$(date -u +%Y%m%dT%H%MZ).txt { date -u ip -s link show dev eth0 # your end: errors and drops ping -n -D -c 200 -i 0.2 "$T" | tail -n 2 tracepath -n "$T" mtr -rn -c 100 "$T" # per-hop loss: read it with the traps ss -tio dst "$T" # RTT and retransmissions on live sessions } > "$out" 2>&1 echo "saved $out" # Optional, root, change ticket first: 60 s of headers plus the first payload bytes. # Send the carrier summaries, not the pcap: payload bytes can hold personal data. # sudo timeout 60 tcpdump -ni eth0 -s 96 -w "${out%.txt}.pcap" host "$T" ``` **Roll back:** nothing changes except the optional capture file; delete captures when the ticket closes. ### Anti-patterns and myths - **"It's the network" or "never the network".** Both are guesses; the isolation table is the answer. - **Ping as the service test.** Routers answer ICMP on a slow, rate-limited path, and many networks filter it; probe the service's protocol and port. - **Treating an emergency mitigation as too urgent to check.** At Cloudflare in July 2020, while relieving backbone congestion, "a one line change started leaking all BGP routes into the backbone"; the outage lasted 27 minutes. The correct change "would have been to deactivate the term instead of the prefix-list", and the fixes include a maximum-prefix limit on backbone BGP sessions.[^cf-2020] The post does not describe Cloudflare's review process; the lesson is this Atlas's inference: a mitigation is a change, so read what it will announce and keep a limit that catches what you miss. - **Myth: a redundant pair at 60% each is safe.** One failure leaves 120% of a link. ## Deciding and adopting ### When to use and when not to use **Build a NetOps function when** you operate your own campus, WAN, data-centre or service-provider network; buy carrier circuits for many sites; owe customers network service levels; or face a regulator. **Invest less when** a provider runs the network (SaaS office, managed SD-WAN, cloud-only): the work becomes managing the provider (service levels, evidence, escalation) and the rest moves to [CloudOps](../cloudops/). **Hidden responsibilities:** contracts and contacts, out-of-band access, inventory and spares, a DDoS runbook agreed with each upstream, a 24×7 rota. ### Decision framework ::: figure Choosing a NOC model, including not building one ```mermaid flowchart TB Q0{"Do you operate the devices and circuits?"}:::decision Q0 -->|"no"| A0["Manage the provider: SLAs, evidence, escalation"]:::neutral Q0 -->|"yes"| Q1{"Does an outage out of hours stop the business?"}:::decision Q1 -->|"no"| A1["Business-hours team plus on-call"]:::neutral Q1 -->|"yes"| Q2{"Can you fund about five people per 24x7 seat?"}:::decision Q2 -->|"no"| A2["Hybrid: managed Tier 1 watch, in-house Tier 2 and 3"]:::core Q2 -->|"yes"| Q3{"Staff in several time zones?"}:::decision Q3 -->|"yes"| A3["Follow-the-sun NOC"]:::core Q3 -->|"no"| A4["In-house 24x7 shifts"]:::core classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 ``` Text equivalent: If you do not operate the devices and circuits, manage the provider through service levels, evidence and escalation. If you do, but an out-of-hours outage does not stop the business, a business-hours team with on-call is enough. If it does and you cannot fund about five people per 24x7 seat, use a hybrid: a managed Tier 1 watch with in-house Tier 2 and Tier 3 engineers. If you can, choose a follow-the-sun NOC when you have staff in several time zones, otherwise in-house 24x7 shifts. ::: The tree assumes the staffing arithmetic in [Cost, capacity and sustainability](#cost-capacity-and-sustainability) and a managed watch that escalates with evidence; one that only forwards alerts adds a hop, not judgement. ### Maturity model This is **this Atlas's model**, not an industry standard. | Level | Observable behaviour | Evidence | Next step | |---|---|---|---| | 0. Reactive | Users report outages; device pings only | Tickets opened by users | Inventory and interface monitoring | | 1. Monitored | Interface state and utilisation graphed; alerts exist | Dashboards | Path probes and runbooks | | 2. Measured | Probes per site; KPIs reviewed; carrier tickets carry evidence | Monthly scorecard | Change pre- and post-checks | | 3. Managed | Few changes per window, verified; out-of-band drilled; alerts reviewed | Drill and change records | Forecasts and automation hand-offs | | 4. Engineered | Capacity ordered from forecasts; evidence collection automated; changes verified upstream in [NetworkOps](../networkops/) | Lead-time compliance | Keep reviewing | ### Adoption roadmap: 30/60/90 days | Window | Actions | Exit criteria | |---|---|---| | Days 1–30 | Inventory devices and circuits with IDs and contracts; SNMPv3 polling of all interfaces; probes to every site | Every site has a probe and an owner | | Days 31–60 | Runbooks for the top ten alerts; inhibition by site; carrier ticket template; handover note | Alert actionability measured | | Days 61–90 | Out-of-band drill; change pre- and post-checks; first capacity forecast per pair | Drill passed; forecasts filed | ## Toolkit ### Checklists **Onboarding a device or circuit** - [ ] In the inventory with site, role, owner; circuits with ID, contract and escalation path - [ ] SNMPv3 (or telemetry) polling and syslog working; one test alert received - [ ] A probe covers the path users take; baseline recorded - [ ] Out-of-band access tested; configuration backup confirmed **Before and after a change window** - [ ] Pre-checks captured; rollback written and timed; silences scoped and expiring - [ ] Few changes, each independently reversible - [ ] Post-checks match pre-checks; observation period completed before closing **Every shift** - [ ] Handover note read and signed; open carrier tickets checked against contract clocks - [ ] No silence outlives its change **Quarterly security and governance audit** - [ ] No SNMPv1/v2c; exporters behind TLS and basic auth; out-of-band accounts named, logged and reviewed - [ ] Every alert rule has an owner, a runbook and a review date; flow, log and capture retention as agreed ### Templates ```text CARRIER TROUBLE TICKET Circuit ID / A-end / B-end: / / Impact: , Start (UTC) / still active: / Evidence: loss RTT p95 over probes; traces both ways Ruled out on our side: Requested: Our contact (24x7): ``` ```text SHIFT HANDOVER (NOC) Shift / engineer: / Open incidents: Carrier tickets: Changes in flight: Active silences: Watch items: ``` ### Deep comparisons | | NetOps / NOC | SOC ([SOC Ops](../soc-ops/)) | Operations bridge ([ITOps](../itops/)) | SRE on-call ([SRE](../sre/)) | |---|---|---|---|---| | Watches | Network paths and devices | Security telemetry | Events across IT services | A service's SLOs | | Success | Availability, service levels | Attacks detected and contained | Incidents routed and restored | Error budget kept | | Typical action | Reroute, repair, escalate to carrier | Contain, investigate | Route, coordinate | Mitigate, fix the service | ### FAQ **Is NetOps the same as NetworkOps or NetDevOps?** In industry usage, largely yes.[^rh-netops] This Atlas splits them editorially: the running network here, the engineering of change (NetDevOps included) on [NetworkOps](../networkops/). **Do I still need SNMP with streaming telemetry?** Often: switches, routers and UPS units are usually SNMP-enabled.[^zabbix] Use SNMPv3 and 64-bit counters.[^rfc4293][^rfc2863] **Can cloud flow logs replace probes?** No. They are delayed, incomplete by design and, in some clouds, sampled.[^aws-flow-basics][^gcp-flow] **Should the NOC also be the SOC?** Small organisations often share a watch floor; MITRE recommends distinct people, tools and funding so availability pressure does not crowd out security.[^mitre] **When does a network fault become an incident?** When customers or several teams are affected, or the fix needs decisions beyond the network team: hand command to [IncidentOps](../incidentops/). ## Going further ### Advanced topics and emerging direction - **Streaming telemetry at scale.** Sub-minute counters expose bursts and flaps but multiply data volume.[^rfc9232] - **Active measurement between devices.** STAMP measures one-way and round-trip delay, delay variation and loss between network endpoints,[^rfc8762] and BFD detects path failures within the network quickly.[^rfc5880] ### Learning path | Stage | Do this | Proof | |---|---|---| | Prerequisites | IP addressing, routing, TCP, a Linux shell | Explain a route lookup | | 1 day | Lab 1; the traceroute traps | Read a real traceroute correctly | | 1 week | Labs 2 and 3; one site monitored with SNMPv3 and a probe | A KPI scorecard for one site | | 30 days | Runbooks for the top alerts; one change window with pre- and post-checks; a carrier ticket with evidence | Records and a drill | | Advanced | Flow analysis, BGP monitoring, then [NetworkOps](../networkops/) | A capacity forecast acted on | ## Wrap-up ### Executive recap and cheat sheet - **Decide:** who runs the network out of hours; which paths matter; what objectives users need. - **Measure:** probes per site (loss, delay p95, PDV), 64-bit interface counters, discards, alert actionability, capacity runway. - **Do:** scope, then isolate bottom-up along the path; evidence before escalation; few verified changes per window; out-of-band drills. - **Watch out for:** cosmetic traceroute spikes, PMTU black holes, 32-bit counters, monitoring that shares the failed path, exporters anyone can query, redundant pairs above 50%. ### A judgement call: fail over to a smaller backup, or ride out a lossy primary? **Situation (illustrative).** At 10:00 a branch's 1 Gbit/s primary circuit starts losing about 2% of packets in bursts; the carrier ticket is open with no estimate. The backup is a 200 Mbit/s circuit; the branch's p95 load is 260 Mbit/s, of which voice and payments are 30 Mbit/s. | Option | For | Against | |---|---|---| | A. Fail everything over | Clean path | 260 Mbit/s into 200: queues and loss on the backup too | | B. Stay on the primary | Full capacity | At 2% loss and 30 ms RTT the Mathis model gives about 3.4 Mbit/s per TCP flow[^mathis]; calls suffer | | C. Move voice and payments only | Critical traffic gets a clean path | Needs policy routing per class, prepared in advance | **Decision:** C if the routers already support class-based routing to the backup: 30 Mbit/s fits easily and bulk traffic keeps the primary's capacity. Without it, choose B and escalate hard: A would push 260 Mbit/s into 200 all afternoon, while the primary loses packets only in bursts. Prepare C before the next fault. **Uncertainty:** the loss model describes Reno-style TCP; CUBIC flows and real-time media degrade differently, and the bursts may worsen. ### Learning objectives and mastery check After this page you can **explain** what NetOps covers and how the label is used; **isolate** a fault layer by layer; **compute** utilisation, loss and delay KPIs; **escalate** to a carrier with evidence; **decide** a NOC model. 1. Traceroute shows 80 ms at hop 6 and 12 ms at hops 7 to 10. Is hop 6 congested? ::: answer Show answer for question 1 Almost certainly not. A delay that does not continue to later hops comes from the router's slow, rate-limited ICMP generation, not from forwarding.[^ras] ::: 2. Users can ping a remote server and log in, but file transfers stall. What do you test first? ::: answer Show answer for question 2 The path MTU: `ping -M do -s 1472` and `tracepath`. Filtered ICMP "fragmentation needed" messages create a black hole that pings do not reveal.[^rfc2923] ::: 3. A 1 Gbit/s interface polled every 5 minutes through `ifInOctets` shows 30 Mbit/s at peak. Why distrust it? ::: answer Show answer for question 3 A 32-bit octet counter wraps in about 34 seconds at 1 Gbit/s,[^rfc2863] so several wraps can hide between polls. Use `ifHCInOctets`. ::: 4. Two 10 Gbit/s links each run at 6 Gbit/s p95. Is the pair redundant? ::: answer Show answer for question 4 No. One failure leaves 12 Gbit/s on a 10 Gbit/s link ([worked calculation 3](#metrics-and-kpis)). ::: 5. Name two design changes that the Rogers assessment recommends for any operator's NOC. ::: answer Show answer for question 5 Separate the management network physically and logically from the data network, and give the NOC backup connectivity from third-party operators.[^crtc] ::: 6. A colleague exposes `snmp_exporter` on all addresses without authentication, "because it only reads". What is the risk? ::: answer Show answer for question 6 Whoever reaches `/snmp` can make it query any device with the stored SNMPv3 credentials. Bind it to the collector's address, admit only Prometheus, and enable TLS and basic authentication ([reference implementation](#reference-implementation-an-snmpv3-and-probe-monitoring-baseline)).[^snmpexp] ::: ### Conclusion NetOps is an old job under a contested label. The job stays: a network fails partially and silently, and somebody must measure the paths users take, read the evidence honestly, and keep a way in when the network itself is down. ## Sources and further reading Claim-by-claim evidence: `research/netops-ledger.md`. Re-check tool versions, cloud flow-log behaviour and AWS bandwidth rules every six months. [^tt-netops2]: [The NetOps 2.0 era is here: A primer for networking pros](https://www.techtarget.com/it-infrastructure/feature/The-NetOps-20-era-is-here-A-primer-for-networking-pros) — Alissa Irei, TechTarget, 2022-03-16. Checked 2026-10-18. [^rh-netops]: [What is NetOps?](https://www.redhat.com/en/topics/automation/what-is-netops) — Red Hat, 2023-08-28. Checked 2026-10-18. [^kentik]: [What is NetOps? (Network Operations)](https://www.kentik.com/kentipedia/what-is-netops-network-operations/) — Kentik. Checked 2026-10-18. [^dodi]: [DoD Instruction 8410.02, NetOps for the Global Information Grid](https://nsarchive.gwu.edu/media/18603/ocr) — US Department of Defense, 2008-12-19 (OCR copy, National Security Archive). Checked 2026-10-18. [^onet]: [Computer Network Support Specialists, 15-1231.00](https://www.onetonline.org/link/summary/15-1231.00) — O*NET OnLine, US Department of Labor, updated 2026. Checked 2026-10-18. [^m3400]: [ITU-T M.3400, TMN management functions](https://www.itu.int/rec/T-REC-M.3400-200002-I/en) — ITU-T, 2000; earlier editions (1992, 1997) on [the Recommendation's page](https://www.itu.int/rec/T-REC-M.3400). Checked 2026-10-18. [^rfc6632]: [RFC 6632, An Overview of the IETF Network Management Standards](https://www.rfc-editor.org/rfc/rfc6632) — IETF (Informational), 2012-06. Checked 2026-10-18. [^att]: [History of Network Management](https://web.archive.org/web/20160113080933/http://www.corp.att.com/history/nethistory/management.html) — AT&T (Internet Archive copy, 2016). Checked 2026-10-18. [^ietf-ops]: [Operations and Management Area](https://datatracker.ietf.org/group/ops/about/) — IETF Datatracker. Checked 2026-10-18. [^mitre]: [11 Strategies of a World-Class Cybersecurity Operations Center](https://web.archive.org/web/20241119095447/https://www.mitre.org/sites/default/files/2022-04/11-strategies-of-a-world-class-cybersecurity-operations-center.pdf) — Knerler, Parker, Zimmerman, MITRE, 2022 (Internet Archive copy). Checked 2026-10-18. [^rfc2863]: [RFC 2863, The Interfaces Group MIB](https://www.rfc-editor.org/rfc/rfc2863) — IETF, 2000-06. Checked 2026-10-18. [^rfc1573]: [RFC 1573, Evolution of the Interfaces Group of MIB-II](https://www.rfc-editor.org/rfc/rfc1573) — IETF, 1994-01. Checked 2026-10-18. [^rfc3410]: [RFC 3410, Introduction and Applicability Statements for Internet Standard Management Framework](https://www.rfc-editor.org/rfc/rfc3410) — IETF (Informational), 2002-12. Checked 2026-10-18. [^rfc4293]: [RFC 4293, MIB for the Internet Protocol](https://www.rfc-editor.org/rfc/rfc4293) — IETF, 2006-04. Checked 2026-10-18. [^rfc7860]: [RFC 7860, HMAC-SHA-2 Authentication Protocols in USM for SNMPv3](https://www.rfc-editor.org/rfc/rfc7860) — IETF, 2016-04. Checked 2026-10-18. [^rfc3954]: [RFC 3954, Cisco Systems NetFlow Services Export Version 9](https://www.rfc-editor.org/rfc/rfc3954) — IETF (Informational), 2004-10. Checked 2026-10-18. [^rfc5101]: [RFC 5101, IPFIX Protocol Specification](https://www.rfc-editor.org/rfc/rfc5101) — IETF, 2008-01 (obsoleted by RFC 7011). Checked 2026-10-18. [^rfc7011]: [RFC 7011, IP Flow Information Export (IPFIX) Protocol](https://www.rfc-editor.org/rfc/rfc7011) — IETF, 2013-09. Checked 2026-10-18. [^sflow]: [sFlow Version 5](https://sflow.org/sflow_version_5.txt) — sFlow.org, 2004-07. Checked 2026-10-18. [^sflow-basics]: [Packet Sampling Basics](https://sflow.org/packetSamplingBasics/index.htm) — Peter Phaal, Sonia Panchen, sFlow.org, undated. Checked 2026-10-18. [^rfc5426]: [RFC 5426, Transmission of Syslog Messages over UDP](https://www.rfc-editor.org/rfc/rfc5426) — IETF, 2009-03. Checked 2026-10-18. [^rfc3416]: [RFC 3416, Version 2 of the Protocol Operations for SNMP](https://www.rfc-editor.org/rfc/rfc3416) — IETF (STD 62), 2002-12. Checked 2026-10-18. [^rfc5425]: [RFC 5425, TLS Transport Mapping for Syslog](https://www.rfc-editor.org/rfc/rfc5425) — IETF, 2009-03. Checked 2026-10-18. [^gnmi]: [gNMI Specification 0.10.0](https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-specification.md) — OpenConfig. Checked 2026-10-18. [^rfc9232]: [RFC 9232, Network Telemetry Framework](https://www.rfc-editor.org/rfc/rfc9232) — IETF (Informational), 2022-05. Checked 2026-10-18. [^rfc7854]: [RFC 7854, BGP Monitoring Protocol](https://www.rfc-editor.org/rfc/rfc7854) — IETF, 2016-06. Checked 2026-10-18. [^rfc8762]: [RFC 8762, Simple Two-Way Active Measurement Protocol](https://www.rfc-editor.org/rfc/rfc8762) — IETF, 2020-03. Checked 2026-10-18. [^rfc5880]: [RFC 5880, Bidirectional Forwarding Detection](https://www.rfc-editor.org/rfc/rfc5880) — IETF, 2010-06. Checked 2026-10-18. [^rfc7679]: [RFC 7679, A One-Way Delay Metric for IPPM](https://www.rfc-editor.org/rfc/rfc7679) — IETF, 2016-01. Checked 2026-10-18. [^rfc7680]: [RFC 7680, A One-Way Loss Metric for IPPM](https://www.rfc-editor.org/rfc/rfc7680) — IETF, 2016-01. Checked 2026-10-18. [^rfc3393]: [RFC 3393, IP Packet Delay Variation Metric for IPPM](https://www.rfc-editor.org/rfc/rfc3393) — IETF, 2002-11. Checked 2026-10-18. [^rfc5481]: [RFC 5481, Packet Delay Variation Applicability Statement](https://www.rfc-editor.org/rfc/rfc5481) — IETF (Informational), 2009-03. Checked 2026-10-18. [^rfc3550]: [RFC 3550, RTP: A Transport Protocol for Real-Time Applications](https://www.rfc-editor.org/rfc/rfc3550) — IETF, 2003-07. Checked 2026-10-18. [^rfc2923]: [RFC 2923, TCP Problems with Path MTU Discovery](https://www.rfc-editor.org/rfc/rfc2923) — IETF (Informational), 2000-09. Checked 2026-10-18. [^rfc8200]: [RFC 8200, Internet Protocol, Version 6 (IPv6) Specification](https://www.rfc-editor.org/rfc/rfc8200) — IETF (STD 86), 2017-07. Checked 2026-10-18. [^rfc8201]: [RFC 8201, Path MTU Discovery for IP version 6](https://www.rfc-editor.org/rfc/rfc8201) — IETF (STD 87), 2017-07. Checked 2026-10-18. [^rfc7454]: [RFC 7454 (BCP 194), BGP Operations and Security](https://www.rfc-editor.org/rfc/rfc7454) — IETF, 2015-02. Checked 2026-10-18. [^rfc7999]: [RFC 7999, BLACKHOLE Community](https://www.rfc-editor.org/rfc/rfc7999) — IETF (Informational), 2016-10. Checked 2026-10-18. [^rfc8955]: [RFC 8955, Dissemination of Flow Specification Rules](https://www.rfc-editor.org/rfc/rfc8955) — IETF, 2020-12. Checked 2026-10-18. [^y1541]: [ITU-T Y.1541, Network performance objectives for IP-based services](https://www.itu.int/rec/T-REC-Y.1541-201112-I/en) — ITU-T, 2011-12. Checked 2026-10-18. [^g114]: [ITU-T G.114, One-way transmission time](https://www.itu.int/rec/T-REC-G.114-200305-I/en) — ITU-T, 2003-05. Checked 2026-10-18. [^mef]: [MEF 10.4, Subscriber Ethernet Service Attributes](https://www.mplify.net/wp-content/uploads/2018/12/MEF-10-4.pdf) — MEF Forum, 2018-12. Checked 2026-10-18. [^manrs]: [MANRS for Network Operators](https://manrs.org/netops/) — MANRS. Checked 2026-10-18. [^sre-mon]: [Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/) — Rob Ewaschuk, *Site Reliability Engineering*, Google, 2016. Checked 2026-10-18. [^sre-oncall]: [Being On-Call](https://sre.google/sre-book/being-on-call/) — Andrea Spadaccini, *Site Reliability Engineering*, Google, 2016. Checked 2026-10-18. [^ranjbar]: [Troubleshooting Methods for Cisco IP Networks](https://www.ciscopress.com/articles/article.asp?p=2273070&seqNum=2) — Amir Ranjbar, Cisco Press sample chapter, 2015-01-14. Checked 2026-10-18. [^ping]: [ping(8)](https://man7.org/linux/man-pages/man8/ping.8.html) — iputils. Checked 2026-10-18. [^tracepath]: [tracepath(8)](https://man7.org/linux/man-pages/man8/tracepath.8.html) — iputils. Checked 2026-10-18. [^traceroute]: [traceroute(8)](https://man7.org/linux/man-pages/man8/traceroute.8.html) — man7.org. Checked 2026-10-18. [^mtr]: [mtr(8)](https://manpages.debian.org/bookworm/mtr-tiny/mtr.8.en.html) — Debian bookworm, mtr 0.95. Checked 2026-10-18. [^ip-route]: [ip-route(8)](https://man7.org/linux/man-pages/man8/ip-route.8.html) — iproute2. Checked 2026-10-18. [^ss]: [ss(8)](https://man7.org/linux/man-pages/man8/ss.8.html) — iproute2. Checked 2026-10-18. [^tcpdump]: [tcpdump(1)](https://man7.org/linux/man-pages/man1/tcpdump.1.html) — tcpdump project. Checked 2026-10-18. [^netem]: [tc-netem(8)](https://man7.org/linux/man-pages/man8/tc-netem.8.html) — iproute2. Checked 2026-10-18. [^proc-environ]: [proc_pid_environ(5)](https://man7.org/linux/man-pages/man5/proc_pid_environ.5.html) — Linux man-pages. Checked 2026-10-18. [^kstats]: [Interface statistics](https://docs.kernel.org/networking/statistics.html) — Linux kernel documentation. Checked 2026-10-18. [^curl]: [curl man page](https://curl.se/docs/manpage.html) — curl project. Checked 2026-10-18. [^wireshark]: [Display Filter Reference: TCP](https://www.wireshark.org/docs/dfref/t/tcp.html) — Wireshark. Checked 2026-10-18. [^ras]: [Traceroute](https://major.io/wp-content/uploads/2012/06/RAS_Traceroute_Book_Format.pdf) — Richard A. Steenbergen, 2011 (copy hosted on major.io). Checked 2026-10-18. [^ripe-atlas]: [RIPE Atlas](https://www.ripe.net/analyse/internet-measurements/ripe-atlas/) — RIPE NCC. Checked 2026-10-18. [^mathis]: [The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm](https://courses.cs.duke.edu/fall24/compsci514/readings/mathis-tcpmodel-ccr97.pdf) — Mathis, Semke, Mahdavi, Ott, ACM CCR 27(3), 1997 (preprint copy). Checked 2026-10-18. [^cubic]: [RFC 9438, CUBIC for Fast and Long-Distance Networks](https://www.rfc-editor.org/rfc/rfc9438) — IETF, 2023-08. Checked 2026-10-18. [^b4]: [B4: Experience with a Globally-Deployed Software Defined WAN](http://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf) — Jain et al., Google, SIGCOMM 2013. Checked 2026-10-18. [^aws-flow-basics]: [Flow log basics](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-basics.html) — AWS. Checked 2026-10-18. [^aws-flow-lim]: [Flow log limitations](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-limitations.html) — AWS. Checked 2026-10-18. [^gcp-flow]: [VPC Flow Logs](https://docs.cloud.google.com/vpc/docs/flow-logs) — Google Cloud. Checked 2026-10-18. [^aws-bw]: [Amazon EC2 instance network bandwidth](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html) — AWS. Checked 2026-10-18. [^aws-ena]: [Monitor network performance for ENA settings](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-network-performance-ena.html) — AWS. Checked 2026-10-18. [^snmpexp]: [Prometheus SNMP Exporter](https://github.com/prometheus/snmp_exporter) — Prometheus project, v0.30.1. Checked 2026-10-18. [^snmpexp-gen]: [SNMP Exporter config generator](https://github.com/prometheus/snmp_exporter/blob/main/generator/README.md) — Prometheus project. Checked 2026-10-18. [^snmpexp-auth]: [snmp-auth-samples.yml](https://github.com/prometheus/snmp_exporter/blob/main/examples/sample-configs/snmp-auth-samples.yml) — Prometheus project. Checked 2026-10-18. [^webcfg]: [Web configuration](https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md) — Prometheus exporter-toolkit. Checked 2026-10-18. [^blackbox]: [Blackbox exporter](https://github.com/prometheus/blackbox_exporter) — Prometheus project, v0.29.0. Checked 2026-10-18. [^prom-basics]: [Querying basics](https://prometheus.io/docs/prometheus/latest/querying/basics/) — Prometheus. Checked 2026-10-18. [^prom-rules]: [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) — Prometheus. Checked 2026-10-18. [^prom-rate]: [Query functions](https://prometheus.io/docs/prometheus/latest/querying/functions/) — Prometheus. Checked 2026-10-18. [^am]: [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) — Prometheus. Checked 2026-10-18. [^junos-desc]: [Physical Interfaces](https://www.juniper.net/documentation/us/en/software/junos/interfaces-fundamentals/topics/topic-map/physical-interface-properties.html) — Juniper Networks, Junos OS documentation. Checked 2026-10-18. [^librenms]: [LibreNMS](https://github.com/librenms/librenms) — LibreNMS project. Checked 2026-10-18. [^zabbix]: [SNMP agent](https://www.zabbix.com/documentation/current/en/manual/config/items/itemtypes/snmp) — Zabbix documentation. Checked 2026-10-18. [^netdisco]: [Netdisco](https://netdisco.org/) — Netdisco project. Checked 2026-10-18. [^nfdump]: [nfdump](https://github.com/phaag/nfdump) — Peter Haag. Checked 2026-10-18. [^pmacct]: [pmacct](http://www.pmacct.net/) — pmacct project. Checked 2026-10-18. [^smokeping]: [SmokePing](https://oss.oetiker.ch/smokeping/) — Tobias Oetiker. Checked 2026-10-18. [^perfsonar]: [perfSONAR documentation](https://docs.perfsonar.net/) — perfSONAR project. Checked 2026-10-18. [^fb]: [More details about the October 4 outage](https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/) — Santosh Janardhan, Meta, 2021-10-05. Checked 2026-10-18. [^aws-2021]: [Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/12721/) — AWS, 2021-12-10. Checked 2026-10-18. [^aws-2025]: [Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/101925/) — AWS, October 2025. Checked 2026-10-18. [^cf-2020]: [Cloudflare outage on July 17, 2020](https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/) — John Graham-Cumming, Cloudflare, 2020-07-18. Checked 2026-10-18. [^crtc]: [Assessment of Rogers Networks for Resiliency and Reliability Following the 8 July 2022 Outage – Executive Summary](https://web.archive.org/web/20251220060847/https://crtc.gc.ca/eng/publications/reports/xona2024.htm) — Xona Partners for the CRTC, July 2024 (Internet Archive copy). Checked 2026-10-18. **Changelog.** 2026-10-18, r0: first version (research, ledger, labs executed where safe). 2026-10-18, r1: revised after the independent review (exporter security, paging rules, DDoS, hardware faults, OSI mapping, targets, staffing, dates). 2026-10-18, post-score polish: IPv6 path MTU, `ifAlias` anchoring, sampling error, FCAPS sources, BGP row, audit checklist, Figure 1 retry branch; repeats trimmed.