--- title: "NetworkOps" slug: networkops category: cloud term_status: overlapping-synonym evidence_status: verified summary: "NetworkOps is an informal variant of NetOps (network operations). This Atlas uses it for network automation, also called NetDevOps: engineering network change through a source of truth, generated configuration, verification before deployment, staged rollout and automatic rollback." audience: [Beginners, Network engineers, Network automation engineers, Architects, Leaders] last_verified: 2026-10-18 key_takeaways: - "'NetworkOps' is a rare, informal variant of NetOps, not a discipline. This Atlas uses it for network automation (NetDevOps), the engineering of change, and NetOps for running the network: an editorial split." - "Automate the whole change path, not just the typing: source of truth, generated configuration, verification, staged rollout, state checks." - "The source of truth holds intended state that people have vetted; NetBox's own documentation strongly discourages importing live state into it." - "Verify before deploying with a model (Batfish) or an emulator (containerlab), then stage the rollout so representative devices are reached early." - "Every deployment needs an exit: a commit-confirmed timer, a management path the change cannot break, and one person who owns reverts." - "Automation multiplies mistakes as well as work: check the rendered diff against declared target and must-not-change lists, and measure change fail rate, not scripts written." related: [netops, automationops, infraops, gitops, cloudops, edgeops, observabilityops, kubernetesops] --- ## What NetworkOps is ### NetworkOps in 30 seconds **NetworkOps**, as this Atlas uses the word, is the engineering of network change: describing the network you want as data, generating device configuration from it, proving the change safe *before* it reaches a router, rolling it out in small waves with an automatic way back, and checking that the network then behaves as intended. Its established names are **network automation** and **NetDevOps**. **In one line: NetworkOps turns "log in and type" into "change the data, let the pipeline prove it, deploy with a timer".** ::: note Honest term status: an overlapping synonym "NetworkOps" is not a recognised discipline, standard, role or certification, and exact matches are rare. In October 2026 a web search returned NetOps pages, and the 26 GitHub repositories named after it were small projects in both senses: automation tools[^nwops-repo], and NOC dashboards or learning sites[^gh-search]. The practice this page covers is better known as **network automation** (the Network Automation Forum runs the AutoCon conferences[^naf]) and **NetDevOps** ("a principle, a philosophy, a culture, a discipline", in Cisco's developer blog[^cisco-netdevops]). Red Hat says **NetOps** "is also referred to as NetOps 2.0 or NetDevOps"[^redhat-netops], so even the synonym partner is used for this practice. **Editorial distinction.** [NetOps](../netops/) is about *operating* the running network: monitoring, fault isolation, the network operations centre (NOC). NetworkOps is about *engineering change*: models, a source of truth, verification, rollback. No standard defines this boundary. ::: **Where to start.** Beginners: the [mental model](#glossary-and-mental-model) and [Lab 1](#lab-1-beginner-from-source-of-truth-to-checked-configuration). Engineers: the [playbooks](#core-practices-and-playbooks) and the [reference implementation](#reference-implementation). Leaders: the [decision framework](#decision-framework) and the [judgement call](#a-judgement-call-replace-the-whole-configuration-or-merge-changes). **Last verified** 2026-10-18; tool versions are dated "as of October 2026". ### At a glance | Aspect | Summary | |---|---| | Goal | Network changes that are verified before they ship, repeatable, small and reversible | | People | Network engineers learning automation, network automation engineers, architects, change managers | | Prerequisites | Routing and switching basics ([NetOps](../netops/)), Git, YAML/JSON, some Python | | In scope | Source of truth, YANG/OpenConfig models, NETCONF, RESTCONF, gNMI, templates, Ansible, Nornir, Batfish, emulation, CI/CD, staged rollout, confirmed commits, drift, intent | | Out of scope | NOC monitoring and fault isolation ([NetOps](../netops/)); cloud VPC design ([CloudOps](../cloudops/#network-foundations)); pod networking ([KubernetesOps](../kubernetesops/)); automation governance in general ([AutomationOps](../automationops/)) | | Outputs | An owned source of truth, generated configs in Git, a verification suite, a pipeline with rollback, a drift report | | Term status | Overlapping synonym of network automation / NetDevOps (note above) | ### The problem it solves Three public reports show outages that started with a change: - **Changes are where failures cluster.** Of over 100 high-impact failure events in Google's network, nearly 70 happened while a management operation was in progress on the affected element, though it was not always the cause[^evolve-or-die]. - **A harmless-looking diff can be fatal.** At Cloudflare in 2022, "a re-ordering of terms caused us to withdraw a critical subset of prefixes" despite a dry run, a stepped rollout and peer review; 19 data centres were offline for up to 75 minutes[^cloudflare-2022]. - **Process checks fail quietly.** At Rogers in 2022, staff removed a policy filter while tidying configuration; core routers were flooded with routes and crashed. The audit missed it, and a risk algorithm rated the step "Low", so no lab test was required. More than 12 million customers lost service, restored gradually over about 26 hours[^crtc-rogers]. Box-by-box CLI work makes this worse: devices drift apart, review means reading CLI text, and nobody can say what the network *should* look like. In 2002 operators already asked to configure "the network as a whole rather than individual devices", to use diff and version control on configurations, and to separate distributing a configuration from activating it[^rfc3535]. **What it does not solve.** Automation spreads a wrong change faster. Google found partial automation "increased availability in general" but could also "cause large failures"[^evolve-or-die]; safety comes from verification, staging and rollback. ### What NetworkOps is not: boundaries and naming - **Not the NOC.** Monitoring, fault isolation and carrier escalation belong to [NetOps](../netops/). NetworkOps configures the telemetry NetOps consumes and owns the rollback of a failed change. - **Not "we wrote some scripts".** Scripts that push commands without a source of truth, verification or rollback are automation debt. Governance of automation in general is on [AutomationOps](../automationops/). - **Not a product.** Controllers and "intent-based" platforms implement parts of the practice; intent terms "are often used loosely and inconsistently"[^rfc9315]. Buying one does not give you owned data or a verification habit. - **Not GitOps by definition.** Network pipelines usually *push* from a runner; pull-based reconciliation ([GitOps](../gitops/)) is possible but is not what makes a change safe. - **Not cloud network design** ([CloudOps](../cloudops/#network-foundations)), though the method applies there too. ## Where it comes from and where it sits ### Origins and evolution The label has no traceable origin. The practice has a clear technical history: - **2002–2003: requirements.** The IAB workshop asked for configuration separate from state, network-wide transactions and diff-able text[^rfc3535]. - **2006–2018: protocols and models.** NETCONF (RFC 4741 in 2006, RFC 6241 in 2011) brought candidate datastores and confirmed commits[^rfc6241]; YANG (2010; 1.1 in 2016)[^rfc7950], RESTCONF (2017)[^rfc8040] and the NMDA datastores (2018)[^rfc8342] followed. An IRTF document defined SDN as "the capacity to initialize, control, change, and manage network behavior dynamically via open interfaces" (2015)[^rfc7426]. - **2015–2017: practitioners' tools.** NetBox was written at DigitalOcean in 2015 and open-sourced in 2016[^netbox-intro]; Cisco released pyATS publicly in late 2017[^cisco-netdevops]; Batfish came from Microsoft Research, UCLA and USC[^batfish]. - **2016 onwards: streaming and provisioning.** Operator-led OpenConfig published the first gNMI specification in 2016 (current v0.10.0, 2023)[^gnmi][^openconfig] and a commit-confirmed extension in 2023[^gnmi-cc]; the IETF added YANG-Push[^rfc8641] and Secure Zero Touch Provisioning[^rfc8572] in 2019. - **2022–2026: intent and community.** RFC 9315 on intent-based networking (2022)[^rfc9315]; the Network Automation Forum's AutoCon events from 2023. NAF admits automation has been discussed "for a long time: 15 years or longer", yet "it's just not taking off like we thought it would"[^naf]. ### Glossary and mental model **Mental model: three views of every device.** The *source of truth* says what the network should be, the *intended configuration* is generated from it, and the *operational state* is what the device does. NetworkOps lives in the gaps between them. ::: figure Three views of a device and where NetworkOps checks the gaps ```mermaid %%{init: {"flowchart": {"wrappingWidth": 170, "nodeSpacing": 14, "rankSpacing": 24, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 SOT["Source of truth: sites, devices, IPs, peerings, policy data, vetted by people"]:::core INT["Intended configuration: generated per device, stored in Git"]:::core G1["Gate 1, before deployment: valid and safe? Lint, model checks, emulation"]:::decision RUN["Running and applied configuration on the device"]:::neutral OPS["Operational state: sessions, routes, counters"]:::neutral G2["Gate 2, after deployment: does behaviour match intent? State checks, telemetry"]:::decision G3["Gate 3, continuously: does running still equal intended? Drift check"]:::decision FIX["On mismatch: fix the data or the device, then re-render"]:::neutral SOT --> INT --> G1 --> RUN RUN --> OPS --> G2 RUN --> G3 --> FIX INT -->|"compared with"| G3 ``` Text equivalent: The source of truth (sites, devices, IP addresses, peerings, policy data, vetted by people) generates an intended configuration per device, stored in Git. Gate 1, before deployment, checks it is valid and safe (lint, model checks, emulation). The device then has a running configuration and an operational state (sessions, routes, counters); gate 2 checks that behaviour matches intent. Gate 3 continuously compares the running configuration with the intended one; on a mismatch, the data or the device is fixed and the configuration re-rendered. ::: | Term | Meaning | |---|---| | Source of truth (SoT) | The authoritative record of *intended* network data; NetBox "intends to represent the desired state of a network versus its operational state"[^netbox-intro] | | Intended configuration | "the configuration that the system attempts to apply" (RFC 8342)[^rfc8342]; here, the generated file in Git | | Applied configuration, operational state | What is in use; operational state adds system state such as counters[^rfc8342] | | Drift | Running configuration that no longer matches intended; RFC 9315's *intent drift* is behaviour ceasing to meet intent[^rfc9315] | | YANG model | A schema for configuration, state, RPCs and notifications[^rfc7950]: vendor-native, IETF or OpenConfig[^openconfig] | | NETCONF, RESTCONF, gNMI | Model-driven management protocols over SSH, HTTP and gRPC[^rfc6241][^rfc8040][^gnmi] | | Candidate datastore | A working copy that "can be manipulated without impacting the device's current configuration"[^rfc6241] | | Confirmed commit | A commit that reverts unless confirmed in time; NETCONF's default is 600 s[^rfc6241], Junos's 10 minutes[^junos-cc] | | Merge, replace | Add lines to the running configuration, or make the device equal the candidate[^napalm-api] | | Wave, blast radius | One step of a staged rollout; the devices and services a change can affect | | AAA | Authentication, authorisation, accounting (TACACS+, RADIUS): who may log in, what they may run, what they did | | IPAM, DCIM | IP address management; data-centre infrastructure management (racks, devices, cables) | | SD-WAN | Software-defined WAN: branch routers run as one overlay by a vendor controller | | TextFSM | A parser that turns CLI output into structured data with templates[^ntc-templates] | | MCP (Cloudflare) | "Multi-Colo PoP", a Cloudflare data-centre design[^cloudflare-2022]; not the Model Context Protocol | | Intent | "operational goals … and outcomes … defined in a declarative manner without specifying how to achieve or implement them"[^rfc9315] | ### How NetworkOps relates to other Ops NetworkOps sits on the change path between design and the running network. These boundaries are **editorial distinctions**. ::: figure NetworkOps and its neighbours ```mermaid %%{init: {"flowchart": {"wrappingWidth": 170, "nodeSpacing": 16, "rankSpacing": 30, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 AUTO["AutomationOps: governance for all automation"]:::neutral INFRA["InfraOps: IaC lifecycle"]:::neutral CLOUD["CloudOps: VPCs, hubs, private access"]:::neutral NWO["NetworkOps: engineer network change (data, generation, verification, rollback)"]:::core NETOPS["NetOps: run the network (NOC, faults, capacity)"]:::neutral OBS["ObservabilityOps: telemetry platform"]:::neutral INC["IncidentOps: incident command"]:::neutral AUTO -->|"guardrails"| NWO INFRA -->|"IaC methods"| NWO CLOUD -->|"cloud constructs"| NWO NWO -->|"verified changes"| NETOPS NWO -->|"telemetry subscriptions"| OBS OBS -->|"signals"| NETOPS NETOPS -->|"change requests, faults"| NWO NETOPS -->|"major incidents"| INC ``` Text equivalent: AutomationOps (guardrails for all automation), InfraOps (infrastructure-as-code methods) and CloudOps (cloud network constructs) feed NetworkOps, which engineers network change. NetworkOps hands verified changes to NetOps, which runs the network, and configures the telemetry subscriptions ObservabilityOps collects. Signals flow to NetOps, which sends change requests and faults back to NetworkOps and escalates major incidents to IncidentOps. ::: | Neighbour | Choose that page for… | |---|---| | [NetOps](../netops/) | Monitoring, fault isolation or capacity on the same network | | [AutomationOps](../automationops/) | Governing automation across domains (ownership, kill switches, audit) | | [InfraOps](../infraops/) | The infrastructure-as-code lifecycle for servers and cloud | | [GitOps](../gitops/) | Pull-based reconciliation from Git | | [CloudOps](../cloudops/) | VPCs, transit hubs and private endpoints | | [ObservabilityOps](../observabilityops/) | Pipelines that store and query telemetry | | [EdgeOps](../edgeops/) | Fleets of edge nodes rather than the network | | [KubernetesOps](../kubernetesops/) | CNI plugins or pod addressing | ## Principles and operating model ### Charter and principles This charter is **this Atlas's proposal**; no body publishes an official one. **Mission.** Make network change boring: described as data, verified before it ships, rolled out in small waves, reversible without a site visit, and confirmed by evidence. 1. **Change the data, not the device.** The unit of change is one pull request: the data change (or its source-of-truth branch), templates, rendered configurations and a change record. 2. **People vet the source of truth.** NetBox says "automated import of live network state is strongly discouraged"[^netbox-intro]. Discovered state feeds a *reconciliation report*, never the SoT, or drift gets written into the record. 3. **Generate; never hand-edit.** Every line on a device comes from a template or model; a hand edit is drift. 4. **Verify with two kinds of evidence.** A model checks the whole network but only the syntax it parses; an emulator runs real software but only at lab scale. Know each one's blind spots. 5. **Take the smallest safe step.** Put a *representative* device in an early wave. Cloudflare's stagger "did not include an MCP data center until the final step"[^cloudflare-2022]. 6. **Every change has an exit.** A confirmed-commit timer, a management path the change cannot break, one owner for reverts. Meta and Rogers both lost that path in their outages[^meta-2021][^crtc-rogers]; designing it is [NetOps](../netops/) work. 7. **Name the blast radius twice.** Declare the devices to change *and* those to leave alone, separately from the data that generates the change, and refuse to run if what would change disagrees. Google adopted this after a wildcard drained "hundreds of devices when the intention was to drain two"[^evolve-or-die]. 8. **Prove the outcome.** Done means sessions, routes and reachability match expectations, not that a command returned success. **Non-goals.** Removing human approval from high-risk changes; replacing the NOC; automating every change. One-off work may be cheaper by hand if its result goes back into the source of truth. ### Fundamentals: the building blocks Seven blocks, connected in the [reference architecture](#reference-architecture); tool options are in the [tooling landscape](#tooling-landscape). | Block | Job | Watch-out | |---|---|---| | Source of truth | Intended data: sites, devices, interfaces, IPs, VLANs, circuits, policies | Ownership and data quality decide success, not the tool | | Generation | Data → configuration text or model payloads (Jinja2, NetBox config templates[^netbox-render]) | Partial templates force merge mode | | Device interface | SSH CLI, NETCONF, RESTCONF or gNMI | CLI output needs parsing (TextFSM, ntc-templates[^ntc-templates]) | | Orchestration | Run tasks across many devices | Parallelism amplifies mistakes | | Verification | Prove a change safe before deployment | Unsupported syntax; lab-scale gaps | | Pipeline | Review, test, approve, deploy | Runner trust and credential scope | | Assurance | Behaviour and drift checks after deployment (streaming telemetry[^rfc8641], NAPALM validate[^napalm-validate]) | Alert fatigue if every difference pages someone | ### Roles, ownership and RACI Usually a capability inside the network team, not a separate team. A typical split (R responsible, A accountable, C consulted, I informed): | Activity | Network engineer | Automation engineer | SoT data steward | Change manager | NOC ([NetOps](../netops/)) | Security | |---|---|---|---|---|---|---| | Edit SoT data for a change | R | C | A | I | I | C (policy objects) | | Change templates or models | C | R/A | I | I | I | C | | Maintain verification suite | C | R/A | I | I | C | C | | Approve a normal change | R (author) | C | I | A | C | C (security-relevant) | | Run deployment | I | A (pipeline) | I | I | R (watches) | I | | Post-change checks | R | C | I | I | R | I | | Decide rollback | C | C | I | I | A (incident lead) | I | | Reconcile drift | R | C | A | I | C | C | Two rules matter most. **One accountable steward per data domain** (IPAM, circuits, BGP policy): an unowned source of truth decays. **The incident lead owns reverts**: at Cloudflare, engineers "walked over each other's changes, reverting the previous reverts"[^cloudflare-2022]. ### Lifecycle: Day 0, Day 1, Day 2 - **Day 0 – design.** Choose the data model and an owner per field, the generation method, the interface per platform and the verification suite. Import the current network *once*, review it by hand, and mark it as the baseline. - **Day 1 – provision.** New devices boot, fetch an image and an initial configuration; SZTP standardises a secure way to do this from "a factory-default state"[^rfc8572]. Vendor implementations differ. - **Day 2 – change and assure.** Most of the work: everyday changes, drift detection, OS upgrades (rings and maintenance groups are on [PatchOps](../patchops/)) and decommissioning, which is also a SoT change. ### Operating model and governance | Change class | Example | Path | Approval | |---|---|---|---| | Standard (pre-approved) | Add a VLAN to an access port, add a BGP peer from an approved template | Pipeline with full verification | Automatic when every check passes and the blast radius is under the limit | | Normal | New routing policy, firewall rule set, fabric expansion | Pipeline, peer review, lab or model evidence attached | Change manager; high-risk changes go to a change board | | Emergency | Mitigate an attack or a carrier failure | Break-glass access, logged | Retrospective review; the source of truth is updated within one working day | Governance rules this Atlas recommends: - **Score risk per change, not per programme.** Rogers' algorithm rated a step "Low" because earlier phases had gone well, so no lab test was required[^crtc-rogers]. Earlier success is not evidence about a different change. - **Cap changes per maintenance window.** Several changes in one window made it "difficult to decide which network change ticket to roll back" at Rogers[^crtc-rogers]. - **Review the evidence, not the CLI.** Reviewers read the verification report (sessions, reachability, filter decisions) beside the diff, because diffs hide meaning. Review cadences are in [metrics](#metrics-and-kpis). The loop mirrors the four phases of NIST's security-focused configuration management: planning, implementing configurations, controlling change, monitoring[^sp800-128]. ## Architecture and workflow ### Reference architecture The reference design has two trust zones. CI runs without device credentials. Inside the management network, only an approval-gated deploy runner can write to devices; a separate drift job only reads. ::: figure Reference architecture for network change ```mermaid %%{init: {"flowchart": {"wrappingWidth": 190, "nodeSpacing": 18, "rankSpacing": 30, "padding": 8, "subGraphTitleMargin": {"top": 4, "bottom": 14}}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 subgraph Z1["Zone 1: no device credentials"] direction TB SOT["Source of truth: data edited on a branch"]:::core GIT["Git: templates, generated configs, change record"]:::core CI["CI: guard, lint, model checks, emulation"]:::core end subgraph Z2["Zone 2: management network"] direction TB DEP["Deploy runner: restricted runner group, environment secrets after approval; commits with a timer, runs post-checks, confirms"]:::risk DRIFT["Drift job: read-only account"]:::neutral DEV["Network devices (candidate, confirmed commit)"]:::neutral TEL["Telemetry collector (gNMI, YANG-Push)"]:::neutral end OOB["Out-of-band access that does not depend on the data network"]:::risk SOT -->|"rendered from the branch"| GIT GIT --> CI CI -->|"approved merge"| DEP DEP --> DEV DRIFT -->|"read running config"| DEV DEV --> TEL OOB -.-> DEV ``` Text equivalent: Zone 1 holds no device credentials: source-of-truth data edited on a branch is rendered into Git (templates, generated configurations, change record), and CI runs the guard, linting, model checks and emulation. After an approved merge, Zone 2, the management network, takes over: a deploy runner in a restricted runner group, given environment secrets only after approval, commits to the devices with a timer, runs the post-checks and confirms. A drift job reads running configurations with a read-only account. Devices stream telemetry to a collector. An out-of-band path reaches the devices independently of the data network. ::: Three design choices carry most of the safety: - **Generated configurations are committed to Git**: reviewers see the real diff, and every deployment has a last-known-good commit. - **Only the deploy runner holds write credentials, and post-checks run there** before it confirms. GitHub warns that self-hosted runners "can be persistently compromised by untrusted code"[^gh-secure]; the [pipeline](#reference-implementation) names the settings that keep other workflows off it. - **The out-of-band path is independent.** Xona Partners' assessment for the CRTC calls for "secure alternative access to crucial remote network elements that is not dependent on the data network"[^crtc-rogers]; [NetOps](../netops/) covers its design. **Alternatives** (vendor controller, YAML-only source of truth, pull-based reconciliation): see [deep comparisons](#deep-comparisons). ### Changing the source of truth In NetBox, an edit is live intent at once: other consumers read it before anyone approves it, reviewers see generated configs without the data change or its author, and a scheduled export bundles several people's edits into one diff. Give data the path code takes: 1. **Stage the edit.** NetBox Labs' Branching plugin "allows you to make copies of NetBox's data model and alter them independently"; API requests pick a branch with the `X-NetBox-Branch` header[^nb-branching]. With a YAML source of truth in Git, the data change is part of the pull request. 2. **Render from the branch**, synced with the main data first so the render does not undo other merged changes. Commit the generated configs to the change's pull request with a change record naming target and must-not-change devices (the [guard](#reference-implementation) checks both). 3. **Merge the data after the last wave is confirmed.** A failed change leaves the branch unmerged, so the source of truth still describes the network; a merged branch can also be reverted until it is archived or deleted[^nb-branching]. Edits made outside a branch show up as drift between the source of truth and Git (playbook 5). ### End-to-end workflow: one change from request to evidence Two properties matter: a failed check before deployment touches no device, and a failed check after deployment needs no human action to revert, because the deployer does not confirm. The incident lead alone decides on further reverts. ::: figure Change workflow with happy and exception paths ```mermaid %%{init: {"flowchart": {"wrappingWidth": 210, "nodeSpacing": 18, "rankSpacing": 24, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 A["Request: ticket or pull request"]:::core B["Edit SoT data or templates on a branch"]:::core C["CI gate: render, lint, model checks, emulation"]:::decision X["Fail: fix data or template; no device touched"]:::risk D["Review diff and verification report; approve by change class"]:::core E["Deploy next wave with a confirmed-commit timer"]:::core F["Post-change checks"]:::decision Y["Fail: do not confirm, device reverts; halt waves; page incident lead"]:::risk G["Pass: confirm commit"]:::core H["More waves?"]:::decision I["Attach evidence; merge the SoT branch; close"]:::core A --> B --> C C -->|"fail"| X C -->|"pass"| D --> E --> F F -->|"fail"| Y F -->|"pass"| G --> H H -->|"yes"| E H -->|"no"| I ``` Text equivalent: A request (ticket or pull request) leads to edits of source-of-truth data or templates on a branch. CI renders, lints, and runs model checks and emulation; any failure stops the change before a device is touched. Otherwise a reviewer reads the diff and verification report, and approval follows the change class. Each wave is deployed with a confirmed-commit timer. If its post-change checks pass, the commit is confirmed and the next wave follows; after the last wave, evidence is attached, the source-of-truth branch merged and the change closed. If checks fail, the commit is not confirmed, so the device reverts; remaining waves halt and the incident lead is paged. ::: ## Practices, frameworks and tools ### Core practices and playbooks | Playbook | Trigger | Steps | Done when | Way back | |---|---|---|---|---| | 1 Baseline the SoT | Programme start, or a new site | Import once; stewards review each field; render; diff against running; fix data or device until the diff is empty | Rendered equals running on every device in scope | Pre-import backup of every config | | 2 Pre-change verification | Every pull request | Render; lint; parse check (Batfish `initIssues`); invariants (sessions up, required flows permitted, forbidden flows denied); differential questions | No new init issues; invariants hold; differences explained in the PR | None needed: nothing deployed | | 3 Staged rollout | Approved change | Canary of 1–2 representative devices, then a site, then a region; confirm each wave only after its checks pass | All waves confirmed; evidence attached | Unconfirmed wave reverts itself; the incident lead rolls back earlier waves | | 4 Confirmed commit | Every deployment, where supported | Commit with a timer (5–10 minutes); post-checks over a path the change cannot break; confirm | Confirmed inside the window | Timer expiry restores the previous configuration[^rfc6241][^junos-cc] | | 5 Drift reconciliation | Scheduled diff finds running ≠ intended | Classify: emergency fix not yet in SoT, unauthorised change, or template gap; update SoT or redeploy | Diff empty, or exception recorded with an expiry date | Redeploy intended config via the pipeline | | 6 Blast-radius guard | Every change | Declare targets and must-not-change devices in the change record; compare with the devices whose rendered configs changed; refuse on any mismatch; cap devices per wave | Guard passed, or a named approver overrode it | Abort before the first wave | **Confirmed commit on each platform.** NETCONF's `` lets a *different* session confirm, which suits automation that reconnects[^rfc6241]. gNMI has a commit-confirmed extension (v0.1.0)[^gnmi-cc] that Nokia SR Linux supports[^srl-gnmi]. NAPALM's `commit_config(revert_in=…)`[^napalm-api] is listed for EOS, Junos and IOS, not the IOS XR or NX-OS drivers[^napalm-support]. Where it is missing, use smaller waves and a tested rollback. **Merge or replace.** Merge adds your lines to whatever is on the device; replace makes the device equal your file. In Ansible, `cli_config` with `replace` replaces the whole running configuration[^ansible-cli-config]; resource-module states (`merged`, `replaced`, `overridden`) make the same choice per resource, such as interfaces or BGP, and `overridden` could "remove your access to the device"[^ansible-resource]. See the [judgement call](#a-judgement-call-replace-the-whole-configuration-or-merge-changes). ### Methods, standards and frameworks Only the IETF RFCs below are standards. OpenConfig is an operator-led open-source project[^openconfig], and RFC 9315 is an informational IRTF document. | Interface or model | Status | Strengths | Limits | |---|---|---|---| | CLI over SSH | Vendor-specific | Works everywhere; what engineers know | Unstructured output; atomicity varies by platform[^napalm-support]; ordering-sensitive | | NETCONF + YANG | IETF standard (RFC 6241, RFC 7950) | Candidate datastore, validate, confirmed commit, locks[^rfc6241] | XML verbosity; per-platform model coverage varies | | RESTCONF | IETF standard (RFC 8040) | HTTP tooling; a subset of NETCONF[^rfc8040] | No explicit locks; each edit is committed at once | | gNMI + OpenConfig/native models | OpenConfig specification | One transaction per SetRequest; streaming in ON_CHANGE, SAMPLE or TARGET_DEFINED mode[^gnmi] | Model and extension support vary by platform | | YANG-Push | IETF standard (RFC 8641) | Subscriptions to datastore changes[^rfc8641] | Check platform and collector support | | Intent-based networking | IRTF concepts (RFC 9315) | Vocabulary: fulfilment, assurance, intent drift[^rfc9315] | Not a protocol; products interpret it freely | **Models change the review question** from "are these commands right?" to "is this data right, and did verification prove its effect?", which is cheaper to answer and easier to automate. **Choosing and reading models.** OpenConfig models are vendor-neutral[^openconfig]; native ones can cover more. Nokia calls its SR Linux models "a more complete representation" of the platform, including features "that the OpenConfig data models do not describe"[^srl-yang]. This Atlas suggests OpenConfig where every platform in a domain implements the module, native models elsewhere; the gNMI Capabilities reply lists the models a device supports[^srl-yang]. Read a path as a walk down the tree: in Lab 3's `/interface[name=ethernet-1/1]/oper-state`, `interface` is a list, `[name=…]` selects one entry by its key, and `oper-state` is the leaf read[^gnmi-path]. **A NETCONF change, step by step.** The safety features are explicit operations: lock the candidate, edit it, validate it, commit with a timer, and confirm later from any session that presents the persist ID[^rfc6241]. _Example status: Python syntax-checked; not executed (needs ncclient 0.7 and a server with `:candidate`, `:validate` and `:confirmed-commit:1.1`)[^ncclient]._ ```python """NETCONF change: lock, edit the candidate, validate, confirmed commit, confirm.""" import os from ncclient import manager CONFIG = """ eth1to core1 """ DEVICE = {"host": "router1.example.net", "port": 830, "hostkey_verify": True, "username": os.environ["NET_USER"], "password": os.environ["NET_PASSWORD"]} with manager.connect(**DEVICE) as m, m.locked("candidate"): m.discard_changes() # start from a clean candidate m.edit_config(target="candidate", config=CONFIG) m.validate(source="candidate") # the server checks the whole candidate m.commit(confirmed=True, timeout="300", persist="chg-0142") # ... run post-checks over a path the change cannot break, then confirm: with manager.connect(**DEVICE) as m: m.commit(persist_id="chg-0142") # without this, the device reverts ``` **Controller-managed domains.** SD-WAN and many data-centre fabrics run through a vendor controller that renders and pushes device configuration itself: the SDN pattern, in which a controller "manipulates the controlled entity via an interface"[^rfc7426]. The controller then owns rendering, transactions and rollback; the source of truth still owns the inputs (sites, prefixes, policy) and feeds the controller's API through the same reviewed pipeline; verification checks behaviour, not device text. RFC 9315 splits intent-based systems into fulfilment and assurance[^rfc9315]: keep some assurance independent of the controller that fulfils. ### Tooling landscape Choose by job, then by platform. Versions as of October 2026[^pypi]. | Job | Representative open-source options | Strengths | Limits | |---|---|---|---| | Source of truth | NetBox[^netbox-intro], Nautobot[^nautobot] | IPAM/DCIM models, APIs, config rendering | Data entry and ownership effort | | Task runner | Ansible (`ansible.netcommon`)[^ansible-cli-config], Nornir[^nornir] | Ansible: declarative states, wide coverage. Nornir: plain, testable Python | Ansible: logic in YAML gets awkward. Nornir: more code to write | | Driver layer | NAPALM[^napalm-api], Netmiko[^netmiko] | Merge, replace, diff, commit confirm; any SSH CLI | Feature gaps per platform[^napalm-support] | | Model-driven clients | gnmic[^gnmic], ncclient[^ncclient] | Transactions, streaming | Requires model knowledge | | Model-based verification | Batfish[^batfish] | Whole-network what-if analysis without devices | Unsupported vendors or syntax; SR Linux is not on its list[^pybatfish] | | Emulation | containerlab[^clab] | Real network operating systems in containers | Lab scale; some images need vendor accounts | | Testing | pyATS[^cisco-netdevops], NAPALM validate[^napalm-validate] | State assertions before and after change | Test upkeep | **Lock-in and upkeep.** Open-source tools move fast. One Nornir inventory plugin for NetBox last released in September 2021, before NetBox's v2 API tokens[^pypi][^netbox-api]. Pin versions, test upgrades in CI, and keep the SoT schema independent of any one runner. ### Automation and AI The evidence on this page is about rule-based automation. AI assistants can draft templates, explain diffs or summarise a verification report; treat their output as an untrusted pull request that passes the same gates, and never give a model write access to devices or the SoT outside the pipeline. Automation "adds an additional layer of complexity"[^evolve-or-die]; AI adds another. Event analysis is on [AIOps](../aiops/). ## NetworkOps in practice ### Reference implementation Beginners can start with the [labs](#hands-on-labs) and return here later. **Assumptions.** About 60 Arista EOS and Juniper Junos devices; NetBox 4.7[^netbox-release] with the Branching plugin; GitHub Enterprise Cloud (see the plan note below); Nornir with NAPALM; Batfish in CI. Replace runs only on platforms whose playbook-1 baseline diff is empty; the rest merge. Syntax-checked, **not executed** except the guard: a shape, not a product. ```text network-repo/ sot-export.sh # NetBox branch -> snapshot/configs/*.cfg, run for each change change.yaml # change record: targets and must_not_change hosts snapshot/configs/ # generated configs, committed so reviewers see the diff guard.py # blast-radius guard: rendered diff vs change.yaml checks/check_change.py # Batfish invariants and differential questions (Lab 2) validate/.yaml # NAPALM post-change expectations per host inventory/hosts.yaml # Nornir hosts: platform, wave, full_template_coverage requirements-*.txt # pinned versions with hashes (pip-compile --generate-hashes) deploy.py # one wave: load with a timer, check, confirm or let it revert .github/workflows/network-change.yml ``` **Export from the source of truth.** NetBox renders each device's configuration from its template and context data, here inside the change's branch (untested; inferred from the rule that the branch "must be specified with each REST API request"[^nb-branching]). A token without "write enabled" can only make GET requests, but `render-config` is a POST[^netbox-api][^netbox-render]: give the export job a write-enabled v2 token whose *user* holds only device view and the `render_config` action, restricted to the job's IP. _Example status: `bash -n` syntax-checked; not executed (requires NetBox 4.5 or later for v2 tokens, the Branching plugin, `curl` and `jq`)._ ```bash #!/usr/bin/env bash # Render every active device's configuration from a NetBox branch into snapshot/configs/. set -euo pipefail : "${NETBOX_URL:?}" "${NETBOX_TOKEN:?}" "${NETBOX_BRANCH:?}" # branch = its schema ID H=(-H "Authorization: Bearer ${NETBOX_TOKEN}" -H "X-NetBox-Branch: ${NETBOX_BRANCH}") mkdir -p snapshot/configs curl -sf "${H[@]}" \ "${NETBOX_URL}/api/dcim/devices/?status=active&brief=true&limit=1000" \ | jq -r '.results[] | "\(.id) \(.name)"' \ | while read -r id name; do curl -sf -X POST "${H[@]}" \ -H "Content-Type: application/json" -H "Accept: text/plain" \ --data '{}' "${NETBOX_URL}/api/dcim/devices/${id}/render-config/" \ > "snapshot/configs/${name}.cfg" done git status --short snapshot/configs ``` **The change record and the guard.** Each pull request carries a `change.yaml` from its author (_example status: YAML parsed_): ```yaml change_id: NW-2026-0142 targets: [spine1, leaf1] # hosts this change is meant to touch must_not_change: [leaf2, border1] # hosts that must keep their config ``` `guard.py` adapts Google's rule of requiring both a positive and a negative list[^evolve-or-die] to what will actually change: the hosts whose rendered configurations differ (derived from the data) must equal the author's `targets`, and none may be in `must_not_change`. A template edit that touches every device fails the pull request. _Example status: executed on 2026-10-18 in a scratch Git repository (Python 3.8.10; PyYAML 5.3.1, again with 6.0.2): `GUARD ok` and exit 0 when the diff matched the record; exit 1 and one `GUARD` line per problem when it did not._ ```python """Blast-radius guard: the rendered diff must match the change record's two lists.""" import pathlib import subprocess import sys import yaml def changed_hosts(base, head): """Hosts whose generated configuration differs between two commits.""" out = subprocess.run( ["git", "diff", "--name-only", base, head, "--", "snapshot/configs"], check=True, capture_output=True, text=True).stdout return {pathlib.Path(p).stem for p in out.splitlines()} def check(base, head, record="change.yaml"): """Return (problems, changed hosts); no problems means the guard passed.""" rec = yaml.safe_load(pathlib.Path(record).read_text()) targets, keep = set(rec["targets"]), set(rec["must_not_change"]) changed = changed_hosts(base, head) problems = [] if changed & keep: problems.append(f"must-not-change hosts would change: {sorted(changed & keep)}") if changed != targets: problems.append(f"diff changes {sorted(changed)}; " f"record declares {sorted(targets)}") return problems, changed if __name__ == "__main__": found, _ = check(sys.argv[1], sys.argv[2]) print("\n".join(f"GUARD {p}" for p in found) or "GUARD ok") sys.exit(1 if found else 0) ``` **The pipeline.** Pull requests are verified on a GitHub-hosted Linux runner without device credentials; the Batfish service container is reached on `localhost`[^gh-services]. Only a push to `main` deploys, from a self-hosted runner in the management network, and four settings make that boundary real: - **Environment secrets.** `NET_USER` and `NET_PASSWORD` belong to the `production` environment, not the repository; environment secrets are "only available to workflow jobs that reference the environment", after approval[^gh-env]. - **Protection rules.** Required reviewers, "prevent self-review", a `main`-only deployment branch rule, administrator bypass disallowed (allowed by default)[^gh-env]. - **A restricted runner group.** Limit it to this repository and to `network-change.yml` at `main` ("Selected workflows")[^gh-runner-groups]; otherwise any workflow on any branch can run code on a runner that reaches every device. Prefer just-in-time runners, which "perform at most one job"[^gh-secure]. - **The plan.** On Free, Pro and Team, required reviewers and the bypass setting work only for public repositories[^gh-env], and workflow-restricted runner groups are documented for Enterprise Cloud[^gh-runner-groups]. A private repository needs GitHub Enterprise, or a deployer outside Actions with its own approval check. _Example status: YAML parsed; not executed. Action SHAs are those of the v7.0.1 and v7.0.0 tags (as of October 2026)._ ```yaml name: network-change on: pull_request: paths: ["snapshot/**", "checks/**", "validate/**", "change.yaml"] push: branches: [main] paths: ["snapshot/**", "change.yaml"] permissions: contents: read jobs: verify: runs-on: ubuntu-latest services: batfish: image: batfish/allinone ports: ["9996:9996"] env: BASE: ${{ github.event.pull_request.base.sha || github.event.before }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.12" - run: pip install --require-hashes -r requirements-ci.txt - run: python guard.py "$BASE" HEAD - run: git worktree add reference "$BASE" - run: > python checks/check_change.py --reference reference/snapshot --candidate snapshot deploy-wave-1: if: github.event_name == 'push' needs: verify runs-on: group: network-deploy environment: production concurrency: network-production env: BASE: ${{ github.event.before }} CHANGE_ID: ${{ github.sha }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 - run: python3 -m venv .venv - run: .venv/bin/pip install --require-hashes -r requirements-deploy.txt - run: .venv/bin/python deploy.py 1 env: NET_USER: ${{ secrets.NET_USER }} NET_PASSWORD: ${{ secrets.NET_PASSWORD }} ``` Later waves copy `deploy-wave-1` and `need` the previous wave; `concurrency` prevents parallel deployments. Actions are pinned to commit SHAs, as GitHub recommends[^gh-secure], and packages install only with matching hashes[^pip-secure]. `deploy.py` repeats the guard and touches only changed hosts in its wave. It loads each configuration with a revert timer, polls the post-checks until a deadline shorter than the timer (new BGP sessions take time), and confirms only if they pass; otherwise the timer restores the previous configuration[^nornir-napalm][^napalm-api]. _Example status: Python syntax-checked; not executed (requires devices whose NAPALM driver supports commit confirm)._ ```python """Deploy one wave: guard, load with a revert timer, check until a deadline, confirm.""" import os import sys import time from nornir import InitNornir from nornir_napalm.plugins.tasks import ( napalm_configure, napalm_confirm_commit, napalm_validate, ) from guard import check CHECK_SECONDS = 180 # sessions can take minutes to come up POLL_SECONDS = 15 REVERT_SECONDS = CHECK_SECONDS + 120 # the timer always outlasts the checks def deploy(task): with open(f"snapshot/configs/{task.host.name}.cfg") as fh: config = fh.read() # Replace only where templates own the whole config (playbook 1 diff is empty). task.run(task=napalm_configure, configuration=config, replace=task.host.get("full_template_coverage", False), revert_in=REVERT_SECONDS, commit_message=os.environ["CHANGE_ID"]) deadline = time.monotonic() + CHECK_SECONDS while True: report = task.run(task=napalm_validate, src=f"validate/{task.host.name}.yaml") if report.result.get("complies"): break if time.monotonic() > deadline: raise RuntimeError("checks failed: not confirming, device will revert") time.sleep(POLL_SECONDS) task.run(task=napalm_confirm_commit) def main(wave): problems, changed = check(os.environ["BASE"], "HEAD") if problems: sys.exit("guard: " + "; ".join(problems)) nr = InitNornir(config_file="nornir.yaml") hosts = nr.inventory.hosts lost = sorted(h for h in changed if h not in hosts or hosts[h].get("wave") is None) if lost: sys.exit(f"guard: changed hosts without inventory entry or wave: {lost}") nr.inventory.defaults.username = os.environ["NET_USER"] nr.inventory.defaults.password = os.environ["NET_PASSWORD"] targets = nr.filter( filter_func=lambda h: h.name in changed and str(h.get("wave")) == wave) names = sorted(targets.inventory.hosts) if not names: print(f"wave {wave}: no changed hosts") return result = targets.run(task=deploy) if result.failed_hosts: sys.exit(f"wave {wave} failed on {sorted(result.failed_hosts)}; halting") print(f"wave {wave} confirmed on {names}") if __name__ == "__main__": main(sys.argv[1]) ``` The validation file states what must be true afterwards, in NAPALM's getter format[^napalm-validate] (_example status: YAML parsed; not executed_): ```yaml - get_bgp_neighbors: global: peers: 10.0.1.0: is_up: true - get_interfaces: Ethernet1: is_up: true ``` ### Hands-on labs Lab 1 was **executed**. Labs 2 and 3 were **not** (they need Docker, not run on the author's shared production host); their commands are syntax-checked against the cited documentation. #### Lab 1 (beginner): from source of truth to checked configuration **Objective.** Generate configurations from data, reject bad data, and block a change that touches more devices than intended. **Prerequisites.** Python 3.11 or later (3.10 and older are past end of life[^python-versions]), `pip install jinja2 pyyaml`. **Cost and risk.** None; no device is touched. _Example status: executed on 2026-10-18 with Python 3.8.10, under Jinja2 2.10.1 with PyYAML 5.3.1 and again under Jinja2 3.1.6 with PyYAML 6.0.2 (identical outputs and files); not run on Python 3.11 or later, which the author's host lacks._ `sot.yaml`, the source of truth for a three-node fabric: ```yaml devices: spine1: asn: 65100 loopback: 10.255.0.1/32 links: - {interface: Ethernet1, ip: 10.0.1.0/31, peer: leaf1} - {interface: Ethernet2, ip: 10.0.1.2/31, peer: leaf2} leaf1: asn: 65101 loopback: 10.255.0.11/32 links: - {interface: Ethernet1, ip: 10.0.1.1/31, peer: spine1} leaf2: asn: 65102 loopback: 10.255.0.12/32 links: - {interface: Ethernet1, ip: 10.0.1.3/31, peer: spine1} ``` `device.j2`, an EOS-style template (Jinja2): ```text hostname {{ name }} ! interface Loopback0 ip address {{ dev.loopback }} {% for l in dev.links %} ! interface {{ l.interface }} description to {{ l.peer }} no switchport ip address {{ l.ip }} {% endfor %} ! router bgp {{ dev.asn }} router-id {{ dev.loopback.split('/')[0] }} {% for n in neighbors %} neighbor {{ n.ip }} remote-as {{ n.asn }} neighbor {{ n.ip }} description {{ n.name }} {% endfor %} network {{ dev.loopback }} ``` `render.py` checks the data (private ASNs, unique IPs, matching /31 ends), derives BGP neighbours, writes only changed files, and refuses to write if more devices would change than `--max-changed` allows: ```python """Render device configs from a YAML source of truth, after pre-flight checks.""" import argparse import ipaddress import pathlib import sys import jinja2 import yaml def check(sot): """Return (errors, neighbours) for every device in the source of truth.""" errors, seen_ips, nbrs = [], {}, {} devs = sot["devices"] for name, dev in devs.items(): nbrs[name] = [] if not 64512 <= dev["asn"] <= 65534: errors.append(f"{name}: ASN {dev['asn']} is not a private 2-byte ASN") for link in dev["links"]: ip = ipaddress.ip_interface(link["ip"]) if str(ip.ip) in seen_ips: errors.append(f"{name}: {ip.ip} already used on {seen_ips[str(ip.ip)]}") seen_ips[str(ip.ip)] = name peer = devs.get(link["peer"]) if peer is None: errors.append(f"{name}: peer {link['peer']} " "is not in the source of truth") continue back = [p for p in peer["links"] if p["peer"] == name] if len(back) != 1: errors.append(f"{name}: {link['peer']} has no single link back") continue pip = ipaddress.ip_interface(back[0]["ip"]) if pip.network != ip.network or pip.ip == ip.ip: errors.append(f"{name}: {ip} and {link['peer']} {pip} are not one /31") continue nbrs[name].append({"ip": str(pip.ip), "asn": peer["asn"], "name": link["peer"]}) return errors, nbrs def main(): ap = argparse.ArgumentParser() ap.add_argument("--sot", default="sot.yaml") ap.add_argument("--out", default="out") ap.add_argument("--max-changed", type=int, default=None, help="refuse to write if more devices would change than this") args = ap.parse_args() sot = yaml.safe_load(pathlib.Path(args.sot).read_text()) errors, nbrs = check(sot) if errors: print("\n".join(f"ERROR {e}" for e in errors)) return 1 env = jinja2.Environment(loader=jinja2.FileSystemLoader("."), undefined=jinja2.StrictUndefined, trim_blocks=True, lstrip_blocks=True, keep_trailing_newline=True) tpl = env.get_template("device.j2") out = pathlib.Path(args.out) rendered = {n: tpl.render(name=n, dev=d, neighbors=nbrs[n]) for n, d in sot["devices"].items()} changed = [n for n, txt in rendered.items() if not (out / f"{n}.cfg").exists() or (out / f"{n}.cfg").read_text() != txt] if args.max_changed is not None and len(changed) > args.max_changed: print(f"BLOCKED {len(changed)} devices would change " f"(limit {args.max_changed}): {', '.join(changed)}") return 2 out.mkdir(exist_ok=True) for n in changed: (out / f"{n}.cfg").write_text(rendered[n]) sessions = sum(len(v) for v in nbrs.values()) // 2 print(f"OK {len(rendered)} devices, {sessions} BGP sessions, " f"{len(changed)} changed: {', '.join(changed) or 'none'}") return 0 if __name__ == "__main__": sys.exit(main()) ``` **Steps.** (1) `python3 render.py`. (2) Run it again. (3) Change the spine's `asn` to 65200 in a copy (`sot-asn.yaml`) and run `python3 render.py --sot sot-asn.yaml --max-changed 1`. (4) In another copy (`sot-bad.yaml`), give leaf2's link the address `10.0.1.1/31` and run `python3 render.py --sot sot-bad.yaml`. **Expected output** (from the run): ```text OK 3 devices, 2 BGP sessions, 3 changed: spine1, leaf1, leaf2 OK 3 devices, 2 BGP sessions, 0 changed: none BLOCKED 3 devices would change (limit 1): spine1, leaf1, leaf2 ERROR spine1: 10.0.1.2/31 and leaf2 10.0.1.1/31 are not one /31 ERROR leaf2: 10.0.1.1 already used on leaf1 ERROR leaf2: 10.0.1.1/31 and spine1 10.0.1.2/31 are not one /31 ``` **Validation.** `out/leaf1.cfg` contains `neighbor 10.0.1.0 remote-as 65100`. Step 3 shows why the guard matters: one spine ASN touches every peer. Exit codes 0, 0, 2 and 1 let CI gate on the result. **Troubleshooting.** `UndefinedError`: the template uses a field the data lacks (`StrictUndefined` makes that fail loudly). **Teardown.** Delete the folder. #### Lab 2 (intermediate): catch a bad change with Batfish **Objective.** Catch two faults no syntax check sees: a BGP neighbour typo and an access-list reordering (an ordering error, like the policy-term reorder in Cloudflare's outage). **Prerequisites.** Docker; Python 3.11 or later (current pybatfish needs 3.10 or later[^pypi]); `python3 -m pip install --upgrade pybatfish`; Batfish started with `docker run --name batfish -v batfish-data:/data -p 8888:8888 -p 9996:9996 batfish/allinone`[^pybatfish]. **Cost and risk.** Local only. The lab uses Cisco IOS syntax, which Batfish supports[^pybatfish]. Create `snapshots/reference/configs/edge1.cfg`: ```text hostname edge1 ! interface GigabitEthernet0/0 ip address 10.0.12.1 255.255.255.252 ip access-group MGMT-IN in ! interface Loopback0 ip address 192.0.2.1 255.255.255.255 ! ip access-list extended MGMT-IN permit tcp 198.51.100.0 0.0.0.255 host 192.0.2.1 eq 22 deny tcp any host 192.0.2.1 eq 22 permit ip any any ! router bgp 65001 bgp router-id 192.0.2.1 neighbor 10.0.12.2 remote-as 65002 address-family ipv4 network 192.0.2.1 mask 255.255.255.255 neighbor 10.0.12.2 activate exit-address-family ! end ``` and `snapshots/reference/configs/core1.cfg`: ```text hostname core1 ! interface GigabitEthernet0/0 ip address 10.0.12.2 255.255.255.252 ! interface Loopback0 ip address 192.0.2.2 255.255.255.255 ! router bgp 65002 bgp router-id 192.0.2.2 neighbor 10.0.12.1 remote-as 65001 address-family ipv4 network 192.0.2.2 mask 255.255.255.255 neighbor 10.0.12.1 activate exit-address-family ! end ``` Copy `reference` to `candidate`. In the candidate `edge1.cfg`, swap the first two `MGMT-IN` lines, and change `neighbor 10.0.12.2 remote-as 65002` to `remote-as 65020`. Then run `checks/check_change.py`: _Example status: Python syntax-checked; not executed. Question names, parameters and columns are from the pybatfish documentation[^pybatfish]._ ```python """Pre-change checks with Batfish: candidate snapshot against the reference.""" import argparse import sys from pybatfish.client.session import Session from pybatfish.datamodel.flow import HeaderConstraints ap = argparse.ArgumentParser() ap.add_argument("--reference", default="snapshots/reference") ap.add_argument("--candidate", default="snapshots/candidate") ap.add_argument("--host", default="localhost") args = ap.parse_args() bf = Session(host=args.host) bf.set_network("networkops-lab") bf.init_snapshot(args.reference, name="reference", overwrite=True) bf.init_snapshot(args.candidate, name="candidate", overwrite=True) failures = [] ref_issues = len(bf.q.initIssues().answer(snapshot="reference").frame()) new_issues = len(bf.q.initIssues().answer(snapshot="candidate").frame()) if new_issues > ref_issues: failures.append(f"init issues rose from {ref_issues} to {new_issues}") sessions = bf.q.bgpSessionStatus().answer(snapshot="candidate").frame() for _, row in sessions[sessions["Established_Status"] != "ESTABLISHED"].iterrows(): failures.append(f"BGP {row['Node']} -> {row['Remote_IP']}: " f"{row['Established_Status']}") ssh = HeaderConstraints(srcIps="198.51.100.0/24", dstIps="192.0.2.1", applications=["ssh"]) denied = bf.q.searchFilters(headers=ssh, action="deny", filters="MGMT-IN") denied = denied.answer(snapshot="candidate").frame() if len(denied): failures.append(f"MGMT-IN denies admin SSH, e.g. {denied.iloc[0]['Flow']}") changed = bf.q.compareFilters().answer(snapshot="candidate", reference_snapshot="reference").frame() print(f"{len(changed)} filter line pair(s) now treat some flows differently") for f in failures: print("FAIL", f) print("PASS" if not failures else f"{len(failures)} check(s) failed") sys.exit(1 if failures else 0) ``` **Expected output (from the documentation, not observed).** Run with `--candidate snapshots/reference`, the script prints `PASS`. Against the candidate it should report `NOT_COMPATIBLE` sessions for edge1 and core1 (one side misconfigured), an `MGMT-IN denies admin SSH` failure with an example flow, at least one changed filter line pair, and exit code 1. **Validation.** Undo one fault at a time; only the other should remain. **Troubleshooting.** Connection refused on 9996: the container is not running (`docker logs batfish`). Rows in `initIssues` mean Batfish did not fully understand a line; read them before trusting other answers. **Teardown.** `docker rm -f batfish && docker volume rm batfish-data`. #### Lab 3 (advanced): model-driven change with commit confirmed **Objective.** Use gNMI against two emulated Nokia SR Linux routers: read state, stream telemetry, and make a change that reverts unless confirmed. **Prerequisites.** A Linux VM with Docker and sudo; containerlab (`curl -sL https://containerlab.dev/setup | sudo -E bash -s "all"`)[^clab]; gnmic (`bash -c "$(curl -sL https://get-gnmic.openconfig.net)"`)[^gnmic]. The SR Linux image is public; default credentials `admin` / `NokiaSrl1!`[^clab]. **Cost and risk.** A lab VM only, never a production host; read installer scripts before running them. _Example status: YAML parsed and `bash -n` syntax-checked; not executed. Commit-confirmed support per the SR Linux 25.10 guide[^srl-gnmi]._ `nwops.clab.yml`: ```yaml name: nwops topology: nodes: srl1: kind: nokia_srlinux image: ghcr.io/nokia/srlinux:25.10.5 srl2: kind: nokia_srlinux image: ghcr.io/nokia/srlinux:25.10.5 links: - endpoints: ["srl1:ethernet-1/1", "srl2:ethernet-1/1"] ``` ```bash #!/usr/bin/env bash set -euo pipefail sudo containerlab deploy -t nwops.clab.yml G=(gnmic -a clab-nwops-srl1 --skip-verify -u admin -p 'NokiaSrl1!' -e json_ietf) IF='/interface[name=ethernet-1/1]' # 1. Read: host name and link state "${G[@]}" get --path /system/name/host-name "${G[@]}" get --path "${IF}/oper-state" # 2. Change with a 120-second rollback timer, then read it back "${G[@]}" set --update-path "${IF}/description" --update-value "to srl2" \ --commit-id lab3a --commit-request --rollback-duration 120s "${G[@]}" get --path "${IF}/description" # 3. Do nothing for over two minutes, then read again: the change should be gone sleep 150 "${G[@]}" get --path "${IF}/description" # 4. Repeat; confirm only if the read-back passes (production: check behaviour too) "${G[@]}" set --update-path "${IF}/description" --update-value "to srl2" \ --commit-id lab3b --commit-request --rollback-duration 120s "${G[@]}" get --path "${IF}/description" --values-only | grep 'to srl2' "${G[@]}" set --commit-id lab3b --commit-confirm ``` In a second terminal, stream state changes on srl1. After step 4, disable the interface and enable it again: ```bash gnmic -a clab-nwops-srl1 --skip-verify -u admin -p 'NokiaSrl1!' -e json_ietf \ subscribe --path '/interface[name=ethernet-1/1]/oper-state' --stream-mode on-change ``` ```bash #!/usr/bin/env bash set -euo pipefail G=(gnmic -a clab-nwops-srl1 --skip-verify -u admin -p 'NokiaSrl1!' -e json_ietf) P='/interface[name=ethernet-1/1]/admin-state' "${G[@]}" set --update-path "$P" --update-value disable # the stream should show down sleep 10 "${G[@]}" set --update-path "$P" --update-value enable # and then up again ``` **Expected output.** Step 1 returns the node's host name and `up` (containerlab enables interfaces named in `links`[^clab]). Step 2 shows the description and step 3 none; in step 4 the read-back finds it, so the script confirms and it persists (a miss would stop the script before the confirm). The subscription prints the current value first, then one update per change, as gNMI requires for ON_CHANGE[^gnmi]: `down` after the disable, `up` after the enable (SR Linux's `admin-state` takes `enable` and `disable`[^srl-gnmi]). **Validation.** During a running timer, send a plain `set`: the target must refuse it with `FAILED_PRECONDITION`[^gnmi-cc], so nobody can stack changes onto an unconfirmed one. A commit cannot be amended once issued, so send the whole change in one request[^gnmi-cc]. **Troubleshooting.** TLS errors: `--skip-verify` is for labs only. `unknown host`: use the address from `containerlab inspect`. **Teardown.** `sudo containerlab destroy -t nwops.clab.yml --cleanup` removes the containers and the lab directory[^clab]. ### Use cases and scenarios **A. Edge routing-policy change (real case: Cloudflare, June 2022).** *Context:* standardising BGP communities on site-local prefixes, with a dry run, a stepped rollout and peer review; yet the steps "weren't small enough"[^cloudflare-2022]. *NetworkOps adds:* a differential check of advertised routes, not just a text diff; a first wave that includes an MCP site; a commit-confirmed timer, which Cloudflare listed as a follow-up. *Success:* no unexpected withdrawals in the differential report. *Pitfall:* trusting reviewers to spot reordering in a diff. **B. Core upgrade under a risk score (real case: Rogers, July 2022).** *Context:* phase six of seven; a filter removed during clean-up, and core routers that normally received about 10,000 routes got "over 900,000" from one distribution router[^crtc-rogers]. *NetworkOps adds:* an invariant ("every distribution-to-core session has an inbound filter and a prefix limit"); lab or model tests required by change *type*, whatever the track record. *Success:* the invariant fails the pull request. *Pitfall:* treating a risk score as evidence ([governance](#operating-model-and-governance)). **C. Branch rollout from the source of truth (illustrative scenario).** *Context:* 120 new branch sites in six months, each with two switches and an SD-WAN edge. *Method:* stewards create the site in the SoT, prefixes are allocated, configs rendered and verified, devices boot with zero-touch provisioning, waves follow opening dates. *Trade-off:* weeks of template work, repaid only if sites are alike. *Success:* over 95 % first-time-right after ten sites. *Pitfall:* local hand-made exceptions, which become drift. **D. Firewall and ACL requests as standard changes (illustrative scenario).** *Context:* 40 rule requests a week, each reviewed by hand. *Method:* an intake form records source, destination, port and owner as SoT policy objects; CI checks "must allow" and "must deny" flows with `searchFilters` and reports changed lines with `compareFilters`[^pybatfish]; additions matching an approved pattern are auto-approved. *Success:* lead time falls from days to hours, fail rate flat. *Pitfall:* rules without an owner or expiry. ## Measure, secure and pay ### Metrics and KPIs Measure outcomes of change, not activity. The first two adapt DORA's definitions to network changes[^dora]; the rest are this Atlas's proposals. | KPI | Formula | Data source | Pitfall | |---|---|---|---| | Change fail rate | Changes needing rollback, hotfix or an incident within 24 h ÷ changes deployed | Change records, pipeline logs | Counting only pipeline changes hides manual ones | | Failed change recovery time | Median time from failed check or alert to service restored | Incident timeline | Confirmed-commit reverts are fast; count them too | | Pre-deployment catch rate | Defects stopped by verification ÷ (stopped + found after deployment) | PR checks, post-incident reviews | False-alarm blocks inflate it; track overrides | | Pipeline coverage | Changes through the pipeline ÷ all changes, including emergency and manual | AAA accounting vs change records | 100 % is not the goal; unrecorded changes are the problem | | Drift rate and age | Devices where running ≠ intended ÷ devices checked; median days drifted | Scheduled diff | A noisy diff (timestamps, counters) gets ignored | | SoT completeness | Devices with every required field valid ÷ devices in the SoT | SoT validation report | Completeness without accuracy; sample-audit against devices | | Standard-change lead time | Median request → verified in production | Ticket and pipeline timestamps | Speed without a fail-rate guard | **Worked calculation (illustrative).** One quarter: 420 changes, 380 through the pipeline and 40 by hand (12 of them emergencies). Nine needed rollback: two pipeline changes and seven manual ones. - Pipeline coverage = 380 ÷ 420 = **90.5 %**. - Change fail rate = 9 ÷ 420 = **2.1 %** overall, but **0.5 %** (2 ÷ 380) through the pipeline and **17.5 %** (7 ÷ 40) by hand. The split tells you where to invest; the blended figure hides it. - Verification blocked 31 pull requests with real defects, and 2 defects escaped: catch rate = 31 ÷ 33 = **93.9 %**. - 14 of 600 devices drifted: drift rate = **2.3 %**. Review the oldest first. **Review cadence.** Weekly: fail rate and escapes. Monthly: drift and SoT completeness. Quarterly: coverage, lead time and the [maturity](#maturity-model) self-assessment. "Number of playbooks" is a vanity metric unless paired with fail rate. ### Quality, security and compliance - **Credentials.** Automation has its own AAA accounts, separate for read (telemetry, drift) and write, so accounting logs show who changed what. Rotate them. - **Runner trust.** Self-hosted runners are "not run in an isolated container, even if they use environments"[^gh-env], so restrict the deploy runner as [the pipeline](#reference-implementation) describes. - **Source-of-truth integrity.** Stewards write; tokens are v2, tied to users with minimal object permissions, IP-restricted and write-disabled wherever GET suffices[^netbox-api]. Keep NetBox patched. - **Management plane.** SSH for NETCONF (mandatory to implement; TLS is an option)[^rfc6241][^rfc7589], TLS for gNMI[^gnmi], HTTPS for RESTCONF[^rfc8040]. Verify host keys and certificates; `--skip-verify` is for labs only. - **Audit evidence.** Each change record links the pull request, verification report, approval, deployment log and post-checks, matching NIST SP 800-128's change-control and monitoring phases[^sp800-128]. Sector rules differ; ask your compliance team. ### Cost, capacity and sustainability - **Cost drivers.** People's time dominates: data modelling, whole-config templates and check upkeep. Tooling is mostly open source; expect CI minutes, a SoT server (NetBox needs PostgreSQL 15+[^netbox-intro]), lab hosts, and vendor accounts for some images (Arista cEOS requires registration[^clab]). - **Unit economics.** Track engineer-hours per change by class. Automation pays where a class is frequent and alike (VLANs, peers, ACL entries), rarely for one-off redesigns. - **Control-plane capacity as an invariant.** Prefix limits, route-table headroom and CPU protection belong in verification. Rogers' core routers "were not configured with such overload protection mechanisms"[^crtc-rogers]. - **Sustainability.** Minor here: tear down emulation labs after use (`containerlab destroy`), and right-size the CI runners. ### Assurance of the operating model Watch the system that makes changes. **Signals:** fail rate by path; share of changes with complete evidence; drift exceptions past expiry; break-glass and bypass events; age of the last rollback drill; verification and guard overrides. **Drill quarterly:** push a known-bad change (for example Lab 2's ACL swap) through the real pipeline against a lab or canary target. Confirm that verification blocks it, and that an unconfirmed commit reverts within its timer. ## When things go wrong ### Common failures and troubleshooting | Symptom | Likely cause | Diagnosis | Remediation | Prevention | |---|---|---|---|---| | Pipeline green, traffic lost after deploy | Missing invariant or model blind spot | Differential routes and flows, reference vs production; `initIssues` | Roll back; add the invariant | One new check per escaped defect | | Commit reverted although the change was fine | Confirmation path crossed the changed link; sessions slower than the check deadline; job timed out | Device commit log; check and job durations vs timer | Re-deploy; confirm out of band or via a persistent session[^rfc6241] | Poll checks to a deadline; derive the timer from it; separate management path | | Someone's pending confirmed commit became permanent | Another tool edited the device; with RESTCONF, "any new commit will act as the confirming commit"[^rfc8040] | AAA accounting for concurrent sessions | Restore the reference config | One writer per device; NETCONF ``, so RESTCONF edits fail with 409[^rfc8040]; gNMI refuses plain Sets during a timer[^gnmi-cc] | | Device rejects rendered config | Syntax differs by platform or version | Device error; compare versions | Fix the template branch | Emulate the exact version in CI | | Drift returns after each deploy | A second writer (controller, script, person) | Accounting logs: who wrote it | Remove or integrate the writer | Single source of change per domain | | Replace removed management access | Template did not cover management config | Out-of-band console | Let the timer revert | Templates own the whole config; never use `overridden` without coverage[^ansible-resource] | | Change hit the wrong devices | Filter, wildcard or template error | Rendered diff vs change record | Stop waves; roll back extras | Guard with target and must-not-change lists[^evolve-or-die] | ### Worked runbook: a wave withdraws prefixes **Illustrative scenario**, modelled on the failure class in Cloudflare's report[^cloudflare-2022]. Wave 2 of a routing-policy change reaches four edge routers at 10:02. At 10:04 reachability probes for three customer prefixes fail. 1. **Stop the line (minute 0–2).** Halt the pipeline so no further wave starts. Do *not* confirm wave 2; its timer runs. 2. **Name one incident lead (minute 1).** Only the lead orders reverts, so nobody reverts a revert ([IncidentOps](../incidentops/)). 3. **Let the timer work, or cancel it (minute 2–7).** Unconfirmed devices restore their previous configuration themselves; with gNMI the lead can cancel by commit ID[^gnmi-cc]. If confirmed wave 1 contributes, roll it back from last-known-good configs in Git. 4. **Verify recovery (minute 7–15).** Probes green, advertised routes equal the reference snapshot, sessions established. 5. **Preserve evidence.** Commit IDs, pipeline and device logs, the verification report that passed. 6. **Fix the system, not the person (days 1–5).** Add the missing invariant ("these prefixes are advertised to every transit peer"), put a router of the new design in wave 1, and record the drill. **Exit criteria.** Service restored, reference and production agree, and the new check fails on the bad change in CI. ### Anti-patterns and myths - **Scripts without a source of truth.** Fast at first; then every script carries its own copy of the network. - **Discovery writes the SoT.** Today's mistakes become tomorrow's intent; NetBox discourages it[^netbox-intro]. - **A bypass everyone uses.** If the emergency path is easier than the pipeline, it becomes the normal path; measure bypasses. - **Big-bang replace.** The whole estate on full-config replace before templates cover everything. - **"Green means safe."** Models miss what they cannot parse, and labs miss scale; staging and timers cover the rest. - **Myth: intent-based networking removes configuration work.** RFC 9315 describes fulfilment *and* assurance, and intent drift to watch[^rfc9315]. Someone still models the intent and checks the outcome. - **Myth: you need software developers.** You need network engineers who read Python and YAML, and a few who write it well. ## Deciding and adopting ### When to use and when not to use | Use it when… | Hold back when… | |---|---| | Changes of the same kind recur weekly (VLANs, peers, ACL entries, sites) | The estate is small and changes rare; backups, peer review and a written procedure may do | | Outages or near-misses trace back to changes | Every device is unique; templates would be as long as the configs | | Several engineers change the same devices | Nobody will own the data; an unowned SoT is worse than none | | You need audit evidence for each change | Platforms have neither commit confirm nor a tested rollback, or will soon be replaced; fix backups and rollback first (missing APIs alone are no reason: template the CLI) | | You are about to grow fast (new sites, fabric expansion) | A vendor controller already runs the domain well; feed it from the SoT | **Hidden responsibilities.** Templates follow every OS upgrade, the verification suite stays current, and the SoT is a production service with backups and access reviews. ### Decision framework ::: figure Should you build a NetworkOps pipeline, and which kind? ```mermaid %%{init: {"flowchart": {"wrappingWidth": 150, "nodeSpacing": 14, "rankSpacing": 24, "padding": 8}}}%% flowchart TB classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 Q1["Are changes frequent, or has a change caused an outage?"]:::decision N1["Not yet: config backups, peer review, written rollback"]:::neutral Q2["Will someone own each data domain?"]:::decision N2["Fix ownership first; a SoT without owners decays"]:::risk Q3["Does a vendor controller already run this domain well?"]:::decision C1["Feed the controller from the SoT; add verification evidence"]:::core Q4["Do devices offer NETCONF or gNMI with confirmed commit?"]:::decision M1["Model-driven pipeline with confirmed commits"]:::core M2["CLI templating via NAPALM or Ansible; small waves; tested rollback"]:::core Q1 -->|"yes"| Q2 Q2 -->|"yes"| Q3 Q3 -->|"no"| Q4 Q4 -->|"yes"| M1 Q1 -->|"no"| N1 Q2 -->|"no"| N2 Q3 -->|"yes"| C1 Q4 -->|"no"| M2 ``` Text equivalent: Are changes frequent, or has a change caused an outage? If not, rely on configuration backups, peer review and a written rollback for now. If so, will someone own each data domain? If not, fix ownership first: a source of truth without owners decays. If owners exist and a vendor controller already runs the domain well, feed it from the source of truth and add verification evidence. Otherwise, if devices offer NETCONF or gNMI with confirmed commit, build a model-driven pipeline with confirmed commits; if not, template the CLI through NAPALM or Ansible, with small waves and a tested rollback. ::: **Assumptions.** "Frequent" means, in this Atlas's suggestion, a change class that recurs at least weekly. Mixed estates choose per platform; one pipeline can serve both. ### Maturity model **This Atlas's model**, not an industry standard. Move up when the exit criterion has held for a quarter. ::: figure NetworkOps maturity progression (Atlas model) ```mermaid %%{init: {"flowchart": {"wrappingWidth": 300, "rankSpacing": 18, "padding": 8}}}%% flowchart TB classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029 classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029 classDef core fill:#e8f3f2,stroke:#16756f,color:#122029 classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029 L0["0 Manual: CLI by hand, configs in personal files"]:::risk E0["Exit: nightly config backups in Git for every device"]:::decision L1["1 Scripted: read-only automation, backups, inventories"]:::neutral E1["Exit: a SoT with named owners covers the in-scope estate"]:::decision L2["2 Generated: configs rendered from the SoT; drift reported"]:::neutral E2["Exit: drift under 5 % and every change via pull request"]:::decision L3["3 Verified: CI checks, staged waves, confirmed commits, evidence"]:::core E3["Exit: fail rate tracked by path; quarterly bad-change drill passes"]:::decision L4["4 Assured: telemetry-based checks, narrow closed-loop fixes"]:::core L0 --> E0 --> L1 --> E1 --> L2 --> E2 --> L3 --> E3 --> L4 ``` Text equivalent: Five levels with exit gates. Level 0, manual (CLI by hand, configs in personal files): exit when every device has nightly config backups in Git. Level 1, scripted (read-only automation, backups, inventories): exit when an owned source of truth covers the in-scope estate. Level 2, generated (configs rendered from it, drift reported): exit when drift is under 5 % and every change goes through a pull request. Level 3, verified (CI checks, staged waves, confirmed commits, evidence): exit when fail rate is tracked by path and the quarterly bad-change drill passes. Level 4, assured: telemetry-based checks, narrow closed-loop fixes. ::: ### Adoption roadmap: 30/60/90 days For a team at level 0–1 starting with one domain, such as data-centre leaf switches. | Window | Actions | Milestone | |---|---|---| | Days 1–30 | Nightly backups to Git; one domain, one change class; stewards named; NetBox up; one reviewed import | Rendered-vs-running diff for the domain | | Days 31–60 | Whole-config templates; Lab 1-style checks and the guard in CI; Batfish snapshot of production; separate CI and deploy credentials | First change class shipped by pull request, verified, pushed by hand | | Days 61–90 | Deploy job with waves, confirmed commits, post-checks; drift report; first bad-change drill; [KPIs](#metrics-and-kpis) | Standard change approved automatically; fail rate measured by path | **Quick wins:** backups in Git; a read-only "what would change" report. **Dependencies:** out-of-band access to every device in scope; AAA accounts for automation. ## Toolkit ### Checklists **Before any change (author)** - [ ] A pull request with the data change (or its SoT branch), not a device session - [ ] Rendered diff reviewed; the guard passed against `targets` and `must_not_change` - [ ] Verification passed: no new parse issues, invariants hold, differences explained - [ ] First wave contains a device of every affected design - [ ] Confirmation path independent of the change - [ ] Rollback source named (last-known-good commit); incident lead on call **Pipeline readiness (automation engineer)** - [ ] No device credentials in CI; device secrets in the `production` environment, not the repository - [ ] Runner group limited to the deploy workflow at `main`; just-in-time runners where possible - [ ] Plan allows required reviewers on a private repository; admin bypass off; self-review prevented - [ ] Actions pinned to commit SHAs; packages installed with hashes - [ ] Revert timer derived from the check deadline; replace only with full template coverage - [ ] One deployment at a time per domain (`concurrency` or lock) - [ ] SoT token scoped, IP-restricted, rotated; out-of-band access tested this quarter **Quarterly governance audit** - [ ] Bad-change drill run and passed - [ ] Drift exceptions past expiry closed - [ ] Break-glass events reconciled into the SoT - [ ] Tool versions pinned and upgraded in CI first ### Templates **Network change record** (one per pull request): ```text Change ID: NW-2026-____ Class: standard | normal | emergency Summary: what changes, in one sentence Blast radius: change.yaml targets + must_not_change; guard result Waves: wave 1 (canary, representative): ...; wave 2: ... Verification: link to CI report (init issues, invariants, differential) Timer: confirmed commit ___ s; confirmation path: ___ Post-checks: sessions, routes, probes expected after each wave Rollback: last-known-good commit ___; incident lead ___ Approvals: reviewer ___; change manager ___ (normal/emergency) Evidence (after): deploy log, post-check results, drift diff = empty ``` **Data ownership charter** (one per data domain): ```text Domain: IPAM | circuits | BGP policy | VLANs | ... Steward (accountable): name / team Writers: who may write, through which path (UI, API, pipeline) Required fields: list; validation rule for each Quality target: completeness >= __ %; monthly sample audit of __ devices Discovery data: allowed only as a reconciliation report, never written directly Review: monthly with the network lead ``` **Architecture decision record: merge or replace for platform X** ```text Context: platform, OS versions, template coverage %, commit-confirm support Options: merge | replace | replace for some sections only Decision: ... Consequences: drift handling, rollback method, lock-out risk and mitigation Review date: ... ``` ### Deep comparisons | Approach | Best for | Strength | Weakness | |---|---|---|---| | CLI templating (Ansible, Nornir + NAPALM or Netmiko) | Mixed and legacy estates | Works with what you have | Ordering and parsing fragility; rollback varies by platform[^napalm-support] | | Model-driven (NETCONF, gNMI with YANG/OpenConfig) | Modern platforms | Transactions, validation, streaming[^rfc6241][^gnmi] | Model coverage gaps; steeper learning | | Vendor controller or intent platform | One vendor's fabric or SD-WAN | Integrated workflows and assurance | Lock-in; SoT and verification still needed upstream | | Pull-based reconciliation ([GitOps](../gitops/) style) | Small, homogeneous estates with strong checks | Continuous drift correction | A bad commit reaches everything unless staged | ### FAQ **Is NetworkOps the same as NetDevOps?** As this Atlas uses it, yes; "NetDevOps" and "network automation" are the common names. **Do I need NETCONF or gNMI to start?** No. CLI templating with good verification and small waves is a valid start; model-driven interfaces add transactions and confirmed commits. **NetBox or YAML files in Git?** YAML suffices for tens of devices and one team. Move to NetBox or Nautobot when several teams edit data, you need IP allocation, or other systems consume it. **Does Batfish replace a lab?** No. It analyses the whole network but only supported syntax[^pybatfish]; a lab runs real software at small scale. They catch different faults. **Can the network fix itself?** For narrow, reversible actions with clear signals (draining a flapping link), at maturity level 4; broad automatic changes are how automation causes large failures. **What if a change must happen during an outage?** Emergency class: break-glass, logged, reconciled into the SoT within a working day, reviewed. ## Going further ### Advanced topics and emerging direction - **Established now:** model-driven configuration with transactions, streaming telemetry, Batfish analysis, container-based emulation, staged rollout with confirmed commits. - **Maturing:** intent assurance (continuous checks for intent drift)[^rfc9315], and emulated twins of production for every change; the Rogers assessment asks for test scenarios that "replicate the production network"[^crtc-rogers]. - **Research:** Google's paper names specifying intent, collecting the network's "ground truth" and reconciling the two as an underexplored area of management-plane design[^evolve-or-die]. - **Vendor marketing to read critically:** "self-driving" and AI-run networks. Ask which actions run without a human, what verifies them first, and how they roll back. ### Learning path | Stage | Do this | Proof of progress | |---|---|---| | Prerequisites | Routing, switching, BGP basics ([NetOps](../netops/)); Git; YAML | You can explain a BGP session state | | Day 1 | Lab 1; read RFC 3535 §3 (two pages)[^rfc3535] | Lab 1 outputs reproduced | | Week 1 | Lab 2; write one invariant for your own network in Batfish | A check that fails on a deliberate mistake | | Month 1 | Lab 3; NetBox with one site; render a config; a Nornir dry run against a lab | A pull request that renders, verifies and deploys to a lab | | Advanced | YANG and OpenConfig models; gNMI streaming; pipeline security | You can defend your rollback design in a review | | Portfolio | A public repository: SoT, templates, checks, lab topology, CI | Someone else can run it from the README | ## Wrap-up ### Executive recap and cheat sheet - **Decide:** one domain and one recurring change class first; data stewards named; merge or replace per platform. - **Build:** SoT → generated configs in Git → verification (model, lab) → staged waves → confirmed commit → state checks. - **Guard:** no device secrets in CI; a restricted deploy runner; one writer per device; the rendered diff checked against two lists; an out-of-band path the change cannot break. - **Measure:** change fail rate split by path, pre-deployment catch rate, drift rate and age, SoT completeness. - **Watch out:** discovery writing the SoT, bypasses becoming normal, risk scores that learn from success, green meaning safe. - **Next action:** nightly config backups in Git this week; everything else builds on them. ### A judgement call: replace the whole configuration or merge changes? **Situation (illustrative).** 300 devices: 200 EOS and Junos devices whose NAPALM drivers support replace and commit confirm, and 100 older ones. Templates cover about 70 % of each configuration; the rest is local history. The automation lead wants full replace (A) to eliminate drift; a senior engineer prefers merge (B) because it cannot delete what templates do not know. **Reasoning.** (1) *Worst failure:* under A, the uncovered 30 % is deleted on first deploy, possibly including management access (Ansible's warning about `overridden`[^ansible-resource]). Under B, drift and stale config persist. (2) *Does the timer rescue A?* Partly: a confirmed commit reverts a lock-out[^rfc6241], but only where supported[^napalm-support] and only if confirmation waits for evidence. (3) *Cost of A:* full template coverage per platform (assumed: weeks each). (4) *What would change the answer:* full coverage and commit confirm everywhere make A clearly better. **Decision: B now, A per platform as coverage reaches 100 %.** Run replace as a nightly *dry run* (`compare_config`); its diff lists the lines templates still miss. **Uncertainty:** how long the older devices stay; if replacement is near, keep them on merge. ### Learning objectives and mastery check After this page you can explain the label and its boundary with NetOps; design a source of truth with owners; choose merge or replace; build a verification gate; roll out in waves with confirmed commits; and measure change outcomes. ::: answer 1. A colleague says "NetworkOps is our NOC team's new name". What is accurate and what is not? The label is informal and overlaps with NetOps, NetDevOps and network automation[^redhat-netops][^cisco-netdevops]. In this Atlas the NOC's work is [NetOps](../netops/) and NetworkOps is change engineering, an editorial boundary, not a standard. ::: ::: answer 2. Why does NetBox discourage importing live network state, and what do you do with discovered data instead? The SoT represents desired state; importing live state records mistakes as intent[^netbox-intro]. Discovery produces a reconciliation report; stewards then fix the data or the device. ::: ::: answer 3. A pipeline deploys and then confirms automatically in the same job, with no checks in between. What is lost? The value of the timer: it helps only if confirmation depends on post-change evidence gathered over a path the change cannot break. Otherwise a lock-out is confirmed too. ::: ::: answer 4. Cloudflare's change had a dry run, peer review and a stepped rollout. Name two controls that would have changed the outcome. A stagger that reached an MCP site early, and an automated commit-confirm rollback; Cloudflare's follow-ups cover both[^cloudflare-2022]. A differential check of advertised prefixes could also have shown the withdrawals before deployment, if the model parsed that policy. ::: ::: answer 5. In a quarter, 6 of 300 pipeline changes and 6 of 30 manual changes failed. What is the fail rate, and what does it suggest? Overall 12 ÷ 330 = 3.6 %; pipeline 2 %, manual 20 %. The blend hides where risk lives: ask why 30 changes bypassed the pipeline before tuning it. ::: ::: answer 6. During a gNMI commit-confirmed window, a teammate's script sends a normal Set. What happens, and why is that good? The target returns FAILED_PRECONDITION[^gnmi-cc], so unreviewed changes cannot be stacked onto an unconfirmed one and make the revert ambiguous. ::: ### Conclusion The word "NetworkOps" adds nothing; the practice behind it adds a great deal. Operators asked for it in 2002[^rfc3535]: treat the network as one system described by data, change it through transactions, keep a way back. The three outages on this page all had review or audit steps. What they lacked was verification of meaning, representative first steps, and an exit that did not depend on the network being changed. Build those, measure what fails, and the tooling choice gets easy. Product facts are dated October 2026; re-check them. ## Sources and further reading [^nwops-repo]: [NetworkOps_Platform](https://github.com/E-Conners-Lab/NetworkOps_Platform) — E-Conners-Lab on GitHub, created 2026-02-18. Checked 2026-10-18. [^gh-search]: [GitHub repository search: "networkops" in repository names](https://api.github.com/search/repositories?q=networkops+in:name) — GitHub REST API, 26 results on 2026-10-18. Checked 2026-10-18. [^naf]: [Network Automation Forum](https://networkautomation.forum/) — NAF (AutoCon events). Checked 2026-10-18. [^cisco-netdevops]: [The Anatomy of NetDevOps](https://blogs.cisco.com/developer/anatomy-of-netdevops) — Siming Yuan, Cisco Blogs, 2019-05-22. Checked 2026-10-18. [^redhat-netops]: [What is NetOps?](https://www.redhat.com/en/topics/automation/what-is-netops) — Red Hat, 2023-08-28. Checked 2026-10-18. [^rfc3535]: [RFC 3535: Overview of the 2002 IAB Network Management Workshop](https://www.rfc-editor.org/rfc/rfc3535) — IETF, May 2003. Checked 2026-10-18. [^rfc6241]: [RFC 6241: Network Configuration Protocol (NETCONF)](https://www.rfc-editor.org/rfc/rfc6241) — IETF, June 2011. Checked 2026-10-18. [^rfc7589]: [RFC 7589: Using the NETCONF Protocol over TLS with Mutual X.509 Authentication](https://www.rfc-editor.org/rfc/rfc7589) — IETF, June 2015. Checked 2026-10-18. [^ncclient]: [ncclient](https://github.com/ncclient/ncclient) — ncclient project on GitHub (v0.7.1 released 2026-03-15). Checked 2026-10-18. [^rfc8040]: [RFC 8040: RESTCONF Protocol](https://www.rfc-editor.org/rfc/rfc8040) — IETF, January 2017. Checked 2026-10-18. [^rfc7950]: [RFC 7950: The YANG 1.1 Data Modeling Language](https://www.rfc-editor.org/rfc/rfc7950) — IETF, August 2016. Checked 2026-10-18. [^rfc8342]: [RFC 8342: Network Management Datastore Architecture (NMDA)](https://www.rfc-editor.org/rfc/rfc8342) — IETF, March 2018. Checked 2026-10-18. [^rfc8641]: [RFC 8641: Subscription to YANG Notifications for Datastore Updates](https://www.rfc-editor.org/rfc/rfc8641) — IETF, September 2019. Checked 2026-10-18. [^rfc8572]: [RFC 8572: Secure Zero Touch Provisioning (SZTP)](https://www.rfc-editor.org/rfc/rfc8572) — IETF, April 2019. Checked 2026-10-18. [^rfc9315]: [RFC 9315: Intent-Based Networking — Concepts and Definitions](https://www.rfc-editor.org/rfc/rfc9315) — IRTF NMRG (informational), October 2022. Checked 2026-10-18. [^rfc7426]: [RFC 7426: SDN Layers and Architecture Terminology](https://www.rfc-editor.org/rfc/rfc7426) — IRTF (informational), January 2015. Checked 2026-10-18. [^gnmi]: [gRPC Network Management Interface (gNMI) specification v0.10.0](https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-specification.md) — OpenConfig, 2023-05-25; first revision committed 2016-11-07 ([file history](https://github.com/openconfig/reference/commits/master/rpc/gnmi/gnmi-specification.md)). Checked 2026-10-18. [^gnmi-cc]: [gNMI Commit Confirmed Extension v0.1.0](https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-commit-confirmed.md) — OpenConfig, 2023-09-20. Checked 2026-10-18. [^gnmi-path]: [Schema path encoding conventions for gNMI v0.4.0](https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-path-conventions.md) — OpenConfig, 2017-06-21. Checked 2026-10-18. [^openconfig]: [OpenConfig](https://www.openconfig.net/) — OpenConfig project. Checked 2026-10-18. [^srl-gnmi]: [SR Linux System Management Guide 25.10: gNMI](https://documentation.nokia.com/srlinux/25-10/books/system-mgmt/gnmi.html) — Nokia. Checked 2026-10-18. [^srl-yang]: [SR Linux System Management Guide 25.10: YANG data models](https://documentation.nokia.com/srlinux/25-10/books/system-mgmt/yang-data-models.html) — Nokia. Checked 2026-10-18. [^junos-cc]: [Commit the Configuration](https://www.juniper.net/documentation/us/en/software/junos/cli/topics/topic-map/junos-configuration-commit.html) — Junos OS CLI User Guide, Juniper Networks. Checked 2026-10-18. [^sp800-128]: [NIST SP 800-128: Guide for Security-Focused Configuration Management of Information Systems](https://csrc.nist.gov/pubs/sp/800/128/upd1/final) — NIST, 2011, updated 2019-10-10. Checked 2026-10-18. [^netbox-intro]: [Introduction to NetBox](https://netboxlabs.com/docs/netbox/introduction/) — NetBox Labs documentation. Checked 2026-10-18. [^netbox-render]: [Configuration Rendering](https://netboxlabs.com/docs/netbox/features/configuration-rendering/) — NetBox Labs documentation. Checked 2026-10-18. [^netbox-api]: [REST API: tokens, write restriction, client IP restriction](https://netboxlabs.com/docs/netbox/integrations/rest-api/) — NetBox Labs documentation. Checked 2026-10-18. [^nb-branching]: [NetBox Branching](https://netboxlabs.com/docs/branching/) and [Using the REST API](https://netboxlabs.com/docs/branching/rest-api/) — NetBox Labs documentation (plugin v1.2.2). Checked 2026-10-18. [^netbox-release]: [NetBox v4.7.3 release](https://github.com/netbox-community/netbox/releases) — netbox-community on GitHub, 2026-10-13. Checked 2026-10-18. [^nautobot]: [Nautobot documentation](https://docs.nautobot.com/projects/core/en/stable/) — Nautobot (v3.2.7 released 2026-10-13). Checked 2026-10-18. [^batfish]: [Batfish](https://batfish.org/) — The Batfish open source project. Checked 2026-10-18. [^pybatfish]: [pybatfish documentation: setup, interacting, questions, supported devices](https://pybatfish.readthedocs.io/en/latest/) — The Batfish project. Checked 2026-10-18. [^nornir]: [Nornir documentation 3.6.0](https://nornir.readthedocs.io/en/latest/) — nornir-automation. Checked 2026-10-18. [^nornir-napalm]: [nornir_napalm tasks](https://github.com/nornir-automation/nornir_napalm/tree/master/nornir_napalm/plugins/tasks) — nornir-automation on GitHub (v0.6.0). Checked 2026-10-18. [^napalm-api]: [NAPALM NetworkDriver API](https://napalm.readthedocs.io/en/latest/base.html) — NAPALM project. Checked 2026-10-18. [^napalm-support]: [NAPALM supported devices and configuration support matrix](https://napalm.readthedocs.io/en/latest/support/) — NAPALM project. Checked 2026-10-18. [^napalm-validate]: [Validating deployments](https://napalm.readthedocs.io/en/latest/validate/index.html) — NAPALM project. Checked 2026-10-18. [^ansible-resource]: [Network resource modules](https://docs.ansible.com/ansible/latest/network/user_guide/network_resource_modules.html) — Ansible documentation. Checked 2026-10-18. [^ansible-cli-config]: [ansible.netcommon.cli_config module (collection 8.7.1)](https://docs.ansible.com/ansible/latest/collections/ansible/netcommon/cli_config_module.html) — Ansible documentation. Checked 2026-10-18. [^netmiko]: [Netmiko](https://github.com/ktbyers/netmiko) — Kirk Byers on GitHub. Checked 2026-10-18. [^ntc-templates]: [ntc-templates](https://github.com/networktocode/ntc-templates) — Network to Code on GitHub. Checked 2026-10-18. [^clab]: [containerlab quick start](https://containerlab.dev/quickstart/), [Nokia SR Linux kind](https://containerlab.dev/manual/kinds/srl/), [Arista cEOS kind](https://containerlab.dev/manual/kinds/ceos/), [destroy command](https://containerlab.dev/cmd/destroy/) — containerlab (v0.81.1 released 2026-10-15). Checked 2026-10-18. [^gnmic]: [gnmic](https://gnmic.openconfig.net/) with [set](https://gnmic.openconfig.net/cmd/set/) and [subscribe](https://gnmic.openconfig.net/cmd/subscribe/) — OpenConfig (v0.49.0 released 2026-09-16). Checked 2026-10-18. [^pypi]: PyPI release pages: [nornir](https://pypi.org/project/nornir/), [napalm](https://pypi.org/project/napalm/), [pybatfish](https://pypi.org/project/pybatfish/), [netmiko](https://pypi.org/project/netmiko/), [nornir-netbox](https://pypi.org/project/nornir-netbox/) — Python Package Index. Checked 2026-10-18. [^meta-2021]: [More details about the October 4 outage](https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/) — Santosh Janardhan, Engineering at Meta, 2021-10-05. Checked 2026-10-18. [^cloudflare-2022]: [Cloudflare outage on June 21, 2022](https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/) — Tom Strickx and Jeremy Hartman, Cloudflare, 2022-06-21. Checked 2026-10-18. [^crtc-rogers]: [Assessment of Rogers Networks for Resiliency and Reliability Following the 8 July 2022 Outage – Executive Summary](https://crtc.gc.ca/eng/publications/reports/xona2024.htm) — Xona Partners for the CRTC, July 2024 (read via the [Internet Archive copy of 2025-12-20](https://web.archive.org/web/20251220060847/https://crtc.gc.ca/eng/publications/reports/xona2024.htm); the live page blocked automated access). Checked 2026-10-18. [^evolve-or-die]: [Evolve or Die: High-Availability Design Principles Drawn from Google's Network Infrastructure](https://research.google/pubs/evolve-or-die-high-availability-design-principles-drawn-from-googles-network-infrastructure/) — Govindan, Minei, Kallahalla, Koley and Vahdat, ACM SIGCOMM 2016. Checked 2026-10-18. [^dora]: [DORA's software delivery performance metrics](https://dora.dev/guides/dora-metrics/) — DORA, updated 2026-01-05. Checked 2026-10-18. [^gh-env]: [Deployments and environments](https://docs.github.com/en/actions/reference/workflows-and-actions/deployments-and-environments) — GitHub Docs. Checked 2026-10-18. [^gh-secure]: [Secure use reference](https://docs.github.com/en/actions/reference/security/secure-use) — GitHub Docs. Checked 2026-10-18. [^gh-runner-groups]: [Managing access to self-hosted runners using groups](https://docs.github.com/en/enterprise-cloud@latest/actions/how-tos/manage-runners/self-hosted-runners/manage-access) — GitHub Docs (Enterprise Cloud). Checked 2026-10-18. [^pip-secure]: [Secure installs](https://pip.pypa.io/en/stable/topics/secure-installs/) — pip documentation. Checked 2026-10-18. [^python-versions]: [Status of Python versions](https://devguide.python.org/versions/) — Python Developer's Guide. Checked 2026-10-18. [^gh-services]: [Use Docker service containers](https://docs.github.com/en/actions/tutorials/use-containerized-services/use-docker-service-containers) — GitHub Docs. Checked 2026-10-18. **Changelog.** 2026-10-18: first draft (r0), researched and written by the managing author; Lab 1 executed, Labs 2–3 and the reference implementation syntax-checked only. 2026-10-18 (r1): revised after the independent review: label status, runner and secret settings, a source-of-truth change path, a guard that checks the rendered diff (executed), post-check polling, NETCONF example, controller domains, transport and chronology fixes, pinned actions. 2026-10-18, post-score polish: Branching revert precondition; Lab 2's Cloudflare comparison made precise; branch rendering marked untested; Lab 3 confirms only after a read-back; OpenConfig-or-native and path-reading paragraph; SDN sentence; beginner pointer; figure 3 label moved into its node (no crossing); Lab 1 and the guard re-run with Jinja2 3.1.6 and PyYAML 6.0.2; restatements trimmed.