NetworkOps

NetworkOps, as this Atlas uses the word, is the engineering of network change: describing the network you want as data, generating device configuration from it, proving the change safe before it reaches a router, rolling it out in small waves with an automatic way back, and checking that the network then behaves as intended. Its established names are network automation and NetDevOps.

In one line: NetworkOps turns “log in and type” into “change the data, let the pipeline prove it, deploy with a timer”.

Where to start. Beginners: the mental model and Lab 1. Engineers: the playbooks and the reference implementation. Leaders: the decision framework and the judgement call. Last verified 2026-10-18; tool versions are dated “as of October 2026”.

Three public reports show outages that started with a change:

• Changes are where failures cluster. Of over 100 high-impact failure events in Google’s network, nearly 70 happened while a management operation was in progress on the affected element, though it was not always the cause^[6].
• A harmless-looking diff can be fatal. At Cloudflare in 2022, “a re-ordering of terms caused us to withdraw a critical subset of prefixes” despite a dry run, a stepped rollout and peer review; 19 data centres were offline for up to 75 minutes^[7].
• Process checks fail quietly. At Rogers in 2022, staff removed a policy filter while tidying configuration; core routers were flooded with routes and crashed. The audit missed it, and a risk algorithm rated the step “Low”, so no lab test was required. More than 12 million customers lost service, restored gradually over about 26 hours^[8].

Box-by-box CLI work makes this worse: devices drift apart, review means reading CLI text, and nobody can say what the network should look like. In 2002 operators already asked to configure “the network as a whole rather than individual devices”, to use diff and version control on configurations, and to separate distributing a configuration from activating it^[9].

What it does not solve. Automation spreads a wrong change faster. Google found partial automation “increased availability in general” but could also “cause large failures”^[6]; safety comes from verification, staging and rollback.

• Not the NOC. Monitoring, fault isolation and carrier escalation belong to NetOps. NetworkOps configures the telemetry NetOps consumes and owns the rollback of a failed change.
• Not “we wrote some scripts”. Scripts that push commands without a source of truth, verification or rollback are automation debt. Governance of automation in general is on AutomationOps.
• Not a product. Controllers and “intent-based” platforms implement parts of the practice; intent terms “are often used loosely and inconsistently”^[10]. Buying one does not give you owned data or a verification habit.
• Not GitOps by definition. Network pipelines usually push from a runner; pull-based reconciliation (GitOps) is possible but is not what makes a change safe.
• Not cloud network design (CloudOps), though the method applies there too.

The label has no traceable origin. The practice has a clear technical history:

• 2002–2003: requirements. The IAB workshop asked for configuration separate from state, network-wide transactions and diff-able text^[9].
• 2006–2018: protocols and models. NETCONF (RFC 4741 in 2006, RFC 6241 in 2011) brought candidate datastores and confirmed commits^[11]; YANG (2010; 1.1 in 2016)^[12], RESTCONF (2017)^[13] and the NMDA datastores (2018)^[14] followed. An IRTF document defined SDN as “the capacity to initialize, control, change, and manage network behavior dynamically via open interfaces” (2015)^[15].
• 2015–2017: practitioners’ tools. NetBox was written at DigitalOcean in 2015 and open-sourced in 2016^[16]; Cisco released pyATS publicly in late 2017^[4]; Batfish came from Microsoft Research, UCLA and USC^[17].
• 2016 onwards: streaming and provisioning. Operator-led OpenConfig published the first gNMI specification in 2016 (current v0.10.0, 2023)^[18][19] and a commit-confirmed extension in 2023^[20]; the IETF added YANG-Push^[21] and Secure Zero Touch Provisioning^[22] in 2019.
• 2022–2026: intent and community. RFC 9315 on intent-based networking (2022)^[10]; the Network Automation Forum’s AutoCon events from 2023. NAF admits automation has been discussed “for a long time: 15 years or longer”, yet “it’s just not taking off like we thought it would”^[3].

Mental model: three views of every device. The source of truth says what the network should be, the intended configuration is generated from it, and the operational state is what the device does. NetworkOps lives in the gaps between them.

%%{init: {"flowchart": {"wrappingWidth": 170, "nodeSpacing": 14, "rankSpacing": 24, "padding": 8}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  SOT["Source of truth: sites, devices, IPs, peerings, policy data, vetted by people"]:::core
  INT["Intended configuration: generated per device, stored in Git"]:::core
  G1["Gate 1, before deployment: valid and safe? Lint, model checks, emulation"]:::decision
  RUN["Running and applied configuration on the device"]:::neutral
  OPS["Operational state: sessions, routes, counters"]:::neutral
  G2["Gate 2, after deployment: does behaviour match intent? State checks, telemetry"]:::decision
  G3["Gate 3, continuously: does running still equal intended? Drift check"]:::decision
  FIX["On mismatch: fix the data or the device, then re-render"]:::neutral
  SOT --> INT --> G1 --> RUN
  RUN --> OPS --> G2
  RUN --> G3 --> FIX
  INT -->|"compared with"| G3

NetworkOps sits on the change path between design and the running network. These boundaries are editorial distinctions.

%%{init: {"flowchart": {"wrappingWidth": 170, "nodeSpacing": 16, "rankSpacing": 30, "padding": 8}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  AUTO["AutomationOps: governance for all automation"]:::neutral
  INFRA["InfraOps: IaC lifecycle"]:::neutral
  CLOUD["CloudOps: VPCs, hubs, private access"]:::neutral
  NWO["NetworkOps: engineer network change (data, generation, verification, rollback)"]:::core
  NETOPS["NetOps: run the network (NOC, faults, capacity)"]:::neutral
  OBS["ObservabilityOps: telemetry platform"]:::neutral
  INC["IncidentOps: incident command"]:::neutral
  AUTO -->|"guardrails"| NWO
  INFRA -->|"IaC methods"| NWO
  CLOUD -->|"cloud constructs"| NWO
  NWO -->|"verified changes"| NETOPS
  NWO -->|"telemetry subscriptions"| OBS
  OBS -->|"signals"| NETOPS
  NETOPS -->|"change requests, faults"| NWO
  NETOPS -->|"major incidents"| INC

This charter is this Atlas’s proposal; no body publishes an official one.

Mission. Make network change boring: described as data, verified before it ships, rolled out in small waves, reversible without a site visit, and confirmed by evidence.

1. Change the data, not the device. The unit of change is one pull request: the data change (or its source-of-truth branch), templates, rendered configurations and a change record.
2. People vet the source of truth. NetBox says “automated import of live network state is strongly discouraged”^[16]. Discovered state feeds a reconciliation report, never the SoT, or drift gets written into the record.
3. Generate; never hand-edit. Every line on a device comes from a template or model; a hand edit is drift.
4. Verify with two kinds of evidence. A model checks the whole network but only the syntax it parses; an emulator runs real software but only at lab scale. Know each one’s blind spots.
5. Take the smallest safe step. Put a representative device in an early wave. Cloudflare’s stagger “did not include an MCP data center until the final step”^[7].
6. Every change has an exit. A confirmed-commit timer, a management path the change cannot break, one owner for reverts. Meta and Rogers both lost that path in their outages^[26][8]; designing it is NetOps work.
7. Name the blast radius twice. Declare the devices to change and those to leave alone, separately from the data that generates the change, and refuse to run if what would change disagrees. Google adopted this after a wildcard drained “hundreds of devices when the intention was to drain two”^[6].
8. Prove the outcome. Done means sessions, routes and reachability match expectations, not that a command returned success.

Non-goals. Removing human approval from high-risk changes; replacing the NOC; automating every change. One-off work may be cheaper by hand if its result goes back into the source of truth.

Seven blocks, connected in the reference architecture; tool options are in the tooling landscape.

Usually a capability inside the network team, not a separate team. A typical split (R responsible, A accountable, C consulted, I informed):

Two rules matter most. One accountable steward per data domain (IPAM, circuits, BGP policy): an unowned source of truth decays. The incident lead owns reverts: at Cloudflare, engineers “walked over each other’s changes, reverting the previous reverts”^[7].

• Day 0 – design. Choose the data model and an owner per field, the generation method, the interface per platform and the verification suite. Import the current network once, review it by hand, and mark it as the baseline.
• Day 1 – provision. New devices boot, fetch an image and an initial configuration; SZTP standardises a secure way to do this from “a factory-default state”^[22]. Vendor implementations differ.
• Day 2 – change and assure. Most of the work: everyday changes, drift detection, OS upgrades (rings and maintenance groups are on PatchOps) and decommissioning, which is also a SoT change.

Governance rules this Atlas recommends:

• Score risk per change, not per programme. Rogers’ algorithm rated a step “Low” because earlier phases had gone well, so no lab test was required^[8]. Earlier success is not evidence about a different change.
• Cap changes per maintenance window. Several changes in one window made it “difficult to decide which network change ticket to roll back” at Rogers^[8].
• Review the evidence, not the CLI. Reviewers read the verification report (sessions, reachability, filter decisions) beside the diff, because diffs hide meaning.

Review cadences are in metrics. The loop mirrors the four phases of NIST’s security-focused configuration management: planning, implementing configurations, controlling change, monitoring^[29].

The reference design has two trust zones. CI runs without device credentials. Inside the management network, only an approval-gated deploy runner can write to devices; a separate drift job only reads.

%%{init: {"flowchart": {"wrappingWidth": 190, "nodeSpacing": 18, "rankSpacing": 30, "padding": 8, "subGraphTitleMargin": {"top": 4, "bottom": 14}}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  subgraph Z1["Zone 1: no device credentials"]
    direction TB
    SOT["Source of truth: data edited on a branch"]:::core
    GIT["Git: templates, generated configs, change record"]:::core
    CI["CI: guard, lint, model checks, emulation"]:::core
  end
  subgraph Z2["Zone 2: management network"]
    direction TB
    DEP["Deploy runner: restricted runner group, environment secrets after approval; commits with a timer, runs post-checks, confirms"]:::risk
    DRIFT["Drift job: read-only account"]:::neutral
    DEV["Network devices (candidate, confirmed commit)"]:::neutral
    TEL["Telemetry collector (gNMI, YANG-Push)"]:::neutral
  end
  OOB["Out-of-band access that does not depend on the data network"]:::risk
  SOT -->|"rendered from the branch"| GIT
  GIT --> CI
  CI -->|"approved merge"| DEP
  DEP --> DEV
  DRIFT -->|"read running config"| DEV
  DEV --> TEL
  OOB -.-> DEV

Three design choices carry most of the safety:

• Generated configurations are committed to Git: reviewers see the real diff, and every deployment has a last-known-good commit.
• Only the deploy runner holds write credentials, and post-checks run there before it confirms. GitHub warns that self-hosted runners “can be persistently compromised by untrusted code”^[30]; the pipeline names the settings that keep other workflows off it.
• The out-of-band path is independent. Xona Partners’ assessment for the CRTC calls for “secure alternative access to crucial remote network elements that is not dependent on the data network”^[8]; NetOps covers its design.

Alternatives (vendor controller, YAML-only source of truth, pull-based reconciliation): see deep comparisons.

In NetBox, an edit is live intent at once: other consumers read it before anyone approves it, reviewers see generated configs without the data change or its author, and a scheduled export bundles several people’s edits into one diff. Give data the path code takes:

1. Stage the edit. NetBox Labs’ Branching plugin “allows you to make copies of NetBox’s data model and alter them independently”; API requests pick a branch with the X-NetBox-Branch header^[31]. With a YAML source of truth in Git, the data change is part of the pull request.
2. Render from the branch, synced with the main data first so the render does not undo other merged changes. Commit the generated configs to the change’s pull request with a change record naming target and must-not-change devices (the guard checks both).
3. Merge the data after the last wave is confirmed. A failed change leaves the branch unmerged, so the source of truth still describes the network; a merged branch can also be reverted until it is archived or deleted^[31].

Edits made outside a branch show up as drift between the source of truth and Git (playbook 5).

Two properties matter: a failed check before deployment touches no device, and a failed check after deployment needs no human action to revert, because the deployer does not confirm. The incident lead alone decides on further reverts.

%%{init: {"flowchart": {"wrappingWidth": 210, "nodeSpacing": 18, "rankSpacing": 24, "padding": 8}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  A["Request: ticket or pull request"]:::core
  B["Edit SoT data or templates on a branch"]:::core
  C["CI gate: render, lint, model checks, emulation"]:::decision
  X["Fail: fix data or template; no device touched"]:::risk
  D["Review diff and verification report; approve by change class"]:::core
  E["Deploy next wave with a confirmed-commit timer"]:::core
  F["Post-change checks"]:::decision
  Y["Fail: do not confirm, device reverts; halt waves; page incident lead"]:::risk
  G["Pass: confirm commit"]:::core
  H["More waves?"]:::decision
  I["Attach evidence; merge the SoT branch; close"]:::core
  A --> B --> C
  C -->|"fail"| X
  C -->|"pass"| D --> E --> F
  F -->|"fail"| Y
  F -->|"pass"| G --> H
  H -->|"yes"| E
  H -->|"no"| I

Confirmed commit on each platform. NETCONF’s <persist> lets a different session confirm, which suits automation that reconnects^[11]. gNMI has a commit-confirmed extension (v0.1.0)^[20] that Nokia SR Linux supports^[32]. NAPALM’s commit_config(revert_in=…)^[24] is listed for EOS, Junos and IOS, not the IOS XR or NX-OS drivers^[33]. Where it is missing, use smaller waves and a tested rollback.

Merge or replace. Merge adds your lines to whatever is on the device; replace makes the device equal your file. In Ansible, cli_config with replace replaces the whole running configuration^[34]; resource-module states (merged, replaced, overridden) make the same choice per resource, such as interfaces or BGP, and overridden could “remove your access to the device”^[35]. See the judgement call.

Only the IETF RFCs below are standards. OpenConfig is an operator-led open-source project^[19], and RFC 9315 is an informational IRTF document.

Models change the review question from “are these commands right?” to “is this data right, and did verification prove its effect?”, which is cheaper to answer and easier to automate.

Choosing and reading models. OpenConfig models are vendor-neutral^[19]; native ones can cover more. Nokia calls its SR Linux models “a more complete representation” of the platform, including features “that the OpenConfig data models do not describe”^[36]. This Atlas suggests OpenConfig where every platform in a domain implements the module, native models elsewhere; the gNMI Capabilities reply lists the models a device supports^[36]. Read a path as a walk down the tree: in Lab 3’s /interface[name=ethernet-1/1]/oper-state, interface is a list, [name=…] selects one entry by its key, and oper-state is the leaf read^[37].

A NETCONF change, step by step. The safety features are explicit operations: lock the candidate, edit it, validate it, commit with a timer, and confirm later from any session that presents the persist ID^[11].

Example status: Python syntax-checked; not executed (needs ncclient 0.7 and a server with :candidate, :validate and :confirmed-commit:1.1)^[38].

"""NETCONF change: lock, edit the candidate, validate, confirmed commit, confirm."""
import os

from ncclient import manager

CONFIG = """<config xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
    <interface><name>eth1</name><description>to core1</description></interface>
  </interfaces>
</config>"""
DEVICE = {"host": "router1.example.net", "port": 830, "hostkey_verify": True,
          "username": os.environ["NET_USER"], "password": os.environ["NET_PASSWORD"]}

with manager.connect(**DEVICE) as m, m.locked("candidate"):
    m.discard_changes()                      # start from a clean candidate
    m.edit_config(target="candidate", config=CONFIG)
    m.validate(source="candidate")           # the server checks the whole candidate
    m.commit(confirmed=True, timeout="300", persist="chg-0142")

# ... run post-checks over a path the change cannot break, then confirm:
with manager.connect(**DEVICE) as m:
    m.commit(persist_id="chg-0142")         # without this, the device reverts

Controller-managed domains. SD-WAN and many data-centre fabrics run through a vendor controller that renders and pushes device configuration itself: the SDN pattern, in which a controller “manipulates the controlled entity via an interface”^[15]. The controller then owns rendering, transactions and rollback; the source of truth still owns the inputs (sites, prefixes, policy) and feeds the controller’s API through the same reviewed pipeline; verification checks behaviour, not device text. RFC 9315 splits intent-based systems into fulfilment and assurance^[10]: keep some assurance independent of the controller that fulfils.

Choose by job, then by platform. Versions as of October 2026^[39].

Lock-in and upkeep. Open-source tools move fast. One Nornir inventory plugin for NetBox last released in September 2021, before NetBox’s v2 API tokens^[39][46]. Pin versions, test upgrades in CI, and keep the SoT schema independent of any one runner.

The evidence on this page is about rule-based automation. AI assistants can draft templates, explain diffs or summarise a verification report; treat their output as an untrusted pull request that passes the same gates, and never give a model write access to devices or the SoT outside the pipeline. Automation “adds an additional layer of complexity”^[6]; AI adds another. Event analysis is on AIOps.

Beginners can start with the labs and return here later.

Assumptions. About 60 Arista EOS and Juniper Junos devices; NetBox 4.7^[47] with the Branching plugin; GitHub Enterprise Cloud (see the plan note below); Nornir with NAPALM; Batfish in CI. Replace runs only on platforms whose playbook-1 baseline diff is empty; the rest merge. Syntax-checked, not executed except the guard: a shape, not a product.

network-repo/
  sot-export.sh            # NetBox branch -> snapshot/configs/*.cfg, run for each change
  change.yaml              # change record: targets and must_not_change hosts
  snapshot/configs/        # generated configs, committed so reviewers see the diff
  guard.py                 # blast-radius guard: rendered diff vs change.yaml
  checks/check_change.py   # Batfish invariants and differential questions (Lab 2)
  validate/<host>.yaml     # NAPALM post-change expectations per host
  inventory/hosts.yaml     # Nornir hosts: platform, wave, full_template_coverage
  requirements-*.txt       # pinned versions with hashes (pip-compile --generate-hashes)
  deploy.py                # one wave: load with a timer, check, confirm or let it revert
  .github/workflows/network-change.yml

Export from the source of truth. NetBox renders each device’s configuration from its template and context data, here inside the change’s branch (untested; inferred from the rule that the branch “must be specified with each REST API request”^[31]). A token without “write enabled” can only make GET requests, but render-config is a POST^[46][27]: give the export job a write-enabled v2 token whose user holds only device view and the render_config action, restricted to the job’s IP.

Example status: bash -n syntax-checked; not executed (requires NetBox 4.5 or later for v2 tokens, the Branching plugin, curl and jq).

#!/usr/bin/env bash
# Render every active device's configuration from a NetBox branch into snapshot/configs/.
set -euo pipefail
: "${NETBOX_URL:?}" "${NETBOX_TOKEN:?}" "${NETBOX_BRANCH:?}"  # branch = its schema ID
H=(-H "Authorization: Bearer ${NETBOX_TOKEN}" -H "X-NetBox-Branch: ${NETBOX_BRANCH}")
mkdir -p snapshot/configs
curl -sf "${H[@]}" \
  "${NETBOX_URL}/api/dcim/devices/?status=active&brief=true&limit=1000" \
  | jq -r '.results[] | "\(.id) \(.name)"' \
  | while read -r id name; do
      curl -sf -X POST "${H[@]}" \
        -H "Content-Type: application/json" -H "Accept: text/plain" \
        --data '{}' "${NETBOX_URL}/api/dcim/devices/${id}/render-config/" \
        > "snapshot/configs/${name}.cfg"
    done
git status --short snapshot/configs

The change record and the guard. Each pull request carries a change.yaml from its author (example status: YAML parsed):

change_id: NW-2026-0142
targets: [spine1, leaf1]               # hosts this change is meant to touch
must_not_change: [leaf2, border1]      # hosts that must keep their config

guard.py adapts Google’s rule of requiring both a positive and a negative list^[6] to what will actually change: the hosts whose rendered configurations differ (derived from the data) must equal the author’s targets, and none may be in must_not_change. A template edit that touches every device fails the pull request.

Example status: executed on 2026-10-18 in a scratch Git repository (Python 3.8.10; PyYAML 5.3.1, again with 6.0.2): GUARD ok and exit 0 when the diff matched the record; exit 1 and one GUARD line per problem when it did not.

"""Blast-radius guard: the rendered diff must match the change record's two lists."""
import pathlib
import subprocess
import sys

import yaml


def changed_hosts(base, head):
    """Hosts whose generated configuration differs between two commits."""
    out = subprocess.run(
        ["git", "diff", "--name-only", base, head, "--", "snapshot/configs"],
        check=True, capture_output=True, text=True).stdout
    return {pathlib.Path(p).stem for p in out.splitlines()}


def check(base, head, record="change.yaml"):
    """Return (problems, changed hosts); no problems means the guard passed."""
    rec = yaml.safe_load(pathlib.Path(record).read_text())
    targets, keep = set(rec["targets"]), set(rec["must_not_change"])
    changed = changed_hosts(base, head)
    problems = []
    if changed & keep:
        problems.append(f"must-not-change hosts would change: {sorted(changed & keep)}")
    if changed != targets:
        problems.append(f"diff changes {sorted(changed)}; "
                        f"record declares {sorted(targets)}")
    return problems, changed


if __name__ == "__main__":
    found, _ = check(sys.argv[1], sys.argv[2])
    print("\n".join(f"GUARD {p}" for p in found) or "GUARD ok")
    sys.exit(1 if found else 0)

The pipeline. Pull requests are verified on a GitHub-hosted Linux runner without device credentials; the Batfish service container is reached on localhost^[48]. Only a push to main deploys, from a self-hosted runner in the management network, and four settings make that boundary real:

• Environment secrets. NET_USER and NET_PASSWORD belong to the production environment, not the repository; environment secrets are “only available to workflow jobs that reference the environment”, after approval^[49].
• Protection rules. Required reviewers, “prevent self-review”, a main-only deployment branch rule, administrator bypass disallowed (allowed by default)^[49].
• A restricted runner group. Limit it to this repository and to network-change.yml at main (“Selected workflows”)^[50]; otherwise any workflow on any branch can run code on a runner that reaches every device. Prefer just-in-time runners, which “perform at most one job”^[30].
• The plan. On Free, Pro and Team, required reviewers and the bypass setting work only for public repositories^[49], and workflow-restricted runner groups are documented for Enterprise Cloud^[50]. A private repository needs GitHub Enterprise, or a deployer outside Actions with its own approval check.

Example status: YAML parsed; not executed. Action SHAs are those of the v7.0.1 and v7.0.0 tags (as of October 2026).

name: network-change
on:
  pull_request:
    paths: ["snapshot/**", "checks/**", "validate/**", "change.yaml"]
  push:
    branches: [main]
    paths: ["snapshot/**", "change.yaml"]
permissions:
  contents: read
jobs:
  verify:
    runs-on: ubuntu-latest
    services:
      batfish:
        image: batfish/allinone
        ports: ["9996:9996"]
    env:
      BASE: ${{ github.event.pull_request.base.sha || github.event.before }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - run: pip install --require-hashes -r requirements-ci.txt
      - run: python guard.py "$BASE" HEAD
      - run: git worktree add reference "$BASE"
      - run: >
          python checks/check_change.py
          --reference reference/snapshot --candidate snapshot
  deploy-wave-1:
    if: github.event_name == 'push'
    needs: verify
    runs-on:
      group: network-deploy
    environment: production
    concurrency: network-production
    env:
      BASE: ${{ github.event.before }}
      CHANGE_ID: ${{ github.sha }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
      - run: python3 -m venv .venv
      - run: .venv/bin/pip install --require-hashes -r requirements-deploy.txt
      - run: .venv/bin/python deploy.py 1
        env:
          NET_USER: ${{ secrets.NET_USER }}
          NET_PASSWORD: ${{ secrets.NET_PASSWORD }}

Later waves copy deploy-wave-1 and need the previous wave; concurrency prevents parallel deployments. Actions are pinned to commit SHAs, as GitHub recommends^[30], and packages install only with matching hashes^[51]. deploy.py repeats the guard and touches only changed hosts in its wave. It loads each configuration with a revert timer, polls the post-checks until a deadline shorter than the timer (new BGP sessions take time), and confirms only if they pass; otherwise the timer restores the previous configuration^[52][24].

Example status: Python syntax-checked; not executed (requires devices whose NAPALM driver supports commit confirm).

"""Deploy one wave: guard, load with a revert timer, check until a deadline, confirm."""
import os
import sys
import time

from nornir import InitNornir
from nornir_napalm.plugins.tasks import (
    napalm_configure,
    napalm_confirm_commit,
    napalm_validate,
)

from guard import check

CHECK_SECONDS = 180                   # sessions can take minutes to come up
POLL_SECONDS = 15
REVERT_SECONDS = CHECK_SECONDS + 120  # the timer always outlasts the checks


def deploy(task):
    with open(f"snapshot/configs/{task.host.name}.cfg") as fh:
        config = fh.read()
    # Replace only where templates own the whole config (playbook 1 diff is empty).
    task.run(task=napalm_configure, configuration=config,
             replace=task.host.get("full_template_coverage", False),
             revert_in=REVERT_SECONDS, commit_message=os.environ["CHANGE_ID"])
    deadline = time.monotonic() + CHECK_SECONDS
    while True:
        report = task.run(task=napalm_validate,
                          src=f"validate/{task.host.name}.yaml")
        if report.result.get("complies"):
            break
        if time.monotonic() > deadline:
            raise RuntimeError("checks failed: not confirming, device will revert")
        time.sleep(POLL_SECONDS)
    task.run(task=napalm_confirm_commit)


def main(wave):
    problems, changed = check(os.environ["BASE"], "HEAD")
    if problems:
        sys.exit("guard: " + "; ".join(problems))
    nr = InitNornir(config_file="nornir.yaml")
    hosts = nr.inventory.hosts
    lost = sorted(h for h in changed if h not in hosts or hosts[h].get("wave") is None)
    if lost:
        sys.exit(f"guard: changed hosts without inventory entry or wave: {lost}")
    nr.inventory.defaults.username = os.environ["NET_USER"]
    nr.inventory.defaults.password = os.environ["NET_PASSWORD"]
    targets = nr.filter(
        filter_func=lambda h: h.name in changed and str(h.get("wave")) == wave)
    names = sorted(targets.inventory.hosts)
    if not names:
        print(f"wave {wave}: no changed hosts")
        return
    result = targets.run(task=deploy)
    if result.failed_hosts:
        sys.exit(f"wave {wave} failed on {sorted(result.failed_hosts)}; halting")
    print(f"wave {wave} confirmed on {names}")


if __name__ == "__main__":
    main(sys.argv[1])

The validation file states what must be true afterwards, in NAPALM’s getter format^[28] (example status: YAML parsed; not executed):

- get_bgp_neighbors:
    global:
      peers:
        10.0.1.0:
          is_up: true
- get_interfaces:
    Ethernet1:
      is_up: true

Lab 1 was executed. Labs 2 and 3 were not (they need Docker, not run on the author’s shared production host); their commands are syntax-checked against the cited documentation.

Objective. Generate configurations from data, reject bad data, and block a change that touches more devices than intended. Prerequisites. Python 3.11 or later (3.10 and older are past end of life^[53]), pip install jinja2 pyyaml. Cost and risk. None; no device is touched.

Example status: executed on 2026-10-18 with Python 3.8.10, under Jinja2 2.10.1 with PyYAML 5.3.1 and again under Jinja2 3.1.6 with PyYAML 6.0.2 (identical outputs and files); not run on Python 3.11 or later, which the author’s host lacks.

sot.yaml, the source of truth for a three-node fabric:

devices:
  spine1:
    asn: 65100
    loopback: 10.255.0.1/32
    links:
      - {interface: Ethernet1, ip: 10.0.1.0/31, peer: leaf1}
      - {interface: Ethernet2, ip: 10.0.1.2/31, peer: leaf2}
  leaf1:
    asn: 65101
    loopback: 10.255.0.11/32
    links:
      - {interface: Ethernet1, ip: 10.0.1.1/31, peer: spine1}
  leaf2:
    asn: 65102
    loopback: 10.255.0.12/32
    links:
      - {interface: Ethernet1, ip: 10.0.1.3/31, peer: spine1}

device.j2, an EOS-style template (Jinja2):

hostname {{ name }}
!
interface Loopback0
   ip address {{ dev.loopback }}
{% for l in dev.links %}
!
interface {{ l.interface }}
   description to {{ l.peer }}
   no switchport
   ip address {{ l.ip }}
{% endfor %}
!
router bgp {{ dev.asn }}
   router-id {{ dev.loopback.split('/')[0] }}
{% for n in neighbors %}
   neighbor {{ n.ip }} remote-as {{ n.asn }}
   neighbor {{ n.ip }} description {{ n.name }}
{% endfor %}
   network {{ dev.loopback }}

render.py checks the data (private ASNs, unique IPs, matching /31 ends), derives BGP neighbours, writes only changed files, and refuses to write if more devices would change than --max-changed allows:

"""Render device configs from a YAML source of truth, after pre-flight checks."""
import argparse
import ipaddress
import pathlib
import sys

import jinja2
import yaml


def check(sot):
    """Return (errors, neighbours) for every device in the source of truth."""
    errors, seen_ips, nbrs = [], {}, {}
    devs = sot["devices"]
    for name, dev in devs.items():
        nbrs[name] = []
        if not 64512 <= dev["asn"] <= 65534:
            errors.append(f"{name}: ASN {dev['asn']} is not a private 2-byte ASN")
        for link in dev["links"]:
            ip = ipaddress.ip_interface(link["ip"])
            if str(ip.ip) in seen_ips:
                errors.append(f"{name}: {ip.ip} already used on {seen_ips[str(ip.ip)]}")
            seen_ips[str(ip.ip)] = name
            peer = devs.get(link["peer"])
            if peer is None:
                errors.append(f"{name}: peer {link['peer']} "
                              "is not in the source of truth")
                continue
            back = [p for p in peer["links"] if p["peer"] == name]
            if len(back) != 1:
                errors.append(f"{name}: {link['peer']} has no single link back")
                continue
            pip = ipaddress.ip_interface(back[0]["ip"])
            if pip.network != ip.network or pip.ip == ip.ip:
                errors.append(f"{name}: {ip} and {link['peer']} {pip} are not one /31")
                continue
            nbrs[name].append({"ip": str(pip.ip), "asn": peer["asn"],
                               "name": link["peer"]})
    return errors, nbrs


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--sot", default="sot.yaml")
    ap.add_argument("--out", default="out")
    ap.add_argument("--max-changed", type=int, default=None,
                    help="refuse to write if more devices would change than this")
    args = ap.parse_args()
    sot = yaml.safe_load(pathlib.Path(args.sot).read_text())
    errors, nbrs = check(sot)
    if errors:
        print("\n".join(f"ERROR {e}" for e in errors))
        return 1
    env = jinja2.Environment(loader=jinja2.FileSystemLoader("."),
                             undefined=jinja2.StrictUndefined,
                             trim_blocks=True, lstrip_blocks=True,
                             keep_trailing_newline=True)
    tpl = env.get_template("device.j2")
    out = pathlib.Path(args.out)
    rendered = {n: tpl.render(name=n, dev=d, neighbors=nbrs[n])
                for n, d in sot["devices"].items()}
    changed = [n for n, txt in rendered.items()
               if not (out / f"{n}.cfg").exists()
               or (out / f"{n}.cfg").read_text() != txt]
    if args.max_changed is not None and len(changed) > args.max_changed:
        print(f"BLOCKED {len(changed)} devices would change "
              f"(limit {args.max_changed}): {', '.join(changed)}")
        return 2
    out.mkdir(exist_ok=True)
    for n in changed:
        (out / f"{n}.cfg").write_text(rendered[n])
    sessions = sum(len(v) for v in nbrs.values()) // 2
    print(f"OK {len(rendered)} devices, {sessions} BGP sessions, "
          f"{len(changed)} changed: {', '.join(changed) or 'none'}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Steps. (1) python3 render.py. (2) Run it again. (3) Change the spine’s asn to 65200 in a copy (sot-asn.yaml) and run python3 render.py --sot sot-asn.yaml --max-changed 1. (4) In another copy (sot-bad.yaml), give leaf2’s link the address 10.0.1.1/31 and run python3 render.py --sot sot-bad.yaml.

Expected output (from the run):

OK 3 devices, 2 BGP sessions, 3 changed: spine1, leaf1, leaf2
OK 3 devices, 2 BGP sessions, 0 changed: none
BLOCKED 3 devices would change (limit 1): spine1, leaf1, leaf2
ERROR spine1: 10.0.1.2/31 and leaf2 10.0.1.1/31 are not one /31
ERROR leaf2: 10.0.1.1 already used on leaf1
ERROR leaf2: 10.0.1.1/31 and spine1 10.0.1.2/31 are not one /31

Validation. out/leaf1.cfg contains neighbor 10.0.1.0 remote-as 65100. Step 3 shows why the guard matters: one spine ASN touches every peer. Exit codes 0, 0, 2 and 1 let CI gate on the result. Troubleshooting. UndefinedError: the template uses a field the data lacks (StrictUndefined makes that fail loudly). Teardown. Delete the folder.

Objective. Catch two faults no syntax check sees: a BGP neighbour typo and an access-list reordering (an ordering error, like the policy-term reorder in Cloudflare’s outage). Prerequisites. Docker; Python 3.11 or later (current pybatfish needs 3.10 or later^[39]); python3 -m pip install --upgrade pybatfish; Batfish started with docker run --name batfish -v batfish-data:/data -p 8888:8888 -p 9996:9996 batfish/allinone^[44]. Cost and risk. Local only. The lab uses Cisco IOS syntax, which Batfish supports^[44].

Create snapshots/reference/configs/edge1.cfg:

hostname edge1
!
interface GigabitEthernet0/0
 ip address 10.0.12.1 255.255.255.252
 ip access-group MGMT-IN in
!
interface Loopback0
 ip address 192.0.2.1 255.255.255.255
!
ip access-list extended MGMT-IN
 permit tcp 198.51.100.0 0.0.0.255 host 192.0.2.1 eq 22
 deny   tcp any host 192.0.2.1 eq 22
 permit ip any any
!
router bgp 65001
 bgp router-id 192.0.2.1
 neighbor 10.0.12.2 remote-as 65002
 address-family ipv4
  network 192.0.2.1 mask 255.255.255.255
  neighbor 10.0.12.2 activate
 exit-address-family
!
end

and snapshots/reference/configs/core1.cfg:

hostname core1
!
interface GigabitEthernet0/0
 ip address 10.0.12.2 255.255.255.252
!
interface Loopback0
 ip address 192.0.2.2 255.255.255.255
!
router bgp 65002
 bgp router-id 192.0.2.2
 neighbor 10.0.12.1 remote-as 65001
 address-family ipv4
  network 192.0.2.2 mask 255.255.255.255
  neighbor 10.0.12.1 activate
 exit-address-family
!
end

Copy reference to candidate. In the candidate edge1.cfg, swap the first two MGMT-IN lines, and change neighbor 10.0.12.2 remote-as 65002 to remote-as 65020. Then run checks/check_change.py:

Example status: Python syntax-checked; not executed. Question names, parameters and columns are from the pybatfish documentation^[44].

"""Pre-change checks with Batfish: candidate snapshot against the reference."""
import argparse
import sys

from pybatfish.client.session import Session
from pybatfish.datamodel.flow import HeaderConstraints

ap = argparse.ArgumentParser()
ap.add_argument("--reference", default="snapshots/reference")
ap.add_argument("--candidate", default="snapshots/candidate")
ap.add_argument("--host", default="localhost")
args = ap.parse_args()

bf = Session(host=args.host)
bf.set_network("networkops-lab")
bf.init_snapshot(args.reference, name="reference", overwrite=True)
bf.init_snapshot(args.candidate, name="candidate", overwrite=True)
failures = []

ref_issues = len(bf.q.initIssues().answer(snapshot="reference").frame())
new_issues = len(bf.q.initIssues().answer(snapshot="candidate").frame())
if new_issues > ref_issues:
    failures.append(f"init issues rose from {ref_issues} to {new_issues}")

sessions = bf.q.bgpSessionStatus().answer(snapshot="candidate").frame()
for _, row in sessions[sessions["Established_Status"] != "ESTABLISHED"].iterrows():
    failures.append(f"BGP {row['Node']} -> {row['Remote_IP']}: "
                    f"{row['Established_Status']}")

ssh = HeaderConstraints(srcIps="198.51.100.0/24", dstIps="192.0.2.1",
                        applications=["ssh"])
denied = bf.q.searchFilters(headers=ssh, action="deny", filters="MGMT-IN")
denied = denied.answer(snapshot="candidate").frame()
if len(denied):
    failures.append(f"MGMT-IN denies admin SSH, e.g. {denied.iloc[0]['Flow']}")

changed = bf.q.compareFilters().answer(snapshot="candidate",
                                       reference_snapshot="reference").frame()
print(f"{len(changed)} filter line pair(s) now treat some flows differently")
for f in failures:
    print("FAIL", f)
print("PASS" if not failures else f"{len(failures)} check(s) failed")
sys.exit(1 if failures else 0)

Expected output (from the documentation, not observed). Run with --candidate snapshots/reference, the script prints PASS. Against the candidate it should report NOT_COMPATIBLE sessions for edge1 and core1 (one side misconfigured), an MGMT-IN denies admin SSH failure with an example flow, at least one changed filter line pair, and exit code 1. Validation. Undo one fault at a time; only the other should remain. Troubleshooting. Connection refused on 9996: the container is not running (docker logs batfish). Rows in initIssues mean Batfish did not fully understand a line; read them before trusting other answers. Teardown. docker rm -f batfish && docker volume rm batfish-data.

Objective. Use gNMI against two emulated Nokia SR Linux routers: read state, stream telemetry, and make a change that reverts unless confirmed. Prerequisites. A Linux VM with Docker and sudo; containerlab (curl -sL https://containerlab.dev/setup | sudo -E bash -s "all")^[45]; gnmic (bash -c "$(curl -sL https://get-gnmic.openconfig.net)")^[43]. The SR Linux image is public; default credentials admin / NokiaSrl1!^[45]. Cost and risk. A lab VM only, never a production host; read installer scripts before running them.

Example status: YAML parsed and bash -n syntax-checked; not executed. Commit-confirmed support per the SR Linux 25.10 guide^[32].

nwops.clab.yml:

name: nwops
topology:
  nodes:
    srl1:
      kind: nokia_srlinux
      image: ghcr.io/nokia/srlinux:25.10.5
    srl2:
      kind: nokia_srlinux
      image: ghcr.io/nokia/srlinux:25.10.5
  links:
    - endpoints: ["srl1:ethernet-1/1", "srl2:ethernet-1/1"]

#!/usr/bin/env bash
set -euo pipefail
sudo containerlab deploy -t nwops.clab.yml
G=(gnmic -a clab-nwops-srl1 --skip-verify -u admin -p 'NokiaSrl1!' -e json_ietf)
IF='/interface[name=ethernet-1/1]'

# 1. Read: host name and link state
"${G[@]}" get --path /system/name/host-name
"${G[@]}" get --path "${IF}/oper-state"

# 2. Change with a 120-second rollback timer, then read it back
"${G[@]}" set --update-path "${IF}/description" --update-value "to srl2" \
  --commit-id lab3a --commit-request --rollback-duration 120s
"${G[@]}" get --path "${IF}/description"

# 3. Do nothing for over two minutes, then read again: the change should be gone
sleep 150
"${G[@]}" get --path "${IF}/description"

# 4. Repeat; confirm only if the read-back passes (production: check behaviour too)
"${G[@]}" set --update-path "${IF}/description" --update-value "to srl2" \
  --commit-id lab3b --commit-request --rollback-duration 120s
"${G[@]}" get --path "${IF}/description" --values-only | grep 'to srl2'
"${G[@]}" set --commit-id lab3b --commit-confirm

In a second terminal, stream state changes on srl1. After step 4, disable the interface and enable it again:

gnmic -a clab-nwops-srl1 --skip-verify -u admin -p 'NokiaSrl1!' -e json_ietf \
  subscribe --path '/interface[name=ethernet-1/1]/oper-state' --stream-mode on-change

#!/usr/bin/env bash
set -euo pipefail
G=(gnmic -a clab-nwops-srl1 --skip-verify -u admin -p 'NokiaSrl1!' -e json_ietf)
P='/interface[name=ethernet-1/1]/admin-state'
"${G[@]}" set --update-path "$P" --update-value disable   # the stream should show down
sleep 10
"${G[@]}" set --update-path "$P" --update-value enable    # and then up again

Expected output. Step 1 returns the node’s host name and up (containerlab enables interfaces named in links^[45]). Step 2 shows the description and step 3 none; in step 4 the read-back finds it, so the script confirms and it persists (a miss would stop the script before the confirm). The subscription prints the current value first, then one update per change, as gNMI requires for ON_CHANGE^[18]: down after the disable, up after the enable (SR Linux’s admin-state takes enable and disable^[32]). Validation. During a running timer, send a plain set: the target must refuse it with FAILED_PRECONDITION^[20], so nobody can stack changes onto an unconfirmed one. A commit cannot be amended once issued, so send the whole change in one request^[20]. Troubleshooting. TLS errors: --skip-verify is for labs only. unknown host: use the address from containerlab inspect. Teardown. sudo containerlab destroy -t nwops.clab.yml --cleanup removes the containers and the lab directory^[45].

A. Edge routing-policy change (real case: Cloudflare, June 2022). Context: standardising BGP communities on site-local prefixes, with a dry run, a stepped rollout and peer review; yet the steps “weren’t small enough”^[7]. NetworkOps adds: a differential check of advertised routes, not just a text diff; a first wave that includes an MCP site; a commit-confirmed timer, which Cloudflare listed as a follow-up. Success: no unexpected withdrawals in the differential report. Pitfall: trusting reviewers to spot reordering in a diff.

B. Core upgrade under a risk score (real case: Rogers, July 2022). Context: phase six of seven; a filter removed during clean-up, and core routers that normally received about 10,000 routes got “over 900,000” from one distribution router^[8]. NetworkOps adds: an invariant (“every distribution-to-core session has an inbound filter and a prefix limit”); lab or model tests required by change type, whatever the track record. Success: the invariant fails the pull request. Pitfall: treating a risk score as evidence (governance).

C. Branch rollout from the source of truth (illustrative scenario). Context: 120 new branch sites in six months, each with two switches and an SD-WAN edge. Method: stewards create the site in the SoT, prefixes are allocated, configs rendered and verified, devices boot with zero-touch provisioning, waves follow opening dates. Trade-off: weeks of template work, repaid only if sites are alike. Success: over 95 % first-time-right after ten sites. Pitfall: local hand-made exceptions, which become drift.

D. Firewall and ACL requests as standard changes (illustrative scenario). Context: 40 rule requests a week, each reviewed by hand. Method: an intake form records source, destination, port and owner as SoT policy objects; CI checks “must allow” and “must deny” flows with searchFilters and reports changed lines with compareFilters^[44]; additions matching an approved pattern are auto-approved. Success: lead time falls from days to hours, fail rate flat. Pitfall: rules without an owner or expiry.

Measure outcomes of change, not activity. The first two adapt DORA’s definitions to network changes^[54]; the rest are this Atlas’s proposals.

Worked calculation (illustrative). One quarter: 420 changes, 380 through the pipeline and 40 by hand (12 of them emergencies). Nine needed rollback: two pipeline changes and seven manual ones.

• Pipeline coverage = 380 ÷ 420 = 90.5 %.
• Change fail rate = 9 ÷ 420 = 2.1 % overall, but 0.5 % (2 ÷ 380) through the pipeline and 17.5 % (7 ÷ 40) by hand. The split tells you where to invest; the blended figure hides it.
• Verification blocked 31 pull requests with real defects, and 2 defects escaped: catch rate = 31 ÷ 33 = 93.9 %.
• 14 of 600 devices drifted: drift rate = 2.3 %. Review the oldest first.

Review cadence. Weekly: fail rate and escapes. Monthly: drift and SoT completeness. Quarterly: coverage, lead time and the maturity self-assessment. “Number of playbooks” is a vanity metric unless paired with fail rate.

• Credentials. Automation has its own AAA accounts, separate for read (telemetry, drift) and write, so accounting logs show who changed what. Rotate them.
• Runner trust. Self-hosted runners are “not run in an isolated container, even if they use environments”^[49], so restrict the deploy runner as the pipeline describes.
• Source-of-truth integrity. Stewards write; tokens are v2, tied to users with minimal object permissions, IP-restricted and write-disabled wherever GET suffices^[46]. Keep NetBox patched.
• Management plane. SSH for NETCONF (mandatory to implement; TLS is an option)^[11][55], TLS for gNMI^[18], HTTPS for RESTCONF^[13]. Verify host keys and certificates; --skip-verify is for labs only.
• Audit evidence. Each change record links the pull request, verification report, approval, deployment log and post-checks, matching NIST SP 800-128’s change-control and monitoring phases^[29]. Sector rules differ; ask your compliance team.

• Cost drivers. People’s time dominates: data modelling, whole-config templates and check upkeep. Tooling is mostly open source; expect CI minutes, a SoT server (NetBox needs PostgreSQL 15+^[16]), lab hosts, and vendor accounts for some images (Arista cEOS requires registration^[45]).
• Unit economics. Track engineer-hours per change by class. Automation pays where a class is frequent and alike (VLANs, peers, ACL entries), rarely for one-off redesigns.
• Control-plane capacity as an invariant. Prefix limits, route-table headroom and CPU protection belong in verification. Rogers’ core routers “were not configured with such overload protection mechanisms”^[8].
• Sustainability. Minor here: tear down emulation labs after use (containerlab destroy), and right-size the CI runners.

Watch the system that makes changes. Signals: fail rate by path; share of changes with complete evidence; drift exceptions past expiry; break-glass and bypass events; age of the last rollback drill; verification and guard overrides. Drill quarterly: push a known-bad change (for example Lab 2’s ACL swap) through the real pipeline against a lab or canary target. Confirm that verification blocks it, and that an unconfirmed commit reverts within its timer.

Illustrative scenario, modelled on the failure class in Cloudflare’s report^[7]. Wave 2 of a routing-policy change reaches four edge routers at 10:02. At 10:04 reachability probes for three customer prefixes fail.

1. Stop the line (minute 0–2). Halt the pipeline so no further wave starts. Do not confirm wave 2; its timer runs.
2. Name one incident lead (minute 1). Only the lead orders reverts, so nobody reverts a revert (IncidentOps).
3. Let the timer work, or cancel it (minute 2–7). Unconfirmed devices restore their previous configuration themselves; with gNMI the lead can cancel by commit ID^[20]. If confirmed wave 1 contributes, roll it back from last-known-good configs in Git.
4. Verify recovery (minute 7–15). Probes green, advertised routes equal the reference snapshot, sessions established.
5. Preserve evidence. Commit IDs, pipeline and device logs, the verification report that passed.
6. Fix the system, not the person (days 1–5). Add the missing invariant (“these prefixes are advertised to every transit peer”), put a router of the new design in wave 1, and record the drill.

Exit criteria. Service restored, reference and production agree, and the new check fails on the bad change in CI.

• Scripts without a source of truth. Fast at first; then every script carries its own copy of the network.
• Discovery writes the SoT. Today’s mistakes become tomorrow’s intent; NetBox discourages it^[16].
• A bypass everyone uses. If the emergency path is easier than the pipeline, it becomes the normal path; measure bypasses.
• Big-bang replace. The whole estate on full-config replace before templates cover everything.
• “Green means safe.” Models miss what they cannot parse, and labs miss scale; staging and timers cover the rest.
• Myth: intent-based networking removes configuration work. RFC 9315 describes fulfilment and assurance, and intent drift to watch^[10]. Someone still models the intent and checks the outcome.
• Myth: you need software developers. You need network engineers who read Python and YAML, and a few who write it well.

Hidden responsibilities. Templates follow every OS upgrade, the verification suite stays current, and the SoT is a production service with backups and access reviews.

%%{init: {"flowchart": {"wrappingWidth": 150, "nodeSpacing": 14, "rankSpacing": 24, "padding": 8}}}%%
flowchart TB
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  Q1["Are changes frequent, or has a change caused an outage?"]:::decision
  N1["Not yet: config backups, peer review, written rollback"]:::neutral
  Q2["Will someone own each data domain?"]:::decision
  N2["Fix ownership first; a SoT without owners decays"]:::risk
  Q3["Does a vendor controller already run this domain well?"]:::decision
  C1["Feed the controller from the SoT; add verification evidence"]:::core
  Q4["Do devices offer NETCONF or gNMI with confirmed commit?"]:::decision
  M1["Model-driven pipeline with confirmed commits"]:::core
  M2["CLI templating via NAPALM or Ansible; small waves; tested rollback"]:::core
  Q1 -->|"yes"| Q2
  Q2 -->|"yes"| Q3
  Q3 -->|"no"| Q4
  Q4 -->|"yes"| M1
  Q1 -->|"no"| N1
  Q2 -->|"no"| N2
  Q3 -->|"yes"| C1
  Q4 -->|"no"| M2

Assumptions. “Frequent” means, in this Atlas’s suggestion, a change class that recurs at least weekly. Mixed estates choose per platform; one pipeline can serve both.

This Atlas’s model, not an industry standard. Move up when the exit criterion has held for a quarter.

%%{init: {"flowchart": {"wrappingWidth": 300, "rankSpacing": 18, "padding": 8}}}%%
flowchart TB
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  L0["0 Manual: CLI by hand, configs in personal files"]:::risk
  E0["Exit: nightly config backups in Git for every device"]:::decision
  L1["1 Scripted: read-only automation, backups, inventories"]:::neutral
  E1["Exit: a SoT with named owners covers the in-scope estate"]:::decision
  L2["2 Generated: configs rendered from the SoT; drift reported"]:::neutral
  E2["Exit: drift under 5 % and every change via pull request"]:::decision
  L3["3 Verified: CI checks, staged waves, confirmed commits, evidence"]:::core
  E3["Exit: fail rate tracked by path; quarterly bad-change drill passes"]:::decision
  L4["4 Assured: telemetry-based checks, narrow closed-loop fixes"]:::core
  L0 --> E0 --> L1 --> E1 --> L2 --> E2 --> L3 --> E3 --> L4

For a team at level 0–1 starting with one domain, such as data-centre leaf switches.

Quick wins: backups in Git; a read-only “what would change” report. Dependencies: out-of-band access to every device in scope; AAA accounts for automation.

Before any change (author)

• A pull request with the data change (or its SoT branch), not a device session
• Rendered diff reviewed; the guard passed against targets and must_not_change
• Verification passed: no new parse issues, invariants hold, differences explained
• First wave contains a device of every affected design
• Confirmation path independent of the change
• Rollback source named (last-known-good commit); incident lead on call

Pipeline readiness (automation engineer)

• No device credentials in CI; device secrets in the production environment, not the repository
• Runner group limited to the deploy workflow at main; just-in-time runners where possible
• Plan allows required reviewers on a private repository; admin bypass off; self-review prevented
• Actions pinned to commit SHAs; packages installed with hashes
• Revert timer derived from the check deadline; replace only with full template coverage
• One deployment at a time per domain (concurrency or lock)
• SoT token scoped, IP-restricted, rotated; out-of-band access tested this quarter

Quarterly governance audit

• Bad-change drill run and passed
• Drift exceptions past expiry closed
• Break-glass events reconciled into the SoT
• Tool versions pinned and upgraded in CI first

Network change record (one per pull request):

Change ID:            NW-2026-____
Class:                standard | normal | emergency
Summary:              what changes, in one sentence
Blast radius:         change.yaml targets + must_not_change; guard result
Waves:                wave 1 (canary, representative): ...; wave 2: ...
Verification:         link to CI report (init issues, invariants, differential)
Timer:                confirmed commit ___ s; confirmation path: ___
Post-checks:          sessions, routes, probes expected after each wave
Rollback:             last-known-good commit ___; incident lead ___
Approvals:            reviewer ___; change manager ___ (normal/emergency)
Evidence (after):     deploy log, post-check results, drift diff = empty

Data ownership charter (one per data domain):

Domain:               IPAM | circuits | BGP policy | VLANs | ...
Steward (accountable): name / team
Writers:              who may write, through which path (UI, API, pipeline)
Required fields:      list; validation rule for each
Quality target:       completeness >= __ %; monthly sample audit of __ devices
Discovery data:       allowed only as a reconciliation report, never written directly
Review:               monthly with the network lead

Architecture decision record: merge or replace for platform X

Context:      platform, OS versions, template coverage %, commit-confirm support
Options:      merge | replace | replace for some sections only
Decision:     ...
Consequences: drift handling, rollback method, lock-out risk and mitigation
Review date:  ...

Is NetworkOps the same as NetDevOps? As this Atlas uses it, yes; “NetDevOps” and “network automation” are the common names.

Do I need NETCONF or gNMI to start? No. CLI templating with good verification and small waves is a valid start; model-driven interfaces add transactions and confirmed commits.

NetBox or YAML files in Git? YAML suffices for tens of devices and one team. Move to NetBox or Nautobot when several teams edit data, you need IP allocation, or other systems consume it.

Does Batfish replace a lab? No. It analyses the whole network but only supported syntax^[44]; a lab runs real software at small scale. They catch different faults.

Can the network fix itself? For narrow, reversible actions with clear signals (draining a flapping link), at maturity level 4; broad automatic changes are how automation causes large failures.

What if a change must happen during an outage? Emergency class: break-glass, logged, reconciled into the SoT within a working day, reviewed.

• Established now: model-driven configuration with transactions, streaming telemetry, Batfish analysis, container-based emulation, staged rollout with confirmed commits.
• Maturing: intent assurance (continuous checks for intent drift)^[10], and emulated twins of production for every change; the Rogers assessment asks for test scenarios that “replicate the production network”^[8].
• Research: Google’s paper names specifying intent, collecting the network’s “ground truth” and reconciling the two as an underexplored area of management-plane design^[6].
• Vendor marketing to read critically: “self-driving” and AI-run networks. Ask which actions run without a human, what verifies them first, and how they roll back.

• Decide: one domain and one recurring change class first; data stewards named; merge or replace per platform.
• Build: SoT → generated configs in Git → verification (model, lab) → staged waves → confirmed commit → state checks.
• Guard: no device secrets in CI; a restricted deploy runner; one writer per device; the rendered diff checked against two lists; an out-of-band path the change cannot break.
• Measure: change fail rate split by path, pre-deployment catch rate, drift rate and age, SoT completeness.
• Watch out: discovery writing the SoT, bypasses becoming normal, risk scores that learn from success, green meaning safe.
• Next action: nightly config backups in Git this week; everything else builds on them.

Situation (illustrative). 300 devices: 200 EOS and Junos devices whose NAPALM drivers support replace and commit confirm, and 100 older ones. Templates cover about 70 % of each configuration; the rest is local history. The automation lead wants full replace (A) to eliminate drift; a senior engineer prefers merge (B) because it cannot delete what templates do not know.

Reasoning. (1) Worst failure: under A, the uncovered 30 % is deleted on first deploy, possibly including management access (Ansible’s warning about overridden^[35]). Under B, drift and stale config persist. (2) Does the timer rescue A? Partly: a confirmed commit reverts a lock-out^[11], but only where supported^[33] and only if confirmation waits for evidence. (3) Cost of A: full template coverage per platform (assumed: weeks each). (4) What would change the answer: full coverage and commit confirm everywhere make A clearly better.

Decision: B now, A per platform as coverage reaches 100 %. Run replace as a nightly dry run (compare_config); its diff lists the lines templates still miss. Uncertainty: how long the older devices stay; if replacement is near, keep them on merge.

After this page you can explain the label and its boundary with NetOps; design a source of truth with owners; choose merge or replace; build a verification gate; roll out in waves with confirmed commits; and measure change outcomes.

The word “NetworkOps” adds nothing; the practice behind it adds a great deal. Operators asked for it in 2002^[9]: treat the network as one system described by data, change it through transactions, keep a way back. The three outages on this page all had review or audit steps. What they lacked was verification of meaning, representative first steps, and an exit that did not depend on the network being changed. Build those, measure what fails, and the tooling choice gets easy. Product facts are dated October 2026; re-check them.

Changelog. 2026-10-18: first draft (r0), researched and written by the managing author; Lab 1 executed, Labs 2–3 and the reference implementation syntax-checked only. 2026-10-18 (r1): revised after the independent review: label status, runner and secret settings, a source-of-truth change path, a guard that checks the rendered diff (executed), post-check polling, NETCONF example, controller domains, transport and chronology fixes, pinned actions. 2026-10-18, post-score polish: Branching revert precondition; Lab 2’s Cloudflare comparison made precise; branch rendering marked untested; Lab 3 confirms only after a read-back; OpenConfig-or-native and path-reading paragraph; SDN sentence; beginner pointer; figure 3 label moved into its node (no crossing); Lab 1 and the guard re-run with Jinja2 3.1.6 and PyYAML 6.0.2; restatements trimmed.