InfraOps

InfraOps is an informal contraction of infrastructure operations: providing and running the servers, networks, storage, databases and cloud accounts that applications depend on. No standards body or certification defines the label.

In this Atlas, InfraOps means running infrastructure through code. Everything is declared in version-controlled files; each change is previewed as a plan, reviewed, applied by a pipeline and then checked against what actually exists.

In one line: InfraOps keeps three things in agreement, the code that says what should exist, the state that records what the tool manages, and the infrastructure that really exists, and it changes them only through reviewed plans.

For engineers who write or inherit infrastructure code, architects who design state layouts and pipelines, and leaders deciding how much of the estate to codify.

Infrastructure changed by hand drifts apart. Martin Fowler’s “snowflake server” is a machine so altered by unrecorded changes that nobody can rebuild it: “good for a ski resort, bad for a data center”.^[4] Configuration tools reduce the problem but, as Kief Morris noted in 2013, “there is considerable scope for configuration drift” even with them.^[5] Four costs follow:

1. Environments cannot be reproduced. DORA asks teams to keep “All cloud configuration files” in version control and to provision any environment automatically.^[6]
2. Changes cannot be reviewed before they happen. A console click has no diff and no reviewer. A plan does.
3. Nobody knows what exists. Inventory lives in memory or in stale spreadsheets.
4. Audit becomes archaeology. Control frameworks expect a documented baseline that “can be changed only through change control procedures”;^[7] reconstructing one after the fact is expensive.

Self-service makes this urgent: Forrester’s Charles Betz observed in 2020 that developers increasingly provision infrastructure themselves with IaC, which puts pressure on the traditional infrastructure-engineering role.^[8]

• Not a tool. Terraform, OpenTofu, Pulumi, CloudFormation and Bicep are engines; the practice is the lifecycle around them.
• Not CloudOps, which decides which provider constructs exist and how they are governed. InfraOps is how any of them is written, changed and checked as code.
• Not GitOps. GitOps adds an agent that pulls and continuously reconciles the declared state. A CI job that runs terraform apply after a merge is IaC with pull-request automation, not GitOps under the OpenGitOps principles.
• Not SysOps or AutomationOps, which own the inside of a host and run-time remediation; InfraOps provisions and replaces the host. PlatformOps wraps its modules behind self-service.
• Not “infrastructure that never changes”. Immutable infrastructure replaces servers instead of editing them; databases and disks hold durable data that must be protected, not replaced.

Mental model: three copies of the truth. The code says what you intend, the state what the tool believes it manages, and the provider what actually exists. A plan refreshes state from reality and compares it with the code. Most InfraOps failures live in a gap between two copies, so name the gap before fixing anything.

%%{init: {"flowchart": {"wrappingWidth": 85, "nodeSpacing": 10, "rankSpacing": 30}}}%%
flowchart TB
  C["Code in Git: what should exist"]
  S["State: what the tool manages"]
  R["Reality: what the provider holds"]
  C -->|"plan compares"| S
  S -->|"refresh reads"| R
  C -.- G1["Gap 1: merged, not applied"]
  S -.- G4["Gap 4: in state, deleted outside"]
  R -.- G2["Gap 2: drift, changed outside"]
  R -.- G3["Gap 3: unmanaged, not imported"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class C,S,R core
  class G1,G2,G3,G4 risk

Gap 3 deserves emphasis. In HashiCorp’s drift tutorial, a hand-made security group appeared in the refresh-only plan only as a new ID inside the instance’s diff; the group itself was not reported and had to be imported.^[23] A clean plan proves that managed objects match the code; it says nothing about unmanaged ones. Those need the provider’s inventory (see assurance).

This charter is the Atlas’s proposal, not an official manifesto. Mission: every infrastructure change is declared, previewed, reviewed, applied by automation and verified afterwards.

1. Code is the only write path. Humans read the console; pipelines write. Emergency changes are allowed and then codified within a fixed time. Trade-off: a slower first response to some incidents.
2. Review the plan, not only the code. A one-line change to a module can replace a database; only the plan shows it.
3. Apply what was reviewed. Applying one saved plan invalidates any other plan made against the same state,^[24] so treat plans as perishable and recompute them after any other apply.
4. Small states, small blast radius. Split state by lifecycle, owner and risk, not by convenience.
5. Replace compute, protect data. Instances come from images; durable stores have explicit deletion protection.
6. Pin everything that can change underneath you: engine, providers, modules and base images.
7. Measure the gaps. Drift, unmanaged resources and image age are tracked like any other operational risk.

Non-goals: codifying throwaway experiments, replacing in-host configuration management, forcing one engine on every team.

R = responsible, A = accountable, C = consulted, I = informed. In small teams one person holds several columns, but the production reviewer is never the author. Organisation-level change approval stays with ITOps.

%%{init: {"flowchart": {"wrappingWidth": 160, "nodeSpacing": 22, "rankSpacing": 30, "padding": 8}}}%%
flowchart TB
  A1["Day 0: state layout, module boundaries"] --> A2["Day 0: backend, CI identities, policies"]
  A2 --> B1["Day 1: write or import"]
  B1 --> B2["Day 1: plan and review"]
  B2 --> B3["Day 1: apply from the pipeline"]
  B3 --> C1["Day 2: detect drift and unmanaged objects"]
  C1 --> C2["Day 2: upgrade engine, providers, images"]
  C2 --> C3["Day 2: refactor or decommission"]
  C3 -.->|"every change is a new plan"| B2
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class A1,A2 neutral
  class B1,B2,B3,C1,C2,C3 core

• Day 0 settles what is hard to change later: state split, naming, tagging, identities, and the state backend itself. Create the backend from a small bootstrap configuration with local state, then move that state into it with terraform init -migrate-state.^[25]
• Day 2 takes most of the effort: drift, upgrades, image rebuilds, refactoring and removal. Slack’s Terraform 0.11 to 0.12 upgrade in 2019 took a quarter,^[26] so plan upgrades as projects.

Three rules make it work. Ownership is written down: a CODEOWNERS entry per directory and a named owner per state. Exceptions expire: every ignore_changes, policy waiver and unmanaged resource has an owner and a review date. Evidence is generated, not written: pull request, plan summary, approval and apply log form the record that ITOps and auditors need.

Four trust zones: authoring needs no cloud credentials; the pipeline holds separate plan (read-only) and apply identities; the stores hold state and versioned artefacts; the provider accounts hold the resources, inventory and audit log.

%%{init: {"flowchart": {"wrappingWidth": 115, "nodeSpacing": 16, "rankSpacing": 28}}}%%
flowchart TB
  DEV["Engineer: code and tests, no cloud keys"] --> PR["Pull request"]
  PR --> CHK["CI: format, validate, lint, scan"]
  REG["Module and image registries"] --> PLAN
  CHK --> PLAN["CI plan job: read-only role"]
  PLAN --> GATE["Plan gate and human approval"]
  GATE --> APPLY["CI apply job: environment role"]
  PLAN --> ST["State: versioned, encrypted, locked"]
  APPLY --> ST
  APPLY --> RES["Provider: managed resources"]
  RES -.->|"read nightly"| DRIFT["Drift job: report only"]
  RES -.-> INV["Provider inventory and audit log"]
  INV --> COV["Coverage report"]
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class DEV,PR,CHK,PLAN,APPLY core
  class GATE decision
  class ST,REG,RES,INV,DRIFT,COV neutral

Alternatives. Managed runners (HCP Terraform and similar) replace the pipeline and often the backend. Kubernetes-hosted reconcilers (Crossplane, tofu-controller)^[18][28] replace push-based applies with continuous reconciliation (GitOps). Provider-native engines (CloudFormation, Bicep) have no state file of yours to protect, but serve one provider.

%%{init: {"sequence": {"actorMargin": 8, "width": 100, "noteMargin": 6, "messageMargin": 30, "diagramMarginX": 8}}}%%
sequenceDiagram
  participant G as Git host, people
  participant CI as Pipeline
  participant STS as Cloud STS
  Note over G: Pull request
  G->>CI: Plan job (PR)
  CI->>STS: Token, PR subject
  STS-->>CI: Read-only session
  CI-->>G: Change list on PR
  Note over G: Review, merge
  G->>CI: Plan job (main)
  CI->>STS: Token, main subject
  STS-->>CI: Read-only session
  CI-->>G: Change list on run
  Note over G: Approver reads it
  G->>CI: Apply job
  CI->>STS: Token, environment
  STS-->>CI: Apply session
  Note over CI: Lock, re-plan
  Note over CI: Compare, apply

Happy path: the pull-request plan, the post-merge plan and the apply-time plan describe the same changes. Exception paths:

• The post-merge plan differs from the pull request’s (something merged or applied in between): the approver, the only check at this step, rejects the run and the change returns to review (principle 3).
• State changed during the approval wait: the comparison fails and nothing is applied; re-run for a fresh plan and approval.
• The lock is held: the apply job waits (-lock-timeout), then fails. Never break a lock you did not take.^[29]
• Apply fails half-way: there is no transaction-style rollback; state records what succeeded. Fix forward with a new plan.
• Policy blocks the plan: fix the code or request a dated exception.

Split state by lifecycle (networks change rarely, services often), ownership (one team per state) and risk (data stores apart from states that change daily). Google Cloud’s guidance is “Don’t include more than 100 resources (and ideally only a few dozen) in a single state”, because every resource is refreshed on every run.^[30] Slack moved from one state per region to one per region in each account, isolating large services because smaller states are faster and safer; by 2022 it had close to 1,400 state files owned by different teams.^[26] Give each environment its own directory and backend key: Terraform’s documentation says CLI workspaces are not a suitable isolation mechanism when environments need different credentials.^[31]

Small states need each other’s values: the service state needs the network’s subnet IDs. terraform_remote_state reads another state’s outputs, but HashiCorp warns that its user “must have access to the entire state snapshot, which often includes some sensitive information”, and recommends publishing shared data separately.^[32] So producers write shared values to a parameter store (for example aws_ssm_parameter), or consumers find objects with provider data sources by name or tag; on HCP Terraform, tfe_outputs reads outputs without full state access. With remote state, the consumer’s plan role needs s3:GetObject on the producer’s state key, and so sees every secret in it. Record dependencies in the root module card (templates).

A module does one job (a network; a database with its alarms), exposes few inputs and validates them. Consumers pin a registry version or a Git tag or commit with ?ref=.^[33] The lock file pins providers, but “Terraform does not remember version selections for remote modules”, so module pins live in the code.^[34] Breaking changes get a major version and an upgrade note.

Run a scheduled plan per root module with -detailed-exitcode (exit code 2 = changes present).^[41] A refresh-only plan shows only what changed outside the tool; apply -refresh-only accepts reality into state without touching infrastructure.^[23] Classify before you act.

%%{init: {"flowchart": {"wrappingWidth": 115, "nodeSpacing": 14, "rankSpacing": 26}}}%%
flowchart TB
  D["Difference found by plan or inventory"] --> Q1{"In state?"}
  Q1 -->|"Yes"| Q2{"Intended?"}
  Q1 -->|"No"| U["Unmanaged: import, hand over or delete"]
  Q2 -->|"Yes, emergency fix"| K["Codify it in a pull request"]
  Q2 -->|"No"| Q3{"Made by a service or controller?"}
  Q3 -->|"No"| Q4{"Actor known?"}
  Q3 -->|"Yes"| I["ignore_changes, with owner and expiry"]
  Q4 -->|"Yes"| V["Revert by applying reviewed code"]
  Q4 -->|"No"| SEC["Security event: preserve evidence first"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  class Q1,Q2,Q3,Q4 decision
  class U,K,I,V core
  class SEC risk

Detection quality varies. CloudFormation compares only explicitly set properties, marks unsupported types NOT_CHECKED and can report false positives such as 1024 MB versus 1 GB;^[42] Bicep’s what-if warns that properties can be “incorrectly reported as deleted”.^[43] Budget triage time for false positives.

Bake operating system, agents and hardening into an image; Packer creates “identical machine images for multiple platforms from a single source configuration”.^[44] Test it, publish it to a channel and roll instances by replacement. AWS’s Well-Architected guidance forbids in-place updates, patches and configuration changes on production workloads under immutable infrastructure, citing reduced drift and the option to disable SSH.^[45] Configuration management still serves inside images and on long-lived hosts (AutomationOps).

Use the same modules everywhere and vary only inputs. The Twelve-Factor App asks for “the same type and version of each of the backing services”.^[46] Parity of versions matters more than of size: a smaller staging database on the same engine version is a fair test; a different engine is not.

Plan the destroy, check dependants, snapshot durable data, apply, then remove code and image channels. Lift deletion protection deliberately, in the same reviewed change.

Automate formatting, validation, plans, policy, drift jobs and image rebuilds in the pipeline; keep a human approval for production deletes and replacements. AI assistants draft modules and explain plans, and vendors expose registry documentation and workspace operations to agents through MCP servers.^[66] Generated code passes the same review and gates, and agents hold no apply credentials.

Assumptions: one AWS account per environment, Terraform 1.16 (not checked with OpenTofu), GitHub Actions with OIDC, the S3 backend with native locking, one root module per component and environment. example-org and ACCOUNT_ID are placeholders. Beginners can do the labs first and return here.

infra-live/                 # modules live in their own repository, by tag
  live/
    prod/network/           # one root module = one state = one plan
    prod/database/
    staging/network/
  tools/plan_gate.py        # Lab 3
  tools/drift.sh            # nightly drift report
  .github/workflows/        # one workflow per root module (or a matrix)
  CODEOWNERS                # owner review: live/prod/**, tools/**,
                            # .github/workflows/**

Example status: not executed and not machine-checked (no Terraform binary on the authoring host); written against the Terraform v1.16 documentation.

# live/prod/network/versions.tf
terraform {
  required_version = ">= 1.11.0, < 2.0.0" # S3 native locking is GA from 1.11

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.69"
    }
  }

  backend "s3" {
    # bucket: versioning on, created by a separate bootstrap stack
    bucket       = "example-tfstate-ACCOUNT_ID"
    key          = "prod/network/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true # replaces the deprecated DynamoDB lock table
  }
}

provider "aws" {
  region = "eu-west-1"
  default_tags {
    tags = { owner = "network-team", cost-centre = "cc-100", managed-by = "terraform" }
  }
}

module "network" {
  # pinned to a tag
  source     = "git::https://git.example.com/infra/modules.git//network?ref=v3.4.1"
  cidr_block = "10.40.0.0/16"
  az_count   = 3
}

The pipeline plans on every pull request and again after merge. The post-merge plan job posts its change list on the run’s summary page, where the production approver reviews the deployment;^[67] the apply job re-plans and applies only if the list still matches. The download step reads this run’s summary by default,^[68] so the code checks post-merge against apply time; post-merge against the pull request is the approver’s check. Plan files never travel between jobs: they hold state and variables,^[24] and repository readers can download artifacts.^[69] Concurrency is per job: a new push cancels its pull request’s stale plan; applies queue and never cancel, because by default a newly queued run cancels the pending one, leaving a merged change unapplied.^[70]

Example status: YAML syntax-checked by the Atlas validator; not executed (needs a GitHub repository, an AWS account and the two roles below). Action versions are the latest majors on 2026-10-16;^[68] pin full commit SHAs in production.

name: prod-network
on:
  pull_request:
    paths: ["live/prod/network/**"]
  push:
    branches: [main]
    paths: ["live/prod/network/**"]
permissions:
  contents: read
env:
  TF_IN_AUTOMATION: "1"
  TF_VERSION: "1.16.6" # one pin for both jobs
  WORKDIR: live/prod/network
jobs:
  plan:
    runs-on: ubuntu-24.04
    concurrency: # a new push to the same pull request cancels its stale plan
      group: plan-prod-network-${{ github.event.pull_request.number || github.run_id }}
      cancel-in-progress: true
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v7
      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: ${{ env.TF_VERSION }}
          terraform_wrapper: false
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::ACCOUNT_ID:role/tf-prod-network-plan
          aws-region: eu-west-1
      - name: Plan without the lock, gate, publish the change list
        working-directory: ${{ env.WORKDIR }}
        run: |
          terraform init -input=false
          terraform validate
          terraform plan -input=false -lock=false -out=tfplan
          terraform show -json tfplan > plan.json
          ALLOW=$(paste -sd, approved-destroys.txt 2>/dev/null || true)
          python3 "$GITHUB_WORKSPACE/tools/plan_gate.py" plan.json \
            --allow "$ALLOW" --summary-out plan-summary.json
          printf 'Change list to approve (full plan in the job log):\n\n' \
            >> "$GITHUB_STEP_SUMMARY"
          sed 's/^/    /' plan-summary.json >> "$GITHUB_STEP_SUMMARY"
          rm -f tfplan plan.json
      - uses: actions/upload-artifact@v7
        with:
          name: plan-summary
          path: ${{ env.WORKDIR }}/plan-summary.json
          retention-days: 7
  apply:
    if: github.event_name == 'push'
    needs: plan
    runs-on: ubuntu-24.04
    environment: production
    concurrency: # queue every merged apply in order; never cancel one
      group: apply-prod-network
      cancel-in-progress: false
      queue: max # up to 100 pending (GitHub docs, October 2026)
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v7
      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: ${{ env.TF_VERSION }}
          terraform_wrapper: false
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::ACCOUNT_ID:role/tf-prod-network-apply
          aws-region: eu-west-1
      - uses: actions/download-artifact@v8
        with:
          name: plan-summary # from this run's post-merge plan job
          path: approved
      - name: Re-plan with the lock, compare with the approved list, apply
        working-directory: ${{ env.WORKDIR }}
        run: |
          terraform init -input=false
          terraform plan -input=false -lock-timeout=5m -out=tfplan
          terraform show -json tfplan > plan.json
          ALLOW=$(paste -sd, approved-destroys.txt 2>/dev/null || true)
          python3 "$GITHUB_WORKSPACE/tools/plan_gate.py" plan.json \
            --allow "$ALLOW" --compare "$GITHUB_WORKSPACE/approved/plan-summary.json"
          terraform apply -input=false -lock-timeout=5m tfplan

Who can reach which credential. OIDC trust, wildcard subjects and GitHub’s 2026 immutable subject format are taught in DevSecOps; three points are IaC-specific.

• Any branch author can use the plan role. It trusts pull requests and main. Forks get no id-token: write unless an administrator allows write tokens for them, but a pull_request run executes the merge commit, edited workflow included,^[71] so anyone who can push a branch can run any command with the role. Plans run code anyway: data sources are read during planning,^[72] and external runs a program^[73] (this page’s inference).
• So the plan role only reads, but that includes every secret in state.^[74] It never locks, so it cannot delete a lock file: plans use -lock=false, which Terraform calls dangerous when others run commands concurrently;^[41] the cost here is a possibly stale plan, caught by the apply job’s locked re-plan.
• The apply role trusts only environment:production, so it is as strong as that environment: required reviewers, “prevent self-review”, a main-only deployment branch rule (which also keeps edited branch workflows out) and no administrator bypass (allowed by default). On GitHub Free, Pro and Team plans, required reviewers work only in public repositories.^[75]

Example status: JSON syntax-checked; not executed. The first document is the apply role’s trust policy. The second is the plan role’s state access: the S3 backend documents s3:ListBucket on the bucket, s3:GetObject and s3:PutObject on the state key, and get/put/delete on the .tflock key;^[27] the plan role keeps only the reads. Its trust policy lists the two subjects repo:example-org/infra-live:pull_request and repo:example-org/infra-live:ref:refs/heads/main. All three subjects use GitHub’s previous format; a repository created, renamed or transferred after 2026-07-15, or opted in, needs the immutable form repo:example-org@OWNER_ID/infra-live@REPO_ID:… instead.^[21]

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TrustProductionEnvironmentOnly",
      "Effect": "Allow",
      "Principal": {
        "Federated":
          "arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub":
            "repo:example-org/infra-live:environment:production"
        }
      }
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListState",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::example-tfstate-ACCOUNT_ID"
    },
    {
      "Sid": "ReadStateOnlyNoLock",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource":
        "arn:aws:s3:::example-tfstate-ACCOUNT_ID/prod/network/terraform.tfstate"
    }
  ]
}

The nightly drift job below reports and never applies. Run it once per environment (a matrix job), each time with that environment’s plan role, so no single identity reads every account’s state. Example status: bash syntax-checked, and run against a stub terraform that returned each exit code in turn; not run against real Terraform.

#!/usr/bin/env bash
# tools/drift.sh ENV: nightly drift report for the root modules of one
# environment, run with that environment's read-only plan role.
# Exit 0 = clean, 2 = changes found, 1 = an error in any module.
set -uo pipefail
env_name=${1:?usage: drift.sh ENVIRONMENT}
root=$(pwd)
status=0
for dir in "live/$env_name"/*/; do
  cd "$root/$dir" || { echo "error    $dir"; status=1; continue; }
  if ! terraform init -input=false -no-color > drift.log 2>&1; then
    echo "error    $dir (init failed, see drift.log)"; status=1; continue
  fi
  terraform plan -input=false -lock=false -detailed-exitcode -no-color \
    -out=drift.tfplan >> drift.log 2>&1
  rc=$?
  case "$rc" in
    0) echo "clean    $dir" ;;
    2) echo "changes  $dir"
       terraform show -json drift.tfplan \
         | python3 "$root/tools/plan_gate.py" /dev/stdin --max-changes 100000 || true
       if [ "$status" -eq 0 ]; then status=2; fi ;;
    *) echo "error    $dir (see drift.log)"; status=1 ;;
  esac
  rm -f drift.tfplan
done
exit "$status"

A golden-image template for Packer, with encrypted volumes, IMDSv2 (the session-oriented version of the EC2 instance metadata service)^[76] and a deprecation date about six months after each build. Example status: not executed and not syntax-checked (no Packer binary); options and functions checked against the Amazon plugin v1.8.3 and Packer documentation;^[77] confirm the current Ubuntu image name pattern with your provider.

packer {
  required_plugins {
    amazon = {
      version = "~> 1.8"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

variable "image_version" {
  type = string
}

source "amazon-ebs" "base" {
  region        = "eu-west-1"
  instance_type = "t3.small"
  ssh_username  = "ubuntu"
  ami_name      = "base-ubuntu-24.04-${var.image_version}"
  encrypt_boot  = true
  imds_support  = "v2.0"
  deprecate_at  = timeadd(timestamp(), "4380h") # about six months
  source_ami_filter {
    filters = {
      name                = "ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    owners      = ["099720109477"]
    most_recent = true
  }
  metadata_options {
    http_endpoint = "enabled"
    http_tokens   = "required"
  }
  tags = {
    image_version = var.image_version
    base_ami_name = "{{ .SourceAMIName }}"
  }
}

build {
  sources = ["source.amazon-ebs.base"]
  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get -y upgrade",
      "sudo apt-get -y install auditd unattended-upgrades",
    ]
  }
}

Labs 1 and 2 use the local provider (no cloud, no cost); Lab 3 needs only Python 3.

• Objective: see the three copies of the truth and a drift, with no cloud.
• Prerequisites: Terraform 1.7 or later (or OpenTofu, untested here), internet access to download the hashicorp/local provider, an empty directory. 20 minutes. Status: not executed by the author.

terraform {
  required_version = ">= 1.7.0"
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.9"
    }
  }
}

resource "local_file" "motd" {
  filename        = "${path.module}/out/motd.txt"
  content         = "Welcome. Owner: payments\n"
  file_permission = "0644"
}

1. Save the block as main.tf and run terraform init; the new .terraform.lock.hcl would be committed.^[34]
2. terraform plan -out=tfplan, read it, then terraform apply tfplan. Passing the saved plan is the approval.^[78]
3. Inspect the copies: cat out/motd.txt (reality), terraform state show local_file.motd (state), main.tf (code).
4. Create drift: echo "edited by hand" >> out/motd.txt.
5. Run terraform plan -refresh-only and read it, then terraform plan -detailed-exitcode; echo "exit=$?".
6. Decide: code wins (leave main.tf as it is), or reality wins (edit content in main.tf to the file’s new text). Either way, run terraform apply, then plan again.

• Expected output: step 2 plans one resource to add. In step 5 the normal plan exits 2 (changes present).^[41] This provider drops a file from state when its checksum differs,^[79] so the refresh-only plan reports it deleted outside Terraform and the normal plan re-creates it rather than updating it. That is why reality wins also needs an apply: the object must return to state, and the apply writes the same text back. Drift looks different in every provider.
• Validation: after the apply in step 6, terraform plan -detailed-exitcode exits 0.
• Troubleshooting: a state-lock error means another terminal is running Terraform; wait. Provider downloads need internet access or a mirror.
• Teardown: terraform destroy, then delete the directory, including terraform.tfstate* (never commit these).

• Objective: validate inputs, test a module with terraform test, and rename it without destroying anything.
• Prerequisites: Lab 1; Terraform 1.7 or later. 40 minutes. Status: not executed by the author.

Create modules/motd/main.tf:

variable "owner" {
  type        = string
  description = "Owning team: lowercase letters and hyphens"
  validation {
    condition     = can(regex("^[a-z][a-z-]{1,30}$", var.owner))
    error_message = "owner must be 2-31 lowercase letters or hyphens."
  }
}

resource "local_file" "motd" {
  filename = "${path.module}/out/motd-${var.owner}.txt"
  content  = "Owner: ${var.owner}\n"
}

output "path" {
  value = local_file.motd.filename
}

and modules/motd/tests/motd.tftest.hcl:

variables {
  owner = "payments"
}

run "content_names_owner" {
  command = plan
  assert {
    condition     = local_file.motd.content == "Owner: payments\n"
    error_message = "The file must name its owner."
  }
}

run "rejects_bad_owner" {
  command = plan
  variables {
    owner = "Payments Team"
  }
  expect_failures = [var.owner]
}

1. In modules/motd, run terraform init and terraform test. Both runs use command = plan, so nothing is created; the second passes because validation fails as expected.^[63]
2. In the parent directory, add a module "motd" block with source = "./modules/motd" and owner = "payments"; init and apply.
3. Rename the block to module "banner" and plan, without applying: one file destroyed, one created.
4. Add a moved block with from = module.motd and to = module.banner, and plan again.^[35]

• Expected output: step 1 reports two passing runs; step 3 shows one to add and one to destroy; step 4 shows the object moving to the new address with nothing added or destroyed.
• Validation: after applying step 4, terraform state list shows module.banner.local_file.motd.
• Troubleshooting: if no tests are found, check the .tftest.hcl suffix and the tests/ directory location.
• Teardown: terraform destroy; delete the directory.

• Objective: block plans that replace protected data stores, miss required tags or exceed a blast-radius limit, and produce the change summary the reference pipeline compares.
• Prerequisites: Python 3.8+. 30 minutes. Status: executed on 2026-10-16 against the hand-written sample below, shaped per Terraform’s documented plan JSON format;^[80] not yet run against real Terraform output.

Example status: executed (Python 3, synthetic input).

#!/usr/bin/env python3
"""Plan gate: read `terraform show -json tfplan` output and block risky plans.

Exit codes: 0 = pass, 1 = policy violation or summary mismatch, 2 = bad input.
"""
import argparse
import json
import sys

PROTECTED = {"aws_db_instance", "aws_rds_cluster", "aws_s3_bucket",
             "aws_dynamodb_table", "aws_kms_key", "aws_efs_file_system"}


def classify(actions):
    if actions in (["delete", "create"], ["create", "delete"]):
        return "replace"
    return actions[0] if len(actions) == 1 else "+".join(actions)


def main():
    p = argparse.ArgumentParser()
    p.add_argument("plan_json")
    p.add_argument("--max-changes", type=int, default=25)
    p.add_argument("--required-tags", default="owner,cost-centre")
    p.add_argument("--allow", default="",
                   help="comma-separated addresses approved for delete/replace")
    p.add_argument("--summary-out", help="write the reviewed change list here")
    p.add_argument("--compare", help="fail unless the change list equals this summary")
    a = p.parse_args()
    try:
        with open(a.plan_json, encoding="utf-8") as f:
            plan = json.load(f)
    except (OSError, ValueError) as e:
        print(f"ERROR cannot read plan JSON: {e}")
        return 2
    allow = {x for x in a.allow.split(",") if x}
    required = [t for t in a.required_tags.split(",") if t]
    changes, problems = [], []
    for rc in plan.get("resource_changes", []):
        if rc.get("mode") != "managed":
            continue
        kind = classify(rc["change"]["actions"])
        if kind == "no-op":
            continue
        addr = rc["address"]
        changes.append([addr, kind])
        protected = rc["type"] in PROTECTED and addr not in allow
        if kind in ("delete", "replace") and protected:
            problems.append(f"BLOCK {kind} of protected {addr} "
                            "(add to --allow only with a written approval)")
        after = rc["change"].get("after") or {}
        # tags_all includes the AWS provider's default_tags
        tags = after.get("tags_all", after.get("tags"))
        if kind in ("create", "update", "replace") and isinstance(tags, dict):
            missing = [t for t in required if t not in tags]
            if missing:
                problems.append(f"BLOCK {addr} lacks required tags: {', '.join(missing)}")
    if len(changes) > a.max_changes:
        problems.append(f"BLOCK {len(changes)} changes exceed the "
                        f"blast-radius limit of {a.max_changes}")
    changes.sort()
    drift = len(plan.get("resource_drift", []))
    counts = {}
    for _, kind in changes:
        counts[kind] = counts.get(kind, 0) + 1
    print(f"changes={len(changes)} {json.dumps(counts, sort_keys=True)} "
          f"drifted_objects={drift}")
    if a.summary_out:
        with open(a.summary_out, "w", encoding="utf-8") as f:
            json.dump(changes, f, indent=1)
    if a.compare:
        with open(a.compare, encoding="utf-8") as f:
            if json.load(f) != changes:
                problems.append("BLOCK the change list differs from the approved "
                                "plan summary: re-review")
    for line in problems:
        print(line)
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main())

Save the sample as sample-plan.json:

{
  "format_version": "1.2",
  "resource_drift": [
    {
      "address": "aws_security_group.web", "mode": "managed",
      "type": "aws_security_group", "name": "web",
      "change": {"actions": ["update"], "before": {}, "after": {}}
    }
  ],
  "resource_changes": [
    {
      "address": "aws_s3_bucket.logs", "mode": "managed",
      "type": "aws_s3_bucket", "name": "logs",
      "change": {
        "actions": ["no-op"],
        "before": {"tags": {"owner": "payments"}},
        "after": {"tags": {"owner": "payments"}}
      }
    },
    {
      "address": "aws_db_instance.main", "mode": "managed",
      "type": "aws_db_instance", "name": "main",
      "change": {
        "actions": ["delete", "create"],
        "before": {},
        "after": {"tags": {"owner": "payments", "cost-centre": "cc-42"}}
      }
    },
    {
      "address": "aws_instance.api[0]", "mode": "managed",
      "type": "aws_instance", "name": "api",
      "change": {
        "actions": ["create"],
        "before": null,
        "after": {"tags": {"owner": "payments"}}
      }
    },
    {
      "address": "aws_sqs_queue.jobs", "mode": "managed",
      "type": "aws_sqs_queue", "name": "jobs",
      "change": {
        "actions": ["create"],
        "before": null,
        "after": {
          "tags": {"Name": "jobs"},
          "tags_all": {"Name": "jobs", "owner": "payments", "cost-centre": "cc-42"}
        }
      }
    },
    {
      "address": "data.aws_ami.base", "mode": "data",
      "type": "aws_ami", "name": "base",
      "change": {"actions": ["read"], "before": null, "after": {}}
    }
  ]
}

1. python3 plan_gate.py sample-plan.json; echo "exit=$?"
2. python3 plan_gate.py sample-plan.json --allow aws_db_instance.main --required-tags owner --summary-out approved.json; echo "exit=$?"
3. python3 plan_gate.py sample-plan.json --allow aws_db_instance.main --required-tags owner --max-changes 2; echo "exit=$?"
4. With a real plan: terraform plan -out=tfplan && terraform show -json tfplan > plan.json, then run step 1 on plan.json.

• Expected output (as executed): step 1 prints changes=3 {"create": 2, "replace": 1} drifted_objects=1, blocks the database replacement and the instance missing cost-centre, and exits 1; the SQS queue passes because the AWS provider merges default tags into tags_all.^[81] Step 2 prints only the summary and exits 0. Step 3 reports the blast-radius limit of 2 and exits 1.
• Validation: approved.json lists three [address, action] pairs; the no-op bucket and the data-source read are excluded.
• Troubleshooting: only a saved plan (-out) can be shown as JSON; tags unknown until apply are absent from after, so the tag rule skips them.
• Limits: the summary compares which objects change and how, not attribute values: a tripwire, not a proof.
• Teardown: delete the files; a real plan.json contains state data, so never commit or upload it.

These metrics are this Atlas’s proposal; no standard defines them. Infrastructure applies also count towards DORA’s delivery metrics (DevOps).

Worked example (illustrative numbers). The provider inventory lists 1,840 in-scope resources and state files list 1,512 of them: coverage = 1,512 ÷ 1,840 = 82.2%. Over 30 days the nightly drift job finds unexplained differences in 11 of 140 root modules: drift incidence = 7.9%. Their ages at resolution, in hours, are 3, 5, 20, 26, 30, 48, 72, 96, 120, 200 and 400: the median is the sixth value, 48 hours, but four exceed a 72-hour target and the worst lasted almost 17 days. 450 of 600 instances run images under 30 days old: 75%. The useful reading is “328 resources nobody reviews, and one drift left for 17 days”, not “82%”.

Vanity metrics: lines of IaC, module counts, and resources tagged managed-by=terraform (tags can be copied by hand).

• State is a secret store you did not mean to build. State and plan files can hold passwords and API tokens.^[74] Restrict readers, encrypt, version, and keep secrets out: ephemeral resources (Terraform 1.10) and write-only arguments (1.11) are not persisted;^[37] OpenTofu (1.7+) encrypts state and plan files, but “cannot read your state file without the correct key”.^[50]
• Supply chain: commit .terraform.lock.hcl (provider hashes);^[34] pin module refs; take base images only from approved owners.
• Least privilege: read-only console access for humans; plan roles that only read; a logged, alerting break-glass role.
• Audit evidence: where NIST SP 800-53 applies, pull request, plan, approval and apply log are CM-3 change records; code and images the CM-2 baseline; dated exceptions CM-6 deviations; state plus the coverage report the CM-8 inventory.^[47] Obligations vary by jurisdiction and contract (not legal advice). Evidence collection: ComplianceOps; security policy: CloudSecOps, DevSecOps.

Tests: apply-mode tests and Terratest create billable resources;^[64] prefer plan-mode tests and mocks. Replacement: blue-green (a new copy beside the old, then a switch) briefly doubles capacity. Review: pull-request cost figures are estimates; spend decisions belong to FinOps. Removal: forgotten resources are usually unmanaged, so the coverage report finds savings. Expiring environments cut cost and idle energy use.

Capacity as code. Editorial split: forecasts and SLO-driven targets belong to SRE, quota policy to CloudOps; this page covers codifying the numbers. Autoscaling bounds (min_size, max_size) and quota values live in reviewed code; the AWS provider’s aws_servicequotas_service_quota requests an increase when the desired value exceeds the current one.^[83] Review a lowered maximum like a delete: it removes headroom.

• Coverage report: monthly; each unmanaged resource gets an owner and a decision.
• Out-of-band write alerts on human identities; an exceptions register with expiry dates.
• Rebuild drill: quarterly, build staging from an empty account; whatever fails is an undocumented dependency.
• State recovery drill: restore an old state version in a sandbox before you need to in an incident.

Blast-radius limits belong in tools. In February 2017 an authorised Amazon S3 engineer “using an established playbook executed a command” with one wrong input and removed far more servers than intended; AWS changed the tool “to remove capacity more slowly” and added minimum-capacity safeguards.^[85] A pipeline that can delete a hundred resources in one apply needs the same limit, as in the plan gate.

Illustrative scenario. A pull request upgrades the AWS provider and renames aws_db_instance.db to aws_db_instance.main. The plan gate prints changes=2 {"create": 1, "delete": 1} drifted_objects=0 and blocks it with BLOCK delete of protected aws_db_instance.db …; the create at .main passes (output from running the Lab 3 gate on a hand-written plan of this shape).

1. Stop. Nothing was applied; the gate worked. Do not allow-list the address to “get it through”.
2. Read the shape. A delete and a create at different addresses is a rename without moved: Terraform deletes the object at the old address.^[35] A ["delete", "create"] pair at one address is a forced replacement (BLOCK replace), and the plan marks the cause # forces replacement.^[41]
3. Fix the rename with a moved block from aws_db_instance.db to aws_db_instance.main; the plan should show a move and zero destroys.^[35]
4. Check the upgrade. If a replace of aws_db_instance.main now appears, the provider upgrade forces it: revert it, read its upgrade guide, and upgrade in a separate pull request.
5. If replacement is truly required (an attribute that cannot change in place), run a migration, not an apply: snapshot, restore to a new instance, switch the application, and delete the old one in a separate, approved change.
6. Verify that the final plan has no destroy or replace of a protected type, and get the summary approved.
7. Follow up: add prevent_destroy and provider deletion protection; move the database to its own state if it shares one.

• Skipping plan review because “the code is the documentation”. Code shows intent; only the plan shows effect.
• One giant state: slow plans, huge blast radius, lock contention.
• Applying from laptops: mixed engine versions, local credentials, no record. Laptops plan; pipelines apply.
• ignore_changes as a mute button. Every ignored attribute is drift you have agreed not to see; give it an expiry.
• Myth: “a clean plan means we know what exists.” It covers only managed objects; unmanaged ones need the inventory.
• Myth: “immutable means no patching.” Patching moves into the image pipeline, which must run on a schedule.

Use it for anything that lives beyond a few weeks, exists in several environments, is shared, or must be audited: networks, identities, data stores, clusters, DNS, monitoring.

Not, or not yet, for sandbox experiments with an expiry date, settings a managed platform owns end to end (codify only its configuration surface), or unstable APIs that would force monthly rewrites. Without version control and code review, start there: IaC without review is faster ClickOps (unreviewed console changes). Hidden costs: state to protect, upgrades to schedule, people who can read plans.

%%{init: {"flowchart": {"wrappingWidth": 110, "nodeSpacing": 14, "rankSpacing": 26}}}%%
flowchart TB
  S["A piece of infrastructure"] --> Q1{"Lives beyond a few weeks, or exists twice?"}
  Q1 -->|"No"| N1["Sandbox by hand, with an expiry date"]
  Q1 -->|"Yes"| Q2{"Fully owned by a managed platform?"}
  Q2 -->|"No"| Q3{"Declarative, stable provider API?"}
  Q2 -->|"Yes"| N2["Codify only the platform's settings"]
  Q3 -->|"No"| N3["Images or config management first"]
  Q3 -->|"Yes"| Y1["Declarative IaC with remote state"]
  Y1 --> Q4{"Many changers, or must self-correct?"}
  Q4 -->|"Yes"| Y2["Add a reconciler: GitOps or controllers"]
  Q4 -->|"No"| Y3["Plan, review and apply pipeline"]
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029
  class Q1,Q2,Q3,Q4 decision
  class Y1,Y2,Y3 core
  class N1,N2,N3 neutral

For the engine, weigh providers, licence (BSL for Terraform from 1.6.0, MPL 2.0 for OpenTofu),^[48][17] language, where state lives, managed services and hiring. Pick one default engine; allow exceptions by decision record.

This maturity model is the Atlas’s proposal, not an industry standard.

Before the first production apply

• State backend versioned, encrypted, locked, readers restricted
• Engine, providers and modules pinned; lock file committed
• Separate plan and apply identities with exact OIDC subjects
• CODEOWNERS and branch protection on production directories, tools/ and .github/workflows/
• Plan gate for deletes, replacements, tags and blast radius
• Deletion protection on every durable data store

Each change

• The plan was read, not just the diff; destroys and replacements explained
• The production approver compared the post-merge change list with the pull request’s
• The applied change list matches the approved one

Monthly

• Coverage reviewed; each unmanaged resource has an owner and a decision
• Drift ages within target; expired exceptions removed or renewed
• Engine, provider and image upgrades scheduled

Break-glass

• Hand change logged with ticket and time, then codified or reverted within the window
• Discussed at the postmortem (IncidentOps)

ROOT MODULE CARD          live/<env>/<component>
Owner team:               <team>        Reviewers: <CODEOWNERS group>
State key:                <bucket>/<env>/<component>/terraform.tfstate
Contains durable data:    yes/no  ->  deletion protection: <how>
Plan role / apply role:   <arn> / <arn>   OIDC subjects: <exact strings>
Depends on:               <published outputs, data sources>  Consumed by: <list>
Drift check:              nightly        Last rebuild drill: YYYY-MM-DD

DRIFT EXCEPTION  DX-NNN
Resource / attribute:  <address> / <attribute path>
Why it differs:        <service mutation | controller | emergency fix | vendor default>
Mechanism:             ignore_changes | removed (destroy=false) | unmanaged
Owner:                 <name>     Created: YYYY-MM-DD     Expires: YYYY-MM-DD
Removal condition:     <what must change for this exception to go away>

ADR: STATE LAYOUT
Context:      <estate, teams, environments, change frequency>
Decision:     one state per <component x environment>; data stores isolated;
              at most ~100 resources per state
Alternatives: single state per environment; CLI workspaces; managed stacks
Consequences: more published outputs between states; smaller blast radius;
              parallel plans
Review:       YYYY-MM-DD

Compared on documented behaviour only, as of October 2026.

• Orchestration across states. Terraform Stacks, generally available in HCP Terraform, deploy one set of components to many environments, each with its own state.^[91]
• Controllers for cloud resources. Crossplane and tofu-controller bring continuous reconciliation, or detection only, to infrastructure.^[18][28]
• AI-assisted IaC. Review, policy and separate apply identities matter more, not less; autonomous infrastructure agents are vendor direction, not established practice.

• Decide: one default engine; state split by lifecycle, owner and risk; which changes need a named approver.
• Practise: plan every pull request; apply only reviewed plans, only from the pipeline; replace compute from images.
• Protect: state readers, exact OIDC subjects, deletion protection on data.
• Measure: coverage, drift incidence and age, out-of-band writes, failed applies, blast radius, image age.
• Watch out for: unmanaged resources behind clean plans, unexpiring ignore_changes, laptop applies, wildcard trust.

Situation (illustrative). A payments platform has 140 root modules and a nightly drift job. An on-call engineer widened a security-group rule by hand during a 02:00 incident; the drift job flagged it and nobody acted for nine days. The security lead wants the drift job to re-apply the code automatically every night. The platform lead prefers alert and assign: an automatic revert at 03:00 could undo an emergency fix mid-incident.

Assumptions: emergency changes are rare but real; plans for most modules are small; the team can staff a daily drift triage.

A defensible decision: neither extreme. Auto-revert only modules without durable data, when the drift plan passes the plan gate and no incident is open; otherwise alert an owner with a 72-hour target and escalate. Tools support both: tofu-controller has a drift-detection-only model besides enforcement,^[28] HCP Terraform health assessments detect without remediating,^[86] and CloudFormation’s drift-aware change sets show whether a deployment would revert out-of-band changes made during incident response.^[89] Uncertainty remains about how often emergency fixes would be reverted; measure it.

Why the attractive choice can be wrong: blanket auto-revert turns the drift job into an unreviewed apply pipeline holding the strongest credentials in the estate. Re-evaluate after a quarter using drift age and reverted emergency fixes.

After this page you should be able to:

1. Explain the three copies of the truth and name the gap behind a given symptom.
2. Design a state layout and a plan-review-apply pipeline with separate, exactly scoped identities.
3. Operate drift detection and classify each difference before acting.
4. Refactor safely with moved, import and removed, and protect durable data.
5. Measure coverage, drift and image age, and decide what to fix first.

“InfraOps” is a thin label over a deep practice. What makes the practice work is not a particular engine; it is the discipline of keeping intent, record and reality in agreement and changing them only through plans that people actually read. The engines will keep changing (licences, managed services, AI assistants), but the hard problems stay the same: state you must protect, plans that run code, data you must not replace, and resources nobody manages. Teams that measure those gaps, and close them deliberately, get what infrastructure as code promised: environments they can rebuild and changes they can explain.

Each source was opened on the date its entry gives. The evidence ledger (research/infraops-ledger.md) maps claims to sources and records the quotation check.

• 2026-10-16: first edition. Lab 3’s plan gate executed against a synthetic plan file; HCL examples written against current documentation but not executed; attributed quotations matched against raw source text.
• 2026-10-17, review round 1: pipeline checks stated precisely (code compares post-merge with apply time; the approver compares pull request with post-merge), per-job concurrency, a plan role that never locks; new sections on sharing data between states and capacity as code; runbook, Lab 1, drift script, Packer date, licences and Bicep row corrected; OIDC basics linked to DevSecOps; figures narrowed for phones.
• 2026-10-17, post-score polish: OIDC example notes GitHub’s immutable subjects for repositories created after 2026-07-15; -target paraphrase matched to its source; figure 4 fits phones and its last note no longer spills; one Terraform pin in the workflow; Lab 3 sample JSON compacted (re-run, same output); “Prevent self-review” sourced to its how-to page; a repeated import caveat and the duplicate prevent_destroy myth cut.