KubernetesOps

KubernetesOps, as this Atlas uses the word, is the work of running Kubernetes clusters and the workloads on them: keeping the control plane, nodes and add-ons healthy and on supported versions, making workloads declare what they need so they schedule, scale and survive maintenance, and finding causes quickly when something breaks.

Kubernetes is a set of control loops: you declare a desired state, and controllers move the real state towards it.^[1] This work keeps the loops current, feeds them correct inputs and notices when they cannot converge. It is for cluster operators, platform engineers, site reliability engineers (SREs), application teams on clusters, and leaders deciding whether to run Kubernetes.

First, Kubernetes moves fast: about three minor releases a year, each patched for roughly 14 months, 12 of standard support and two of maintenance mode for security (CVE) and critical fixes.^[2][3] On 2026-10-16 the project maintained release branches for 1.37, 1.36 and 1.35, and 1.34 was in maintenance mode until its end of life on 2026-10-27.^[4][3]

Second, Kubernetes acts on what you declare. The scheduler places pods by their requests and refuses a full node “although actual memory or CPU resource usage on nodes is very low”;^[5] node autoscalers also ignore real usage.^[6] Inflated requests waste money; missing ones let a noisy pod starve its neighbours and leave the Horizontal Pod Autoscaler (HPA) blind.^[7]

Third, diagnosis. A failure may sit in a node, the control plane, the image, a probe, a Service selector, DNS, the Container Network Interface (CNI) plugin or a cloud quota; teams without a method restart things until the symptom fades. Adoption is broad: the Cloud Native Computing Foundation (CNCF) reports that “82% of container users now run Kubernetes in production”.^[8]

“KubernetesOps” is not a recognised discipline. We found no foundation, standard, certification or book that uses the one-word label, and it collides with three things:

1. kOps, officially “kOps - Kubernetes Operations” and “kubectl for clusters”: a tool that creates, upgrades and maintains clusters and their cloud infrastructure (AWS and GCE officially supported as of October 2026),^[9] one installer among several.^[10]
2. A consultancy’s methodology page, “KubernetesOps (K8sOps)”, that “automates critical cluster management tasks”:^[11] one firm’s framing, not an industry term.
3. KubeOps, “a Kubernetes Operator SDK designed for .NET developers”.^[12]

The practice behind the label is established as Kubernetes administration or cluster operations. The Linux Foundation’s performance-based Certified Kubernetes Administrator (CKA) exam (based on v1.37 as of October 2026) weights Troubleshooting 30 %, Cluster Architecture, Installation and Configuration 25 %, Services and Networking 20 %, Workloads and Scheduling 15 % and Storage 10 %.^[13]

Its boundaries with neighbouring pages are editorial distinctions, not industry rules (neighbour table); release strategy belongs to ReleaseOps.

Kubernetes descends from Google’s cluster managers Borg and Omega,^[14] a lineage its engineers described in “Borg, Omega, and Kubernetes” (ACM Queue, 2016).^[15] The first commit landed on 6 June 2014; version 1.0 shipped on 21 July 2015, when Google announced it would donate the project to the newly formed CNCF.^[14] Kubernetes 1.24 (2022) removed dockershim, so runtimes plug in through the Container Runtime Interface; kubeadm stopped applying the node-role.kubernetes.io/master label to new control-plane nodes, and kubeadm upgrade apply removed it from existing ones, a change behind one large outage (use case 1).^[16][17]

Mental model: a cluster is a database of desired state (etcd behind the API server) plus control loops. Operations asks three questions: is the machinery healthy and current, are the declarations correct, and why is a loop not converging?

The Atlas splits the container stack by layer: an editorial distinction, since one platform team often does all of it.

flowchart TB
  PO["PlatformOps: the platform as a product"]:::neutral
  KS["KubernetesSecOps: RBAC, admission, runtime"]:::risk
  GO["GitOps: delivery from Git"]:::neutral
  KO["KubernetesOps: clusters, nodes, add-ons, upgrades, scaling"]:::core
  CO["ContainerOps: images and runtimes"]:::neutral
  OB["ObservabilityOps: telemetry"]:::neutral
  ST["StorageOps: CSI and volumes"]:::neutral
  SO["SysOps and CloudOps: hosts, networks, accounts"]:::neutral
  PO --> KO
  KS -.->|hardens| KO
  GO -.->|delivers to| KO
  KO --> CO
  KO -.->|sends telemetry| OB
  KO -.->|takes volumes from| ST
  CO --> SO
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

No body publishes a KubernetesOps charter; this is the Atlas’s proposal.

Mission: keep every cluster on a supported version, able to absorb planned and unplanned disruption, and clear enough that any on-call engineer can find why a workload is not converging.

Principles

1. Declare, then verify convergence. An apply only records intent; check rollout status, Ready conditions and events.
2. Requests are a contract, declared for CPU and memory from measured need.
3. Upgrade continuously, one minor at a time, on a calendar rather than as projects.
4. Disruption is budgeted, not forbidden. Every replicated workload has a budget that allows at least one eviction.
5. Everything installed has an owner and a version, including add-ons, custom resource definitions (CRDs) and operators.
6. Clusters are rebuildable from code, and the rebuild or restore has been rehearsed.
7. Diagnose from evidence. Events, conditions and logs first; restarts and deletions only once the cause is known.

Non-goals: application code, security policy design (KubernetesSecOps), Kubernetes as the default for every workload. Example goals: version lag ≤ 1; no unapproved extended support; every production workload with requests, a readiness probe and a budget allowing one disruption; a restore rehearsed every six months.

Gateway API names three roles that generalise well: the infrastructure provider, the cluster operator (“policies, network access, application permissions”) and the application developer (“application-level configuration and Service composition”).^[31] The RACI below, this Atlas’s proposal, builds on that split.

A = accountable, R = responsible, C = consulted, I = informed.

• Day 0 (design): managed or self-managed control plane; cluster count; network plugin and IP plan; node pools; add-ons; version policy; backup scope.
• Day 1 (build): clusters and add-ons from code at pinned versions; the workload baseline; first workloads; a proven drain and restore before production.
• Day 2 (operate): calendar upgrades; add-on updates; rightsizing; certificate rotation; incidents; restore drills; retiring old versions, APIs and clusters.

• Version policy, owned by the cluster operator: target version, maximum lag, exception approver, and each cluster’s exit date from its minor. The upstream cadence means one hop per cluster roughly every four months.
• Change path: cluster and add-on changes go through version control and review, applied by a pipeline or reconciler (GitOps); emergency changes are written back to code.
• Admission rules such as “pods must declare requests”, designed with KubernetesSecOps.
• Cadence: the monthly review (toolkit checklist) and a post-incident review for every cluster-level incident (IncidentOps).

Certified Kubernetes offerings^[33] share the same logical parts and differ in who operates which part (figure below); self-managed, you run everything, including etcd.

flowchart LR
  U["kubectl, Helm, GitOps, CI"]:::neutral
  subgraph CP["Control plane: provider-run if managed"]
    CTRL["scheduler, controller-managers"]:::core
    API["kube-apiserver"]:::core
    ETCD[("etcd: all cluster state")]:::risk
  end
  ADD["Add-ons, yours: DNS, gateway, metrics, autoscaler"]:::neutral
  NODE["Worker nodes, usually yours: kubelet, runtime, CNI, proxy"]:::neutral
  U --> API
  CTRL --> API
  ADD --> API
  API --> ETCD
  API <--> NODE
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Pod networking is not built in: a CNI plugin assigns pod IPs and implements the flat pod network, and a NetworkPolicy has no effect unless the network implementation supports it.^[34][35]

Operating the CNI plugin. It has its own version matrix: kubeadm says “Manually upgrade your CNI provider plugin”,^[36] and AWS lists a VPC CNI version per EKS minor and recommends upgrading the add-on one of its own minor versions at a time.^[37] Pod addresses are capacity. Where the controller-manager allocates per-node pod ranges, each node takes a /24 by default,^[38] so a /16 cluster range holds 256 nodes. VPC-native plugins such as the Amazon VPC CNI give each pod a subnet address,^[37] and when subnets run out, “You will not be able to create new Pods or nodes”.^[39] Count addresses for pods, surge nodes and managed upgrades,^[40] and alert on free ones.

Design choices with trade-offs

The version skew policy sets the rules: the API server must not skip minors; highly available (HA) API servers stay within one minor of each other; kubelets are never newer than the API server and at most three minors older; the order is API server, then controller-manager, scheduler and cloud-controller-manager, then kubelets, then kube-proxy; and a node is drained before its kubelet moves to a new minor.^[43][36] Providers can be stricter. Amazon Elastic Kubernetes Service (EKS) says managed and Fargate nodes must match the control plane’s minor before the next control-plane update (the same guide elsewhere allows kubelets three minors behind: plan for the stricter rule).^[40] Add-on order varies too: kubeadm applies new DNS and kube-proxy manifests once the last control plane is upgraded, before kubelets; EKS updates nodes first, add-ons after.^[36][40]

flowchart TB
  T["Trigger: next minor available, or current minor near end of support"]:::neutral
  P["Preflight: deprecated APIs, add-on compatibility, skew, PDBs, backups"]:::core
  B{"Blockers found?"}:::decision
  F["Fix, or record an approved exception"]:::risk
  C["Upgrade control plane by exactly one minor"]:::core
  H{"Control plane healthy?"}:::decision
  RB["Re-run (kubeadm), use the provider's rollback window, or restore"]:::risk
  A["Add-ons (CNI, DNS, proxy): kubeadm now, EKS after nodes"]:::core
  N["Replace or upgrade node pools: surge, cordon, drain, uncordon"]:::core
  S{"Drain stalls or SLOs burn?"}:::decision
  X["Pause; fix the blocking PDB or workload; never force-delete blindly"]:::risk
  V["Validate: nodes Ready, rollouts complete, error rates normal"]:::core
  R["Record duration and findings; schedule the next hop"]:::neutral
  T --> P --> B
  B -->|yes| F --> P
  B -->|no| C --> H
  H -->|no| RB
  H -->|yes| A --> N --> S
  S -->|yes| X --> N
  S -->|no| V --> R
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Actors and outputs. The cluster operator runs it, consulting application owners whose budget or API use blocks it; the cloud team checks quotas and IP space (EKS needs up to five free subnet addresses^[40]). The output is a findings record.

Rollback is narrow, so plan it first. kubeadm upgrade apply rolls a control-plane component back if it fails to come up, keeps etcd and manifest backups under /etc/kubernetes/tmp, and can be re-run (it is idempotent).^[36] EKS cannot pause an upgrade once started, reverts if its health checks fail and, as of October 2026, allows rollback to the previous minor within seven days.^[40] Elsewhere the fallback is a restore or a replacement cluster.

Six playbooks carry most of the work; the cluster operator owns all but rightsizing (application teams, with the operator’s data).

Playbook 1: find deprecated API use before it breaks. Beta API versions stop being served nine months or three minors after deprecation, whichever is longer, and the migration guide lists each release’s removals (for example autoscaling/v2beta2 HPA in 1.26).^[44][45] Evidence comes from client warnings, the API server metric apiserver_requested_deprecated_apis (labelled removed_release) and audit events annotated k8s.io/deprecated;^[46] EKS upgrade insights scan a rolling 30-day window.^[40] The metric covers only requests since each API-server instance started: query every instance over a window longer than your slowest client’s cycle (monthly jobs, quarterly reports), or use audit events. Fix manifests and charts, then controllers. Done when no client requested a removed version during that window.

Example status: first query adapted from the Kubernetes blog’s example, second written for this page; neither executed (requires Prometheus scraping every API server).

# Deprecated APIs requested since each API-server instance started, with request counts:
apiserver_requested_deprecated_apis{removed_release="1.38"}
  * on(group, version, resource, subresource) group_right()
  apiserver_request_total

# Requested on any instance at any time in the last 35 days (needs 35 days of retention):
max by (group, version, resource, subresource) (
  max_over_time(apiserver_requested_deprecated_apis{removed_release="1.38"}[35d])
)

Playbook 2: drain a node without surprises. kubectl drain evicts through the Eviction API, so budgets and graceful termination are respected.^[27] A refused eviction returns 429 and is retried; a 500 can mean two budgets select one pod.^[47] Drains stick when a budget allows zero disruptions or pods never become Ready; the documentation recommends unhealthyPodEvictionPolicy: AlwaysAllow (the default is IfHealthyBudget).^[26][48] For stuck evictions it advises pausing and investigating; deleting the pod, which bypasses the budget, is its last resort, so agree it with the owner.^[47] Done when only DaemonSet pods remain and every evicted workload is back at its Ready count.

Playbook 3: rightsize requests. Run the Vertical Pod Autoscaler (VPA), an add-on, in Off mode, which writes recommendations without touching pods;^[49] agree changes with the owning team and ship them in code. Since v1.35, CPU and memory can be resized in place through the pod resize subresource, but the QoS class cannot change and memory decreases are best effort.^[50]

Playbook 4: autoscale in the right order. The HPA sets replica counts from metrics; the node autoscaler adds nodes only for Pending pods, and both it and consolidation judge by requests.^[7][6]

flowchart TB
  M["Metrics API: metrics-server or custom metrics"]:::neutral
  H["HPA sets replica count"]:::core
  V["VPA or people set requests"]:::core
  Q{"Do new pods fit on existing nodes, by requests?"}:::decision
  R["Pods scheduled and serving"]:::core
  P["Pods Pending"]:::risk
  N["Node autoscaler adds nodes sized for the requests"]:::core
  C["Consolidation removes nodes whose pods fit elsewhere"]:::neutral
  M --> H --> Q
  V --> Q
  Q -->|yes| R
  Q -->|no| P --> N --> R
  R --> M
  R --> C
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Two rules follow. HPA on CPU utilisation needs CPU requests, or “the autoscaler will not take any action for that metric”.^[7] And the VPA project says VPA “should not be used with” HPA on the same resource metric (CPU or memory); VPA on memory with HPA on CPU, or HPA on custom metrics, is allowed.^[51] A common pattern: HPA on load, VPA in Off mode for advice.

Playbook 5: back up what a managed service will not. Self-managed clusters snapshot etcd regularly; it holds all cluster state and “access to etcd is equivalent to root permission”.^[41] Volume data belongs to BackupOps. Either way, keep manifests in Git and a tested rebuild of namespaces, CRDs and add-ons.

Playbook 6: rotate certificates. kubeadm client certificates expire after a year; each control-plane upgrade renews them. Check with kubeadm certs check-expiration; after a manual kubeadm certs renew all, restart the control-plane pods.^[52]

Vendor-neutral categories first; named tools are examples checked in October 2026, not endorsements.

Controllers are already automation. Operators encode a human operator’s knowledge (backups, upgrades) in a controller with custom resources,^[61] each one a dependency to upgrade and trust. Declarative cluster lifecycle (Cluster API, provider APIs, infrastructure as code in InfraOps) makes creation and upgrade reviewed changes.

AI assistants can summarise events, propose a diagnosis or draft a manifest. Give them read access without Secrets (the built-in view role excludes them because they hold service-account credentials^[62]) and no exec or port-forward rights; require human approval for mutating commands; check claims against events, conditions and logs. Permission design belongs to IAMOps and KubernetesSecOps, automation governance to AutomationOps.

A baseline for one stateless HTTP service in a shared cluster. Assumptions: Kubernetes 1.35 or later; a Gateway public in gateway-system accepting routes from this namespace; metrics-server; nodes labelled topology.kubernetes.io/zone.

Example status: YAML syntax-checked by the Atlas validator; not schema-validated or applied to a cluster.

# Namespace guardrails: quota plus defaults, so pods without requests are still admitted with sane values.
apiVersion: v1
kind: Namespace
metadata:
  name: shop
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute
  namespace: shop
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.memory: 24Gi
    pods: "60"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: container-defaults
  namespace: shop
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        memory: 256Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: shop
  labels:
    app.kubernetes.io/name: web
spec:
  # No spec.replicas: the HPA owns the replica count.
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: web
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: web
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: web
      containers:
        - name: web
          image: registry.example.com/shop/web:1.4.2   # pin by digest in production
          ports:
            - name: http
              containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi          # no CPU limit: see the note below
          startupProbe:              # up to 24 x 5 s = 120 s to start
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 5
            failureThreshold: 24
          readinessProbe:            # may check dependencies; failing removes the pod from endpoints
            httpGet:
              path: /ready
              port: http
            periodSeconds: 5
          livenessProbe:             # process-local only; failing restarts the container
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 10
            failureThreshold: 3
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
  namespace: shop
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app.kubernetes.io/name: web
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
  namespace: shop
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 3
  maxReplicas: 12
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
---
apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: shop
spec:
  selector:
    app.kubernetes.io/name: web
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web
  namespace: shop
spec:
  parentRefs:
    - name: public
      namespace: gateway-system
  hostnames:
    - shop.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: web
          port: 80

Why each choice.

• No spec.replicas: with an active HPA, every apply of a manifest that sets it resets the count; removing it from an existing Deployment can briefly drop it to one replica, so follow the documented migration.^[7]
• Probes split by purpose: startup gives slow starters time (24 × 5 s); readiness controls traffic; liveness restarts only on unrecoverable, process-local failure, because wrong liveness probes “can lead to cascading failures”.^[25]
• maxUnavailable: 1 with AlwaysAllow: a drain evicts one pod at a time while the rest are healthy, and unhealthy pods never block it.^[26][48]
• ScheduleAnyway spread: the default DoNotSchedule guarantees spread but can leave pods Pending, partly because tainted nodes (cordoned or control-plane) count as domains unless nodeTaintsPolicy: Honor is set (stable since v1.33).^[63]
• Memory limit, no CPU limit: memory limits act through OOM kills, CPU limits through throttling.^[5] CPU limits are debated: in a 2020 post a Buffer engineer reported that removing them cut latency, blamed a kernel bug fixed in 4.19, and in an update advised upgrading the kernel first.^[64] With accurate CPU requests, this baseline sets no CPU limit; multi-tenant or licence-bound clusters may need one. ContainerOps explains throttling and why hosts without a scheduler keep CPU limits.

Cluster-side artefacts for self-managed kubeadm clusters. The first script wraps the documented commands for one minor hop: run it on each node in the documented order (first control plane, other control planes, workers), drain each node before its kubelet step, and first switch the node’s pkgs.k8s.io repository to the target minor.^[36]

Example status: bash -n passed; branches tested with a stub sudo (hold restored after a failed install; cgroup v1 host refused); never run against a cluster.

#!/usr/bin/env bash
# One-minor kubeadm upgrade helper (Debian/Ubuntu). Usage: TARGET=1.37.1 ./kubeadm-hop.sh first|node|kubelet
set -euo pipefail
TARGET="${TARGET:?set TARGET to the latest patch of the next minor, for example 1.37.1}"

install_pinned() {   # upgrade one package to TARGET; keep it on hold even if the install fails
  sudo apt-mark unhold "$1"
  if ! { sudo apt-get update && sudo apt-get install -y "$1=${TARGET}-*"; }; then
    sudo apt-mark hold "$1"
    return 1
  fi
  sudo apt-mark hold "$1"
}

case "${1:-}" in
  first)    # first control-plane node only; then upgrade the CNI plugin per its docs
    install_pinned kubeadm
    sudo kubeadm upgrade plan
    sudo kubeadm upgrade apply "v${TARGET}"
    ;;
  node)     # every other control-plane node and every worker
    install_pinned kubeadm
    sudo kubeadm upgrade node
    ;;
  kubelet)  # after: kubectl drain <node> --ignore-daemonsets ; then: kubectl uncordon <node>
    if [ "$(stat -fc %T /sys/fs/cgroup/)" != "cgroup2fs" ]; then
      echo "cgroup v1 on this node: kubelet 1.37+ will not start by default" >&2
      exit 1
    fi
    install_pinned kubelet
    install_pinned kubectl
    sudo systemctl daemon-reload
    sudo systemctl restart kubelet
    ;;
  *)
    echo "usage: TARGET=1.37.1 $0 first|node|kubelet" >&2
    exit 64
    ;;
esac

The second takes and verifies an etcd snapshot with the documented commands. The snapshot holds every Secret, so it is written with restrictive permissions and must leave the node encrypted. Prerequisites: root on a control-plane node, and the separate etcdctl and etcdutl tools (“you may need both”) at your cluster’s etcd version.^[41]

Example status: bash syntax-checked; not executed.

#!/usr/bin/env bash
# Take and verify an etcd snapshot on a control-plane node. Certificate paths are inputs
# (on kubeadm clusters, look under /etc/kubernetes/pki).
set -euo pipefail
: "${ETCD_CACERT:?path to the etcd CA certificate}"
: "${ETCD_CERT:?path to a client certificate trusted by etcd}"
: "${ETCD_KEY:?path to its private key}"
OUT_DIR="${OUT_DIR:-/var/backups/etcd}"
SNAP="${OUT_DIR}/etcd-$(date -u +%Y%m%dT%H%M%SZ).db"
umask 077
mkdir -p "${OUT_DIR}"
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert="${ETCD_CACERT}" --cert="${ETCD_CERT}" --key="${ETCD_KEY}" \
  snapshot save "${SNAP}"
etcdutl --write-out=table snapshot status "${SNAP}"
sha256sum "${SNAP}" > "${SNAP}.sha256"
echo "written ${SNAP}; copy it off this node, encrypted, and record it in the backup log"

To restore: stop all API servers, restore every etcd member, restart the API servers, then the scheduler, controller-manager and kubelets, so none acts on stale data.^[41]

Labs 1 and 2 run free on a laptop with kind, which runs Kubernetes nodes as containers.^[66] Their kubectl steps were not executed for this page (no container runtime on the authoring host); expected outputs come from the documentation. Lab 3 was executed.

• Objective: diagnose three common “my pod does not work” failures from evidence.
• Prerequisites: Docker or Podman, kind, and kubectl within one minor of the cluster;^[43] about 30 minutes; no risk outside the local cluster.

1. Create a cluster: kind create cluster --name ops-lab. kubectl now uses the context kind-ops-lab.^[66]
2. Save the manifests below as lab1.yaml and apply them with kubectl apply -f lab1.yaml. They contain three deliberate faults.

Example status: YAML syntax-checked; not applied.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bad-image
spec:
  replicas: 1
  selector:
    matchLabels: {app: bad-image}
  template:
    metadata:
      labels: {app: bad-image}
    spec:
      containers:
        - name: app
          image: nginx:this-tag-does-not-exist
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: too-big
spec:
  replicas: 1
  selector:
    matchLabels: {app: too-big}
  template:
    metadata:
      labels: {app: too-big}
    spec:
      containers:
        - name: app
          image: nginx:1.27
          resources:
            requests: {cpu: "64", memory: 128Mi}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels: {app: web}
  template:
    metadata:
      labels: {app: web}
    spec:
      containers:
        - name: app
          image: nginx:1.27
          ports: [{containerPort: 80}]
          readinessProbe:
            httpGet: {path: /, port: 80}
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector: {app: wbe}   # deliberate typo
  ports: [{port: 80, targetPort: 80}]

3. Look before you act: kubectl get pods, kubectl describe pod <name> for each unhealthy pod, then kubectl get events --field-selector type=Warning.^[67]
4. Check the Service: kubectl get endpointslices -l kubernetes.io/service-name=web.^[29]
5. Fix each fault in lab1.yaml (a real tag, a request that fits, the selector app: web) and re-apply.

• Expected output: bad-image waits in ErrImagePull or ImagePullBackOff.^[67] too-big stays Pending with a FailedScheduling warning about insufficient CPU, like the documented “0/42 nodes available: insufficient cpu”.^[5] The web pods are Ready, but the Service’s EndpointSlice lists no endpoints, because its selector matches no pods.^[29]
• Validation: after the fixes all pods are Ready, the EndpointSlice lists two addresses, and kubectl run tmp --rm -it --image=busybox:1.36 --restart=Never -- wget -qO- http://web prints the nginx welcome page.
• Troubleshooting: wrong cluster? kubectl config use-context kind-ops-lab. kind fails? Check the container runtime.
• Teardown: kind delete cluster --name ops-lab.

• Objective: see a disruption budget let a drain proceed, and a strict one block it.
• Prerequisites: as Lab 1; about 45 minutes; a three-node kind cluster from this file, saved as kind-3node.yaml:

Example status: YAML syntax-checked; not applied.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

1. kind create cluster --name ops-drain --config kind-3node.yaml.
2. kubectl create deployment web --image=nginx:1.27 --replicas=4, then kubectl create poddisruptionbudget web --selector=app=web --max-unavailable=1.
3. Note which worker hosts which pods: kubectl get pods -o wide.
4. Drain one worker: kubectl drain ops-drain-worker --ignore-daemonsets --timeout=120s.
5. All four pods now run on ops-drain-worker2. Uncordon the first worker (kubectl uncordon ops-drain-worker), make the budget impossible (kubectl delete pdb web, then kubectl create poddisruptionbudget web --selector=app=web --min-available=4) and drain the second worker: kubectl drain ops-drain-worker2 --ignore-daemonsets --timeout=60s.
6. Restore: kubectl uncordon ops-drain-worker2, then recreate the budget with --max-unavailable=1.

• Expected output: in step 4 pods are evicted one at a time while replacements become Ready on the other worker; the drain finishes and the node shows SchedulingDisabled. In step 5 every eviction is refused (429^[47]); kubectl retries until the timeout and leaves the node cordoned.
• Validation: kubectl get pdb web shows ALLOWED DISRUPTIONS 1 in step 4 and 0 in step 5.
• Troubleshooting: if drain refuses pods with emptyDir data, add --delete-emptydir-data only once the data is known to be disposable. In production, deleting a stuck pod bypasses the budget: a last resort (Playbook 2).^[26][47]
• Teardown: kind delete cluster --name ops-drain.

• Objective: turn the skew policy and disruption budgets into an automated go/no-go check before a minor hop.
• Prerequisites: Python 3.8+ (standard library only); read-only cluster access for three exports. Risk: none; it reads files.

1. Export the inputs: kubectl version -o json > version.json, kubectl get nodes -o json > nodes.json, kubectl get pdb -A -o json > pdbs.json.
2. Save the script as upgrade_preflight.py and run python3 upgrade_preflight.py --target 1.37 version.json nodes.json pdbs.json.

Example status: executed on 2026-10-16 with Python 3.8.10 against the illustrative JSON below, not a live cluster. The kubelet rule is the “three minor versions older” limit for kubelets 1.25 and later (older ones allow two).^[43]

#!/usr/bin/env python3
"""Read-only upgrade preflight for one Kubernetes cluster.

Inputs are JSON files you capture first:
  kubectl version -o json    > version.json
  kubectl get nodes -o json  > nodes.json
  kubectl get pdb -A -o json > pdbs.json
Usage: python3 upgrade_preflight.py --target 1.37 version.json nodes.json pdbs.json
Exit code: 0 no blockers, 1 blockers found, 2 unreadable input.
"""
import argparse
import json
import re
import sys

VERSION = re.compile(r"v?(\d+)\.(\d+)")


def minor_of(text):
    """Return (major, minor) from strings such as 'v1.36.5' or '1.37'."""
    match = VERSION.match(text or "")
    if not match:
        raise ValueError(f"cannot parse version {text!r}")
    return int(match.group(1)), int(match.group(2))


def load(path):
    with open(path, encoding="utf-8") as handle:
        return json.load(handle)


def check(server, target, nodes, pdbs):
    blockers, warnings = [], []
    if target[0] != server[0] or target[1] - server[1] not in (0, 1):
        blockers.append(f"control plane {server[0]}.{server[1]} -> {target[0]}.{target[1]}: "
                        "kube-apiserver must not skip minor versions")
    for node in nodes:
        name = node["metadata"]["name"]
        kubelet = minor_of(node["status"]["nodeInfo"]["kubeletVersion"])
        lag = target[1] - kubelet[1]
        if kubelet > target:
            blockers.append(f"node {name}: kubelet 1.{kubelet[1]} is newer than the target")
        elif lag > 3:
            blockers.append(f"node {name}: kubelet 1.{kubelet[1]} would be {lag} minors behind "
                            "(maximum 3); replace or upgrade this node first")
        elif lag == 3:
            warnings.append(f"node {name}: kubelet 1.{kubelet[1]} reaches the 3-minor limit; "
                            "it must move before the next control-plane hop")
        if node.get("spec", {}).get("unschedulable"):
            warnings.append(f"node {name}: already cordoned; find out why before you start")
    for pdb in pdbs:
        ref = f'{pdb["metadata"]["namespace"]}/{pdb["metadata"]["name"]}'
        status = pdb.get("status", {})
        if status.get("expectedPods", 0) == 0:
            warnings.append(f"pdb {ref}: selects no pods; check its selector")
        elif status.get("disruptionsAllowed", 0) == 0:
            blockers.append(f"pdb {ref}: allows 0 disruptions (healthy "
                            f"{status.get('currentHealthy')}/{status.get('expectedPods')}, needs "
                            f"{status.get('desiredHealthy')}); node drains will stall")
    return blockers, warnings


def main(argv=None):
    parser = argparse.ArgumentParser(description="Kubernetes upgrade preflight (read-only)")
    parser.add_argument("--target", required=True, help="target minor version, for example 1.37")
    parser.add_argument("version")
    parser.add_argument("nodes")
    parser.add_argument("pdbs")
    args = parser.parse_args(argv)
    try:
        server = minor_of(load(args.version)["serverVersion"]["gitVersion"])
        target = minor_of(args.target)
        nodes = load(args.nodes)["items"]
        pdbs = load(args.pdbs)["items"]
    except (OSError, KeyError, ValueError) as exc:
        print(f"ERROR unreadable input: {exc}", file=sys.stderr)
        return 2
    blockers, warnings = check(server, target, nodes, pdbs)
    for line in blockers:
        print("BLOCKER", line)
    for line in warnings:
        print("WARN   ", line)
    print(f"summary: 1.{server[1]} -> {args.target}; {len(nodes)} nodes, {len(pdbs)} PDBs; "
          f"{len(blockers)} blockers, {len(warnings)} warnings")
    return 1 if blockers else 0


if __name__ == "__main__":
    sys.exit(main())

Illustrative inputs, trimmed to the fields the script reads:

Example status: JSON syntax-checked; these exact files were used for the executed run.

{"clientVersion": {"major": "1", "minor": "37", "gitVersion": "v1.37.1"},
 "serverVersion": {"major": "1", "minor": "36", "gitVersion": "v1.36.5"}}

{"items": [
  {"metadata": {"name": "cp-1"}, "spec": {}, "status": {"nodeInfo": {"kubeletVersion": "v1.36.5"}}},
  {"metadata": {"name": "pool-a-1"}, "spec": {}, "status": {"nodeInfo": {"kubeletVersion": "v1.36.5"}}},
  {"metadata": {"name": "pool-b-1"}, "spec": {"unschedulable": true}, "status": {"nodeInfo": {"kubeletVersion": "v1.34.12"}}},
  {"metadata": {"name": "legacy-1"}, "spec": {}, "status": {"nodeInfo": {"kubeletVersion": "v1.33.9"}}}
]}

{"items": [
  {"metadata": {"namespace": "shop", "name": "web"}, "status": {"expectedPods": 3, "currentHealthy": 3, "desiredHealthy": 2, "disruptionsAllowed": 1}},
  {"metadata": {"namespace": "payments", "name": "ledger"}, "status": {"expectedPods": 2, "currentHealthy": 2, "desiredHealthy": 2, "disruptionsAllowed": 0}},
  {"metadata": {"namespace": "batch", "name": "old-report"}, "status": {"expectedPods": 0, "currentHealthy": 0, "desiredHealthy": 1, "disruptionsAllowed": 0}}
]}

• Expected output (actual output of the executed run; exit code 1):

BLOCKER node legacy-1: kubelet 1.33 would be 4 minors behind (maximum 3); replace or upgrade this node first
BLOCKER pdb payments/ledger: allows 0 disruptions (healthy 2/2, needs 2); node drains will stall
WARN    node pool-b-1: kubelet 1.34 reaches the 3-minor limit; it must move before the next control-plane hop
WARN    node pool-b-1: already cordoned; find out why before you start
WARN    pdb batch/old-report: selects no pods; check its selector
summary: 1.36 -> 1.37; 4 nodes, 3 PDBs; 2 blockers, 3 warnings

• Validation: run it again with --target 1.38; it must add a blocker that the API server cannot skip a minor. A missing input file must exit with code 2 (both checked in the executed run).
• Troubleshooting: provider suffixes such as v1.36.5-eks-… are fine; the parser reads the leading major.minor. Budget status is a snapshot. The script applies the upstream skew limit; on EKS, treat any managed or Fargate node behind the control plane as a blocker (workflow).
• Extension: add the Playbook 1 query and a cgroup v1 check. Teardown: delete the exports; they contain node names and addresses.

1. A minor upgrade that broke pod networking (documented case). Two independent summaries of Reddit’s postmortem describe its 14 March 2023 outage: during a 1.23 → 1.24 upgrade of its oldest, largest cluster, the node-role.kubernetes.io/master label disappeared from control-plane nodes, the network plugin’s route reflectors selected nodes by that label, and cross-node pod traffic failed. Recovery took about five hours, from an etcd backup.^[68][69] Reddit’s own account, quoted in one summary, says the restore procedure “hadn’t been kept fully up to date” and had never been used against a production cluster.^[69] The removal had been announced.^[16][17] Lessons: search add-on configuration for labels and APIs the next release removes; rehearse upgrades and restores on a production-like cluster. Success metric: zero upgrade-caused incidents; restore drills within the recovery objective.

2. Retiring Ingress NGINX (real event, illustrative organisation). Twelve clusters run Ingress NGINX, which gets no releases or security fixes after March 2026; its repository was archived on 2026-03-24.^[70][71] Inventory with kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx; generate Gateway API resources with ingress2gateway print --providers=ingress-nginx --all-namespaces; review every warning, because the tool is “a migration assistant, not a one-shot replacement” and some annotations (proxy-body-size) have no equivalent;^[72] run old and new entry points side by side and move DNS gradually. Trade-off: Gateway API splits ownership between cluster operators (Gateways) and application teams (routes).^[31] Success metric: every hostname on Gateway API with unchanged error rates and latency; the old controller removed by a set date.

3. A noisy neighbour (illustrative). Batch pods without requests crowd out a latency-sensitive service. Give every pod requests (quota plus LimitRange), keep the service within its requests, and give it a higher PriorityClass: under node pressure the kubelet evicts pods above their requests first, by Priority.^[32][20] Success metric: no node-pressure evictions of the service in a month.

4. A GPU node pool for inference (illustrative). A tainted pool, the vendor’s device plugin, and GPUs in limits (requests, if set, must equal limits);^[73] for sharing, Dynamic Resource Allocation (advanced topics). Serving belongs to MLOps, GPU cost to FinOps for AI. Success metric: GPUs requested ÷ present above target; no CPU-only pods on GPU nodes.

Worked example 1: HPA arithmetic. The HPA computes desiredReplicas = ceil(currentReplicas × currentMetric ÷ targetMetric) and skips scaling within the tolerance, 0.1 by default.^[7] With 4 replicas at 90 % average CPU and a 60 % target, the ratio is 1.5: ceil(4 × 1.5) = 6 replicas. At 64 % the ratio is 1.067, inside the tolerance: no change. Scaling down waits for the 300-second stabilisation window.^[7]

Worked example 2: allocation versus usage (illustrative data). The documentation’s example node, 16 CPUs and 32 Gi, keeps 14.5 CPUs allocatable after reservations.^[24] Ten nodes give 145. Pods request 120 CPUs: allocation 120 ÷ 145 = 82.8 %. Measured p95 usage is 42 CPUs, a usage-to-request ratio of 35 %: 78 requested CPUs, over five nodes’ worth, sit idle. Consolidation will not help: the Cluster Autoscaler by default considers a node for removal only when its requests are below 50 % of allocatable.^[6][75] If teams rightsize requests to p95 × 1.3 ≈ 55 CPUs and you aim for 75 % allocation, you need 55 ÷ (14.5 × 0.75) ≈ 5.1, so six nodes instead of ten by CPU. Repeat for memory (28.5 Gi allocatable per node) and pods per node; the largest count wins.

Targets and cadence. Set targets per service class (say, usage-to-request 50–80 % for steady services); review monthly. “Number of clusters” and “pods running” are vanity metrics.

Security design belongs to KubernetesSecOps; these duties stay here. Patch currency: AWS notes that after about a year “the Kubernetes community stops releasing common vulnerabilities and exposures (CVE) patches” for a version,^[74] and add-ons age too (use case 2). etcd and its backups hold every Secret: restrict, encrypt, log access. Compliance evidence (upgrade records, drills, audit logs) feeds ComplianceOps.

Node count follows requests, so request accuracy is the main lever (worked example 2). Other drivers: idle headroom per pool, many small clusters each with its own control plane and add-ons, and version debt: EKS extended support costs $0.60 per cluster-hour instead of $0.10, $4,380 more per cluster-year (prices as of October 2026).^[76] OpenCost^[77] allocates cluster cost by namespace and label for FinOps; fewer idle nodes also mean less energy (GreenOps).

Leading indicators: version lag ≤ 1; no zero-disruption budgets outside approved windows; under 2 % of production pods without requests; inventory matching clusters. Lagging: upgrade-caused incidents, node-pressure evictions, forced drains. Check them automatically (Lab 3, admission rules, GitOps drift reports) and settle exceptions monthly.

Start one level up: when nodes or the control plane fail, workload symptoms are echoes; triage the cluster first, then the workload (Lab 1).

flowchart TB
  S["Cluster symptom: kubectl get nodes"]:::neutral
  A{"Does the API server answer?"}:::decision
  CP["Self-managed: control-plane static pods, their logs, certificates, etcd. Managed: provider status"]:::risk
  WR{"Writes rejected across namespaces?"}:::decision
  WH["Admission webhook unreachable (failurePolicy Fail), or etcd space alarm"]:::risk
  N{"Any node NotReady or Unknown?"}:::decision
  K["On the node: kubelet, container runtime, network"]:::risk
  PR{"Memory, disk or PID pressure?"}:::decision
  PE["Free disk, fix requests, reserve system resources"]:::risk
  OK["Nodes healthy: use the workload tree"]:::core
  S --> A
  A -->|no| CP
  A -->|yes| WR
  WR -->|yes| WH
  WR -->|no| N
  N -->|yes| K
  N -->|no| PR
  PR -->|yes| PE
  PR -->|no| OK
  CP ~~~ WH ~~~ K ~~~ PE
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

A NotReady or Unknown node is tainted, and a default toleration keeps its pods bound for 300 seconds before eviction (DaemonSet pods stay).^[78][22] On kubeadm clusters the control-plane components are static pods defined in /etc/kubernetes/manifests; read their logs with crictl on the node when kubectl cannot answer.^[79][80] While the API server is down, existing pods and Services keep working unless they depend on the Kubernetes API, but nothing can be started, stopped or updated.^[78]

Example status: commands from the Kubernetes documentation; not executed here.

# From an admin workstation
kubectl get nodes
kubectl describe node NODE_NAME                 # Conditions, taints, events
kubectl get events -A --field-selector involvedObject.kind=Node
# On the node (SSH or console)
systemctl status kubelet
journalctl -u kubelet --since "30 min ago" | tail -n 50
sudo crictl ps -a                               # is the runtime up, which containers exited?
sudo crictl logs CONTAINER_ID                   # for example a control-plane static pod
ls /etc/kubernetes/manifests                    # kubeadm control-plane manifests

flowchart TB
  S["Workload does not serve: describe the Deployment, get pods"]:::neutral
  C{"Pods created?"}:::decision
  FC["FailedCreate: quota exceeded, or a webhook rejects or is down"]:::risk
  Q{"Scheduled?"}:::decision
  P["FailedScheduling: requests, taints, affinity, volume"]:::risk
  WN{"ContainerCreating?"}:::decision
  NW["FailedMount: missing Secret or ConfigMap volume, or a volume that will not attach. Else pod network: CNI pod, free pod IPs"]:::risk
  WP{"Image or config error?"}:::decision
  WI["Image pull, or missing ConfigMap or Secret for env"]:::risk
  WL{"CrashLoopBackOff?"}:::decision
  WC["logs --previous, exit code, OOMKilled"]:::risk
  R{"Ready?"}:::decision
  R1["Readiness probe: path, port or dependency"]:::risk
  E{"Service endpoints?"}:::decision
  E1["Selector does not match pod labels"]:::risk
  E2["Check DNS, NetworkPolicy, Gateway route"]:::core
  S --> C
  C -->|no| FC
  C -->|yes| Q
  Q -->|no| P
  Q -->|yes| WN
  WN -->|yes| NW
  WN -->|no| WP
  WP -->|yes| WI
  WP -->|no| WL
  WL -->|yes| WC
  WL -->|no| R
  R -->|no| R1
  R -->|yes| E
  E -->|no| E1
  E -->|yes| E2
  FC ~~~ P ~~~ NW ~~~ WI ~~~ WC ~~~ R1 ~~~ E1
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Illustrative scenario, modelled on use case 1. As a new pool’s first nodes join, cross-node pod traffic times out; same-node traffic works.

1. Stabilise (first 10 minutes). Stop the node rollout and draining automation; old nodes are your working capacity. Declare an incident (IncidentOps).
2. Scope. Confirm: cross-node fails, same-node works, only new nodes are involved.
3. Ask what changed. Diff a new and an old node: labels, taints, kubelet version, network add-on pods.
4. Find the dependency. Search add-on configuration for selectors on differing labels.
5. Mitigate with the smallest reversible fix: cordon new nodes and move workloads back, or patch the add-on selector (hand-labelling nodes is only a stopgap).
6. Verify from a debug pod on each pool, then resume one node at a time.
7. Prevent: a preflight that diffs labels and selectors against the next release’s removals.

Example status: commands syntax-checked; not executed.

kubectl get nodes -L kubernetes.io/os,node.kubernetes.io/instance-type --show-labels > nodes-labels.txt
kubectl -n kube-system get daemonsets -o wide
kubectl -n kube-system get pods -o wide --field-selector spec.nodeName=NEW_NODE_NAME
kubectl get events -A --field-selector type=Warning --sort-by=.metadata.creationTimestamp | tail -n 30
kubectl debug node/NEW_NODE_NAME -it --image=busybox:1.36
# afterwards: kubectl delete pod <the node-debugger-NEW_NODE_NAME-xxxxx pod it created>

kubectl debug node/… creates a node-debugger-… pod with the node’s root filesystem at /host, in the host network, IPC and PID namespaces:^[88] effectively host access, so it needs rights to create pods and stays until you delete it.

Use it when many services or teams share infrastructure, you want one deployment model across clouds or on-premises, or need fine scheduling control (GPUs, data locality, edge), and can staff upgrades and on-call.

Avoid it, or take the most managed form, when a few stateless services would fit a managed container or serverless platform (ServerlessOps), nobody can own three upgrade cycles a year, or the driver is fashion. The hidden costs are the version treadmill, the add-on inventory, request hygiene and tested restores; a managed control plane removes etcd and API server operations, not the rest (NoOps estimates what remains).

flowchart TB
  A{"Many services or teams, portability or special scheduling needs?"}:::decision
  X["Do not adopt: use a managed container or serverless platform"]:::neutral
  B{"Can you staff about three upgrade cycles a year and on-call?"}:::decision
  Y["Adopt only the most managed form, or wait"]:::risk
  C{"Need control of the control plane: air-gap, edge, custom builds?"}:::decision
  Z["Managed control plane; you own nodes, add-ons, workloads"]:::core
  W["Self-managed: kubeadm, Cluster API, kOps or Kubespray; you own etcd"]:::core
  A -->|no| X
  A -->|yes| B
  B -->|no| Y
  B -->|yes| C
  C -->|no| Z
  C -->|yes| W
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

To compare managed and self-managed options, score each 1–5 on operating effort (40 %), control and placement needs (25 %), cost including staff (20 %) and support obligations (15 %), weights this Atlas suggests, and record an architecture decision (template below).

This Atlas’s model, not an industry standard; each exit criterion is observable.

• Days 1–30: see the estate. Inventory clusters, versions, end dates, add-ons and owners; run Lab 3 everywhere; write the version policy. Quick win: fix zero-disruption budgets.
• Days 31–60: make change routine. Cluster and add-on config in code; publish the workload baseline; rehearse one hop on a production-like cluster; restore one backup. Milestone: first hop done from the runbook.
• Days 61–90: measure and enforce. Admission rule for requests (with KubernetesSecOps); KPI dashboard; first rightsizing round; a twelve-month upgrade calendar. Milestone: version lag ≤ 1 in production.

Upgrade preflight

• Target is one minor above the control plane; release notes and deprecated-API guide read
• No requests to APIs removed in the target, on any API server, over a window longer than the slowest client’s cycle
• Add-ons pinned to versions compatible with the target
• Kubelets within skew after the hop; nodes three minors behind go first (on EKS: managed nodes on the control plane’s minor)
• Before 1.37: every node on cgroup v2 (stat -fc %T /sys/fs/cgroup/ prints cgroup2fs)
• No zero-disruption budgets; surge capacity and free IP addresses available
• Backup taken; restore path and platform rollback limits written down
• Window, owner and communication agreed

Workload readiness

• Requests on every container; memory limit set; CPU limit decision documented
• Startup, readiness and liveness probes with distinct purposes
• PDB allowing at least one disruption; spread across nodes and zones
• HPA with no spec.replicas in the manifest, or a fixed count by design

Monthly review

• Version lag, add-on drift and days to end of support per cluster
• Usage-to-request ratios and idle capacity
• Blocked or forced drains, node-pressure evictions, restart hot spots
• Exceptions approved, closed or escalated

Template status: illustrative; adapt names and numbers.

# Add-on inventory (one entry per installed component; keep it in the cluster-config repository)
cluster: prod-eu-1
kubernetes_minor: "1.36"
addons:
  - name: cni-plugin
    version: "<pinned version>"
    owner: platform-network
    compatible_minors: ["1.35", "1.36", "1.37"]
    upgrade_notes: "check node selectors and labels before each hop"
  - name: gateway-controller
    version: "<pinned version>"
    owner: platform-edge
    compatible_minors: ["1.36", "1.37"]
    upgrade_notes: "Gateway API CRD version must match the controller"

CLUSTER UPGRADE PLAN
Cluster / environment:            Current -> target minor:
Owner:                            Window (UTC):          Communication channel:
Preflight result (attach):        Blockers fixed:        Exceptions (approver, expiry):
Add-ons to change (name: from -> to):
Node strategy: replace nodes | upgrade drained nodes      Max nodes in flight:
Rollback or restore path and its time limit:
Validation: nodes Ready | rollouts complete | error rate within SLO for 30 min
Result and duration:              Findings for next preflight:

ARCHITECTURE DECISION RECORD: CLUSTER TOPOLOGY
Context: workloads, teams, regions, compliance scope
Options considered: (A) ... (B) ...
Scores (effort 40 %, control 25 %, cost 20 %, support 15 %):
Decision and consequences (who owns what, upgrade cadence, cost):
Review date:

Neighbours and control planes are compared in the overlap table and the reference architecture.

Is “KubernetesOps” a job title or certification? Not a standard one; titles vary (platform engineer, SRE, cluster operator), and the certification for this work is the CKA.^[13]

Can we skip minor versions to catch up? Not on the API server, and kubeadm does not support it;^[43][36] the alternative is a new cluster on the target version (judgement call).

Do budgets protect against every disruption? No. Node-pressure eviction ignores them and preemption respects them only on a best effort basis.^[20][21]

Do we need etcd backups on a managed service? Providers set their own duties: EKS says it “regularly backs up all managed clusters” and points to AWS Backup for customer-controlled backups of cluster state and volumes;^[40] check your provider. You still need configuration in Git, data backups (BackupOps) and a tested rebuild.

Is Ingress going away? The API is frozen, with “no plans to remove” it; the project recommends Gateway API.^[30]

Established, but new to many teams:

• Accelerators. Dynamic Resource Allocation lets workloads claim and share devices through device classes and claims (stable since v1.35).^[95]
• Resizing without restarts: in-place pod resize is stable since v1.35, and VPA’s in-place modes build on it.^[50][49]

Treat “self-driving cluster” claims as marketing until they show what happens to nodes, add-ons and deprecated APIs.

• Prerequisites: Linux, networking basics, containers (ContainerOps).
• One day: build a kind cluster; deploy the reference baseline; break and fix it (Lab 1).
• One week: Labs 2 and 3; the skew policy and deprecated-API guide; describe, logs --previous, crictl.
• Thirty days: a kubeadm cluster on VMs; a minor upgrade; an etcd snapshot and restore; the CKA domains as syllabus.^[13]
• Specialise: networking, autoscaling and capacity, fleet lifecycle, GPUs, or security (KubernetesSecOps).
• Portfolio ideas: a fleet upgrade preflight; a KPI dashboard; a timed restore drill write-up.

Example status: commands from the Kubernetes documentation; not executed here.

kubectl get pods -A --field-selector status.phase=Pending
kubectl describe pod POD -n NS
kubectl logs POD -n NS -c CONTAINER --previous
kubectl get events -n NS --field-selector type=Warning
kubectl get endpointslices -n NS -l kubernetes.io/service-name=SVC
kubectl drain NODE --ignore-daemonsets --timeout=10m
kubectl uncordon NODE
kubectl get pdb -A
sudo kubeadm upgrade plan
sudo kubeadm certs check-expiration

Situation (illustrative; prices and dates real as of October 2026). Six EKS clusters run 1.33, whose standard support ended 2026-07-29; extended support runs to 2027-07-29 at an extra $0.50 per cluster-hour.^[74][76] The team must at least leave extended support.

Reasoning. A looks cheap next to engineer time, and that is the trap: the fee buys time, not progress, and ends in the least controlled upgrade possible. Between B and C, statefulness and drift decide: stateless clusters rebuilt from code make C one migration that also proves the rebuild; persistent volumes and hand-made integrations favour B, starting now. Default: C for stateless clusters, B for stateful ones, A only as a dated exception covering the migration. Uncertainty: effort per hop is unknown until the first; measure it and revisit.

You can now explain the label, plan an upgrade within the skew policy, write a workload baseline, diagnose from nodes down to Services, size requests from data, and argue for or against adoption.

1. A kubeadm cluster runs 1.35 and the team wants 1.37 next week. What is the plan?

2. A drain has been “evicting” the same pod for 20 minutes. Name two likely causes and the safe fix.

3. Nodes are 85 % allocated by requests but CPU usage is 30 %. Why does the node autoscaler not remove nodes?

4. A worker shows NotReady, yet its pods still appear on it three minutes later. Is the control plane broken? What do you check?

5. Your organisation calls its cluster team “KubernetesOps”. What should a new joiner’s reading list include, and why is the label ambiguous?

Under an ambiguous label sits a well-documented practice: feeding control loops correct declarations on versions the project still patches. Four habits carry it. Upgrade on a calendar, not in a crisis. Treat requests as the contract every scheduler and autoscaler believes. Give every workload a budget that lets maintenance happen. Diagnose from nodes and events before restarting anything. Without these habits a managed control plane will not save you, because the work that remains is exactly this work.

Sources were opened on 2026-10-16, or on 2026-10-17 where marked; Kubernetes docs are undated live pages for v1.37.

Changelog: 2026-10-16 — first version. 2026-10-16 — r1 after independent review: node and control-plane troubleshooting, priority and preemption, cgroup v2 and EKS upgrade rules, corrections. 2026-10-17 — r2 after the final score: admission, quota and CNI paths in both triage trees, CNI operation, attribution and link fixes. 2026-10-17, post-score polish: volume causes of ContainerCreating, OpenCost source, VPC CNI wording, trims.