KubernetesSecOps

KubernetesSecOps, as this Atlas uses the word, is the day-to-day work of keeping Kubernetes clusters and their workloads secure. It covers who may change the cluster (authentication and RBAC), what may run in it (Pod Security and admission policy), what may talk to what (NetworkPolicy), where secrets live (encryption and external stores), and how attacks get noticed (audit logs and runtime detection).

Its centre of gravity is the two moments when a cluster can say no or raise its hand: admission, before an object is stored, and runtime, while a container runs.

Kubernetes defaults favour running workloads. Secret objects “are stored unencrypted in etcd” unless you configure encryption,^[1] every pod accepts all connections until a NetworkPolicy selects it,^[2] and the API server logs nothing without an audit policy file.^[3] The kubelet’s command-line defaults accept anonymous requests and authorise everything;^[4] its configuration-file defaults differ (the defaulting code turns anonymous access off and sets Webhook authorisation),^[5][6] and pods can reach the cloud metadata API unless you block it.^[7] Installers and managed services change several of these.

Add-ons hold privileges too: in March 2025 one flaw in ingress-nginx’s admission webhook, CVE-2025-1974 (rated 9.8), combined with four sibling flaws, could let “anything on the Pod network” take over a cluster, and the controller could by default read all Secrets cluster-wide (use case 1).^[8]

In Red Hat’s 2024 survey of 600 DevOps, engineering and security professionals (all respondents, self-reported), 67 % had delayed or slowed deployment of container-based applications over security concerns, and nearly 9 in 10 organisations reported at least one container or Kubernetes security incident in the previous 12 months, spanning runtime (45 %), build and deployment (44 %) and detected misconfigurations (40 %).^[9]

“KubernetesSecOps” is not a recognised discipline. Searches on 2026-10-16 for it, “KubeSecOps” and “K8sSecOps” found only guides to “Kubernetes DevSecOps” and “Kubernetes security”. The practice is established under that second name: the Kubernetes documentation’s Security section,^[10] the CIS Benchmark,^[11] the OWASP Kubernetes Top 10^[12] and the CKS exam, which certifies “best practices for securing Kubernetes platforms”.^[13] Hence overlapping synonym: the page teaches Kubernetes security under the Atlas’s name for it.

The 4Cs are an older map. Older Kubernetes documentation taught “the 4C’s of Cloud Native security” (Cloud, Clusters, Containers, Code), where “each layer of the Cloud Native security model builds upon the next outermost layer”.^[14] Current English documentation follows the CNCF lifecycle phases: Develop, Distribute, Deploy, Runtime.^[15] The 4Cs say where a control sits, the phases say when it acts; this page dwells on Deploy (admission) and Runtime.

Boundaries, an editorial distinction of the Atlas, not an industry standard: DevSecOps produces scans and signatures, which this page verifies at admission because kubectl apply, operators and Helm can bypass a pipeline; the neighbour table has every other handoff.

Misconceptions. A namespace scopes names, RBAC and policy but is not a hard isolation boundary: pods share nodes and kernels, and cluster-level isolation “remains the strictest workload isolation mechanism”.^[16] Base64 “is not an encryption method”.^[1]

Controls moved into the API server (from webhooks to built-in admission) while threats moved out to the supply chain and the add-ons around the cluster.

The mental model is the request path: every change is an API request, each step is a control you own, and a few paths skip the API server. Admission never sees reads, which “bypass the admission control layer”, so only RBAC protects Secrets from being listed.^[24]

flowchart TB
  C["Client: person, CI job or controller"]:::neutral --> AN["1. Authentication: who is calling?"]:::core
  AN --> AZ["2. Authorisation (RBAC): is this verb allowed? Reads stop here"]:::core
  AZ -->|"writes, exec"| MU["3. Mutating admission: add defaults"]:::core
  MU --> VA["4. Validating admission: Pod Security, CEL policies, webhooks"]:::decision
  VA --> ST["5. Stored in etcd, encrypted at rest"]:::core
  ST -->|"scheduled; the kubelet watches the API server"| KL["6. Kubelet starts the pod: seccomp, capabilities, user namespace"]:::core
  KL --> RT["7. Runtime detection watches the running container"]:::decision
  RT ~~~ AU["Audit log: API requests, steps 1 to 5"]:::neutral
  AU ~~~ BE["Bypass, not audited: direct etcd access"]:::risk
  BE -.-> ST
  BE ~~~ BK["Bypass, not audited: kubelet API, runtime socket, static pods"]:::risk
  BK -.-> KL
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

The project documents these bypasses: direct kubelet and etcd access is “not subject to admission control and is not logged”,^[22] and write access to etcd “is equivalent to gaining root on the entire cluster”.^[7]

The Atlas’s proposal (no foundation publishes one), distilled from the project’s checklist and good-practice pages.

Mission: clusters refuse insecure workloads by default, exceptions are visible and temporary, and runtime anomalies reach a human fast enough to contain them.

1. Deny by default where it is cheap: default-deny networking, the strictest Pod Security level a namespace can bear, no standing cluster-admin for people or pipelines.
2. Check in the pipeline, guarantee in the cluster: the same rules run in CI (DevSecOps) for feedback and at admission as the guarantee.
3. Every control has an owner, a mode (audit, warn, enforce) and an expiring exception path.
4. Design for the control failing: decide each webhook’s failure mode per tier in advance (judgement call) and rehearse recovery.
5. Prefer built-in mechanisms: the project “recommends that you use CEL-based admission control when possible”.^[28]
6. Assume something gets in: audit logs, runtime detection and a rehearsed containment runbook.
7. Treat add-ons as privileged tenants: inventory, least privilege, CVE and maintenance tracking.

Non-goals: pipeline scanners, cloud IAM design, staffing the SOC, Restricted everywhere regardless of the workload.

R = responsible, A = accountable, C = consulted, I = informed; small companies merge the platform and security columns.

Who may accept residual risk is set by the exception policy SecOps governs; the business owner signs. The exception path is the most important interface: without a fast, fair answer, teams route around the controls.

• Day 0 (design). Threat-model tenants, exposure and data; choose the identity provider, a network plugin that enforces NetworkPolicy, an encryption provider, an audit-log destination and, if needed, a policy engine.
• Day 1 (build). Harden control plane and kubelets, run the CIS checks, create namespaces from a baseline template, start admission rules in audit mode and runtime detection.
• Day 2 (operate). Promote rules, run exceptions, review RBAC, tune detections, respond to security releases, rehearse runbooks: most of the work and the value.

Policies live in a reviewed, tested repository like application code; Kyverno runs the same policies at admission and as a CLI scanner.^[30]

• Exception register: rule, scope, reason, compensating control, risk owner, expiry; expired entries are removed or escalated.
• Break-glass: a logged emergency bypass with a named approver and automatic review.
• Cadence: weekly rejection and alert review; monthly exception review; quarterly RBAC review and benchmark scan (the Atlas’s suggestion; the checklist’s floor is “no more than 24 months apart”^[29]); a yearly tabletop with the SOC.

On a managed service the control-plane parts are provider settings (CloudOps teaches the split).

flowchart TB
  IDP["People via OIDC"]:::neutral --> API
  CI["CI with its own workload identity"]:::neutral --> API
  API["API server: RBAC,<br/>Pod Security, CEL rules"]:::core
  API --> WH["Policy webhook,<br/>highly available"]:::decision
  API --> ETCD["etcd: API server only,<br/>mutual TLS"]:::core
  API --> KUB["Kubelets: authenticated,<br/>Webhook authz"]:::core
  WH -.-> REG["Registry: image signatures"]:::neutral
  ETCD ~~~ KMS["KMS v2 plugin and key,<br/>used by the API server"]:::core
  API -.-> KMS
  KUB --> POD["Pods: restricted,<br/>default-deny network"]:::core
  POD --> SEN["Runtime sensor on every node"]:::decision
  SEN --> SIEM["SIEM and SOC: sensor alerts<br/>and API audit log"]:::neutral
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Trust boundaries where real incidents start:

• Human access: “For production clusters with multiple users directly accessing the Kubernetes API, it is recommended to use external authentication sources such as OIDC”; client certificates “cannot be individually revoked”.^[31]
• Pod network: often reachable from far beyond the cluster; treat in-cluster services, webhooks included, as exposed.^[8]

Most of the work is changing rules for many teams at once; the Atlas recommends this path for every new guardrail.

flowchart TB
  P["Proposal: threat, audit finding or incident action"]:::neutral --> W["Write the rule as code, with tests and an owner"]:::core
  W --> A["Audit mode; count violations per namespace"]:::core
  A --> F{"Violations fixed or excepted?"}:::decision
  F -->|"no"| T["Teams fix manifests"]:::core
  T --> F
  T -->|"cannot comply"| RQ["Request an exception"]:::core
  RQ --> X{"Risk owner accepts?"}:::decision
  X -->|"yes, with expiry"| E["Record in the exception register"]:::neutral
  X -->|"no"| T
  E --> F
  F -->|"yes"| WN["Warn for two weeks; publish the enforce date"]:::core
  WN --> EN["Enforce; watch rejections and webhook errors"]:::core
  EN --> R{"Unexpected rejections?"}:::decision
  R -->|"yes"| RB["Back to warn; fix the rule"]:::risk
  R -->|"no"| M["Steady state: monthly exception review"]:::core
  RB --> WN
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

Use case 2 shows the happy path; the exception path is a log-collection DaemonSet that needs hostPath: a dedicated namespace at a lower level, an owner, compensating controls, an expiry.

Owners: the RACI; checks: the fundamentals table. The playbooks:

RBAC. The rule of thumb: a permission is dangerous when it lets a subject become someone else or read what someone else may read. Grant at namespace level, avoid wildcards, keep cluster-admin for named break-glass use, and never add users to system:masters, which “bypasses all RBAC rights checks”.^[32] Treat as privileged: list/watch on Secrets; writing pods or any workload resource, which “implicitly grants access to many other resources in that namespace, such as Secrets, ConfigMaps” and every service account there; exec, attach and ephemeral containers; escalate, bind, impersonate, nodes/proxy and token minting; certificate signing request (CSR) approval, which mints client certificates; webhook configurations; and patch on namespaces, which can relax Pod Security labels (Lab 2 checks each).^[32] Set automountServiceAccountToken: false where a pod never calls the API.^[23]

Pod Security. Two traps: enforcement applies to pods, not to Deployments (warn and audit do check workload resources), and exempting a controller’s service account “would implicitly exempt any user that can create the corresponding workload resource”.^[20] Restricted adds non-root, no privilege escalation, a seccomp profile, a short list of volume types, and dropping ALL capabilities, with only NET_BIND_SERVICE allowed back.^[33]

Network. A deny-all egress policy “also blocks DNS traffic”;^[2] for logging, explicit deny or encryption, see the FAQ. Block the cloud metadata address in egress rules.^[29]

Secrets. Use a KMS v2 provider (KMS v1 is deprecated since v1.28 and disabled by default since v1.29);^[27] aescbc is rated “Weak”, and a locally stored key “fails to protect against a host compromise”.^[34] External Secrets Operator writes external values into a Kubernetes Secret,^[35] while the Secrets Store CSI Driver mounts them as a volume.^[36]

Admission and provenance. Prefer images referenced by digest and, for sensitive namespaces, verify signatures at deploy time^[29] against a pinned signer identity (use case 3).

Control plane and nodes. The API server’s own flag defaults are permissive (AlwaysAllow authorisation, anonymous requests and profiling on),^[37] and installers differ, so check the running flags (reference implementation). etcd should use mutual TLS with a certificate authority “unique to etcd”,^[29] because any certificate from a CA that etcd trusts “allows full access to the data stored inside etcd”.^[22] Restrict the etcd and kubelet ports to trusted addresses,^[22] enable NodeRestriction and consider DenyServiceExternalIPs.^[29] Re-run the CIS checks (kube-bench, as a Job) after every build and upgrade.^[38]

Audit and runtime. Ship audit logs where cluster users cannot change them (“protected from general access”).^[29][3] Managed defaults differ (October 2026): EKS sends no control-plane logs until each type is enabled, at CloudWatch prices;^[39] GKE always keeps Admin Activity audit logs, but Data Access logs are off by default.^[40] Falco loads only stable rules by default; start runtime detection there and tune by scoped overrides, never by disabling rules (Lab 5).^[41]

None of these is law; auditors’ expectations are a ComplianceOps question.

Statuses checked on 2026-10-16 and 2026-10-17; commercial suites (CNAPP, CWPP) are compared in CloudSecOps.

Policy as code is automation by design; automated response needs care. Automate evidence capture, relabelling and cordoning; leave deletion to a person, because it destroys evidence and does not revoke every token (runbook). Language models can draft policies or summarise alerts; test their output like code; no agent gets standing cluster-admin (AIOps for Security).

Assumptions: Kubernetes v1.36.5, v1.37.1 or later (patched for CVE-2026-2270, in which namespace-scoped write on StatefulSets and ControllerRevisions could create pods in other namespaces;^[54] as of October 2026), a network plugin that enforces NetworkPolicy, a cluster DNS whose pods carry the label k8s-app: kube-dns (check yours), and an OIDC group team-shop.

Example status: YAML syntax-checked; not applied to a cluster.

# 1. Namespace: enforce Restricted, pinned to your cluster's minor (v1.37 here);
#    warn and audit on latest.
apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.37
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    security.example.com/tier: prod
---
# 2. Default deny for every pod in the namespace, both directions.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: shop
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
# 3. Allow DNS to the cluster DNS pods only; quarantined pods excluded (see the runbook).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: shop
spec:
  podSelector:
    matchExpressions:
      - {key: quarantine, operator: DoesNotExist}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - {protocol: UDP, port: 53}
        - {protocol: TCP, port: 53}
---
# 4. The web tier: ingress only from the ingress controller's namespace; TCP 443 out to
#    any address except the cloud metadata endpoint. 0.0.0.0/0 also covers your pod and
#    Service CIDRs: add them to `except` if this must stay Internet-only.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-flows
  namespace: shop
spec:
  podSelector:
    matchLabels: {app: web}
  policyTypes: ["Ingress", "Egress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: gateway-system
      ports:
        - {protocol: TCP, port: 8080}
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except: ["169.254.169.254/32"]
      ports:
        - {protocol: TCP, port: 443}
---
# 5. The team manages its workloads. It cannot list Secrets or exec into pods, but its
#    Deployment rights let it mount, and so read, any Secret in shop (see the note below).
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: shop-developer
  namespace: shop
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["pods", "pods/log", "services", "configmaps", "events"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: shop-developers
  namespace: shop
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: shop-developer
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: team-shop
---
# 6. A workload that satisfies Restricted and needs no API token.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: web
  namespace: shop
automountServiceAccountToken: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: shop
spec:
  replicas: 2
  selector:
    matchLabels: {app: web}
  template:
    metadata:
      labels: {app: web}
    spec:
      serviceAccountName: web
      securityContext:
        runAsNonRoot: true
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: web
          image: registry.example.com/shop/web@sha256:<64-hex-digest-from-your-pipeline>
          ports: [{containerPort: 8080}]
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: ["ALL"]}
          resources:
            requests: {cpu: 100m, memory: 128Mi}
            limits: {memory: 128Mi}

The Role is not a Secret boundary: Deployment rights create pods that can mount any Secret or ConfigMap in shop and run as any service account there,^[32] and pods/log returns what such a pod prints; keep Secrets the team must not see, and powerful service accounts, in namespaces it cannot deploy to. The namespace selectors use kubernetes.io/metadata.name, an immutable label set by the control plane.^[55] The memory limit equals the request, as the checklist suggests.^[29]

The control-plane and node side, for clusters you configure yourself:

Example status: YAML syntax-checked; not applied. Flag names from the Kubernetes reference; file paths are examples.

# /etc/kubernetes/encryption.yaml  (kube-apiserver --encryption-provider-config)
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - kms:
          apiVersion: v2
          name: cloud-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - identity: {}   # lets the API server read data written before encryption
---
# /etc/kubernetes/audit-policy.yaml  (kube-apiserver --audit-policy-file)
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  - level: None                      # noisy, low-value system traffic first
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: ""
        resources: ["endpoints", "services"]
  - level: Metadata                  # never log Secret or ConfigMap bodies
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  - level: RequestResponse           # who changed which permission, in full (writes only)
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
  - level: Metadata                  # everything else, including pods/exec requests
---
# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous: {enabled: false}   # config-file default; the flag default is true
  webhook: {enabled: true}
  x509: {clientCAFile: /etc/kubernetes/pki/ca.crt}
authorization:
  mode: Webhook                 # config-file default; the flag default is AlwaysAllow
readOnlyPort: 0
seccompDefault: true   # RuntimeDefault for every workload; test first, some break
---
# kubeadm ClusterConfiguration excerpt (kubeadm init --config). Mount the files it
# names into the API server with apiServer.extraVolumes (not shown).
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
apiServer:
  extraArgs:
    - {name: authorization-mode, value: "Node,RBAC"}       # flag default: AlwaysAllow
    # authentication-config must not be combined with --anonymous-auth
    - {name: authentication-config, value: /etc/kubernetes/authn.yaml}
    - {name: enable-admission-plugins, value: "NodeRestriction,DenyServiceExternalIPs"}
    - {name: profiling, value: "false"}                    # flag default: true
    - {name: service-account-extend-token-expiration, value: "false"}   # see below
    - {name: audit-policy-file, value: /etc/kubernetes/audit-policy.yaml}
    - {name: encryption-provider-config, value: /etc/kubernetes/encryption.yaml}
# etcd: kubeadm's local etcd has its own CA (pki/etcd/ca.crt) for client and peer TLS.
---
# /etc/kubernetes/authn.yaml: anonymous requests reach health endpoints only.
apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
anonymous:
  enabled: true
  conditions:
    - path: /livez
    - path: /readyz
    - path: /healthz

The first matching audit rule sets the level.^[3] seccompDefault: true applies RuntimeDefault everywhere; some workloads then fail.^[56][57] The kubelet block spells out configuration-file defaults because the same options default the other way as flags.^[5][6] identity keeps old plaintext readable until all Secrets are rewritten.^[34] The authentication file (stable since v1.34) limits anonymous access to health checks.^[58] With expiry extension on (the default), injected service-account tokens last up to a year;^[37] before turning it off, check that clients reload their tokens (EKS keeps a 90-day extension and marks stale tokens in its audit log).^[59] Whoever runs the control plane, review bindings to system:anonymous and system:unauthenticated yourself.^[32]

The labs use kind, which enforces NetworkPolicy since v0.24.0 (v0.33.0 was current when checked).^[60][61] Only Lab 2 was executed; expected output for Labs 1, 3 and 4 is summarised from the Kubernetes documentation, and Lab 5’s from the Falco documentation and rules file.

• Objective: see how labels change admission and why an enforced level can leave a Deployment with no pods.
• Prerequisites: Docker or Podman, kind, kubectl; about 30 minutes. Risks: none outside the local cluster.

Example status: syntax-checked, not executed (requires a local kind cluster).

kind create cluster --name pss-lab
kubectl create namespace example
kubectl label --overwrite ns example \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/enforce-version=latest \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/warn-version=latest
kubectl apply -n example -f https://k8s.io/examples/security/example-baseline-pod.yaml
# Now tighten enforcement and create a Deployment that runs nginx as root:
kubectl label --overwrite ns example pod-security.kubernetes.io/enforce=restricted
kubectl create deployment root-nginx -n example --image=nginx
kubectl get deploy,rs,pods -n example
kubectl describe rs -n example -l app=root-nginx | tail -5

• Expected output: the first apply warns that the pod would violate restricted:latest (privilege escalation, capabilities, runAsNonRoot, seccomp) and prints pod/nginx created.^[62] Relabelling warns that the existing pod violates Restricted.^[63] create deployment warns and succeeds, but the Deployment stays at 0/1: ReplicaSet events show pods forbidden by PodSecurity.^[20]
• Validation: only the original nginx pod exists, still running: enforcement does not evict existing pods.
• Troubleshooting: no warning means the label did not apply (--show-labels) or the cluster predates v1.25.
• Teardown: kind delete cluster --name pss-lab.

• Objective: turn the RBAC good-practice list into a first-pass automated review (a clean run is not a clean cluster).
• Prerequisites: Python 3.8+; read access to RBAC objects; 30 minutes. Risks: read-only, but the export maps every permission in the cluster: handle it as sensitive.

Example status: executed with Python 3.8.10 against the sample file below on 2026-10-17; not run against a live cluster.

#!/usr/bin/env python3
"""rbac_risks.py - flag risky RBAC grants in `kubectl get ... -o json` output.

kubectl get clusterroles,roles,clusterrolebindings,rolebindings -A -o json > rbac.json
python3 rbac_risks.py rbac.json      # exit 0 = clean, 1 = findings, 2 = bad input
Built-in bindings (label kubernetes.io/bootstrapping=rbac-defaults) are skipped
for system subjects unless --include-system is given. Anyone who can create
bindings can forge that label, and a "system:" name proves nothing.
A first pass, not proof: API groups and resourceNames are ignored.
"""
import json
import sys

# (verbs that trigger, resources, reason) - from Kubernetes "RBAC good practices".
WRITE = {"create", "update", "patch"}
WORKLOADS = {"pods", "deployments", "replicasets", "statefulsets", "daemonsets",
             "jobs", "cronjobs", "replicationcontrollers"}
CHECKS = [
    ({"get", "list", "watch"}, {"secrets"}, "reads Secret values"),
    (WRITE, WORKLOADS, "writes {res} (any SA, any Secret)"),
    # get is enough for WebSocket exec and attach before Kubernetes v1.35
    ({"get", "create"}, {"pods/exec", "pods/attach"}, "{res}: runs commands in pods"),
    (WRITE, {"pods/ephemeralcontainers"}, "adds debug containers to pods"),
    ({"escalate"}, {"roles", "clusterroles"}, "escalate on {res}"),
    ({"bind"}, {"roles", "clusterroles"}, "bind on {res}"),
    ({"impersonate"}, {"users", "groups", "serviceaccounts"}, "impersonates {res}"),
    ({"get", "create"}, {"nodes/proxy"}, "nodes/proxy: kubelet API, no audit"),
    ({"create"}, {"serviceaccounts/token"}, "mints service account tokens"),
    (WRITE, {"certificatesigningrequests/approval"}, "approves CSRs (client certs)"),
    ({"approve"}, {"signers"}, "approves CSRs for a signer"),
    (WRITE, {"validatingwebhookconfigurations", "mutatingwebhookconfigurations"},
     "controls admission webhooks"),
    ({"create"}, {"persistentvolumes"}, "creates PersistentVolumes (hostPath)"),
    ({"update", "patch"}, {"namespaces"}, "relabels namespaces (Pod Security)"),
]
BROAD_GROUPS = {
    "system:anonymous": "bound to anonymous requests",
    "system:unauthenticated": "bound to unauthenticated users",
    "system:authenticated": "bound to every signed-in user",
}
DEFAULTS_LABEL = ("kubernetes.io/bootstrapping", "rbac-defaults")


def findings_for_rules(rules):
    out = []
    for rule in rules or []:
        verbs = set(rule.get("verbs", []))
        resources = set(rule.get("resources", []))
        if "*" in verbs and "*" in resources:
            out.append("wildcard: every verb on every resource")
            continue
        for trig, targets, why in CHECKS:
            if not (verbs & trig or "*" in verbs):
                continue
            hits = targets if "*" in resources else targets & resources
            out.extend(why.format(res=res) for res in hits)
    return sorted(set(out))


def is_system(subj):
    if subj.get("kind") == "ServiceAccount":
        return subj.get("namespace") == "kube-system"
    return subj.get("name", "").startswith("system:")


def main(argv):
    include_system = "--include-system" in argv
    paths = [a for a in argv[1:] if not a.startswith("--")]
    if len(paths) != 1:
        print(__doc__.strip(), file=sys.stderr)
        return 2
    try:
        with open(paths[0], encoding="utf-8") as fh:
            items = json.load(fh)["items"]
    except (OSError, ValueError, KeyError) as exc:
        print(f"cannot read RBAC list: {exc}", file=sys.stderr)
        return 2
    roles = {}
    for it in items:
        if it.get("kind") in ("Role", "ClusterRole"):
            ns = it["metadata"].get("namespace", "")
            roles[(it["kind"], ns, it["metadata"]["name"])] = it.get("rules")
    count = 0
    for it in items:
        kind = it.get("kind")
        if kind not in ("RoleBinding", "ClusterRoleBinding"):
            continue
        meta = it["metadata"]
        labels = meta.get("labels") or {}
        builtin = labels.get(DEFAULTS_LABEL[0]) == DEFAULTS_LABEL[1]
        ns = meta.get("namespace", "")
        ref = it["roleRef"]
        role_ns = ns if ref["kind"] == "Role" else ""
        if kind == "ClusterRoleBinding":
            label = f"CRB {meta['name']}"
        else:
            label = f"RB {ns}/{meta['name']}"
        if ref["name"] == "cluster-admin":
            reasons = ["cluster-admin"]
        else:
            reasons = findings_for_rules(roles.get((ref["kind"], role_ns, ref["name"])))
        for subj in it.get("subjects") or []:
            if builtin and is_system(subj) and not include_system:
                continue
            prefix = f"{subj['namespace']}/" if subj.get("namespace") else ""
            who = f"{subj['kind']} {prefix}{subj['name']}"
            extra = [BROAD_GROUPS[subj["name"]]] if subj["name"] in BROAD_GROUPS else []
            for reason in reasons + extra:
                high = kind == "ClusterRoleBinding" or extra
                print(f"{'HIGH' if high else 'MEDIUM':6} {label} -> {who}: {reason}")
                count += 1
    print(f"{count} finding(s)")
    return 1 if count else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))

The sample sample.json holds the built-in cluster-admin binding, a look-alike named system:backdoor and six other grants:

{"items": [
  {"kind": "ClusterRoleBinding",
   "metadata": {"name": "cluster-admin",
                "labels": {"kubernetes.io/bootstrapping": "rbac-defaults"}},
   "roleRef": {"kind": "ClusterRole", "name": "cluster-admin"},
   "subjects": [{"kind": "Group", "name": "system:masters"}]},
  {"kind": "ClusterRoleBinding", "metadata": {"name": "ci-deployer-admin"},
   "roleRef": {"kind": "ClusterRole", "name": "cluster-admin"},
   "subjects": [{"kind": "ServiceAccount", "namespace": "ci", "name": "deployer"}]},
  {"kind": "ClusterRoleBinding", "metadata": {"name": "system:backdoor"},
   "roleRef": {"kind": "ClusterRole", "name": "cluster-admin"},
   "subjects": [{"kind": "User", "name": "mallory"}]},
  {"kind": "ClusterRole", "metadata": {"name": "secret-reader"},
   "rules": [{"resources": ["secrets"], "verbs": ["get", "list", "watch"]}]},
  {"kind": "ClusterRoleBinding", "metadata": {"name": "monitoring-secrets"},
   "roleRef": {"kind": "ClusterRole", "name": "secret-reader"},
   "subjects": [{"kind": "ServiceAccount", "namespace": "monitoring", "name": "agent"}]},
  {"kind": "Role", "metadata": {"namespace": "shop", "name": "shop-developer"},
   "rules": [{"resources": ["deployments"], "verbs": ["get", "create", "patch"]},
             {"resources": ["pods", "pods/log"], "verbs": ["get", "list"]}]},
  {"kind": "RoleBinding", "metadata": {"namespace": "shop", "name": "shop-developers"},
   "roleRef": {"kind": "Role", "name": "shop-developer"},
   "subjects": [{"kind": "Group", "name": "team-shop"}]},
  {"kind": "Role", "metadata": {"namespace": "shop", "name": "debugger"},
   "rules": [{"resources": ["pods/exec"], "verbs": ["create"]}]},
  {"kind": "RoleBinding", "metadata": {"namespace": "shop", "name": "shop-debug"},
   "roleRef": {"kind": "Role", "name": "debugger"},
   "subjects": [{"kind": "User", "name": "alice@example.com"}]},
  {"kind": "Role", "metadata": {"namespace": "shop", "name": "log-reader"},
   "rules": [{"resources": ["pods/log"], "verbs": ["get"]}]},
  {"kind": "RoleBinding", "metadata": {"namespace": "shop", "name": "shop-logs"},
   "roleRef": {"kind": "Role", "name": "log-reader"},
   "subjects": [{"kind": "Group", "name": "system:unauthenticated"}]},
  {"kind": "ClusterRole", "metadata": {"name": "csr-approver"},
   "rules": [{"resources": ["certificatesigningrequests/approval"], "verbs": ["update"]},
             {"resources": ["signers"], "verbs": ["approve"]}]},
  {"kind": "ClusterRoleBinding", "metadata": {"name": "csr-approvers"},
   "roleRef": {"kind": "ClusterRole", "name": "csr-approver"},
   "subjects": [{"kind": "ServiceAccount", "namespace": "certs", "name": "approver"}]}
]}

1. Save both files, run python3 -I rbac_risks.py sample.json, then run it on a cluster export (command in the docstring).
2. For each HIGH finding, confirm what the subject can really do: kubectl auth can-i --list --as=system:serviceaccount:ci:deployer (needs impersonation rights).^[64]

• Expected output for the sample (the labelled built-in binding is skipped; the look-alike is not):

HIGH   CRB ci-deployer-admin -> ServiceAccount ci/deployer: cluster-admin
HIGH   CRB system:backdoor -> User mallory: cluster-admin
HIGH   CRB monitoring-secrets -> ServiceAccount monitoring/agent: reads Secret values
MEDIUM RB shop/shop-developers -> Group team-shop: writes deployments (any SA, any Secret)
MEDIUM RB shop/shop-debug -> User alice@example.com: pods/exec: runs commands in pods
HIGH   RB shop/shop-logs -> Group system:unauthenticated: bound to unauthenticated users
HIGH   CRB csr-approvers -> ServiceAccount certs/approver: approves CSRs (client certs)
HIGH   CRB csr-approvers -> ServiceAccount certs/approver: approves CSRs for a signer
8 finding(s)

• Validation: exit code 1 with findings, 0 when clean, 2 for bad input (all observed in the test run).
• Troubleshooting: export all four kinds in one command, or bindings lose their roles. get on pods/exec is flagged because WebSocket exec needed only get before v1.35.^[65] Blind spots: API groups and resourceNames are ignored; indirect paths through operators are invisible.
• Teardown: delete rbac.json.

• Objective: prove that deny-all works on your plugin, restore only the flows you need, and see what happens to an open connection.
• Prerequisites: kind v0.24.0 or later, kubectl; 30 minutes. Risks: none outside the local cluster.

Example status: syntax-checked, not executed (requires a local kind cluster).

kind create cluster --name netpol-lab
kubectl create namespace shop
kubectl run web -n shop --image=nginx --labels=app=web --port=80 --expose
kubectl run client -n shop --image=busybox:1.36 --labels=app=client -- sleep 3600
kubectl wait -n shop --for=condition=Ready pod/web pod/client --timeout=120s
kubectl exec -n shop client -- wget -qO- -T 3 http://web | head -1
# Second terminal: kubectl exec -it -n shop client -- nc web 80   (leave it open)
kubectl apply -n shop \
  -f https://k8s.io/examples/service/networking/network-policy-default-deny-all.yaml
kubectl exec -n shop client -- wget -qO- -T 3 http://web
# Second terminal: type GET / HTTP/1.0 and press Enter twice. Does nginx still answer?
# Save allow-dns-egress from the reference implementation as dns.yaml, then:
kubectl apply -n shop -f dns.yaml -f flows.yaml
kubectl exec -n shop client -- wget -qO- -T 3 http://web | head -1
kubectl run other -n shop --image=busybox:1.36 --rm -it --restart=Never -- \
  wget -qO- -T 3 http://web

Example status: YAML syntax-checked; not applied. The two allow rules, saved as flows.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: web-from-client, namespace: shop}
spec:
  podSelector: {matchLabels: {app: web}}
  policyTypes: ["Ingress"]
  ingress:
    - from: [{podSelector: {matchLabels: {app: client}}}]
      ports: [{protocol: TCP, port: 80}]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: client-to-web, namespace: shop}
spec:
  podSelector: {matchLabels: {app: client}}
  policyTypes: ["Egress"]
  egress:
    - to: [{podSelector: {matchLabels: {app: web}}}]
      ports: [{protocol: TCP, port: 80}]

• Expected output: first <!DOCTYPE html>; after deny-all, a name-resolution error or a timeout, because egress, DNS included, is blocked.^[2] With dns.yaml and flows.yaml, client gets the page again and other times out.
• Validation: other failing shows the rules are specific to app=client. Whether the open nc session still answered after deny-all is “implementation defined”;^[2] note your plugin’s behaviour for the runbook.
• Troubleshooting: traffic still flowing after deny-all means the plugin does not enforce NetworkPolicy; fix that first.
• Teardown: kind delete cluster --name netpol-lab.

• Objective: write a webhook-free rule that requires images pinned by digest, observe warn mode, then enforce.
• Prerequisites: a kind cluster running Kubernetes v1.30 or later; 45 minutes. Risks: local only; on a shared cluster a Deny binding blocks deploys in every matching namespace.

Example status: YAML syntax-checked; not applied.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-image-digest
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  matchConditions:   # skip updates that keep the images (label patches on old pods)
    - name: create-or-image-change
      expression: >-
        request.operation == 'CREATE' ||
        object.spec.containers.map(c, c.image) !=
          oldObject.spec.containers.map(c, c.image) ||
        (has(object.spec.initContainers) &&
         object.spec.initContainers.map(c, c.image) !=
         oldObject.spec.initContainers.map(c, c.image))
  validations:
    - expression: >-
        object.spec.containers.all(c, c.image.contains('@sha256:')) &&
        (!has(object.spec.initContainers) ||
         object.spec.initContainers.all(c, c.image.contains('@sha256:')))
      message: "Every container image must be pinned by digest (name@sha256:...)."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-image-digest-prod
spec:
  policyName: require-image-digest
  validationActions: ["Warn", "Audit"]
  matchResources:
    namespaceSelector:
      matchLabels:
        security.example.com/tier: prod

1. Apply both objects, create namespace lab labelled security.example.com/tier=prod, and run kubectl run old -n lab --image=nginx.
2. Note the digest the node resolved: kubectl get pod old -n lab -o yaml | grep imageID.
3. Change validationActions to ["Deny", "Audit"] (Deny and Warn may not be combined)^[25] and run kubectl run new -n lab --image=nginx.
4. Run kubectl run pinned -n lab --image=<imageID from step 2>, then kubectl label pod old -n lab reviewed=yes.

• Expected output: step 1 creates old with a warning carrying the policy’s message; step 3 is denied with it; step 4 creates pinned without warnings, and the label patch on old passes because the match condition skips updates that keep the images. Updates can change a pod’s images, so UPDATE stays matched.^[66]
• Validation: under Deny, a tag-only Deployment is still accepted but its pods are rejected; match apps resources for earlier feedback.
• Troubleshooting: no effect usually means a missing binding; policy and binding “must be defined for a policy to have an effect”.^[25]
• Teardown: delete the binding, then the policy, or delete the kind cluster.

• Objective: deploy a sensor, trigger the “Terminal shell in container” rule with an exec (the weekly synthetic alert) and a shell entrypoint, and tune it with a scoped override, not by switching it off.
• Prerequisites: a kind cluster on a Linux host (kind nodes share the host kernel, which must expose BTF and support the BPF ring buffer for the default modern eBPF probe),^[67] Helm; 45 minutes. Risks: the Falco DaemonSet runs privileged; local cluster only.

Example status: commands from the Falco documentation^[68][69] (chart 9.2.0, Falco 0.45.0 when checked); syntax-checked, not executed (no cluster on this host). Expected output is taken from the rule’s output string, not captured from a run.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install --replace falco --namespace falco --create-namespace --set tty=true \
  falcosecurity/falco
kubectl wait pods --for=condition=Ready --all -n falco --timeout=300s
kubectl create namespace shop
kubectl run canary -n shop --image=busybox:1.36 -- sleep 3600
kubectl wait -n shop --for=condition=Ready pod/canary --timeout=120s
# Trigger 1: an exec. This is also the weekly synthetic alert.
kubectl exec -it -n shop canary -- sh -c 'id; exit'
# Trigger 2: no exec at all; the pod's entrypoint is a shell with a TTY. --attach=false
# skips the attach that -i implies (no race with the shell, no pods/attach record).
kubectl run tty-shell -n shop --image=busybox:1.36 -it --attach=false --restart=Never \
  -- sh -c 'id; sleep 30'
# Without an attach, kubectl run returns at once: wait until the shell is running.
kubectl wait -n shop --for=condition=Ready pod/tty-shell --timeout=120s
# With a label selector, kubectl logs shows only the last 10 lines unless --tail is set.
kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --tail=-1 \
  | grep 'attached terminal'

The tuning file replaces the rule’s empty tuning macro with a scoped condition. Falco sees the process and its container, not who ran kubectl, so the exception keys on the break-glass toolbox image (an admission rule limits who may run that image); identity checks stay in the SIEM, with the audit log.

Example status: YAML syntax-checked; not applied.

# values-break-glass.yaml. The chart writes customRules files under /etc/falco/rules.d,
# which its default rules_files order loads after falco_rules.yaml, so the override
# applies.
customRules:
  break-glass.yaml: |-
    - macro: user_expected_terminal_shell_in_container_conditions
      condition: (container.image.repository = "registry.example.com/ops/break-glass")
      override:
        condition: replace

Apply it with helm upgrade falco -n falco --set tty=true -f values-break-glass.yaml falcosecurity/falco, then repeat trigger 1 and run a shell in the toolbox image.

• Expected output: both triggers print a line with the time, Notice, “A shell was spawned in a container with an attached terminal” and fields such as process=sh and command=.^[70] Trigger 1 has a matching pods/exec audit entry; trigger 2 only the pod’s create (--attach=false cancels the attach that -i implies;^[71] that the rule fires is inferred from its description and condition, not observed). After the override, the toolbox shell is silent and the canary exec still alerts.
• Validation: the weekly canary exec must reach the SOC queue within the agreed time, with its audit-log exec entry beside it; the SIEM does that correlation, as the rule’s description suggests (“correlate with k8saudit exec logs”).^[70]
• Troubleshooting: no alert: look for a driver-load error (a kernel without BTF) and confirm tty=true, which flushes logs at once;^[72] the rule is maturity_stable, loaded by default.^[41] Override ignored: the custom file must load after falco_rules.yaml (the chart’s default order ends with rules.d).^[72][73]
• Teardown: helm uninstall falco -n falco, then kind delete cluster.

1. Responding to an add-on CVE: ingress-nginx and CVE-2025-1974 (sourced real case). Problem: on 2025-03-24 the Kubernetes Security Response Committee announced five ingress-nginx vulnerabilities; combined, they could let anything on the pod network take over a cluster through the controller’s admission webhook.^[8] Method: find exposure with kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx; upgrade to v1.12.1 or v1.11.5, or disable the admission webhook meanwhile.^[8] Lessons: inventory add-ons with cluster-wide rights; restrict who can reach in-cluster webhooks; follow the official CVE feed (Kubernetes-project components, ingress-nginx included)^[74] and other add-ons’ own advisories. Ingress NGINX was then retired with no fixes after March 2026 (migration: KubernetesOps); the Steering and Security Response Committees warned that “unless you proactively check, you may not know you are affected until you are compromised”.^[75][76] Measure: hours from announcement to a complete exposure list.

Worked example (illustrative). A cluster has 46 workload namespaces: 40 enforce Baseline or stricter, 22 enforce Restricted and 31 have deny-all policies in both directions.

• Enforcement coverage = 40 ÷ 46 = 87.0 %; Restricted coverage = 22 ÷ 46 = 47.8 %; default-deny coverage = 31 ÷ 46 = 67.4 %.
• The runtime sensor raised 1,240 alerts last month; 62 led to an action: 62 ÷ 1,240 = 5.0 %. 1,150 came from the “Terminal shell in container” rule, whose description suggests correlating with audit logs.^[70] Excluding shells that match an on-call engineer’s audited exec leaves 140 alerts, 58 actionable: 58 ÷ 140 = 41.4 % (the four actionable alerts dropped by the exclusion were investigated separately).

Targets are local; track trends, not one “security score”.

The practice produces the evidence audits ask for: benchmark and policy reports, the exception register, RBAC reviews and protected audit logs; mapping them to a framework belongs to ComplianceOps, and nothing here is legal advice. Privacy: Request and RequestResponse audit levels record bodies,^[3] so log Secrets and ConfigMaps, whose values may hold personal data, at Metadata.

• Audit logs cost API-server memory^[3] and SIEM ingest; budget with SecDataOps.
• Webhooks add latency to matching requests; they should answer “typically in milliseconds”, and the timeout defaults to 10 seconds.^[28][26]
• Runtime sensors run on every node; measure their overhead.
• Keys: on EKS (1.28+), envelope encryption of all API data is on by default with an AWS-owned key at no extra cost; a customer-managed key costs normal KMS prices, and deleting it or revoking its grants makes the cluster unrecoverable.^[80]
• Isolation: separate clusters isolate best^[16] and cost most.

Check that controls work, not only that they exist: keep audit and warn on after enforcing so drift still shows; diff RBAC against the policy repository; fire the synthetic exec alert weekly (Lab 5’s canary); run the policy-engine outage drill quarterly; have someone outside the platform team review exceptions yearly.

Policy-engine outage drill (quarterly, in staging): scale the engine to zero; confirm the symptom per tier; check that the fail-open counter and policy reports catch what passed; verify the engine’s own namespace is excluded so it can restart; use break-glass once; restore; review the timings.

flowchart TB
  A["Runtime alert: terminal shell in a container"]:::neutral --> B{"Ticketed exec, attach, debug or shell-pod create by on-call in the audit log?"}:::decision
  B -->|"no"| D{"Any API record: a session, or a create with a TTY shell entrypoint?"}:::decision
  D -->|"yes, other identity"| IR["Incident response: pod in scope"]:::risk
  D -->|"no record at all"| BY["Suspect a bypass: node in scope too"]:::risk
  IR --> E["Isolate: relabel the pod, cordon the node, cut live connections"]:::core
  BY --> E
  E --> F["Preserve: pod spec, logs, events, audit entries, node evidence"]:::core
  F --> G["Revoke: delete the pod, drop external trust, rotate readable Secrets"]:::core
  G --> H["Scope: audit-log search for the service account, user and node"]:::core
  H --> I["Recover and learn: rebuild, replace the node if needed, postmortem"]:::neutral
  I ~~~ C["Close as expected; note it"]:::core
  B -->|"yes"| C
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

1. Triage (SOC, about 10 minutes). Find the API request that opened the session: exec, attach, an ephemeral debug container, or nodes/proxy on the node (on a managed service, in the provider’s log store).

   Example status: syntax-checked, not executed (requires the API server’s audit log in JSON lines and jq 1.6+).

   jq -c 'select((.objectRef.namespace == "payments"
                  and .objectRef.name == "api-7d9c4-x2kq"
                  and (.objectRef.subresource
                       | IN("exec", "attach", "ephemeralcontainers")))
                 or (.objectRef.resource == "nodes" and .objectRef.name == "node-17"
                     and .objectRef.subresource == "proxy"))
          | {ts: .requestReceivedTimestamp, user: .user.username, verb: .verb,
             sub: .objectRef.subresource, ip: .sourceIPs}' audit.log
   # No session found? Check whether the pod's own entrypoint is an interactive shell.
   kubectl get pod -n payments api-7d9c4-x2kq -o json \
     | jq -c '.spec.containers[] | {name, command, args, tty}'
   

   A ticketed session by an on-call engineer closes the alert; any other identity is an incident. No record needs one more check: the rule fires for a terminal shell whose parent is the container runtime (or gone), so a pod whose entrypoint is an interactive shell (kubectl run -it --attach=false … -- sh,^[71] a debug image, an init script that opens a TTY) fires it with only a create record; a shell spawned by a compromised application is the job of rules such as “Run shell untrusted”.^[70] Only a shell that neither an API session nor the pod’s own command explains points to a bypass of the API server (kubelet API, runtime socket, stolen node credential),^[22] which puts node-17 in scope too. Escalate to ResponseOps.

2. Isolate without destroying evidence. The aim is a pod that can no longer talk but still exists for forensics. NetworkPolicies are additive,^[2] so a “quarantine” policy cannot cancel allow rules. Instead, remove the label those rules and the Service select: the ReplicaSet releases the pod and starts a replacement.^[81] The suspect then falls under the namespace’s deny-all (payments must have one), which is why the baseline’s DNS rule skips pods labelled quarantine. Whether an open connection, such as a reverse shell, survives is “implementation defined” by the plugin (Lab 3 tests yours);^[2] for a sure cut, use the plugin’s own deny policy or block on the node, or capture evidence and delete the pod. Cordon the node; do not drain it yet.

   Example status: syntax-checked, not executed.

   kubectl label pod -n payments api-7d9c4-x2kq app- quarantine=true
   kubectl cordon node-17
   mkdir -p evidence
   kubectl get pod -n payments api-7d9c4-x2kq -o yaml > evidence/pod.yaml
   kubectl logs -n payments api-7d9c4-x2kq --all-containers --timestamps \
     > evidence/logs.txt
   kubectl get events -n payments -o yaml > evidence/events.yaml
   

3. Revoke. Once the incident lead has the evidence, delete the pod. The API then rejects its bound token, but systems that validate tokens through OIDC discovery accept them until expiry,^[23] up to a year for an injected token (reference implementation). If other workloads use the same service account, or the token may have been copied out, delete and re-create the ServiceAccount: its UID changes and the API rejects every token issued for it.^[82] Remove the service account from external trust policies (CloudSecOps), which should accept only tokens requested for their own audience.^[23] Rotating the signing key is the last resort, and it ends old tokens only once the old verification key leaves the API server’s key set.^[37] Drop RBAC it did not need; rotate every Secret it could read.

4. Scope. Search the audit log for the service account (system:serviceaccount:payments:api) and any human identity since the first suspicious event: new bindings, pods, exec into other pods, Secret reads. After a suspected bypass, add the node’s kubelet and runtime logs, credentials and other pods.

5. Recover and learn. Redeploy from a known-good, signed image; replace the node if an escape is suspected; feed rule and policy changes into DetectionOps and the postmortem into IncidentOps.

• “Enable every sample policy in enforce mode.” An outage, and teams learn to ask for blanket exemptions.
• “failurePolicy: Ignore everywhere, so nothing breaks.” A silent bypass is worse than a visible outage for sensitive tiers; decide per tier (judgement call).
• “cluster-admin for CI so deploys never fail.” A compromised pipeline is then a compromised cluster.
• “Block every image with a CVE at admission.” Scanners report unfixable and unreachable findings; the Kubernetes blog’s reviewers recommend exceptions, risk-based overrides and build-time checks.^[16]

Invest deliberately when clusters are shared by several teams, run Internet-facing or regulated workloads, host third-party operators, or hold credentials to other systems. Keep it light when one small team runs a few workloads on a managed service: enforce Baseline, keep RBAC tight, use the provider’s audit logging and encryption, and stop there until the risk grows. Node-less modes keep the in-cluster controls yours; node hardening, and often DaemonSet sensors, become the provider’s. Container services without Kubernetes belong to ServerlessOps; not running Kubernetes where you do not need it is a control in itself (NoOps).

Hidden costs: policy engines become critical services; exceptions, triage and upgrades take staff.

flowchart TB
  Q1{"Pod hardening that the Pod Security Standards cover?"}:::decision
  Q1 -->|"yes"| PSA["Pod Security Admission labels"]:::core
  Q1 -->|"no"| Q2{"Can CEL decide it from the object, old object, namespace, parameters and RBAC checks?"}:::decision
  Q2 -->|"yes"| VAP["Validating or MutatingAdmissionPolicy, written directly or generated by Kyverno or Gatekeeper"]:::core
  Q2 -->|"no: signatures, registries, other objects"| WH["Webhook engine: Kyverno, Gatekeeper or policy-controller"]:::neutral
  WH --> Q3{"Is an unverified workload worse than a blocked deploy for this tier?"}:::decision
  Q3 -->|"yes"| FC["Fail closed: high availability, own namespace excluded, break-glass"]:::risk
  Q3 -->|"no"| FO["Fail open: audit mode on, alert on fail-open calls, policy reports"]:::neutral
  classDef core fill:#e8f3f2,stroke:#16756f,color:#122029
  classDef risk fill:#fbeceb,stroke:#b4332b,color:#122029
  classDef decision fill:#fdf3e3,stroke:#c9800f,color:#122029
  classDef neutral fill:#f1f4f6,stroke:#5b7a87,color:#122029

The Atlas’s model, not an industry standard.

• Days 1–30: see. Inventory clusters, add-ons with cluster-wide rights and cluster-admin subjects (Lab 2); Baseline warn and audit everywhere; audit logging on; benchmark run; subscribe to security announcements.^[83][74]
• Days 31–60: guard. Enforce Baseline where dry runs are clean; default-deny pilot in two namespaces; encryption at rest; runtime alerts to the SOC; exception register open.
• Days 61–90: verify. Restricted for new namespaces; digest rule from warn to deny in production; a signature pilot; rehearse the containment runbook and the policy-engine outage drill; publish the first scorecard.

Before switching any rule to enforce

• Rule in audit or warn mode for two weeks or more, with violation counts per namespace
• Every remaining violation fixed or covered by an expiring exception
• Enforce date announced; rollback (back to warn) tested
• Webhooks: two or more replicas, own namespace and kube-system excluded, timeout and failure policy set, error and fail-open alerts

Quarterly review

• cluster-admin subjects and Lab 2 findings reviewed and signed off
• Namespaces without an enforce label: none (kubectl get ns --selector='!pod-security.kubernetes.io/enforce')^[63]
• Expired exceptions removed; benchmark re-run after the last upgrade; add-ons checked for CVEs and maintenance status

First hour of a suspected compromise

• Audit log searched for exec, attach, debug and nodes/proxy; pod spec checked for a TTY shell entrypoint; neither found means the node is in scope; incident lead named
• Pod relabelled out of service, node cordoned, live connections cut if needed, evidence collected before deletion
• Service account’s external trust and RBAC reviewed; readable Secrets queued for rotation

Template status: illustrative; adapt fields to your tooling.

POLICY EXCEPTION REQUEST
Rule:                 e.g. Pod Security Restricted / require-image-digest
Scope:                namespace(s), workload(s), images
Requested by / owner: team, named person
Reason:               why the workload cannot comply now
Residual risk:        what an attacker could do with this exception
Compensating control: e.g. dedicated nodes, signed image, extra runtime rule
Expiry:               date (maximum 90 days); review trigger
Assessed by:          security engineer, date
Risk accepted by:     business owner, date (per the SecOps exception policy)
Register entry:       ID in the shared exception register

GUARDRAIL CHANGE RECORD
Rule and version:     name, repository commit
Mode change:          audit -> warn -> enforce (dates)
Violations at change: count by namespace; exceptions granted
Failure mode:         fail closed / fail open; break-glass procedure
Rollback:             command or commit that restores the previous mode
Owner and reviewers:  names

Is “KubernetesSecOps” a job title or a certification? Not that we found; the relevant certification is the CKS.^[13]

If we enforce Pod Security, do we still need a policy engine? Often not at first: Pod Security plus CEL policies cover many rules without a webhook; add an engine for signatures, generated resources or rules that look up other objects.

We use a managed service. What is still our job? Everything inside the cluster: RBAC, Pod Security, NetworkPolicy, admission and audit review. Control-plane settings become provider options with their own defaults (EKS encrypts API data by default but sends no audit logs until enabled).^[80][39]

Are NetworkPolicies enough, or do we need a service mesh? NetworkPolicy allow-lists by labels and addresses; it cannot do anything TLS-related, for which the docs point to a service mesh or ingress controller, nor logging or explicit deny.^[2] In the Atlas’s summary, a mesh adds mTLS and request-level policy, while logging and explicit deny depend on the network plugin (Cilium, Calico and others provide them); each is another critical component to run.

Can namespaces separate untrusted tenants? Only with Restricted pods, default-deny networking, quotas and strict RBAC, and they still share kernels; hostile tenants need sandboxed runtimes (RuntimeClass with gVisor or Kata) on separate nodes, or separate clusters.^[16]

• Newly stable (v1.36): user namespaces, where pod root is unprivileged on the host and Pod Security relaxes runAsNonRoot checks; they need idmap-capable filesystems and containerd 2.0+ or CRI-O 1.25+.^[86] MutatingAdmissionPolicy brings defaults without webhooks.^[44]
• Moving fast: engines shifting to CEL (Kyverno’s docs mark its older ClusterPolicy deprecated);^[30] in-kernel eBPF enforcement that can kill a process mid-syscall, which needs the care of any automated response.^[52]
• Treat with caution: claims of autonomous AI remediation; ask what it may change.

• Prerequisites: KubernetesOps and ContainerOps basics.
• One day: the Security Checklist and Pod Security Standards; Lab 1.
• One week: RBAC good practices and Lab 2; NetworkPolicy and Lab 3; write an audit policy.
• Thirty days: Labs 4 and 5; draft an exception process; map detections to ATT&CK techniques such as T1609 and T1611.^[42]
• Specialise: the CKS (after the CKA);^[13] supply-chain depth in DevSecOps.
• Portfolio: a repository with a namespace baseline, tested policies and a runbook, applied to kind in CI.

Assumptions. Three engine replicas across zones; several deploys a day; one audited payments tier, the rest ordinary web services; 15-minute page response.

What the evidence says. The webhook API defaults to fail closed,^[26] but engines set their own: Kyverno’s resource webhooks fail closed, Gatekeeper’s fail open.^[84][85] Kyverno warns that a fail-closed pod rule, a total engine outage and no exclusion for its namespace may leave deleting webhook configurations by hand as the only recovery;^[84] hence the exclusion. CEL cannot reach registries or transparency logs, so a webhook stays.

The attractive wrong answer is one global setting: fail closed everywhere turns every engine incident into a platform outage; fail open everywhere makes the control decorative where it matters.

A defensible choice. Fail closed for payments, with high availability, the engine’s namespace and kube-system excluded, a short timeout, an error alert and a rehearsed break-glass. Fail open elsewhere, with audit mode on, an alert on the API server’s fail-open counter (metrics)^[78] where the provider exposes it, and policy reports to find pods admitted during an outage (on a managed service without control-plane metrics, the reports are the alert). Uncertainty: if the engine proves unstable, payments feels it first; count outage minutes.

You should now be able to trace a request through the control points, roll out a guardrail, review RBAC for escalation paths, read a runtime alert against the audit log, contain a suspicious pod without destroying evidence, and argue a failure mode.

1. A monitoring group has list on Secrets “only to count them”. What is the risk?

2. Why does a quarantine NetworkPolicy not isolate a suspicious pod, and what does relabelling it achieve?

3. Falco reports a terminal shell in a pod, and the audit log has no exec, attach or debug request for it. What does that suggest?

4. After deleting a compromised pod, is its service-account token useless everywhere?

5. A team proposes a ValidatingAdmissionPolicy that rejects any list of Secrets outside kube-system. Will it work?

KubernetesSecOps is a name almost nobody uses for work almost every Kubernetes team must do: limit who can change the cluster, decide at admission what may run, make the network deny by default, keep secrets out of reach, and watch running containers with evidence good enough to act on. The hard part is operational: rules that break deployments, webhooks that become single points of failure, exceptions that never expire, add-ons that quietly hold the keys. Treat each guardrail as a service with an owner, a failure mode and an exit, and the cluster says no at the right moments and tells you fast when something got through.

Further reading

• Kubernetes for Security Operations Centres — ControlPlane with JP Morgan CyberOps, 2023 (page checked 2026-10-16; guide not reviewed).

Changelog: 2026-10-16 — first version (only Lab 2 executed, against a sample file). 2026-10-17 — revision 1 after the first review (archived): runbook, reads and webhook defaults corrected; control-plane and managed audit settings added; Lab 2 extended and re-run. 2026-10-17 — revision 2 after review KS2: Lab 5 (Falco, synthetic alert, scoped override); TTY-entrypoint check before suspecting a bypass; ServiceAccount re-creation; kubelet-default, CVE, egress, service-mesh and metrics precisions; neighbour figure and restatements cut; fifth mastery question; back inside the length band. 2026-10-17, post-score polish: Lab 5 entrypoint trigger without attach, labelled an inference; defaults.go pinned to v1.37.1; code lines and figure labels wrapped (runbook entrypoint check now uses jq).

Re-check cadence: the version-bound facts here (Kubernetes v1.37 and patched minimums, kind v0.33.0, CIS Benchmark 2.1.0, Gatekeeper v3.23, Kyverno v1.19, Falco 0.45.0 and chart 9.2.0, EKS and GKE defaults) should be re-verified at each Kubernetes minor release, roughly every four months.