Hard Difficulty Questions

The kubelet is configured to always mount tokens

Why it fails: There is no kubelet configuration that forces service account token mounting for all pods; the kubelet only honors the `automountServiceAccountToken` field in the pod spec (or the effective setting from the service account). Even if such a flag existed, it would contradict Kubernetes' per-pod security model and would require a feature gate, not a simple kubelet flag. The presence or absence of the token is therefore determined entirely by pod and service account configuration, not by kubelet settings.

The namespace has a default automountServiceAccountToken: true

Why it fails: Kubernetes does not have a namespace-level `automountServiceAccountToken` default; that field exists only on `PodSpec` and `ServiceAccount`. The pod's own setting takes precedence over any inherited or default value, so if the pod spec explicitly set `automountServiceAccountToken: false`, the token would not be mounted regardless of any hypothetical namespace default. Because no such namespace mechanism exists, this cannot explain why the token is present.

The pod is using a custom service account with automountServiceAccountToken: true

Why it fails: When both the pod spec and the service account specify `automountServiceAccountToken`, the pod-level value wins; a service account's `true` cannot override a pod's `false`. The kubelet reads the effective `PodSpec` field after merging service account settings, so a custom service account with `automountServiceAccountToken: true` would only matter if the pod did not explicitly set the field. This option would imply a precedence that Kubernetes does not implement, making it an incorrect explanation for the observed token mount.

The pod must run as privileged to use AppArmor

Why it fails: AppArmor does not require privileged containers; in fact, privileged containers typically bypass Linux security module (LSM) restrictions. A container is confined by having the AppArmor profile loaded on the host and referencing it via the Kubernetes annotation, regardless of privilege level. Running privileged would disable AppArmor confinement, making the workload less secure. Thus, the pod does not need to be privileged; the issue lies elsewhere.

The annotation is missing the 'localhost/' prefix

Why it fails: The annotation value already includes the required 'localhost/' prefix, which indicates a custom profile loaded on the node. Without this prefix, Kubernetes would interpret the profile as one of the runtime's built-in profiles and could fail to apply the intended policy. Because the prefix is present and the annotation is correctly formatted, a formatting mistake cannot explain the lack of enforcement. The real problem is that the profile is loaded in complain mode, not enforce mode.

The profile is not loaded on the node

Why it fails: The scenario explicitly states that the node has the AppArmor profile loaded, so the kernel already has the profile's rule set available. If the profile were absent, the container runtime would either reject pod creation or fail to start with a 'profile not found' error. Since the profile is loaded, the container's lack of restriction cannot be due to a missing profile. The failure to enforce restrictions points directly to the profile being in complain mode rather than its absence.

kubectl taint nodes control-plane node-role.kubernetes.io/control-plane-

Why it fails: Removing the taint with the trailing dash (`kubectl taint nodes control-plane node-role.kubernetes.io/control-plane-`) unsets the node's NoSchedule taint, which is a cluster-wide change affecting all future scheduling decisions. This deliberately weakens the security boundary of the control-plane node by allowing any pod without a matching toleration to be placed there, and it does not target the specific pod or workload; the correct approach is to add a toleration to the pod spec, not diminish the node's protections.

kubectl cordon control-plane

Why it fails: Cordoning a node marks it as unschedulable, which actually prevents new pods (including the stuck one) from being scheduled there entirely. This is a maintenance or node-drain operation and does not remove or modify any existing taints; in fact, it adds a second scheduling barrier, ensuring the pod remains Pending. It is the opposite of what is needed to place a workload on the control-plane node.

kubectl delete pod --all

Why it fails: Deleting all pods does not change taints, tolerations, or scheduler constraints, so the replacement pods will be created with the same missing tolerations and will become Pending again immediately. This action is also highly disruptive because it terminates every workload in the namespace or cluster, including healthy ones, and it fails to address the root cause: the pod lacks the toleration required to schedule on the tainted control-plane node.

CHOWN, DAC_OVERRIDE, FOWNER

Why it fails: CHOWN, DAC_OVERRIDE and FOWNER concern file ownership and bypassing filesystem permission checks, which do not grant the kernel-level access a container escape requires. It is tempting because they are commonly dropped in hardened profiles, and doing so is correct when the aim is limiting file permission manipulation rather than kernel capabilities.

KILL, SETPCAP, SYS_CHROOT

Why it fails: KILL, SETPCAP and SYS_CHROOT relate to signalling processes, altering capability sets and changing root directories, none of which are the privileged kernel operations exploited during a container escape. It is tempting because they appear in default Docker capability lists, and dropping them is correct when tightening general process and filesystem privileges.

NET_RAW, NET_ADMIN, NET_BIND_SERVICE

Why it fails: Dropping these three network capabilities leaves kernel-level escape vectors such as SYS_ADMIN, SYS_PTRACE and SYS_MODULE intact, so host kernel access remains possible. It is tempting because these capabilities are commonly dropped for network hardening, which is the correct choice when the goal is limiting network attack surface, not kernel escapes.

The imagePullPolicy is set to Always, causing Kyverno to ignore the registry check

Why it fails: The existence of `imagePullPolicy: Always` tells the kubelet to always fetch the image from the registry at runtime, but it has no influence on Kubernetes admission control. Kyverno acts as an admission webhook, evaluating the Pod spec as submitted; the pull policy is merely a field in the manifest and neither enables nor disables validation. Furthermore, Kyverno's image checks inspect the `spec.containers[].image` string, which is independent of how the image is pulled later. Thus, setting Always does not cause Kyverno to skip the registry check; such a check would still be enforced if the policy matches.

Kyverno cannot validate images with a tag

Why it fails: Kyverno fully supports validating images that include tags; the image reference parser extracts registry, repository, tag, and digest in `kyverno.io/v1` policies. Rules can use fields like `image.tag` to enforce conditions, such as forbidding the `latest` tag or requiring specific version prefixes. Moreover, Kyverno can also evaluate image digests for immutable deployment verification. Hence, the presence of a tag on the image does not prevent validation; the policy would still be evaluated against the full reference. The assertion that Kyverno cannot validate tagged images is false.

The policy is missing the 'validate' rule type

Why it fails: A `validate` rule is indeed the standard mechanism for admission validation in Kyverno, but the policy being described is already a validation policy—the user intends it to validate image sources. If the policy were missing a `validate` rule, Kyverno would not perform any admission check at all; however, the symptom here is that the check is silently not triggering, which points to a mismatch in the image pattern rather than a missing rule type. In practice, omitting `validate` would cause the policy to have no effect, but the problem statement implies the policy exists and is misconfigured. Therefore, this answer does not explain the situation.

Implement network policies to restrict egress from pods to the registry.

Why it fails: Network policies operate at the network layer, filtering traffic based on IP addresses, ports, and labels; they cannot inspect the contents or provenance of container images. Restricting egress to the registry only limits where pods can connect, not which images are allowed to run, so a malicious image already pulled and launched would remain unaffected. Admission-time verification of signed attestations is required to establish trust in image provenance before a workload is scheduled.

Grant registry write access only to the CI system's service account.

Why it fails: Granting registry write access solely to the CI system's service account follows least privilege, but it still leaves the CI pipeline as a single point of failure. If an attacker compromises the CI system—for example, via a malicious dependency or stolen credentials—they can push arbitrary images to the registry, and the cluster will blindly run them. This control reduces the attack surface but provides no cryptographic proof that the images were built by the intended, unmodified pipeline.

Use a static analysis tool to check the Dockerfile before building.

Why it fails: Static analysis of a Dockerfile examines the declared build steps, such as base images and RUN commands, but it cannot inspect the fully built image or its layers. Problems like a vulnerable base image that is pulled at build time, a compromised package repository, or a tampered build cache will not be detected by Dockerfile linting alone. Such tools are useful for security hygiene but do not create verifiable evidence of the image's origin or content.

Use a unique registry path and restrict access via firewall rules.

Why it fails: Using a unique registry path and firewall rules provides access control based on network location and obscurity, but it does not ensure that images pushed to that path are legitimate. If the CI credentials or a developer's token are stolen, the attacker can push a malicious image to that same path, and the firewall will allow it because it originates from a permitted network. This approach cannot detect or prevent tampering; it only reduces the chance of outside access.

Use a validating webhook instead of ImagePolicyWebhook

Why it fails: Wrong. Substituting ImagePolicyWebhook with a validating webhook does not change the order; mutating webhooks still run first.

The mutating webhook should be configured to skip pods with certain images

Why it fails: Wrong. Configuring the mutating webhook to skip certain images is a potential workaround but does not address the fundamental ordering issue.

The ImagePolicyWebhook should be placed before the mutating webhook in the webhook configuration

Why it fails: Wrong. In Kubernetes, the order of admission controllers is fixed: mutating controllers run before validating ones. You cannot reorder ImagePolicyWebhook before the mutating webhook in the plugin chain.

kube-bench run --targets=controlplane

Why it fails: kube-bench does not recognize the target name 'controlplane'. The CLI rejects it, so no checks execute and the command exits with an invalid target error. The CIS Kubernetes Benchmark groups all control plane components under the exact target 'master', so you must use that supported name to trigger any control plane audit.

kube-bench run --targets=node

Why it fails: The 'node' target runs only benchmarks for worker-node components such as kubelet and kube-proxy, which are not the focus of a control plane audit. On a control plane node, this skips the API server, controller-manager, and scheduler checks, leaving critical security settings unexamined. Even though a control plane node may run kubelet, the node target does not substitute for the master target.

kube-bench run --targets=etcd

Why it fails: The 'etcd' target restricts kube-bench to the etcd-only checks in the CIS benchmark, but the control plane includes more than etcd. This command would not audit kube-apiserver, kube-scheduler, or kube-controller-manager, so a major portion of the control plane remains unchecked. Since the prompt asks for control plane checks, limiting the scan to etcd is insufficient and incorrect.

PodSecurityPolicy (deprecated)

Why it fails: PodSecurityPolicy (PSP) was a deprecated admission controller that only enforced security context constraints such as users and capabilities. It could not add labels to pod metadata because it lacked any mutation mechanism; PSP was entirely validation-oriented and was removed in Kubernetes 1.25, making it irrelevant for labeling tasks.

ResourceQuota

Why it fails: ResourceQuota is an API object that limits aggregate resource consumption—like CPU, memory, and storage—within a namespace. It operates on resource quantities and rejects pods that would exceed quotas, but it never inspects or modifies pod labels. Admission for ResourceQuota is a validating controller, so it cannot add labels to a pod during creation.

ValidatingAdmissionPolicy

Why it fails: ValidatingAdmissionPolicy is a declarative admission policy using CEL expressions that only checks requests and returns an admission decision. It has no mutation phase, so while it can validate whether a pod has a required label, it cannot insert or change labels. The policy engine is designed solely for validation, not for modifying resources during admission.

Allow privileged containers for system-critical workloads.

Why it fails: Privileged containers essentially disable all Linux security controls, including seccomp, AppArmor, and capability dropping, giving them near-host-level access. They can access host devices, manipulate the iptables/netfilter stack, and even run privileged operations that normally require root on the host. For system-critical workloads, there are always better alternatives: either run them as non-privileged with the specific capabilities they need (such as NET_ADMIN for kube-proxy) or use dedicated node agent patterns. Allowing privileged containers violates the principle of least privilege and is explicitly prevented by baseline and restricted PSS profiles.

Allow containers to use hostNetwork for easier service discovery.

Why it fails: Setting hostNetwork: true places the container directly in the node's network namespace, so it shares the host IP and network stack. This breaks network isolation, creates port collision risks with other daemons, and exposes the container to host-level network policies and raw traffic sniffing. It also increases the blast radius of a container compromise, as an attacker gains visibility into all host traffic. Kubernetes-native service discovery via DNS and ClusterIP Services makes hostNetwork unnecessary for normal workloads; only a few node-level agents like cAdvisor or kube-proxy might need it, and those should be tightly controlled.

Set --anonymous-auth=true to allow all users

Why it fails: Setting --anonymous-auth=true makes the kube-apiserver accept unauthenticated requests, attributing them to the synthetic system:anonymous user. While RBAC might restrict this user, any unauthenticated attacker who can reach the API server endpoint (e.g., via a misconfigured NetworkPolicy, a compromised pod, or a public ingress) gains the ability to enumerate endpoints, read resources if RBAC allows, or potentially exploit additional misconfigurations. This flag should be false in production, forcing every request to carry valid credentials, otherwise authentication becomes a mere formality and the cluster's security boundary collapses.

Enable audit logging to detect unauthorized attempts

Why it fails: Audit logging is a detective control that records a chronological trail of API requests, including who made them, what they accessed, and the outcome. It does not intercept or block malicious actions at execution time; if an attacker has already passed authentication and authorization, they can still perform their actions before any log-based alert triggers. Hardening measures must be preventive, such as NodeRestriction for limiting node privileges or client TLS certificates for establishing identity, while audit logs serve only for post-incident analysis and compliance, not as a barrier against the initial threat.

Disable all authentication mechanisms and rely on network policies

Why it fails: Disabling all authentication mechanisms and relying solely on network policies fundamentally misunderstands the threat model: network policies filter pod-to-pod and pod-to-service traffic at the CNI layer, but they do not police who can issue HTTP requests to the kube-apiserver, which is a workload itself often exposed on a cluster-facing port. Without authentication, every request to the API server is effectively anonymous, so any process with network access—including a compromised pod in a different namespace or a rogue container on a host—can call admin endpoints. Network policies also cannot distinguish between legitimate and malicious API verbs, making this an unworkable substitute for identity-based security.

Seccomp profiles can only be applied to privileged containers

Why it fails: This statement is false: seccomp is a kernel-level syscall filter that applies equally to privileged and non-privileged containers. The privileged flag in Kubernetes expands capabilities and device access, but it does not control or require seccomp filters. In fact, seccomp is most beneficial for unprivileged containers because it reduces attack surface, and privileged containers can also be constrained by seccomp to mitigate their elevated privileges.

Seccomp can only restrict system calls, not allow them

Why it fails: Seccomp profiles are actually BPF programs that can both prohibit and permit individual system calls. A common pattern is a whitelist profile, where all syscalls are denied by default and only specific ones are explicitly allowed. Profiles can also return custom error codes like EPERM or EACCES, so they do more than simply restrict execution; they give fine-grained control over the syscall interface.

Deny all ingress and egress traffic to/from the pod's namespace

Why it fails: This approach targets the entire namespace by effectively selecting all pods, not just the compromised one. Because NetworkPolicy applies to all pods matching its podSelector, a namespace-wide deny-all would cut off traffic for every workload in that namespace, including legitimate services and the forensic tools you may need. It also fails to preserve the isolation boundary at the pod level, so the compromised pod remains reachable from pods in other namespaces unless cluster-wide policies exist. Thus it is over-broad and ignores the requirement to isolate a single pod.

Add a label to the pod and create a NetworkPolicy allowing only traffic from a forensic pod

Why it fails: This approach fails because it explicitly allows traffic to and from a forensic pod, meaning the compromised pod is not fully isolated—it can still communicate, which could let an attacker continue to exfiltrate data or pivot through that allowed path. Additionally, labeling a running pod requires manually modifying its metadata, and any NetworkPolicy that permits traffic must define specific rules, leaving the rest of the pod's connections denied, but the allowed connection itself creates a gap. Full isolation requires denying all traffic, not merely restricting to a trusted counterpart.

Delete the pod

Why it fails: Deleting the pod destroys the very evidence you need to investigate the compromise—filesystem artifacts, process memory, network connections, and logs—which is essential for forensic analysis. Moreover, if the pod is managed by a controller like a Deployment or ReplicaSet, the controller may immediately recreate a new pod with the same image and labels, and without a NetworkPolicy that new pod will have the same exposure. Thus deletion neither isolates nor remediates; it loses data and potentially lets the attacker redeploy a fresh instance.

--plugin-dir=/admission

Why it fails: The --plugin-dir flag is not a valid kube-apiserver flag for enabling admission plugins. The correct way to enable admission controllers is via --enable-admission-plugins; plugin directories are not used by the API server to discover admission plugins. The flag appears may be confused with --plugin-dir for certain CLI tools, but it has no effect on admission control.

--admission-control=NodeRestriction

Why it fails: The --admission-control flag was the old, deprecated flag used in older Kubernetes versions (pre-1.10) to enable admission controllers. In modern Kubernetes, it has been replaced by --enable-admission-plugins, and using the deprecated flag with a newer API server will not enable the controller; it may either be ignored or cause an error depending on the version. Thus, this flag is no longer valid for enabling NodeRestriction.

Open all ports on nodes to allow easy debugging

Why it fails: Opening all ports on nodes removes the network access controls that protect node services and pod traffic. This practice directly contradicts the security principle of default deny: it allows any network attacker to reach any service on the node, including kubelet's unauthenticated read-only port (10255) or other sensitive management interfaces. For debugging, Kubernetes provides safer alternatives such as kubectl port-forward, ephemeral containers, or temporarily creating a NetworkPolicy to allow specific traffic. Exposing all ports also bypasses the Egress/Access control that firewalls and security groups provide, making the node trivially reachable from the internet or other untrusted networks.

Run all containers as root user

Why it fails: Running containers as root is the reverse of least privilege and is especially dangerous on Kubernetes. Although processes inside a container are namespaced, the root user still has root-equivalent capabilities (like CAP_SYS_ADMIN, CAP_NET_ADMIN) over the container's kernel namespace, and a kernel vulnerability can let them break out to the host. Kubernetes best practices require setting securityContext.runAsNonRoot: true and runAsUser to a non-root UID to reduce privilege; many images also run with read-only root filesystems. Root containers also make filesystem writes more impactful—for example, filling /var/log or /var/lib/kubelet could disrupt the whole node.

Configure RBAC for etcd

Why it fails: Kubernetes RBAC applies to the Kubernetes API server, not directly to etcd, which is a separate datastore. etcd does offer its own user/role object model, but in standard cluster deployments access is enforced through mutual TLS client certificate authentication, not RBAC. Configuring RBAC is therefore not one of the required controls to secure etcd itself.

Allow anonymous access for monitoring

Why it fails: Allowing anonymous access to etcd means any unauthenticated client can read or modify the entire cluster state, including secrets, without presenting credentials. This completely undermines the confidentiality and integrity of the datastore. Monitoring should be performed using authenticated read-only accounts or dedicated agent certificates, never by permitting anonymous connections, making this practice inherently insecure.

--admission-control=NodeRestriction on the kubelet

Why it fails: The --admission-control flag is deprecated and was historically used on the kube-apiserver, not the kubelet. The kubelet has no admission control chain and cannot enforce NodeRestriction. Admission controllers are components of the API server that intercept requests before they are persisted. Therefore, specifying this flag on the kubelet is both syntactically and architecturally incorrect.

--enable-admission-plugins=NodeRestriction on the kubelet

Why it fails: The --enable-admission-plugins flag exists only on the kube-apiserver, where it lists the admission controllers that should be activated. The kubelet is a node-level agent that does not process API admission webhooks or abstract authorization decisions. NodeRestriction is an API server admission plugin that must run in the kube-apiserver's request pipeline. Passing it to the kubelet would either be ignored or cause a startup failure.

--node-restriction=true on the kubelet

Why it fails: There is no --node-restriction=true flag in any Kubernetes component. NodeRestriction is not a boolean toggle; it is an admission controller that must be included in the --enable-admission-plugins list on the kube-apiserver. Even if such a flag existed on the kubelet, it would not activate the API server-side admission logic. This option confuses a feature with a simple configuration switch and mistakenly targets the wrong component.

kubectl cp <pod>:/ -c <container> /tmp/forensic && kubectl logs <pod> --previous

Why it fails: The correct approach is to use `kubectl cp` to copy the entire container filesystem to a local directory, which creates a static, bit-for-bit snapshot without injecting processes or altering the container's runtime state. Specifying `-c <container>` is mandatory for multi-container pods to target the exact compromised container. Adding `kubectl logs <pod> --previous` retrieves logs from the previously terminated container instance, which may hold the attacker's commands or crash output that the current instance no longer contains. This combination preserves both disk state and historical log evidence for offline analysis.

kubectl exec <pod> -- cat /proc/1/cmdline && kubectl exec <pod> -- ls -la /

Why it fails: Executing `cat /proc/1/cmdline` and `ls -la /` via `kubectl exec` spawns new processes inside the running container, which changes its process table, updates access times, and may trigger the attacker's monitoring or anti-forensic defenses. This live interaction contaminates the scene and violates forensic best practices, as even read-only commands can alter the system state at the OS level. The output also depends on the container's available binaries and does not capture deleted files, open file descriptors, or memory contents, making it a shallow and unreliable source of evidence.

kubectl exec -it <pod> -- bash && kubectl exec <pod> -- cat /var/log/syslog

Why it fails: Launching an interactive shell with `kubectl exec -it <pod> -- bash` is especially intrusive because it creates a persistent process, writes shell history, allocates a pseudo-TTY, and modifies the container's runtime environment in ways that can destroy volatile evidence. Following this with another `kubectl exec` to read `/var/log/syslog` adds yet another process and may overwrite important log entries due to normal system activity. In a compromised container, executing a shell could also trigger malicious behaviors or give the attacker interactive access, so this approach is both forensically unsound and operationally dangerous.

containers: - name: app securityContext: runAsNonRoot: true volumeMounts: - mountPath: /tmp name: scratch volumes: - name: scratch emptyDir: {}

Why it fails: The runAsNonRoot field only verifies that the container process uses a non-zero user ID; it does not restrict filesystem write access. The container's root filesystem remains fully writable, so a compromised process could modify binaries, configuration, or other persistent data. The emptyDir volume provides temporary writable storage at /tmp, but it does not make the main filesystem immutable.

containers: - name: app securityContext: readOnlyRootFilesystem: false

Why it fails: This option explicitly sets readOnlyRootFilesystem to false, which is the default behavior and leaves the container's overlay filesystem writable. With this setting, any process in the container can write to the root filesystem, so the filesystem is not immutable at all. Additionally, there is no volume mounted for temporary writes, so the solution fails the requirement on both immutability and writable scratch space.

containers: - name: app securityContext: readOnlyRootFilesystem: true volumeMounts: - mountPath: /tmp name: scratch volumes: - name: scratch hostPath: path: /tmp

Why it fails: The hostPath volume mounts the node's /tmp, so writes escape the container and persist on the host, defeating immutability and leaking data between pods. An emptyDir volume would give a pod-scoped scratch space. The readOnlyRootFilesystem setting is correct, but the volume choice breaks the requirement.

Use 'kubectl exec -it <pod> -- bash' to inspect the container, then kill the malicious process from within

Why it fails: Executing into a container with kubectl exec is not a reliable containment or investigation method because many production images lack a shell such as bash, and the malicious process may be running as PID 1 where a SIGTERM could simply cause the container runtime to restart it per the pod restart policy. Moreover, interactive access alters the container's filesystem and process state, contaminating memory and disk forensic evidence that the incident response team would need to analyze the C2 traffic and the attacker's tooling.

Scale the 'payment-processor' Deployment to 0 replicas to immediately stop all pods

Why it fails: Scaling the Deployment to zero replicas is an indiscriminate action that halts all instances of the payment-processor service, not just the compromised pod, causing an immediate availability outage and disrupting legitimate traffic processing. This abrupt termination also discards the running container's memory snapshot, open network connections, and file descriptors, which are critical for analyzing the attack, and it prevents you from isolating which specific process or artifact was compromised.

Check the Falco logs for an alert containing the pod name, then use 'kubectl delete pod' to remove the compromised pod

Why it fails: Depending on Falco alerts is uncertain because the default rule set is focused on syscall anomalies and may not flag outbound C2 traffic, and even if an alert exists it might be too generic to correlate with this particular pod. Deleting the pod is also ineffective as the owning ReplicaSet immediately schedules a replacement, which could land on the same node and inherit the same vulnerable image or configuration, so the attacker's foothold is not removed. Additionally, pod deletion erases the container's writable layer and logs, destroying evidence of the intrusion.