Hard Difficulty Questions

The kubelet is running out of disk space.

Why wrong: If the host runs out of disk space, the kubelet will typically trigger disk pressure taints or fail during write-heavy operations like container image pulling and logging. It would not produce a filesystem 'permission denied' error during the initial read of its static configuration files.

The kubelet is not able to reach the API server.

Why wrong: A failure to reach the Kubernetes API server due to network partitions, incorrect API endpoints, or firewall rules results in connection timeouts, 'connection refused' errors, or TLS handshake failures. It does not manifest as a local filesystem permission denied error when the kubelet process starts.

The kubelet binary is missing.

Why wrong: If the kubelet binary is absent from the host's executable paths, the systemd service manager will fail to launch the process entirely, reporting a 'file not found' or 'no such file or directory' error. It cannot execute far enough to encounter or report a permission denied error on its configuration.

kubeadm reset and re-init

Why wrong: `kubeadm reset` destroys the current control plane state, including etcd data, and `kubeadm init` creates a brand-new cluster with empty storage. It does not ingest a snapshot, so all events, resources, and secrets from the snapshot are lost. This is a rebuild, not a restore, and is only useful if you intend to re-bootstrap from scratch rather than recover a snapshot.

kubectl apply -f snapshot.yaml

Why wrong: `kubectl apply` only accepts Kubernetes manifests and submits them to the API server via the control plane, which itself relies on etcd for storage. An etcd snapshot file is binary and contains internal metadata, not YAML resources, so it cannot be applied; the command would either error or be silently ignored as an invalid object. There is no mechanism in `kubectl` for reading or writing etcd snapshots.

The pod has a resource request that exceeds available CPU on all nodes

Why wrong: This option is only partially true. The scheduler reported that three nodes have insufficient CPU, meaning those nodes lack enough allocatable CPU for the pod's request, but the remaining node's problem is an untolerated taint, not a CPU shortage. Therefore the claim that all nodes lack CPU is false; one node may actually have sufficient free CPU but is unavailable due to a taint. The option overstates the resource constraint.

The pod does not tolerate any taints, and all nodes have taints

Why wrong: This misstates the event detail. Only one node in the cluster has a taint relevant to the pod; the other three nodes are not tainted but have insufficient CPU. Moreover, 'does not tolerate any taints' is inaccurate because the pod could tolerate other taints, just not the specific one present. The problem is not that every node is tainted.

The cluster has only one node with sufficient CPU, but it is cordoned

Why wrong: Cordoning is a manual node maintenance state where the scheduler avoids placing new pods, but the described events mention nothing about a node being cordoned. Instead, the events specifically identify an untolerated taint on one node and insufficient CPU on three nodes. If a node were cordoned, 'kubectl describe node' would show SchedulingDisabled, but the pod's events would typically say 'node(s) cordoned.' Since these messages are absent, cordoning is not part of this diagnosis.

Upgrade kubeadm on the control plane node, then upgrade kubelet and kubectl on worker nodes, then upgrade kubelet and kubectl on control plane node.

Why wrong: Upgrading worker nodes before the control plane components introduces significant version skew, potentially causing `kubelet` on the workers to be incompatible with an older control plane API server. This can lead to worker nodes failing to register or function correctly, disrupting cluster operations. Furthermore, this approach neglects to drain worker nodes, resulting in service disruption as pods are abruptly terminated during `kubelet` restarts without proper eviction.

Drain all nodes, upgrade kubelet and kubectl on all nodes, then upgrade kubeadm on the control plane node.

Why wrong: Draining all nodes simultaneously is highly disruptive and will lead to a complete cluster outage, as no pods will be available to serve traffic during the upgrade. Additionally, attempting to upgrade `kubelet` and `kubectl` on all nodes before upgrading `kubeadm` and the control plane components violates the recommended upgrade order. This can result in an unmanageable cluster state due to severe version mismatches between components and the orchestration tool.

Upgrade kubelet and kubectl on all nodes first, then upgrade kubeadm on the control plane node.

Why wrong: Upgrading `kubelet` and `kubectl` on all nodes first, before `kubeadm` on the control plane, is fundamentally incorrect and risks severe cluster instability. `kubeadm` is the orchestrator for the upgrade process and must be updated first to correctly manage the new version's component deployments and configurations. Proceeding this way creates immediate version skew between the `kubelet` and the older control plane, potentially rendering the cluster unmanageable and unstable.

The Kubernetes API server is down

Why wrong: If API server is down, kubectl commands would fail with connection errors, not metrics errors.

The kubelet is not running on all nodes

Why wrong: If kubelet is not running, nodes would be NotReady, not just metrics unavailable.

One or more nodes are in NotReady state

Why wrong: NotReady nodes might not report metrics, but the error message 'metrics not available yet' specifically points to Metrics Server absence.

The Job will restart the completed pods

Why wrong: Pods that exit successfully with a zero status code are transitioned to the Succeeded phase and are never restarted by the Job controller. The Job's restart policy, which must be set to either OnFailure or Never, only applies to failed containers or pods. Completed pods remain intact in the cluster to allow administrators to inspect their logs.

The Job will continue to run new pods indefinitely

Why wrong: The Job controller regulates pod creation based on the completions parameter in the Job specification. Because this parameter defines a strict, finite target of successful pod completions, the controller will stop spawning new pods as soon as that target is met. It does not behave like a Deployment or ReplicaSet, which maintain a continuous running state indefinitely.

The Job will be marked as Failed because parallelism is not fully utilized

Why wrong: The parallelism field defines the maximum number of pods that can run concurrently, acting as a ceiling rather than a strict requirement. If the actual number of active pods falls below this limit—due to scheduling constraints or because the completion target is nearly reached—the Job is not considered failed. It will successfully complete as long as the target completions are met.

The node has a taint that prevents scheduling

Why wrong: A taint is a scheduling attribute placed on a Node object to repel Pods; it does not alter the Node's reported readiness. The kubelet continues sending heartbeats and the Node condition remains 'Ready' even when a taint like 'node.kubernetes.io/unschedulable' is present. Node readiness is derived from kubelet-reported conditions such as memory pressure, pid pressure, disk pressure, and runtime status, not from taints. Taints affect where new Pods are scheduled, but they never cause the Node itself to be marked NotReady.

The node has insufficient disk space

Why wrong: Insufficient disk space triggers the kubelet to set the NodeCondition 'DiskPressure' to True, which is a distinct condition from 'Ready'. DiskPressure is an eviction signal that causes the kubelet to reclaim disk by evicting pods, and the Node may be given the taint 'node.kubernetes.io/disk-pressure', but the 'Ready' condition remains True as long as the kubelet is healthy and can report status. A node with DiskPressure can still accept new pods unless another mechanism like a taint prevents it; it is not marked NotReady. The confusion arises because DiskPressure is a type of node pressure, but Kubernetes deliberately separates resource pressure from readiness.

The node's IP address has changed

Why wrong: A node IP address change does not directly set the NotReady condition; NotReady is caused by the Node controller not receiving heartbeats or by kubelet-reported health failures. If the IP changes, the API server may lose the old endpoint used to reach the kubelet, but the kubelet will continue trying to update its Node object and can re-register with the new address. In practice, a changed IP can lead to the Node being deleted and re-created, or to the kubelet being unable to connect to the API server, but the Node object itself is not marked NotReady by this event. The NotReady status reflects an inability to communicate current health status, not an address mismatch.

The scheduler is not running.

Why wrong: If the kube-scheduler control plane component were completely inactive, the Pod would remain in a Pending state indefinitely without any scheduling decision events. The kubectl describe output would show no scheduling attempts or events at all, rather than specific scheduling failure events pointing to taints or toleration mismatches.

The Pod requests more CPU than any node can provide.

Why wrong: When a Pod cannot be scheduled due to insufficient CPU, the scheduler emits a specific FailedScheduling event indicating 'Insufficient cpu' on the candidate nodes. In this scenario, the scheduling failure event explicitly references taints and tolerations rather than resource exhaustion or capacity limits.

The Pod has a node selector that doesn't match any node.

Why wrong: If a Pod specifies a nodeSelector or nodeAffinity that cannot be satisfied by any active node, the scheduler reports a FailedScheduling event with a 'node(s) didn't match Pod affinity/selector' message. This is distinct from taint-related rejections, which are triggered by the node's taints repelling the Pod.

Restart the kube-apiserver on the control plane

Why wrong: The kube-apiserver is a control-plane component that serves the Kubernetes API and persists node objects in etcd; it does not control the kubelet or the container runtime on any worker node. Restarting it cannot fix a node that is NotReady, because the API server only receives NodeStatus updates from the kubelet, it does not create or heal them. Moreover, restarting the API server is risky and unnecessary when the failure is at the node agent or runtime layer, and it would not regenerate any missing kubelet registration.

Run 'kubectl delete node <node-name>' to re-register

Why wrong: Running 'kubectl delete node <node-name>' only removes the Node object from the cluster's etcd-backed state; it does not restart or repair the kubelet or container runtime on the node itself. When the kubelet comes back online, it typically creates a new Node object automatically, so a delete is not a meaningful 're-registration' and won't cure an underlying service failure. Deleting a Node can also disrupt workloads that were scheduled on it and is not considered a troubleshooting step for a failing kubelet or runtime.

Drain the first node

Why wrong: Draining a node is a critical step performed immediately before upgrading its local kubelet and container runtime to safely evict running workloads. Since the first control plane node's core components have already been upgraded, draining this node is premature before the remaining control plane nodes are brought up to the target version.

Upgrade the worker nodes

Why wrong: In a highly available kubeadm cluster, all control plane nodes must be fully upgraded to the target version before any worker nodes are modified. Upgrading worker nodes prematurely violates Kubernetes version skew policies and can lead to control plane instability or communication failures between newer worker kubelets and older control plane components.

Run kubeadm upgrade plan

Why wrong: Running the upgrade plan command is a preliminary, non-mutating step used to check for available versions and validate that the cluster is ready for an upgrade. Because the upgrade process has already been initiated and the first control plane node is upgraded, running this planning command at this stage is redundant and does not advance the upgrade sequence.

Apply a NetworkPolicy with ingress rule allowing from a specific pod only

Why wrong: This alone does not deny other traffic; you need a default deny policy.

Apply a NetworkPolicy with empty podSelector and ingress rule allowing all

Why wrong: This would allow all ingress to all pods.

Apply a NetworkPolicy with podSelector: matchLabels: { app: myapp } and ingress rule with namespaceSelector: { matchLabels: { name: monitoring } }

Why wrong: This selects pods in monitoring namespace, not the monitoring pods themselves.

The secret type is wrong; it should be kubernetes.io/dockercfg.

Why wrong: The correct and modern secret type for Docker registry authentication credentials is `kubernetes.io/dockerconfigjson`, not `kubernetes.io/dockercfg`. While using an incorrect secret type would indeed prevent successful image pulls, the more fundamental problem here, as highlighted by the correct option, is that the ServiceAccount isn't even referencing *any* secret for image pulls. The type becomes relevant only once a secret is actually linked.

The ServiceAccount does not have the imagePullSecrets field configured.

Why wrong: For a Pod to automatically inherit image pull credentials from its associated ServiceAccount, that ServiceAccount must explicitly list the relevant `Secret` in its `imagePullSecrets` array. If this field is absent or empty within the ServiceAccount definition, the Kubernetes admission controller will not inject the necessary authentication tokens into the Pod, leading to `ImagePullBackOff` errors due to unauthorized access to private registries. This is the standard and most secure method for managing registry credentials at scale.

The Pod spec must set the secret in imagePullSecrets directly.

Why wrong: While it is technically possible to specify `imagePullSecrets` directly within a Pod's `spec`, this approach is generally discouraged for security and manageability reasons, especially when ServiceAccounts are available. The recommended practice is to configure the `imagePullSecrets` on the ServiceAccount, allowing all Pods using that ServiceAccount to automatically inherit the credentials without needing individual Pod modifications. This centralizes credential management and reduces the risk of exposing secrets in Pod definitions.

Run 'kubeadm join' again with the correct token.

Why wrong: Running 'kubeadm join' again with a new or correct token is not the right fix because the original join command already executed successfully enough for node2's kubelet to start and generate the pending CSR. The presence of a pending CSR indicates that the bootstrap token phase worked; what is missing is the certificate approval step, not the token. Re-running join would not approve the existing CSR, and it may fail if the node is already partially registered or cause a duplicate join attempt. The correct action is to approve the pending CSR, not to re-initiate the join procedure.

Add the node's IP to the API server's TLS certificate.

Why wrong: The API server's TLS certificate is used to prove the API server's identity to clients, and it should contain the DNS names or IP addresses that clients (including kubectl, kubelets, and the API server itself) will use to connect to it. Worker node IPs are not required in the API server certificate because the kubelet on node2 uses its own client certificate—issued from the CSR—to authenticate to the API server. Modifying the API server certificate would not affect whether node2's CSR is approved; that CSR is for node2's client identity, not for the API server's serving identity. This option confuses server-side TLS with client-side TLS and is unnecessary for fixing the pending CSR.

Restart the kubelet on node2.

Why wrong: Restarting the kubelet on node2 will cause it to re-read its configuration and re-submit its certificate signing request, but a restart does not and cannot approve the pending CSR—approval is an administrative action performed through the Kubernetes API. The current issue is not that the kubelet is malfunctioning or missing a state; it simply has a CSR stuck in the 'Pending' phase because no approver has signed it. Restarting will just produce a new or renewed request for the same certificate, and the node will remain NotReady until someone runs 'kubectl certificate approve'. This option attempts to solve a permission approval problem with a process-level action, which is ineffective.

Modify the CoreDNS configuration to use a different DNS policy.

Why wrong: CoreDNS configuration primarily governs how DNS queries are resolved within the cluster, mapping service names and pod hostnames to IP addresses. Since pods can successfully resolve the database service and individual pod hostnames, DNS resolution is functioning as expected. Modifying CoreDNS settings, such as changing the `policy` or `forward` directives, would not address a connectivity issue that arises *after* a successful DNS lookup, which points to network-level blocking, not name resolution failures.

Remove the headless service and use a regular ClusterIP service for pod discovery.

Why wrong: A headless service (ClusterIP: None) is crucial for StatefulSets because it allows DNS resolution to return the IP addresses of individual pods, enabling direct pod-to-pod communication and stable network identities (e.g., `db-0.db.mynamespace.svc.cluster.local`). Replacing it with a regular ClusterIP service would only provide a single, stable IP for the service, load-balancing traffic across pods but preventing direct, individual pod addressing required for certain database clustering or peer-to-peer communication patterns. This would fundamentally break the existing pod identity mechanism.

Change the ClusterIP service 'db' to NodePort to expose the database on each node.

Why wrong: Changing a service type to NodePort exposes the service on a static port on each node's IP address, primarily for external access to the cluster. This modification does not alter or bypass internal Kubernetes NetworkPolicies that govern direct pod-to-pod communication within the cluster. The issue described is about pods communicating directly with other pods' IPs internally, not about external access to a service via node IPs, making NodePort irrelevant to the problem's scope and resolution.

Completed

Why wrong: "Completed" is not a valid value for the Pod's status.phase field. The Kubernetes API only defines Pending, Running, Succeeded, Failed, and Unknown as valid phases. When a container exits with code 0 under restartPolicy: Never, the kubelet marks the phase as Succeeded, not "Completed," so this label is incorrect.

Failed

Why wrong: "Failed" describes a Pod phase where a container exits with a non-zero exit code or where the scheduler/kubelet cannot start the container (e.g., image pull failure). In this scenario, the container has exited with status 0, which is a successful termination; thus the phase cannot be Failed. Only a non-zero exit would transition the pod to Failed under restartPolicy Never.

Running

Why wrong: "Running" is a transient phase that means at least one container is still executing or is in the process of starting or restarting. For a Pod with restartPolicy: Never, once the container exits, even with status 0, the phase immediately moves away from Running to a terminal phase. Because the container has already exited, the Running phase is impossible, so this option is wrong.

Run 'kubeadm init phase certs kubelet-client'

Why wrong: While the 'kubeadm init phase' command suite is used during cluster bootstrapping to generate initial assets, there is no sub-command 'kubeadm init phase certs kubelet-client' designed for renewing an existing certificate. Attempting to run this command will result in an error, as certificate renewal tasks are properly managed under the dedicated 'kubeadm certs renew' command suite.

Renew all certificates using 'kubeadm certs renew all'

Why wrong: Although executing this command successfully renews every control plane certificate managed by kubeadm, it is inefficient and incorrect when the objective is to target only the kubelet client certificate. In a production environment or a CKA exam scenario, you should follow the principle of least disruption by renewing only the specific expired certificate identified.

Delete the old certificate and restart kubelet

Why wrong: Deleting the certificate files manually and restarting the kubelet service will not automatically regenerate the required client certificates on a kubeadm-managed control plane node. This destructive action can lead to API server communication failures and cluster instability, whereas kubeadm provides safe, built-in commands designed specifically to handle certificate lifecycles gracefully.

EndpointSlices are created manually by the user.

Why wrong: EndpointSlices are not meant to be created manually; the endpoint controller automatically creates and manages them for every Service that has a selector. When Pods matching the Service's selector change, the controller reconciles the corresponding EndpointSlices, adding or removing endpoints as needed. Attempting to create them manually is unnecessary and would likely be overwritten or conflict with the controller's generated slices, so the statement is false.

Each EndpointSlice can contain only one endpoint.

Why wrong: An EndpointSlice does not represent a single endpoint but is a collection that can hold multiple endpoints, with a default maximum of 100 per slice. When a Service has more than 100 matching Pods, the controller automatically creates additional EndpointSlices, each containing a subset of the endpoints. This batching is exactly what provides scalability advantages over the monolithic Endpoints object, and limiting a slice to one endpoint would defeat the purpose.

EndpointSlices are only used with ClusterIP services.

Why wrong: EndpointSlices are used with all Service types that have a selector, not just ClusterIP. Services of type NodePort and LoadBalancer also rely on the same EndpointSlice objects because their traffic must be forwarded to the backing Pods; only Services without selectors (like ExternalName) do not generate EndpointSlices. The endpoint controller watches every selected Service regardless of its type, so the claim that EndpointSlices are exclusive to ClusterIP is incorrect.

Restore the snapshot on every etcd member in the cluster

Why wrong: In a multi-node etcd cluster, you do not simply restore the snapshot on one node and expect it to automatically sync to others. Instead, you must restore the snapshot on all members individually using the same snapshot file while carefully configuring unique peer URLs and cluster membership flags before starting the services.

Run 'kubeadm init' to reinitialize the control plane

Why wrong: Running 'kubeadm init' is used exclusively for bootstrapping a brand-new Kubernetes control plane from scratch. It is not part of the etcd disaster recovery workflow, as re-running it on an existing cluster would overwrite configuration files, generate new certificates, and disrupt the existing control plane components.

The nodes have insufficient resources

Why wrong: Insufficient resources would surface in the scheduler's events as a FailedScheduling reason with messages like '0/3 nodes are available: 1 Insufficient cpu, 2 Insufficient memory.' That event explicitly names the missing resource, whereas anti-affinity failures report 'didn't match pod anti-affinity rules.' Also, resource constraints can often be resolved by moving other workloads, but the pod's own resource requests don't make the entire cluster ineligible unless every node lacks capacity.

The pod has a taint tolerance issue

Why wrong: A taint/toleration mismatch would generate a completely different scheduler event: '3 node(s) had untolerated taint {key=value:NoSchedule}' (or NoExecute). That means the nodes are tainted and the pod lacks the corresponding toleration; the scheduler skips those nodes regardless of their available resources. This is unrelated to label-based pod relationships, and the fix is adding a toleration to the pod spec, not changing the cluster's node labels or topology.

The nodes are all cordoned

Why wrong: When nodes are cordoned, kubectl get nodes shows them as SchedulingDisabled, and the scheduler reports '0/3 nodes are available: 3 node(s) were unschedulable.' Cordon is an administrative action that sets the node's unschedulable flag, preventing any new pods from being placed there, but it does not consider pod affinity or toleration rules. The pod remains Pending because the node is intentionally marked unavailable, and it will only schedule after an admin runs kubectl uncordon.

An ingress rule with namespaceSelector matching the same namespace.

Why wrong: This would allow traffic from all pods in the namespace, not just those with label 'app: web'.

An egress rule with namespaceSelector and podSelector.

Why wrong: Egress rules are for outbound traffic; also namespaceSelector is not needed here.

An egress rule with podSelector matching 'app: web' and ports.

Why wrong: Egress rules control outbound traffic, not inbound.