Troubleshooting practice questions

The NodePort is conflicting; change the service type to ClusterIP.

Why wrong: NodePort services expose applications on a port across all cluster nodes, which is handled by kube-proxy at the node level. A container-level port binding conflict occurs within the pod's network namespace, meaning changing the Service type to ClusterIP will not resolve an internal container process failing to bind to its local port.

The container is missing an environment variable required for startup; add it via ConfigMap.

Why wrong: While missing environment variables can cause application startup failures or database connection errors, they do not result in 'address already in use' bind errors. A bind error specifically indicates that the network socket is already occupied by another process within the network namespace, which cannot be resolved by injecting configuration data.

The pod has insufficient memory; increase memory limits in the deployment.

Why wrong: If a pod exceeds its allocated memory limits, the Linux kernel's Out-Of-Memory (OOM) killer terminates the process, resulting in an OOMKilled status (Exit Code 137). This is fundamentally different from a port binding conflict, which is a network socket allocation failure and does not trigger memory-related termination events.

The pod's dnsPolicy is set to Default instead of ClusterFirst.

Why wrong: The scenario establishes that the pod's dnsPolicy is already correctly configured to ClusterFirst, which instructs the kubelet to configure the pod to use the in-cluster DNS service. If it were set to Default, the pod would inherit the node's name resolution configuration instead. Therefore, this option is incorrect because the policy is already set to the desired ClusterFirst value.

The node's network plugin is misconfigured, blocking UDP port 53.

Why wrong: While a misconfigured Container Network Interface (CNI) plugin blocking UDP port 53 would indeed disrupt DNS traffic, such a fundamental network plugin failure would typically cause broader node-to-node or pod-to-pod communication failures across the entire cluster. Because the issue is isolated specifically to DNS name resolution rather than general network connectivity, a complete CNI misconfiguration is less likely than a localized CoreDNS service outage.

The pod's /etc/resolv.conf contains incorrect nameserver entries.

Why wrong: When a pod is created with the ClusterFirst DNS policy, the local kubelet automatically and reliably populates the pod's /etc/resolv.conf file with the correct ClusterIP of the kube-dns service and the appropriate search domains. Manual misconfiguration of this file is rare because it is managed dynamically by the control plane. Thus, incorrect entries in this file are highly unlikely to be the root cause under standard Kubernetes operations.

Run 'kubectl logs <pod-name>' to view application logs.

Why wrong: The `kubectl logs` command only reads stdout/stderr from a container that exists and is currently running — a Pending pod has no container yet. The command will fail with an error like 'container not found' or 'pod is not running' because the kubelet has not created any container. Application logs simply do not exist until the Pod is scheduled and started, making this useless for pending-pod troubleshooting.

Run 'kubectl exec -it <pod-name> -- /bin/sh' to inspect the container.

Why wrong: `kubectl exec` requires a running container with which to attach the interactive process; a Pending pod has not been scheduled to a node and thus has no running container. The kubelet will not have started anything, so exec returns an error immediately. Interactive debugging only becomes possible after the pod reaches the Running phase, so this is not a valid method for investigating scheduling failures.

Run 'kubectl top pod <pod-name>' to check resource usage.

Why wrong: `kubectl top pod` displays real-time resource consumption obtained from the metrics server for pods that are currently Running. A Pending pod is not scheduled, has no CPU/memory metrics, and the command typically reports 'no resources found' or the pod is simply not listed. Furthermore, the metrics-server itself must be deployed and fully operational; otherwise the command fails globally. Thus it cannot diagnose why a pod is pending.

kubectl logs deployment/web-app

Why wrong: kubectl logs deployment/web-app streams the logs of a single pod in the Deployment, so it tells you whether the container is crashing or logging requests. It reveals nothing about the Service's selector, targetPort, or endpoint population; a healthy application can still be unreachable if the Service labels don't match the pods. This command is premature until you have verified that the Service actually has endpoints, so it is not the best first diagnostic step.

kubectl get events --sort-by='.lastTimestamp'

Why wrong: kubectl get events --sort-by='.lastTimestamp' gives a time-ordered view of every cluster event, but a Service selector mismatch usually produces no Event at all. The kube-controller-manager might log warnings when endpoints are empty, but those are often in controller logs rather than in Event objects, so sorting events can still hide the exact Service-to-pod relationship. Event history is valuable for broader troubleshooting after you know what to look for; as a first step it casts too wide a net and dilutes the focused signal that a Service description provides.

kubectl describe ingress web-app

Why wrong: kubectl describe ingress web-app inspects Layer 7 HTTP routing rules, but an Ingress backend merely points to a Service name and port; if that Service has no endpoints, the Ingress can look perfectly configured while the application is still unreachable. Additionally, the workload may not be exposed through an Ingress at all, so running this command first could return an Ingress that does not exist and waste time at the wrong network layer. The correct first move is to verify the Service itself, not the Ingress controller's routing configuration, because the Service is the glue between an Ingress backend and the pods.

The Service selector does not match the pod labels.

Why wrong: If the Service selector did not match the Pod labels, the Kubernetes control plane would fail to populate the Service's Endpoints object. Consequently, internal traffic to the Service IP would also fail, which contradicts the scenario where internal access succeeds.

The NodePort service type is not enabled in the cluster.

Why wrong: NodePort is a standard, built-in Kubernetes Service type that is enabled by default in all compliant clusters. Furthermore, a LoadBalancer service automatically provisions and builds upon NodePort functionality under the hood, meaning NodePort does not need to be independently enabled.

The Ingress resource is missing or misconfigured.

Why wrong: A Service of type LoadBalancer exposes applications directly to the internet via a cloud provider's network load balancer. It operates independently of an Ingress resource, which is a separate API object used for HTTP/HTTPS routing and requires an Ingress controller to function.

kubectl logs <pod-name>

Why wrong: This command shows logs from the current container instance, but since the pod is crashing, the crash output is in the previous instance's logs.

kubectl describe pod <pod-name>

Why wrong: This command shows pod specifications and status, but not the container logs.

kubectl logs --previous <pod-name>

Why wrong: Correct. The --previous flag is equivalent to -p and retrieves logs from the previous instance.

The kubelet is not installed

Why wrong: This is incorrect. The scenario states the kubelet is running, which presumes the kubelet binary is installed and the service is active. If the kubelet were not installed, you would not see a running kubelet process, and the node would likely not even register with the cluster. A missing kubelet would cause a completely different symptom, such as the node never appearing or being in a perpetual register/unregister loop, not a transition from Ready to NotReady.

The node has been cordoned

Why wrong: This is incorrect because cordoning a node (kubectl cordon) only marks it as unschedulable, meaning new Pods will not be scheduled onto it. Cordon does not alter or affect the node's Ready condition; a cordoned node with a healthy kubelet and API connectivity will still show Ready. Node readiness is determined entirely by the API server's ability to receive heartbeats from the kubelet, not by the node's scheduling state.

The API server is down

Why wrong: This is incorrect. If the API server were completely down, you would not be able to query node status at all because kubectl commands would fail to reach the API server. Moreover, an API server outage would affect all nodes in the cluster, not just a single node, causing every node's status to become stale or unknown, rather than a single node being marked NotReady while others remain Ready. The described symptom—a single node NotReady with its kubelet running—specifically points to a node-level communication problem, not a control-plane-wide failure.

The pod needs a toleration for node.kubernetes.io/not-ready

Why wrong: This is not the root cause because the node.kubernetes.io/not-ready taint is automatically added by the node controller when a node becomes NotReady. Even if the pod tolerated this taint, the kubelet on an unhealthy node cannot fetch/start containers, and the scheduler still considers the node unschedulable due to the NotReady condition. Tolerating the taint does not make the node Ready; it only grants permission to schedule onto an already compromised node.

The pod has a resource request that cannot be met

Why wrong: Although unschedulable requests are a frequent cause of Pending, resource insufficiency typically appears as events like 'Insufficient cpu' or 'Insufficient memory' on a Ready node. Here, all nodes are NotReady, so the scheduler filters them out due to taints and node conditions before it even evaluates resource requests. Without at least one Ready node with available capacity, resource requests are never the limiting factor.

The scheduler is not running

Why wrong: If the scheduler were not running, no pod in the cluster would be assigned to a node, and this would affect all workloads, but node health would remain unchanged. In this scenario, the nodes themselves are NotReady, which alone is sufficient to leave the pod pending even when the scheduler is healthy. A missing or crashed scheduler would also produce different diagnostic signs, such as no 'schedule' attempts in scheduler logs and a separate control-plane failure, but it does not explain taint-related pending behavior.

Check kube-proxy logs on the nodes

Why wrong: Checking kube-proxy logs is a later diagnostic step, not a first-step check. Kube-proxy continuously reconciles iptables or IPVS rules based on Service and EndpointSlice objects; if the Service's selector matches no pods, the Endpoints object is empty, and no amount of kube-proxy log inspection will reveal a rule problem. Even if kube-proxy were misconfigured, you'd first want to confirm that backends exist. In a troubleshooting workflow, verifying Endpoints is faster, has no logs to parse, and covers the most common failure mode.

Check DNS resolution

Why wrong: Checking DNS resolution addresses a symptom where a Service name cannot be resolved, but the question describes an unreachable Service, not a name-resolution failure. If the Service itself has no endpoints, ClusterIP traffic will be dropped regardless of whether DNS correctly returns the service IP. DNS checks are relevant only when the client gets 'host not found' errors, which is a different failure domain (CoreDNS vs. endpoint selection). Therefore, inspect the Service's Endpoints before investigating DNS.

Restart the service

Why wrong: Restarting the Service object is not a meaningful action because Services are declarative control-plane objects with no embedded state or process to restart; the Kubernetes API doesn't act on a 'restart' for Services. Even if you delete and recreate the Service, the same selector and missing endpoints will reproduce the identical failure. The real issue is almost always the backing pods or their readiness, and a restart masks the symptom while adding no diagnostic value. Instead, you should inspect the Endpoints and the pods' labels and readiness status.

kubectl delete pod -n kube-system -l k8s-app=kube-dns

Why wrong: Deleting the CoreDNS pods is a recovery/restart action, not a debugging technique, and it actively destroys the very diagnostic evidence you need — such as crash-loop stack traces, concatenated error logs, and pod-level events. If a pod is in a CrashLoopBackOff, deleting it may merely reschedule a new instance, and the same underlying misconfiguration (Corefile syntax error, missing ConfigMap, conflicting port bindings) will immediately reappear. This command can also cause temporary DNS outages for all cluster workloads, so you should only ever consider it as a last-resort remediation after inspecting `kubectl describe pod` and `kubectl logs`.

kubectl scale deployment coredns --replicas=2

Why wrong: Scaling the `coredns` Deployment changes the desired replica count, so it is a capacity or availability adjustment, not a diagnostic step. It provides no information about why CoreDNS is not responding — for example, it cannot reveal a ZoneNotAuthoritative plugin error, a bad forward directive, or a readiness probe failure. Running this command may actually mask a horizontal scaling misconfiguration or worsen resource contention, and the correct debugging practice is to use `kubectl get deployment coredns -n kube-system` and `kubectl describe` to inspect the deployment's status, events, and rollout conditions before considering any scaling action.

kubectl run -it --rm test --image=busybox -- nslookup kubernetes.default

Why wrong: This command launches an ephemeral busybox pod and runs `nslookup kubernetes.default`, which is a client-side end-to-end validation of DNS resolution from inside the cluster, not a method for debugging the CoreDNS pods themselves. A successful lookup only indicates that the DNS server answered; a failure could be caused by many layers unrelated to CoreDNS, including CNI network policies, kube-proxy iptables rules, service discovery search domains, or even the test pod's own `/etc/resolv.conf`. It is useful as a smoke test after applying a fix, but it gives no visibility into CoreDNS internals, so it is not an appropriate debugging command for diagnosing a broken CoreDNS pod.

kubectl logs <pod>

Why wrong: kubectl logs <pod> exposes the container's stdout/stderr output, which may reveal an application-level failure, but it cannot show the kubelet's perspective: the restart count, exit code from the last terminated instance, or the Events stream that records failed liveness probe or container create failures. Since CrashLoopBackOff is a scheduling/restart state set by kubelet rather than an app message, logs alone are insufficient to identify the underlying cause.

kubectl get pod <pod> -o yaml

Why wrong: kubectl get pod <pod> -o yaml retrieves the live pod manifest and its status section, which contains container states, restart counts, and last termination details, but it lacks the ordered Events that explain the sequence of failures. The YAML output is also overly verbose, mixing spec, metadata, and status, and its status conditions update slowly; it may not show transient probe failures or image pull backoff reasons that describe surfaces from the event log. Thus it is inferior for diagnosing the root cause of CrashLoopBackOff.

kubectl top pod <pod>

Why wrong: kubectl top pod <pod> reads metrics from the Metrics API (heapster/metrics-server) and reports current CPU and memory usage of the pod's containers, which is irrelevant for CrashLoopBackOff since the pod is restarting before steady-state usage. It provides no access to container state, restart history, exit codes, or events, so it cannot inform why the crash loop is happening. Resource exhaustion can cause crashes, but top shows usage, not OOM kill reasons or liveness probe failures.

Check kubelet logs on the node

Why wrong: Checking kubelet logs on the node is a useful secondary step, but it's not the definitive first choice because it requires network access to the node and parsing unstructured journald or log files across many components (sandbox, runtime, CNI). Kubelet logs can reveal the exact error—for example, a failed CRI call or a failed Exec probe—but they give you no direct view of the Node object's condition history or events that describe node-level state changes. As a first step, describe node provides the authoritative condition and heartbeat data from the apiserver, guiding whether you even need to inspect local logs.

Restart kubelet

Why wrong: Restarting the kubelet is a reactive remediation action, not a diagnostic step, and blindly restarting it can mask the root cause—for example, a persistent disk pressure condition or a broken container runtime will simply recur after restart. It also doesn't help if the node is NotReady because the control plane has not heard a heartbeat, since a restart of kubelet may actually interrupt the Kubernetes-managed heartbeat loop and delay recovery. You should always ascertain the underlying cause with describe node and relevant logs before taking any disruptive action, especially in a production cluster.

Check API server logs

Why wrong: Checking API server logs is an incorrect choice because the kube-apiserver does not compute or store node-level conditions; it merely persists the Node object status that the kubelet reports via PATCH updates. The node-lifecycle controller on the kube-controller-manager (not the API server) is responsible for marking a node NotReady after node-monitor-grace-period with no heartbeats, and that decision appears in the Node's status, not in API server audit logs. API server logs would show requests from the kubelet (like heartbeat calls), but they don't include the node's local runtime or resource pressure information that would explain why the node became NotReady.

Image pull failure

Why wrong: Image pull failure occurs after the sandbox (pause container) is successfully created, because the kubelet first asks the container runtime to create the pod sandbox and only then pulls the application images. A failure to pull an image surfaces as an ErrImagePull or ImagePullBackOff event on the pod, not as a CRI 'CreateContainerSandbox' error. The sandbox creation phase does not depend on the application image being available locally or in the registry.

Node disk pressure

Why wrong: Node disk pressure is a node-level condition that causes the kubelet to evict pods or refuse admission of new pods, but it does not specifically trigger a sandbox creation failure. If disk pressure is present, the kubelet may mark the node as 'MemoryPressure' or 'DiskPressure' and prevent new pods from scheduling, but the underlying CRI sandbox creation call is not directly rejected because the disk is full. To diagnose sandbox creation failures you need to inspect events and logs, not rely on pressure conditions.

Incorrect security context

Why wrong: Incorrect security contexts apply to the container process, not to the creation of the sandbox itself. While the pod's security context (e.g., runAsUser, seccomp, capabilities) is eventually passed to the container runtime, a mistake in it typically causes a 'CreateContainer' error rather than a 'CreateContainerSandbox' error. The sandbox is created before the individual containers are started, and the sandbox only gets basic namespace and network configuration, so a security-context issue is a much later failure point.

kubectl logs <pod>

Why wrong: kubectl logs <pod> is wrong because it streams the container's stdout/stderr, which only exists if the container image was successfully pulled and the process actually started. With ImagePullBackOff the container never leaves the Waiting state, so there are usually no application logs to inspect; the decision to pull and the failure reason are handled by the kubelet, not by the container runtime's output stream.

kubectl get events

Why wrong: kubectl get events is wrong because it returns a cluster/namespace-wide list of events that are time-stamped and can be pruned or truncated; it may not clearly correlate to the specific pod's current ImagePullBackOff state, especially if the pull attempts happened many times and older events were compacted or evicted. kubectl describe pod scopes the relevant events to that pod and pairs them with the live status, making it the more targeted and reliable diagnostic command.

kubectl exec -it <pod> -- sh

Why wrong: kubectl exec -it <pod> -- sh is wrong because exec requires a running container with a shell installed, but a pod in ImagePullBackOff has no container process at all because the kubelet could not pull the image. Even if you tried to exec, the API server would return an error saying the container is not running or is waiting to start, so this command cannot provide any information about why the image pull failed.

DNS resolution failure

Why wrong: DNS resolution failure is a symptom of a broader networking issue, not a cause of ClusterIP unreachability. ClusterIPs are virtual IPs programmed into iptables or IPVS by kube-proxy; even if DNS works perfectly, a Service's ClusterIP remains unreachable when kube-proxy's rules are absent or stale. DNS failures typically stem from CoreDNS pods being down, which is independent of the data path that routes ClusterIP traffic.

Ingress controller not set up

Why wrong: An Ingress controller operates at Layer 7 and provides HTTP/HTTPS routing based on hostnames and paths, but it has no role in making a ClusterIP reachable from within the cluster. ClusterIP traffic is handled entirely at Layer 4 by service proxy rules; Ingress is only relevant for external access to Services, often via NodePort or LoadBalancer, and cannot fix or cause a direct ClusterIP connectivity failure.

Service type should be NodePort

Why wrong: The Service type does not affect ClusterIP reachability because every Kubernetes Service, regardless of whether it is ClusterIP, NodePort, or LoadBalancer, is assigned a ClusterIP and is reachable at that address from inside the cluster. NodePort merely adds a static port on every node for external traffic; choosing NodePort would not resolve an internal ClusterIP connectivity problem. Changing the type would only expose the Service externally and would not repair the underlying missing or broken forwarding rules.

kubectl get nodes

Why wrong: kubectl get nodes only shows the cluster's view of node readiness, which is based on the kubelet's last heartbeat. It does not expose any local kubelet state, so if the kubelet has crashed or lost connectivity, this command merely displays the symptom (NotReady) while offering no diagnostic details about why the kubelet stopped reporting; it is useful for confirming the problem, not for debugging it.

Check API server logs

Why wrong: API server logs, while they could show missing kubelet heartbeats, are stored on control plane nodes and are not the most direct way to diagnose a worker node's NotReady status. Searching them is indirect and slow, since the absence of a heartbeat is a downstream effect of the kubelet's local failure; the definitive evidence lives in the kubelet's own systemd state and logs on the affected node, not in the API server's request history.

kubectl get pods -o wide

Why wrong: kubectl get pods -o wide is wrong for diagnosing CrashLoopBackOff because it only expands the output to include the pod's node IP, host IP, and assigned node, none of which explain why the container is crashing. The pod's STATUS will simply show CrashLoopBackOff, but this command does not fetch logs, show termination reasons, or list events, so it gives no insight into the root cause. It is useful for verifying scheduling or network placement, not for debugging an application that continuously restarts.

kubectl edit deployment <deploy>

Why wrong: kubectl edit deployment <deploy> is wrong as a diagnostic step because it mutates the deployment's desired state without first identifying the cause of the crash. When a pod is in CrashLoopBackOff, the issue is usually inside the container image or its runtime configuration, so blindly editing the deployment risks introducing unrelated changes or restarting the rollout without learning what went wrong. After you diagnose the problem—for example, a wrong command or a bad image—this command can be used to apply a fix, but it is not a diagnostic tool for the crash itself.

kubectl logs <pod-name>

Why wrong: kubectl logs is ineffective because the container has not started; there is no container to stream logs from, and the command will fail with an error indicating the container isn't running. Logs only capture output from a successfully launched container, so this does not reveal why the container runtime failed to create or start the container, such as image pull issues or volume mount failures.

kubectl get events --watch

Why wrong: kubectl get events --watch is wrong because it is a long-running, blocking command that streams only new events from all namespaces and does not surface the historical events that caused the ContainerCreating state. By the time you run it, the relevant kubelet events have already been recorded, so you would either miss them or be flooded with unrelated cluster events. The --watch flag does not replay existing events, making kubectl describe pod the targeted way to see that pod's event history.

kubectl exec -it <pod-name> -- /bin/sh

Why wrong: kubectl exec is invalid for a Pod in ContainerCreating because the container does not yet exist or is not running, so there is no process namespace to attach to and no shell to start. Exec requires a running container and a working container runtime, both of which are absent when the pod is still being created. Therefore it would simply return an error instead of providing any diagnostic information.

kubectl delete node <node-name>

Why wrong: Deleting the Node object with kubectl only removes it from the cluster's control plane registry; the physical machine remains untouched and kubelet will recreate the Node resource on its next sync. This action doesn't free any blocks on the underlying filesystem, so the kubelet's eviction manager will still observe the same high disk usage and re-enter DiskPressure. It also risks separating workloads from the node without safely draining them.

kubectl drain <node-name> --ignore-daemonsets

Why wrong: kubectl drain --ignore-daemonsets gracefully evicts pods and marks the node unschedulable, but it doesn't remove container images, logs, or other local storage. Eviction can actually write additional data to disk, such as checkpointed logs or temporary files, during the pod termination process. The filesystem usage that triggered DiskPressure remains largely unchanged, so the node will stay in DiskPressure after the drain completes.

Restart the kubelet service

Why wrong: Restarting the kubelet service forces a fresh evaluation of the node's conditions, but it does not alter the actual disk-usage statistics collected from the container runtime and OS. If the underlying filesystem is still above the eviction hard/soft thresholds, kubelet will quickly set DiskPressure again. A restart may also unmount volumes briefly and disrupt running pods, while failing to address the root cause.

The CoreDNS pod is down

Why wrong: If CoreDNS were actually down, all DNS resolution in the cluster would fail, not just the lookup for one Service name. The pod would be unable to resolve any cluster domain, including kubernetes.default.svc.cluster.local, and would almost certainly emit generic timeouts or 'no such host' errors for every name it tries to resolve. The fact that the error specifically points to a missing Service named 'service' indicates the DNS resolver is functioning and responding with NXDOMAIN, which is why a CoreDNS outage cannot be the cause of this particular CrashLoopBackOff.

The pod's DNS policy is set to 'None'

Why wrong: Setting dnsPolicy: None in the pod spec disables the cluster's default DNS configuration entirely, forcing the pod to supply its own resolv.conf through dnsConfig. That would break resolution for all hostnames, both internal and external, because the pod would not be using CoreDNS at all; it would not produce a targeted error about a specific Service in the 'default' namespace. Moreover, dnsPolicy: None is an explicit, intended configuration chosen by the workload author, and it would not manifest as a repeated crash loop caused by one missing Service record while other DNS lookups remain successful.

A network policy is blocking UDP port 53

Why wrong: A NetworkPolicy that blocks UDP port 53 would prevent the pod's DNS queries from ever reaching CoreDNS, causing all name resolution attempts to time out or fail with a generic network error. The application would not be able to resolve any cluster service, so it would not be able to distinguish that a Service named 'service' is missing—it would simply see that every DNS query is unreachable. Since the observed error is specific to the non-existence of the Service record, a network-level block on DNS traffic cannot be the explanation for this CrashLoopBackOff.