Troubleshooting Scenario Questions

The node has disk pressure

Why wrong: Disk pressure is a specific node condition indicating that the node's disk space is running low, which can trigger eviction of pods to free up resources. While it affects pod scheduling and can lead to performance issues, it does not inherently cause the node to transition to a NotReady state. A node experiencing disk pressure typically remains Ready but with the DiskPressure=True condition, signaling a resource constraint rather than a complete loss of functionality or connectivity.

A pod on the node is consuming excessive memory

Why wrong: A single pod consuming excessive memory will likely be terminated by the kernel's Out-Of-Memory (OOM) killer or by Kubernetes if it exceeds its memory limits. While this affects the specific pod, it is an isolated event within the node's resource management. The node itself, and the kubelet running on it, typically remain healthy and Ready, continuing to report their status to the API server, unless the excessive memory consumption is so severe it destabilizes the entire operating system.

kubectl delete pod pod-name

Why wrong: kubectl delete pod pod-name is not a useful diagnostic step because it merely removes the current pod object. If the pod is controlled by a Deployment, ReplicaSet, StatefulSet, or DaemonSet, a new pod is created with a fresh backoff timer, but the old pod's events, logs, and crash history are discarded. You lose the very information needed to understand why the container failed, and the new pod will likely fall into the same CrashLoopBackOff.

kubectl exec -it pod-name -- sh

Why wrong: kubectl exec -it pod-name -- sh is ineffective because CrashLoopBackOff means the container is repeatedly crashing and restarting, so there is no stable running container to attach to. Even during the brief moments the container starts, exec may fail immediately if the process dies before the shell is established, and the underlying issue is in the startup path rather than an interactive environment. The command also requires at least one running container, which usually does not exist in this state.

kubectl rollout restart deployment

Why wrong: kubectl rollout restart deployment is not the appropriate first step because CrashLoopBackOff is a pod-level runtime failure, not a rollout or deployment configuration issue. Restarting the Deployment forces a new ReplicaSet and new pods, but if the underlying container command, image, or environment is broken, the new pods will still enter CrashLoopBackOff. It also obscures the diagnostic trail and is irrelevant when the pod is not managed by a Deployment or is not yet ready for a rollout-level intervention.

Run 'kubectl get events --all-namespaces'

Why wrong: `kubectl get events --all-namespaces` retrieves cluster-scoped events, such as scheduling failures, image pulls, or node pressure, but it does not fetch the container's stdout/stderr stream. If `kubectl logs` yields no output, events can show why a pod was never scheduled or was evicted, yet they cannot reveal application output that was produced after startup. Events are also ephemeral and may rotate out before you inspect them, making them a supplementary diagnostic rather than a direct replacement for log retrieval.

Run 'kubectl top pod pod-name' to check resource usage

Why wrong: `kubectl top pod pod-name` reports live CPU and memory utilization from the metrics-server, which is useful for identifying resource saturation or throttling, but it never exposes the container log stream. A pod with zero log output might be OOMKilled or stuck in a pending state, and while resource pressure might be a contributing cause, `top` will not show the actual stdout/stderr lines or whether the process wrote anything before failing. Additionally, `top` requires metrics-server to be deployed; in a minimal environment it may fail entirely, so it is not a reliable log-diagnosis step.

Run 'kubectl exec pod-name -- cat /var/log/container.log'

Why wrong: `kubectl exec pod-name -- cat /var/log/container.log` executes a command inside the container, but it does not access the container runtime's managed log stream. Unless the application explicitly writes its own log file to `/var/log/container.log`, that path usually does not exist, and `cat` will return an error such as `No such file or directory` instead of log content. Moreover, `exec` requires a currently running container with a shell or suitable binary, which is often impossible for a container that has crashed or is in a pending state. This approach conflates application-level file logs with the runtime's captured stdout/stderr, and it is not the standard way to retrieve container logs.

kubectl get nodes

Why wrong: kubectl get nodes fetches the Node object from the API server, which reflects the control plane's view of node readiness based on heartbeats and status reported by the kubelet. It will confirm that the node is indeed NotReady, but it provides no diagnostic details about why the kubelet failed to report ready status; the underlying fault could be in kubelet's communication with the container runtime, network issues, or resource pressure, all of which require inspecting logs on the node itself.

journalctl -u docker

Why wrong: journalctl -u docker examines the systemd journal for the Docker daemon, which is the container runtime, not the kubelet. While a node being NotReady can be caused by the container runtime being unhealthy, Docker logs do not reveal the kubelet's own status reporting, registration, or heartbeat failures. Furthermore, many modern Kubernetes nodes use containerd or CRI-O, so Docker-specific logs may be entirely irrelevant if the cluster does not use Docker as the runtime.

kubectl describe pod

Why wrong: kubectl describe pod is meant for inspecting individual pod objects, including their events, scheduled node, containers, and recent state, which is useful for pod-level failures but useless for a node-level NotReady issue. Node readiness is a status recorded by the kubelet about the node's overall health (e.g., disk pressure, PID pressure, memory pressure, kubelet readiness probes), and no amount of pod introspection will reveal why the kubelet is failing to report a Ready status. The troubleshooting target should be the kubelet service on the affected node, not any particular pod.

kubectl exec -it mypod -- sh

Why wrong: The `kubectl exec -it mypod -- sh` command is ineffective for a pod in `CrashLoopBackOff` because `exec` requires a container to be in a stable `Running` state to establish an interactive shell. Since the container is repeatedly crashing and restarting, it is not running long enough or stably enough to successfully execute a command inside it, making this approach unsuitable for diagnosing the initial crash.

kubectl delete pod mypod && kubectl create -f mypod.yaml

Why wrong: Executing `kubectl delete pod mypod && kubectl create -f mypod.yaml` will simply terminate the existing pod and launch a new one. While this might resolve transient issues, it does not provide any diagnostic information about *why* the original pod entered `CrashLoopBackOff`. If the underlying problem, such as an application bug or misconfiguration, persists, the newly created pod will likely encounter the exact same crashing behavior without revealing the root cause.

kubectl describe pod mypod

Why wrong: The `kubectl describe pod mypod` command provides valuable metadata, events, and status information about the pod and its containers from the Kubernetes API perspective. However, while it can show events indicating that a container has crashed or restarted, it does not expose the internal application logs or error messages generated *from within* the container that directly explain the cause of the application's failure, which is essential for debugging a `CrashLoopBackOff`.

kubectl exec pod-a -- nslookup service-b

Why wrong: This command only tests if Pod A can resolve the DNS name `service-b` to its corresponding ClusterIP address via CoreDNS. While successful DNS resolution is a prerequisite for network communication, it does not verify that a TCP connection can be established to the service's port 80, nor does it confirm that the application behind the service is actively listening and responding. A network policy, firewall rule, or an unresponsive application could still prevent actual HTTP connectivity even with correct DNS.

curl http://<node-ip>:<nodeport>

Why wrong: This command tests external connectivity to a service exposed via a NodePort, routing traffic from outside the Kubernetes cluster through a specific node's IP and port. It does not simulate the internal cluster network path from Pod A to Service B, which involves internal DNS resolution, ClusterIP routing, and kube-proxy's iptables/IPVS rules within the cluster's CNI network. Therefore, it fails to diagnose an issue specific to inter-pod communication within the cluster.

ssh node-ip 'curl http://<pod-b-ip>:80'

Why wrong: Executing `curl` directly on the node via SSH bypasses the network namespace and routing context of Pod A entirely. This test verifies connectivity from the node's perspective to Pod B's IP, which might differ significantly from Pod A's network path due to network policies, CNI plugin configurations, or specific routing rules applied within Pod A's isolated network environment. It does not accurately reflect the network conditions or potential issues experienced by Pod A.

The pod's /etc/resolv.conf has incorrect search domains

Why wrong: Incorrect search domains in a pod's `/etc/resolv.conf` would lead to DNS queries failing to resolve specific names, resulting in an `NXDOMAIN` response or a lookup failure after exhausting search paths. However, the initial attempt to connect to the `nameserver` IP specified in `resolv.conf` would still succeed if the CoreDNS service is operational. A timeout specifically indicates that the DNS server itself is unreachable or unresponsive, not merely that the name could not be found.

A network policy is blocking traffic to the kube-dns service

Why wrong: While a network policy could indeed block traffic to the `kube-dns` service, causing DNS queries to time out due to dropped packets, this is generally a less common root cause than CoreDNS pods being unhealthy. If a network policy were the culprit, it would explicitly deny egress from the requesting pod to the `kube-dns` service IP or ingress to the CoreDNS pods. However, the fundamental unavailability of the DNS server (CoreDNS pods not running) is a more direct and frequent cause of connection timeouts.

The DNS name does not exist

Why wrong: If the DNS name being queried does not exist, a functional DNS server, such as CoreDNS, would process the request and explicitly respond with an `NXDOMAIN` (Non-Existent Domain) status. This response confirms that the DNS server is reachable and operational, having authoritatively determined the name is not registered. A connection timeout, conversely, signifies that the DNS server could not be reached at all, preventing any response, including an `NXDOMAIN`.

ssh to the node and run 'docker ps'

Why wrong: In modern Kubernetes environments (v1.29+), Docker Engine is typically not the container runtime; containerd is the default Container Runtime Interface (CRI) implementation. Therefore, running 'docker ps' on a node would likely yield an empty list or an error, as no Docker daemon is managing the Kubernetes pods. More importantly, this command only lists containers managed by Docker and provides no insight into the operational status or health of the critical 'kubelet' service itself, which is responsible for node registration and pod lifecycle management.

kubectl describe node <node-name>

Why wrong: The 'kubectl describe node <node-name>' command provides a high-level overview of the node's resources, events, and conditions reported by the kubelet to the API server. While it can show conditions like 'Ready', 'MemoryPressure', or 'DiskPressure', these are reported states from the kubelet, not direct indicators of whether the kubelet service itself is running, stopped, or encountering startup failures on the host operating system. It doesn't provide access to the kubelet's local service status or its internal logs.

kubectl logs kubelet -n kube-system

Why wrong: The 'kubelet' is a critical agent that runs directly on each node as a host-level service, typically managed by 'systemd' on Linux distributions. It is not a pod running within the Kubernetes cluster itself, nor is it part of the 'kube-system' namespace. Consequently, attempting to retrieve logs for a "kubelet" pod using 'kubectl logs kubelet -n kube-system' will fail, as no such pod exists for the API server to query. Its logs are managed by the host's logging system.

The pod's /etc/resolv.conf has incorrect nameservers

Why wrong: If a pod's /etc/resolv.conf contained incorrect nameservers, resolution attempts would likely result in immediate connection errors like 'connection refused' or 'host unreachable' if the specified IP address does not host a DNS server or is entirely invalid. A DNS timeout, however, implies that the query successfully reached a server that was expected to respond but failed to do so within the configured timeframe. This points to the DNS service itself being unresponsive rather than the initial target IP being fundamentally wrong.

The node's network plugin is misconfigured

Why wrong: A misconfigured node network plugin (e.g., CNI) would typically manifest as broader network connectivity issues, affecting all pod-to-pod or pod-to-service communication on that node or across the cluster. While DNS relies on network connectivity, a specific DNS timeout suggests the network path to the DNS service IP is functional, but the service at that IP is not responding to DNS queries. If the network plugin were truly broken, the pod would likely be unable to establish any network connections, including reaching the DNS service IP.

The pod's DNS policy is set to 'None'

Why wrong: When a pod's `dnsPolicy` is set to 'None', Kubernetes configures the pod's `/etc/resolv.conf` using the `dnsConfig` specified in the pod definition, completely bypassing the cluster's default DNS. If the `dnsConfig` points to an invalid or non-existent external nameserver, resolution attempts would typically fail immediately with 'Host not found' or 'NXDOMAIN' if the server responds negatively, or 'connection refused' if the server is unreachable. A timeout implies the query was sent to a server that did not respond, which is less common for 'None' unless the explicitly configured external server is unresponsive.

The registry is unreachable due to network issues

Why wrong: If the registry were unreachable due to network issues, the container runtime would typically report errors like 'dial tcp: lookup registry.example.com: no such host' for DNS resolution failures, or 'connection refused' if the host is reachable but the port is closed. The 'manifest not found' error, however, clearly indicates successful communication with the registry but an inability to locate the requested image metadata, not a network connectivity problem.

The image registry requires authentication and the imagePullSecret is missing

Why wrong: Authentication failures, such as a missing or incorrect imagePullSecret, would typically result in HTTP 401 Unauthorized or 403 Forbidden errors from the registry. These messages explicitly state that access to the resource is denied due to insufficient credentials. The 'manifest not found' error, conversely, implies successful authentication (or no authentication required for the resource) but an inability to find the requested image data.

The image has been deleted from the registry

Why wrong: While a deleted image could indeed lead to a 'manifest not found' error, the more common and precise interpretation when troubleshooting is that the specified tag '1.0' was either never pushed to the registry, was mistyped in the Pod definition, or was explicitly removed. If an image were truly deleted, the tag would effectively cease to exist, making this option a less direct cause than the tag simply not being present.

The pod's liveness probe is misconfigured

Why wrong: A misconfigured liveness probe does not directly cause a 'connection refused' error inside the application's logs. When a liveness probe fails, the kubelet restarts the container, which would eventually manifest as CrashLoopBackOff, but the visible symptom would be probe timeout or HTTP error from the kubelet, not a failed outbound database connection. You would also see probe-related events in 'kubectl describe pod', whereas a TCP connection error is an application-level startup dependency failure.

The pod's container image is missing

Why wrong: If the container image were truly missing, the container runtime would never start the process; the pod would remain in ImagePullBackOff or ErrImagePull state, with events about a manifest unknown or repository not found. In contrast, a pod in CrashLoopBackOff has executed the container at least once, meaning the image was successfully pulled and started. A missing image would prevent the application binary from running, so it could not produce a database connection attempt at all.

The pod has a memory limit that is too low

Why wrong: If the container's memory limit were too low, the kernel's OOM killer would terminate the process abruptly, and 'kubectl get pod' would report OOMKilled in the last termination reason, not a connection refused message. Memory exhaustion also tends to happen after some runtime usage, whereas this failure appears during startup when the app tries to reach the database. The exit code and the specific network error in the logs are definitive evidence that resource limits are not the culprit.

Run 'kubectl get nodes' to check node status

Why wrong: Node status reflects the overall health of the Kubernetes node (Kubelet, CRI, and kube-proxy) but does not expose the state of pod-to-pod communication paths. A pod can be unable to reach another pod's Service even when all nodes show Ready, because the fault may lie in the Service's endpoint selector, a NetworkPolicy, or iptables/IPVS forwarding rules. Checking node status is a coarse, low-yield diagnostic that won't identify the specific break in the Service-to-pod chain.

Run 'kubectl logs' on the target pod to check application logs

Why wrong: Pod logs contain what the application writes to stdout/stderr, which reflects application-level behavior and not the underlying network forwarding or Service routing. A connectivity failure often occurs before a packet reaches the target container—for example, a NetworkPolicy drop, a Service with no endpoints, or a wrong port—so the target application may never even receive a request and thus log nothing. Logs are useful only after you have confirmed that packets can reach the pod; they cannot reveal why the Service is not routing traffic or why DNS resolution failed.

Add a node selector to the pod spec to target a specific node

Why wrong: Node selector does not bypass taints; it only selects nodes with matching labels.

Increase the memory request to prioritize scheduling

Why wrong: Increasing request makes it harder to fit on nodes.

Add resource limits without changing requests

Why wrong: Limits alone do not affect scheduling; requests matter.

journalctl -u docker

Why wrong: While Docker (or any container runtime like containerd or CRI-O) is essential for running containers on a node, its logs are not the primary place to start when the node itself is `NotReady`. The Kubelet is the component that communicates the node's health to the Kubernetes control plane. If the Kubelet itself is failing to start, communicate, or detect the runtime, its own logs will reveal that failure, even before delving into the container runtime's specific operational issues.

journalctl -u kube-apiserver

Why wrong: The `kube-apiserver` is a central component of the Kubernetes control plane, typically running on master nodes, responsible for exposing the Kubernetes API. While the Kubelet communicates *with* the API server, checking the API server's logs would primarily reveal issues with the control plane itself or requests *from* the Kubelet, not the root cause of a `NotReady` state on a specific worker node. The problem is localized to the worker node, making its local Kubelet logs the appropriate diagnostic starting point.

journalctl -u kube-controller-manager

Why wrong: The `kube-controller-manager` runs various controllers that watch the shared state of the cluster through the API server and make changes to move the current state towards the desired state. For instance, the Node Controller handles node registration and monitors node health. While it might mark a node as `NotReady` based on Kubelet's reports, its logs would only show *that* it detected a problem, not *why* the Kubelet on the specific node is failing. Troubleshooting a `NotReady` node requires investigating the agent *on that node* directly.

Check if the kube-proxy is running on the node hosting pod-b

Why wrong: Checking kube-proxy on the node hosting pod-b is a red herring because kube-proxy only manages ClusterIP service traffic via iptables or IPVS rules. Direct pod-to-pod communication uses the pod's IP address and is forwarded by the CNI plugin (e.g., Calico, Cilium, Flannel) through a virtual bridge or overlay network, not through kube-proxy. Since the user can already ping the Kubernetes service IP, kube-proxy is demonstrably alive and functional; even if it were not running, it would not affect a direct connection to pod-b's IP on port 5432.

Check if the container runtime is Docker

Why wrong: The container runtime (whether Docker, containerd, or CRI-O) is irrelevant to pod-to-pod connectivity at the layer this symptom describes. Kubernetes delegates all networking to the CNI plugin, which is responsible for allocating pod IPs, creating virtual interfaces, and enforcing routing rules; the runtime merely launches containers and attaches them to the network namespace the CNI prepared. Docker's own bridge network is never used for pod IPs when Docker is run via cri-dockerd, and the runtime choice does not affect NetworkPolicy enforcement, route tables, or iptables rules.

Check if the DNS resolution for pod-b's service is correct

Why wrong: DNS alone cannot be the culprit because the failure is observed at the connection level to a specific port (5432), not at the name-resolution stage. If the user is connecting directly by pod-b's IP or via a service, an incorrect DNS record would typically produce an NXDOMAIN error or a resolution to the wrong ClusterIP, not a refused or timed-out TCP connection. Moreover, even if the service name resolves correctly, the subsequent packets still pass through kube-proxy DNAT into pod-b's namespace, where a NetworkPolicy can still drop the traffic based on the source namespace 'foo'.