Troubleshooting Scenario Questions

Check kube-proxy logs on the nodes

Why it fails: Checking kube-proxy logs is a later diagnostic step, not a first-step check. Kube-proxy continuously reconciles iptables or IPVS rules based on Service and EndpointSlice objects; if the Service's selector matches no pods, the Endpoints object is empty, and no amount of kube-proxy log inspection will reveal a rule problem. Even if kube-proxy were misconfigured, you'd first want to confirm that backends exist. In a troubleshooting workflow, verifying Endpoints is faster, has no logs to parse, and covers the most common failure mode.

Check DNS resolution

Why it fails: Checking DNS resolution addresses a symptom where a Service name cannot be resolved, but the question describes an unreachable Service, not a name-resolution failure. If the Service itself has no endpoints, ClusterIP traffic will be dropped regardless of whether DNS correctly returns the service IP. DNS checks are relevant only when the client gets 'host not found' errors, which is a different failure domain (CoreDNS vs. endpoint selection). Therefore, inspect the Service's Endpoints before investigating DNS.

Restart the service

Why it fails: Restarting the Service object is not a meaningful action because Services are declarative control-plane objects with no embedded state or process to restart; the Kubernetes API doesn't act on a 'restart' for Services. Even if you delete and recreate the Service, the same selector and missing endpoints will reproduce the identical failure. The real issue is almost always the backing pods or their readiness, and a restart masks the symptom while adding no diagnostic value. Instead, you should inspect the Endpoints and the pods' labels and readiness status.

The PV's access mode is ReadOnlyMany

Why it fails: A ReadOnlyMany access mode does not block a volume from being mounted; it merely restricts the mounted volume to read-only access for multiple nodes. If the PV were ReadOnlyMany and compatible with the PVC's requested access mode, the kubelet would still perform the attach/mount operation successfully and the pod would run with a read-only filesystem. The error in question is a mount-stage failure, which is independent of access-mode permissions; an incompatible access mode would instead prevent the PVC from binding in the first place, not fail during mounting.

The PV is not created

Why it fails: If the PV were not created, the PVC could never bind, and the pod would be stuck in Pending at scheduling time with an event like '0/1 nodes are available: 1 persistentvolumeclaim ... is unbound' or 'no persistent volumes available for this claim'. The pod would never reach the container-starting phase, so it would not trigger a storage device mount attempt. Since the observed failure is specifically a mount failure that happens after the pod has been scheduled to a node, it proves the PV exists and is already bound to the PVC.

The PVC is unbound

Why it fails: An unbound PVC causes the scheduler to leave the pod unscheduled because the volume cannot be resolved before placement; Kubernetes would report a FailedBinding event or 'pod has unbound immediate PersistentVolumeClaims' rather than attempting to mount anything. The PVC status must be Bound for the kubelet to receive the volume's device path and call mount operations. Because the failure occurs during the mount phase, the PVC has necessarily already been bound to the PV and the pod has been scheduled to a node.

kubectl describe node | grep Network

Why it fails: kubectl describe node fetches the Node object's status and metadata, and piping to grep Network simply isolates lines containing the word "Network," such as pod CIDR assignments or the internal IP field. This is a static snapshot of node configuration, not an active probe: it cannot tell you whether packets can actually traverse the CNI, whether a remote pod is listening, or if network policies permit traffic. Therefore it does not test pod-to-pod connectivity and is not a valid troubleshooting command for this scenario.

kubectl logs pod-a | grep network

Why it fails: kubectl logs pod-a streams the container's captured stdout/stderr from its runtime; grepping "network" merely filters those log lines for the literal string, which may reveal application-level errors if the app logs network failures. However, it generates no traffic, sends no packets to another pod, and cannot test reachability, DNS, routing, or service connectivity. At best it is a passive observation aid, so it is not a valid command to actively troubleshoot network connectivity between pods.

Change the Service type to NodePort

Why it fails: Changing the Service type to NodePort merely exposes the application on a static port across each cluster node's IP address. It does not alter the underlying load-balancing behavior of kube-proxy, meaning client requests will still be distributed across all backend pods randomly or via round-robin, failing to solve the session stickiness issue.

Remove the ClusterIP Service and use headless service

Why it fails: Utilizing a headless service bypasses the ClusterIP proxying mechanism entirely, forcing clients to resolve pod IPs directly via DNS. This approach often exacerbates session issues because client-side DNS caching can lead to uneven traffic distribution, and it lacks any native mechanism to guarantee that a client consistently connects to the same pod replica.

Add a readiness probe to the pods

Why it fails: While readiness probes are crucial for ensuring that traffic is only routed to healthy pods that have finished initializing, they do not manage traffic routing logic for active sessions. Implementing a readiness probe will not prevent kube-proxy from routing successive requests from a single client to different, healthy backend replicas, leaving the session-state issue unresolved.

Use kubectl delete svc and recreate

Why it fails: Deleting and recreating the Service is not a diagnostic step—it only applies the same YAML again, so any misconfigured selector, missing labels, or incorrect port mapping will remain unchanged. Worse, it actively disrupts clients by briefly dropping the Virtual IP and any existing connections, and it can mask useful state like the generated ClusterIP or the Endpoints currently attached to the Service. Troubleshooting should start with read-only inspection, not destructive re-creation.

Use kubectl logs on the service pod to see application logs

Why it fails: Application logs from the backend pods will only show the application's own output, such as request handling errors or startup failures, but they reveal nothing about the Service object itself or the kube-proxy forwarding rules. If the Service has no endpoints because of a selector mismatch, the packets are discarded before they ever reach a pod, so nothing would appear in the application logs at all. This option is a good follow-up after you have confirmed the Service has valid endpoints, but it is not a primary Service troubleshooting action.

Use kubectl top nodes to check resource usage

Why it fails: `kubectl top nodes` reports CPU and memory utilization from the metrics server, not the state of Service networking, endpoint slices, or kube-proxy. While severe node resource pressure could theoretically affect kube-proxy or pod scheduling, resource usage alone will not explain why a Service stops routing traffic—especially when the Service has a valid ClusterIP and Endpoints. Service reachability issues are far more commonly caused by selector mismatches, missing endpoints, or failed kube-proxy health checks, none of which `top nodes` can show.

The pod's YAML has a syntax error

Why it fails: If the Pod's YAML manifest contained a syntax error, the Kubernetes API server would reject the creation request entirely during the validation phase. In this scenario, the Pod object would never be instantiated in the cluster. This differs from a Pod that successfully registers but is blocked from starting due to an unbound storage dependency.

The pod is using a hostPath volume

Why it fails: A hostPath volume mounts a file or directory from the host node's filesystem directly into the Pod. This type of volume bypasses the PersistentVolumeClaim and PersistentVolume subsystem entirely, as it is defined directly within the Pod's volume specification. Therefore, a PVC issue cannot block a Pod that relies solely on hostPath storage.

The node has insufficient CPU resources

Why it fails: Insufficient CPU resources on a node will cause the scheduler to mark the Pod as unschedulable, resulting in a FailedScheduling event. However, this resource constraint does not impact the lifecycle or binding process of a PersistentVolumeClaim. PVC binding occurs independently of node CPU capacity, meaning a PVC can successfully bind even if no nodes have the resources to run the Pod.

kubectl delete node master && kubeadm reset

Why it fails: This is a destructive, disproportionate response to a transient status condition. `kubectl delete node master` removes the node object from the cluster's etcd, and `kubeadm reset` tears down the local control-plane components, deletes etcd data, and resets the kubelet configuration. The NotReady state is almost never caused by a corrupted node; it is typically a health-check failure from the kubelet being unable to reach the container runtime. Reinitializing the node would require regenerating bootstrap tokens, re-joining the control plane, and restoring the cluster state — far more disruption than simply checking and restarting the runtime service.

systemctl status kubelet && systemctl restart kubelet

Why it fails: The kubelet is the agent that reports node status, but it is a client of the container runtime through the CRI (Container Runtime Interface). If containerd is stopped or hung, the kubelet cannot create pod sandboxes, so it repeatedly fails its runtime readiness probe and transitions the node to NotReady. Restarting the kubelet service alone does not start the runtime; it only re-runs the same failing CRI connection attempts, and the node will remain NotReady until containerd is operational. The proper troubleshooting sequence is to verify the runtime first, then check kubelet logs after the runtime is healthy.

systemctl status docker && systemctl restart docker

Why it fails: This option incorrectly assumes Docker is the configured CRI runtime. Modern kubeadm clusters use containerd by default, and dockershim was removed in Kubernetes 1.24, so Docker is not a drop-in replacement for containerd at the CRI level. Even if Docker is installed on the node, the kubelet will not use it unless explicitly configured through the `--container-runtime-endpoint` flag or the CRI configuration; in a kubeadm setup, that endpoint points to containerd. Restarting Docker when the actual runtime is containerd does nothing to fix the CRI connection broken by containerd being down, so the node would remain NotReady.

kubectl cluster-info

Why it fails: kubectl cluster-info displays the URL of the Kubernetes API server and the dashboard (if present), along with component statuses via the /healthz endpoint. It does not retrieve or present Event objects, which are ephemeral records of cluster changes. Therefore, this command cannot help you identify recent pod failures, evictions, or other problem-indicating events.

kubectl logs <pod-name>

Why it fails: kubectl logs <pod-name> streams the stdout/stderr output written by the main container(s) within the specified pod. Logs show application-level messages, but Kubernetes Events are a separate API resource recorded by the control plane (e.g., kubelet, scheduler) about state transitions. A container may crash without writing any log output, yet an event would still be generated for the crash loop; thus logs do not expose the troubleshooting event information the question asks about.

kubectl top pods

Why it fails: kubectl top pods reports real-time CPU and memory usage metrics for the pods in a namespace, using the metrics pipeline (metrics-server or similar). It is useful for capacity analysis or detecting resource saturation, but it does not list Event objects or describe pod lifecycle transitions. Since events like node pressure, image pull failures, or backoff restarts are not usage metrics, top pods cannot show the cluster events needed for troubleshooting.

kubectl get pv

Why it fails: While this command lists the cluster-wide PersistentVolumes, it does not target the specific PersistentVolumeClaim (my-pvc) or reveal why it remains unbound. It lacks the granular event logs and status conditions of the claim itself, which are critical for diagnosing provisioning or binding failures.

kubectl get pvc my-pvc -o yaml

Why it fails: This command outputs the current configuration and status block of the PVC in YAML format, but it lacks the real-time event stream generated by the control plane. To diagnose why a claim is stuck in a Pending state, you need the detailed scheduler or volume-provisioner events, which are omitted from this structured YAML output.

kubectl describe pod my-pvc

Why it fails: This command fails because it attempts to query a Pod resource named my-pvc instead of the PersistentVolumeClaim. Since my-pvc is a storage claim and not a workload instance, the API server will return a 'not found' error, providing no diagnostic information about the storage issue.

The pod's readiness probe is failing

Why it fails: A failing readiness probe does not terminate the container; it only pauses traffic routing to the pod via the Service. The pod remains Running and the container continues to execute, so logs would show the application running normally, not a crash loop. CrashLoopBackOff is caused by the container exiting, which readiness probes cannot do.

The pod's liveness probe is misconfigured

Why it fails: A misconfigured liveness probe might cause the kubelet to restart the container if the probe endpoint is unreachable, but the logs would typically be silent about the probe itself—the container would run until killed by the kubelet. Also, if the container is crash-looping from a DB error, the liveness probe may never even be checked because the container exits before the probe completes. Misconfiguration would not explain a database connection error in the logs.

The pod is out of memory

Why it fails: An OOM kill is enforced by the kernel and does not appear in `kubectl logs`; the container's stdout/stderr ends abruptly without an application-level error message. You would see the container status as `OOMKilled` in `kubectl describe pod`, not a DB connection failure in the logs. Also, OOM would cause immediate termination, but the logs would show the last lines of output before the kill, not a specific connection error.

kubectl describe node <node-name>

Why it fails: While `kubectl describe node` is a useful first step to inspect high-level node conditions and events from the control plane, it cannot be run locally on the node if the API server is unreachable, nor does it provide the detailed, real-time systemd logs of the kubelet service itself. To diagnose deep root causes like configuration errors or runtime failures, you must inspect the actual service logs directly on the host.

kubectl logs kubelet -n kube-system

Why it fails: The `kubectl logs` command is designed exclusively to retrieve logs from containerized workloads managed by the Kubernetes API server. Since the kubelet is a system daemon running directly on the host OS bare-metal or VM layer, it does not run as a standard pod in the `kube-system` namespace. Consequently, attempting this command will result in an error indicating that no such pod exists.

systemctl restart kubelet

Why it fails: Although restarting the service with `systemctl restart kubelet` might temporarily resolve transient issues, it is an action rather than a diagnostic tool. Executing this command blindly without checking the logs first can overwrite valuable historical error context in the journal. Furthermore, if there is a persistent configuration error, restarting the service will simply cause it to crash again without revealing why.

Check if the kube-proxy is running on the node hosting pod-b

Why it fails: Checking kube-proxy on the node hosting pod-b is a red herring because kube-proxy only manages ClusterIP service traffic via iptables or IPVS rules. Direct pod-to-pod communication uses the pod's IP address and is forwarded by the CNI plugin (e.g., Calico, Cilium, Flannel) through a virtual bridge or overlay network, not through kube-proxy. Since the user can already ping the Kubernetes service IP, kube-proxy is demonstrably alive and functional; even if it were not running, it would not affect a direct connection to pod-b's IP on port 5432.

Check if the container runtime is Docker

Why it fails: The container runtime (whether Docker, containerd, or CRI-O) is irrelevant to pod-to-pod connectivity at the layer this symptom describes. Kubernetes delegates all networking to the CNI plugin, which is responsible for allocating pod IPs, creating virtual interfaces, and enforcing routing rules; the runtime merely launches containers and attaches them to the network namespace the CNI prepared. Docker's own bridge network is never used for pod IPs when Docker is run via cri-dockerd, and the runtime choice does not affect NetworkPolicy enforcement, route tables, or iptables rules.

Check if the DNS resolution for pod-b's service is correct

Why it fails: DNS alone cannot be the culprit because the failure is observed at the connection level to a specific port (5432), not at the name-resolution stage. If the user is connecting directly by pod-b's IP or via a service, an incorrect DNS record would typically produce an NXDOMAIN error or a resolution to the wrong ClusterIP, not a refused or timed-out TCP connection. Moreover, even if the service name resolves correctly, the subsequent packets still pass through kube-proxy DNAT into pod-b's namespace, where a NetworkPolicy can still drop the traffic based on the source namespace 'foo'.

The node has disk pressure

Why it fails: Disk pressure is a specific node condition indicating that the node's disk space is running low, which can trigger eviction of pods to free up resources. While it affects pod scheduling and can lead to performance issues, it does not inherently cause the node to transition to a NotReady state. A node experiencing disk pressure typically remains Ready but with the DiskPressure=True condition, signaling a resource constraint rather than a complete loss of functionality or connectivity.

A pod on the node is consuming excessive memory

Why it fails: A single pod consuming excessive memory will likely be terminated by the kernel's Out-Of-Memory (OOM) killer or by Kubernetes if it exceeds its memory limits. While this affects the specific pod, it is an isolated event within the node's resource management. The node itself, and the kubelet running on it, typically remain healthy and Ready, continuing to report their status to the API server, unless the excessive memory consumption is so severe it destabilizes the entire operating system.

kubectl delete pod pod-name

Why it fails: kubectl delete pod pod-name is not a useful diagnostic step because it merely removes the current pod object. If the pod is controlled by a Deployment, ReplicaSet, StatefulSet, or DaemonSet, a new pod is created with a fresh backoff timer, but the old pod's events, logs, and crash history are discarded. You lose the very information needed to understand why the container failed, and the new pod will likely fall into the same CrashLoopBackOff.

kubectl exec -it pod-name -- sh

Why it fails: kubectl exec -it pod-name -- sh is ineffective because CrashLoopBackOff means the container is repeatedly crashing and restarting, so there is no stable running container to attach to. Even during the brief moments the container starts, exec may fail immediately if the process dies before the shell is established, and the underlying issue is in the startup path rather than an interactive environment. The command also requires at least one running container, which usually does not exist in this state.

kubectl rollout restart deployment

Why it fails: kubectl rollout restart deployment is not the appropriate first step because CrashLoopBackOff is a pod-level runtime failure, not a rollout or deployment configuration issue. Restarting the Deployment forces a new ReplicaSet and new pods, but if the underlying container command, image, or environment is broken, the new pods will still enter CrashLoopBackOff. It also obscures the diagnostic trail and is irrelevant when the pod is not managed by a Deployment or is not yet ready for a rollout-level intervention.

kubectl get nodes

Why it fails: kubectl get nodes fetches the Node object from the API server, which reflects the control plane's view of node readiness based on heartbeats and status reported by the kubelet. It will confirm that the node is indeed NotReady, but it provides no diagnostic details about why the kubelet failed to report ready status; the underlying fault could be in kubelet's communication with the container runtime, network issues, or resource pressure, all of which require inspecting logs on the node itself.

journalctl -u docker

Why it fails: journalctl -u docker examines the systemd journal for the Docker daemon, which is the container runtime, not the kubelet. While a node being NotReady can be caused by the container runtime being unhealthy, Docker logs do not reveal the kubelet's own status reporting, registration, or heartbeat failures. Furthermore, many modern Kubernetes nodes use containerd or CRI-O, so Docker-specific logs may be entirely irrelevant if the cluster does not use Docker as the runtime.

kubectl describe pod

Why it fails: kubectl describe pod is meant for inspecting individual pod objects, including their events, scheduled node, containers, and recent state, which is useful for pod-level failures but useless for a node-level NotReady issue. Node readiness is a status recorded by the kubelet about the node's overall health (e.g., disk pressure, PID pressure, memory pressure, kubelet readiness probes), and no amount of pod introspection will reveal why the kubelet is failing to report a Ready status. The troubleshooting target should be the kubelet service on the affected node, not any particular pod.