The Troubleshooting domain makes up 30% of the CKA exam, which means knowing how to diagnose and fix broken clusters is more important than how to build them. This chapter teaches you the detective skills to find why a node has gone offline, why a pod won't start, or why the entire cluster is unresponsive, using the same logical process a plumber uses to find a leak.
Jump to a section
A simple way to picture Troubleshooting Cluster and Node Issues
Your Kubernetes cluster is an apartment building, and you are the building superintendent. The control plane is the building's main office, the nodes are the individual apartments, and the pods are the tenants living in those apartments. Your job is to keep everything running smoothly.
One morning, a tenant in apartment 3B calls to say they have no water. You don't immediately go to 3B. First, you check the main water pipe in the basement (the kubelet service on the control plane). If the main pipe is fine, you then check the water pipes for that specific floor (the node's kubelet status). If the floor's pipes are clogged, you fix that. If the floor's pipes are fine, the problem is inside the apartment itself (the pod). You might need to evict that tenant (delete the pod) so a new one can move in.
This is exactly how you troubleshoot a Kubernetes cluster. You don't just guess. You follow a logical path from the most central component (the control plane) outward to the most local component (the individual pod). If the entire building has no power (the cluster's API server is down), it doesn't matter if one apartment has a leaky faucet. You must fix the power first. The superintendant's tools are 'kubectl get nodes' to see if any apartments are marked 'NotReady', 'systemctl status kubelet' to check if the building's maintenance unit is running, and 'journalctl -u kubelet' to read the building's maintenance logbook for error messages.
Troubleshooting a Kubernetes cluster means methodically finding and fixing problems with the machines (nodes) and the software that runs the cluster (cluster components). A node is a physical or virtual server — think of it as a worker computer in a team. A cluster is the whole team of these computers working together. The control plane is the team leader that decides where work goes. The kublet is a small program that runs on each worker computer and reports back to the team leader.
When something goes wrong, you must start at the biggest, most central part and move outward, like a mechanic checking a car's battery before its spark plugs. Here is the hierarchy of what to check:
First, check the control plane components. The control plane runs on a set of nodes (usually three or five) and includes the API server (the front door of the cluster), the controller manager (a bookkeeper that watches for problems), and the scheduler (the person who assigns tasks to workers). If the API server is down, nothing else works. You can check its status using 'kubectl get componentstatuses' or, in newer versions, by checking the kube-system namespace pods with 'kubectl get pods -n kube-system'.
Second, check the node health. A node can be 'Ready' (healthy), 'NotReady' (unreachable or broken), or 'Unknown' (the cluster has lost contact for some time). You run 'kubectl get nodes' to see a list of all nodes and their status. If a node shows 'NotReady', you need to SSH into that specific node (using 'ssh user@node-ip') and check if the kubelet service is running with 'systemctl status kubelet'.
Third, check the kubelet logs. The kubelet is the heartbeat of each node. If it stops, the node is dead to the cluster. Use 'journalctl -u kubelet -f' to watch the kubelet's log messages in real time. Common errors include 'failed to get node status' (network issue), 'no kubelet pods directory' (configuration problem), or 'tls handshake error' (certificate problem).
Fourth, check the container runtime. The container runtime (usually containerd or CRI-O) is the program that actually runs your applications inside containers. If it is not running, the kubelet can't start pods. Check it with 'systemctl status containerd'. If it is dead, start it with 'systemctl start containerd'.
Fifth, check the pod health. Once you confirm the node is ready and the runtime is running, look at why individual pods are failing. Use 'kubectl describe pod <pod-name>' to see events like 'ImagePullBackOff' (wrong image name), 'CrashLoopBackOff' (the app inside keeps crashing), or 'Pending' (no node can fit the pod).
Why does this layered approach exist? Because if the team leader (control plane) is asleep, no amount of fixing workers (nodes) will help. Similarly, if a worker (node) is dead, worrying about its pods is pointless. Always start from the top of the chain and work down. This is not just a good idea for the exam — it is how every experienced Kubernetes administrator works in real production systems.
Identify the Scope
Run 'kubectl get nodes' to see which nodes are NotReady or Unknown. Also run 'kubectl get pods --all-namespaces' to spot pending or crash-looping pods. This tells you if the problem is a single machine or the entire cluster.
Check the Control Plane
If all nodes are unreachable (all NotReady), the control plane may be down. Check the API server, etcd, and controller manager pods in the kube-system namespace. Use 'kubectl get pods -n kube-system' to see if they are running.
Investigate the Unhealthy Node
If one node is NotReady, SSH into that node using 'ssh <node-ip>' or 'kubectl node-shell <node-name>'. Check the kubelet service with 'systemctl status kubelet'. If it is inactive, start it. If it is running but the node is still NotReady, check the logs.
Analyse the Kubelet Logs
Run 'journalctl -u kubelet -n 100' to read the last 100 lines of the kubelet log. Look for keywords like 'error', 'failed', 'certificate', 'timeout'. This will tell you if the problem is a certificate, network, or configuration issue.
Fix the Root Cause
Based on the log error, take action: renew expired certificates, restart the container runtime if it is dead, correct a misconfigured CNI plugin, or clear disk space if there is DiskPressure. Then restart the kubelet: 'systemctl restart kubelet'.
Verify the Fix
After applying the fix, wait 30-60 seconds and run 'kubectl get nodes' from a control plane node. Confirm the node status changes to 'Ready'. Also check that affected pods are rescheduled with 'kubectl get pods -o wide'.
Imagine you work for an online retailer, and it is Black Friday. Your entire e-commerce site runs on a Kubernetes cluster. Suddenly, customers start reporting that the site is loading slowly and sometimes returning an error page. You are the on-call engineer.
First, you open your terminal and run 'kubectl get nodes'. The output shows one node is 'NotReady'. That is the most obvious issue — a single worker machine has dropped off the network. You do not yet know why, but you know that node is hosting customer-facing application pods.
Step two: you SSH into the failed node using its IP address. You run 'systemctl status kubelet'. The output shows 'active (running)' — so the kubelet is alive. That is good. But the node is still NotReady. Next, you check the network connectivity by pinging the control plane IP address. If the ping fails, you know there is a network cable or firewall problem. In this scenario, the ping succeeds.
Step three: you check the kubelet logs with 'journalctl -u kubelet -n 50' to see the last 50 log lines. You see repeated error messages: 'failed to update node status: context deadline exceeded'. This usually means the kubelet cannot reach the API server, even though the ping worked on a basic level. That suggests a certificate issue or a port blocking issue.
You check the certificates in /etc/kubernetes/pki/. The kubelet uses a certificate to authenticate to the API server. You run 'openssl x509 -in /etc/kubernetes/pki/kubelet-client-current.pem -text -noout'. The certificate has expired. There is the root cause. The node cannot authenticate to the control plane, so it is marked NotReady.
You rotate the certificates by running 'kubeadm certs renew kubelet-client' on the control plane node and then scp the new certificate to the broken node. After restarting the kubelet with 'systemctl restart kubelet', the node comes back to 'Ready' within 30 seconds. The customers' traffic is restored.
Without this methodical approach, you might have spent hours looking at pod configurations or restarting the application, wasting precious time on a holiday. The real-world sequence is always: control plane, node health, kubelet, runtime, pods. Each layer filters out simpler possibilities, so you only dig deep when necessary.
The CKA exam will test your ability to diagnose and fix node and cluster issues using command-line tools only. There is no GUI. You must memorise the exact commands and their flags. Here is what you must know inside and out:
'kubectl get nodes': shows the status of all nodes. A node in 'NotReady' state is the most common exam scenario. You must know to SSH into that node and check the kubelet.
'systemctl status kubelet': checks if the kubelet daemon is running. If it is not, start it with 'systemctl start kubelet'. If it fails to start, check its logs with 'journalctl -u kubelet'.
'journalctl -u kubelet': shows the kubelet's log output. Exam questions often hide the error in the logs, like a missing configuration file or a certificate mismatch.
'kubectl describe node <node-name>': shows detailed conditions, including 'Ready', 'MemoryPressure', 'DiskPressure', and 'PIDPressure'. If a node has 'DiskPressure', for example, you must clear disk space on that node.
'kubectl cordon <node-name>' and 'kubectl drain <node-name>': used to safely evict pods from a node before performing maintenance. The exam will ask you to perform node maintenance without causing application downtime. You must use '--ignore-daemonsets' flag when draining, because DaemonSets (pods that must run on every node) cannot be evicted.
'kubectl logs <pod-name>': gets logs from a specific pod. If a pod is crash-looping, use 'kubectl logs --previous <pod-name>' to see the logs of the last crashed instance.
'kubectl get events --sort-by='.metadata.creationTimestamp': shows chronological events in the cluster. This can reveal a cascading failure, like one node failing and the scheduler trying to schedule pods onto already overloaded nodes.
Common exam traps include:
A node is NotReady because the kubelet is not installed (nothing to start). You must install it with 'apt install kubelet' or 'yum install kubelet'.
A node has 'DiskPressure' but no disk space left on root. You must find large log files with 'du -sh /var/log/*' and clean them.
A pod is 'Pending' because there are no nodes with enough CPU or memory. Use 'kubectl top nodes' to check resource usage, then describe the pod to see the exact resource request.
Questions that give an incorrect node IP or hostname in the CNI configuration. You must edit the configuration file (usually /etc/cni/net.d/) to fix the IP.
Key definitions to memorise:
Kubelet: the primary agent on each node that ensures containers are running.
Container Runtime: the software that actually runs containers (containerd, CRI-O, Docker).
Control Plane: the set of components (API server, etcd, scheduler, controller manager) that manage the cluster.
API Server: the front door of the cluster. All commands go through it.
etcd: the cluster's database that stores all state.
Always start troubleshooting from the most central component (control plane) and move outward to nodes, then pods.
Run 'kubectl get nodes' first to identify which nodes are NotReady — this is the most common exam scenario.
To check a node's kubelet, SSH into it and run 'systemctl status kubelet' — if it is dead, start it with 'systemctl start kubelet'.
The kubelet logs, accessed via 'journalctl -u kubelet', contain the real error messages needed to diagnose node failures.
A Pod stuck in 'Pending' likely means no node has enough resources — use 'kubectl describe pod' and 'kubectl top nodes' to find the bottleneck.
Before rebooting a node for maintenance, use 'kubectl drain <node-name> --ignore-daemonsets' to safely evict pods.
If a pod is CrashLoopBackOff, read its previous logs with 'kubectl logs --previous <pod-name>' to see why it crashed.
The container runtime (containerd or CRI-O) must be running on every node — check it with 'systemctl status containerd'.
A node can be Ready but still have issues if its kubelet is misconfigured — always verify with logs, not just status.
The API server is the single point of failure — if it is unresponsive, run 'systemctl status kube-apiserver' on the control plane node.
These come up on the exam all the time. Here's how to tell them apart.
kubectl describe node
Shows node conditions like Ready, DiskPressure, MemoryPressure.
Displays events related to the node (e.g., failed to attach volume).
Does not show real-time CPU/memory usage metrics.
kubectl top node
Shows current CPU and memory usage for each node.
Requires metrics-server to be installed in the cluster.
Useful for deciding which node has capacity for new pods.
Cordon a Node
Marks the node as unschedulable for new pods.
Existing pods continue running on the node.
Useful to prevent new deployments while investigating an issue.
Drain a Node
Marks the node unschedulable AND evicts existing non-mirror pods.
Safely moves workloads to other nodes before maintenance.
Must use --ignore-daemonsets to avoid evicting system pods.
Pod CrashLoopBackOff
The container starts but crashes immediately due to application error.
Check logs with 'kubectl logs --previous pod-name'.
Fix: correct application code or configuration.
Pod ImagePullBackOff
The container image cannot be pulled from the registry.
Common causes: wrong image name, registry authentication, or network.
Fix: correct the image tag, add registry credentials, or check network.
Kubelet Stopped
Service is inactive: 'systemctl status kubelet' shows inactive.
Fix by starting the service: 'systemctl start kubelet'.
Easy to identify because the node is completely unresponsive.
Kubelet Misconfigured
Service is running, but node is still NotReady.
Configuration file (kubelet config.yaml) contains errors.
Fix by checking logs and correcting the configuration file.
Mistake
If a node shows NotReady, you must immediately delete all pods on that node by hand.
Correct
You should first investigate the cause. The kubelet might be restarting due to a temporary issue, and pods will be rescheduled automatically if they were managed by a ReplicaSet or Deployment.
Beginners panic when they see a red node and think manual deletion is the only way, not realising that controllers handle restarts.
Mistake
Checking kubelet logs with journalctl is optional; the node status tells you everything.
Correct
Node status only tells you the symptom (NotReady). The kubelet logs contain the exact error message that reveals the root cause (certificate expired, missing configuration, etc.).
Many beginners assume the output of 'kubectl get nodes' is enough diagnostic data, when it is only the starting point.
Mistake
You can fix a NotReady node by restarting the API server.
Correct
The API server runs on the control plane, not on worker nodes. A node being NotReady is almost always caused by the kubelet on that node, not the API server. Restarting the API server will not help.
Newcomers confuse the roles of control plane components and worker node components.
Mistake
If a pod is CrashLoopBackOff, you should edit the pod YAML to fix it.
Correct
You should first check the pod logs with 'kubectl logs <pod-name>' to see why the application is crashing. You cannot edit most fields of a running pod; you must delete it and let the controller recreate it with a corrected YAML.
People familiar with containers often think of directly editing a running instance, but Kubernetes pods are mostly immutable.
Mistake
A DiskPressure condition on a node means you should delete pods to free space.
Correct
DiskPressure means the node's disk is almost full. You should clean up unnecessary files, logs, or images on the node itself (using 'docker image prune' or 'crictl rmi'), not just delete pods.
Beginners assume deleting pods fixes everything, but pod deletion doesn't reclaim disk space occupied by container images or log files.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
The kubelet might be running, but it cannot communicate with the API server due to a certificate mismatch, network firewall, or wrong configuration. Check 'journalctl -u kubelet' for specific errors.
Use 'kubectl drain <node-name> --ignore-daemonsets --delete-local-data'. The --ignore-daemonsets flag ensures critical system pods like network plugins are not evicted, and --delete-local-data empties local storage pods safely.
It means the pod YAML specifies 'runAsNonRoot: true' in its security context, but the application image is built to run as the root user. You must change the image or remove the security constraint.
Yes. On the control plane node, run 'kubeadm certs renew kubelet-client' to generate a new certificate, then copy it to the worker node and restart the kubelet. The node will rejoin the cluster.
'kubectl describe node' shows status conditions (Ready, DiskPressure) and events. 'kubectl top node' shows current CPU and memory usage for each node. Use describe for conditions, top for resource consumption.
The pod might request more CPU or memory than any node has available, or it may have a nodeSelector that does not match any node. Use 'kubectl describe pod <pod-name>' to see the exact scheduling error.
You've finished Troubleshooting Cluster and Node Issues. Continue through the CKA study guide to build a complete picture of the exam.
Done with this chapter?