Managing Cluster Upgrades: the single most stressful operation you will perform as a Kubernetes administrator. You have a running, working cluster handling real user traffic, and you need to update the very software that keeps it all together without breaking anything. For the CKA exam, understanding how to upgrade a cluster using kubeadm is not just a theoretical concept – it is a practical, hands-on skill they will test directly.
Jump to a section
A simple way to picture Managing Cluster Upgrades
You live on the 5th floor of a 10-floor apartment building. The building manager decides to replace all the old, rickety elevators with new, faster ones. This is a major upgrade, not just a repair.
The problem with upgrading a building is that if you upgrade the elevator in the middle of the day, everyone living on floors 6 through 10 will be trapped. There is no way to get to their apartments. Your building manager has a plan. First, she tells everyone that the work will happen on a Saturday. Then, she upgrades the elevator on the 10th floor first. The residents on floors 1-9 can still use the old elevator to get to the 10th floor. Then, she upgrades the elevator on the 9th floor. Now, residents on floors 1-8 can still use the old elevator to reach the 9th floor, and then the new elevator from the 9th floor up to the 10th. She repeats this process, moving down one floor at a time.
The crucial rule is that the difference between the highest floor with a new elevator and the lowest floor with an old elevator can never be more than one floor. If she skipped from 10 down to 8, the 9th floor would be isolated. This precise, floor-by-floor, controlled upgrade is exactly how Kubernetes cluster upgrades work. You cannot upgrade the entire cluster at once. You upgrade one node (floor) at a time, and you must never let the version difference between the control plane (your building manager) and the nodes (the floors) get too wide. The entire system stays functional for the residents (your applications) throughout the process.
Let's break down what a 'cluster upgrade' actually means. Kubernetes is a piece of software, and like all software, it gets new features, bug fixes, and security patches. Your cluster is made up of several machines (servers). The 'brain' of the cluster is called the control plane. The 'workers' that run your applications are called worker nodes. When you upgrade, you are updating the Kubernetes software on these machines to a newer version.
Before we go further, you need to understand version skew. Kubernetes has a very specific policy: you can only have a difference of one minor version between certain components. A minor version is the middle number in the version string, like 1.28 or 1.29. A patch version is the last number, like 1.28.1 or 1.28.2. You can always upgrade patch versions (1.28.1 to 1.28.2) without issues. The problem comes with minor versions.
The golden rule of version skew is this: the kube-apiserver (the main part of the control plane) must be at a higher or equal version than everything else. The control plane can be one minor version ahead of the nodes. The nodes can also be one minor version behind the control plane. This is called the 'n-1' rule. This gives you a safe window to perform the upgrade.
So, how does an upgrade actually happen? Let's use the real tool, kubeadm. The process has two major phases: upgrading the control plane, and then upgrading the worker nodes.
Phase 1: Upgrade the Control Plane This is where all the planning happens. You pick a target version, say 1.29.0, and you upgrade the control plane component by component. - Step 1: Drain the control plane node. 'Draining' means gently moving all the programs (pods) running on that node to other nodes, so you can take it down for maintenance without causing downtime. - Step 2: SSH into the control plane machine. You run 'kubeadm upgrade plan' first. This command checks if your target version is valid and shows you exactly what will change. It's like an impact assessment. - Step 3: Run 'kubeadm upgrade apply v1.29.0'. This is the core command. It upgrades the control plane components (the API server, the scheduler, the controller manager) to the new version. - Step 4: Update the kubelet (the agent that runs on every node). You must manually install the new kubelet and kubeadm packages on the control plane node, then restart the kubelet service. - Step 5: Uncordon the node. 'Uncordoning' is the opposite of draining; it tells Kubernetes that the node is healthy and ready to accept new pods again.
The control plane is now upgraded. The cluster is still running, but it is now running a 'mixed' version state: a new control plane with old worker nodes. This is perfectly fine according to the version skew policy.
Phase 2: Upgrade the Worker Nodes This is the repetitive part. You have to do this for every single worker node in your cluster. You cannot just upgrade them all at once, because the control plane would lose contact with all of them simultaneously. You do them one by one. - Step 1: For one worker node, run 'kubectl drain <node-name>'. This moves all pods from that specific node to other nodes in the cluster. - Step 2: SSH into the worker node. Run 'kubeadm upgrade node'. This command tells kubeadm to upgrade the configuration on this specific worker node to match the new control plane version. - Step 3: Manually install the new kubelet and kubeadm packages on this worker node, then restart the kubelet service. - Step 4: Run 'kubectl uncordon <node-name>' to bring the node back into service. - Step 5: Repeat steps 1-4 for the next worker node.
Why does this process exist? It replaces the old, dangerous 'big bang' approach of taking the entire cluster offline, upgrading everything, and then turning it back on. This is called a 'rolling upgrade'. It ensures your applications remain available to users throughout the entire upgrade process. The cluster is never fully offline.
In summary, the entire upgrade is a controlled, step-by-step dance that respects version limits. You upgrade the brain first, then upgrade each worker one at a time, always ensuring the brain is never more than one minor version ahead of the workers.
1. Check the Current Version and Plan the Upgrade
Run 'kubeadm upgrade plan' on the control plane node. This command prints the current cluster version, the target version you can upgrade to, and a detailed list of changes. This is your safety check to avoid upgrading to a version that might break things.
2. Drain the Control Plane Node
Run 'kubectl drain <control-plane-node-name> --ignore-daemonsets'. This moves all pods (except system daemonsets) to other nodes, so the control plane can be taken offline without disrupting applications. The node is marked as 'SchedulingDisabled'.
3. Upgrade the Control Plane Components
Run 'kubeadm upgrade apply v1.29.0'. This command updates the static pod manifests for the API server, controller manager, and scheduler to the new version. The components are restarted automatically.
4. Update kubelet and Uncordon the Control Plane Node
Install the new kubelet and kubeadm versions using your package manager (e.g., 'apt-get install kubelet=1.29.0-00 kubeadm=1.29.0-00'). Restart the kubelet with 'systemctl restart kubelet'. Then run 'kubectl uncordon <control-plane-node>' to make it schedulable again.
5. Drain and Upgrade Each Worker Node
For each worker node, run 'kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data'. SSH into the node and run 'kubeadm upgrade node'. Then update the kubelet package and restart the service. Finally, uncordon the node. Repeat for every worker node.
6. Verify the Cluster Health After the Upgrade
Run 'kubectl get nodes' to confirm all nodes are 'Ready'. Run 'kubectl version' to check the server version is now the target version. Deploy a test pod to confirm the cluster can still run workloads. This step validates the upgrade was successful.
Imagine you are a platform engineer at a mid-sized e-commerce company. Your Kubernetes cluster runs the entire website: product catalogue, shopping cart, payment processing, and user login. It is a Friday afternoon, and your security team has announced a critical vulnerability in the current Kubernetes version (1.28.0). You have received approval to upgrade to 1.29.0 next Tuesday during your scheduled maintenance window.
Your real-world actions begin on Tuesday morning at 6:00 AM.
Verify the current state: You SSH into a control plane node and run 'kubectl get nodes'. You see 1 control plane node and 5 worker nodes. You run 'kubeadm upgrade plan' to confirm the path to 1.29.0 is clear and see no blockers.
Perform the control plane upgrade with caution: You know the control plane is the single point of failure. You drain the control plane node: 'kubectl drain master1 --ignore-daemonsets'. This command carefully evicts all pods, except daemonsets (which should run everywhere). You wait until the node status shows 'SchedulingDisabled'. Then you run 'kubeadm upgrade apply v1.29.0'. It takes about 30 seconds. You watch the output to ensure no errors.
Update the local binaries: You install the new 'kubelet' and 'kubectl' packages on the control plane node using your system package manager (apt or yum). You restart the kubelet: 'sudo systemctl restart kubelet'. Then you uncordon: 'kubectl uncordon master1'.
Test the control plane: You quickly run 'kubectl get pods --all-namespaces' to ensure the API server is responding and all system pods (CoreDNS, etc.) are running. Everything looks green.
Upgrade the worker nodes one at a time: This is the most time-consuming part. You have 5 worker nodes. For each one:
You run 'kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data'. The '--delete-emptydir-data' flag is necessary because some pods might be using temporary local storage.
You SSH into worker1 and run 'kubeadm upgrade node'.
You install the new kubelet and kubeadm packages and restart kubelet.
You run 'kubectl uncordon worker1'.
You wait a few minutes for the node to become healthy and for pods to be rescheduled onto it.
Rollback planning: In your back pocket, you have the old kubeadm and kubelet binaries ready on a USB drive. If something goes catastrophically wrong with a node, you know you can manually revert the package version and restart the kubelet. You also have a backup of the etcd database (the cluster's brain data), taken before the upgrade started.
Final verification: After all nodes are upgraded, you run 'kubectl get nodes' to confirm all are in 'Ready' state. You run 'kubectl version' to show the client and server versions are now 1.29.0. You then run a quick smoke test by deploying a test pod to ensure the cluster can still run a container normally.
The whole operation, from first command to final verification, takes about 2 hours. The website never goes down. The shopping cart never empties. The users never notice a thing. This is the value of a well-executed rolling upgrade.
The CKA exam loves to test your knowledge of the 'kubeadm upgrade' workflow because it is a specific, procedural task. You will not be asked 'How do you upgrade Kubernetes?'. You will be asked to actually do it on a live, broken, or partially upgraded cluster. The exam environment is a terminal in a browser, and you have to type the commands correctly.
Here are the exact concepts they test and the traps they set:
Version skew policy: They will give you a cluster where the control plane is on version 1.28 and the nodes are on 1.29. This violates the policy (nodes can only be one minor version BEHIND the control plane, not ahead). You must recognise this is an illegal state and know how to fix it (downgrade the nodes or upgrade the control plane further).
The order of operations: You MUST drain a node before you upgrade it. A common trap is to upgrade a node without draining it first, which will cause the pods to be killed. They will give you a scenario where the cluster is healthy but a specific node is 'NotReady' because it was upgraded incorrectly. The fix is to uncordon it.
The difference between 'kubeadm upgrade apply' and 'kubeadm upgrade node': 'apply' is for the first control plane node only. 'node' is for all subsequent control plane and worker nodes. If you run 'apply' on a worker node, it will fail. They will test this distinction.
Cordon and uncordon: Knowing these commands is non-negotiable. 'Cordon' marks a node as unschedulable (no new pods), but existing pods stay running. 'Drain' does cordon + evicts pods. 'Uncordon' makes it schedulable again. They love to create a situation where a node is cordoned but not drained, and you have to uncordon it to fix it.
Package management commands: The exam expects you to know how to install or update packages on Ubuntu (apt) or CentOS/RHEL (yum). You will need to run 'apt-get install kubelet=1.29.0-00' or similar. The exact version string format (with the '-00') is important.
Checking versions: You must know 'kubectl version' (shows client and server versions) and 'kubeadm upgrade plan' (shows the upgrade path and available versions). They might give you a cluster where the node version differs from the control plane version, and ask you to identify the skew.
Common exam traps:
They will present a cluster where the control plane has already been upgraded, and one worker node is still on the old version. Your task is to upgrade just that one node. You must remember to drain it first.
They will have a node that is 'NotReady' because the kubelet service is not running or is running the wrong version. You need to restart the service or install the correct version.
They will ask you to upgrade a cluster from version 1.28 to 1.29, but the cluster is running 1.26. You must know that you cannot skip minor versions. You would need to go 1.26 -> 1.27 -> 1.28 -> 1.29.
Key definitions to memorise: - 'kubeadm upgrade plan': The read-only command to see the upgrade path. - 'kubeadm upgrade apply <version>': The write command to upgrade the first control plane. - 'kubeadm upgrade node': The command for all other nodes. - 'kubectl drain': The command to evict pods from a node. - 'kubectl uncordon': The command to make a node schedulable again.
The version skew policy allows the control plane to be one minor version ahead of the nodes, but nodes can never be ahead of the control plane.
You must run 'kubeadm upgrade apply' only on the first control plane node; all other nodes use 'kubeadm upgrade node'.
Draining a node is mandatory before any upgrade to ensure pods are gracefully moved away and your applications stay available.
The 'kubeadm upgrade plan' command is a dry-run that shows you exactly what will change before you commit to the upgrade.
After running kubeadm, you must manually install the new kubelet and kubeadm packages using the system package manager (apt or yum) and restart the kubelet service.
You cannot skip minor versions during an upgrade; each minor version jump requires a separate upgrade cycle (e.g., 1.27 to 1.28 to 1.29).
These come up on the exam all the time. Here's how to tell them apart.
kubeadm upgrade apply
Used on the first control plane node only
Updates control plane component manifest files
Can be run only once per cluster per version
kubeadm upgrade node
Used on all subsequent control plane and worker nodes
Updates the kubelet configuration for that specific node
Can be run on any number of nodes after the apply step
Drain a Node
Marks node as unschedulable AND evicts all pods
Requires the '--ignore-daemonsets' flag for DaemonSet pods
Used before maintenance to free the node completely
Cordon a Node
Only marks node as unschedulable; pods remain running
Does not evict any running pods
Used to gently prevent new workloads during light maintenance
Minor Version Upgrade (e.g., 1.28 to 1.29)
Requires careful planning and version skew compliance
Changes the middle number; can introduce new features
Must be done sequentially; skipping versions is not allowed
Patch Version Upgrade (e.g., 1.28.1 to 1.28.2)
Is safe to perform without draining nodes
Changes the last number; only fixes bugs and security issues
Can be applied to all nodes simultaneously in many cases
Mistake
You can upgrade all worker nodes simultaneously and the cluster will automatically rebalance.
Correct
You must upgrade worker nodes one at a time. Upgrading all at once would cause the control plane to lose connection to every worker, leading to total application downtime.
New users think of nodes as independent entities, but the control plane needs to maintain a connection to every node. If you take them all offline, you break the cluster.
Mistake
The 'kubeadm upgrade apply' command also updates the kubelet binary automatically.
Correct
The 'kubeadm upgrade apply' command only updates the static pod manifests. You must manually install the new kubelet and kubeadm packages using the package manager (apt or yum) and restart the kubelet service.
People expect a magical 'one command to do everything' solution. But kubeadm cannot install system packages; that is the job of the OS package manager.
Mistake
If a node becomes 'NotReady' after an upgrade, you must reinstall the entire operating system.
Correct
A 'NotReady' node is usually caused by a version mismatch, a crashed kubelet service, or a failed configuration. You can usually fix it by checking the kubelet logs, restarting the service, or manually updating the configuration.
This misconception comes from fear of the unknown. The exam environment is designed to be fixable with simple commands, not full reinstallation. 'NotReady' is a symptom, not a death sentence.
Mistake
You must upgrade the control plane and worker nodes to the exact same version at the same time.
Correct
It is standard practice for the control plane to be one version ahead of the worker nodes during the upgrade process. This is allowed by the version skew policy (n-1). You can have two different versions running simultaneously for hours.
Beginners think of Kubernetes as a monolithic system where all components must match. The version skew policy exists precisely to allow safe, rolling upgrades with mixed versions.
Mistake
The 'drain' command immediately deletes pods and they are lost forever.
Correct
The 'drain' command gracefully evicts pods, allowing them to complete their current operations. The pods are then rescheduled onto other healthy nodes by the scheduler. No data is lost unless the pod is a stateful application tied to a specific node.
The word 'drain' sounds destructive. In reality, it is a polite, controlled process designed to maintain application availability.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
The command will fail with an error because 'kubeadm upgrade apply' is designed to update the control plane static pods. On a worker node, you must use 'kubeadm upgrade node' instead.
No, you cannot skip minor versions. You must upgrade sequentially: 1.27 to 1.28, then 1.28 to 1.29. Skipping versions would violate the version skew policy and can cause cluster instability.
Draining ensures that all pods running on that node are gracefully terminated and rescheduled to other nodes before the node is taken offline. This prevents application downtime during the upgrade.
Cordon marks a node as unschedulable so no new pods are assigned to it, but existing pods continue running. Drain does a cordon first and then evicts all existing pods from the node.
The flag tells kubectl to skip pods that are managed by DaemonSets (like the networking agent). DaemonSet pods are designed to run on every node, so they stay on the node during the drain. You cannot evict them.
Use 'kubectl version' to see the client and server versions. Use 'kubectl get nodes' to see the kubelet version on each node. Use 'kubeadm upgrade plan' to see the full upgrade path.
You've finished Managing Cluster Upgrades. Continue through the CKA study guide to build a complete picture of the exam.
Done with this chapter?