If you lose the central brain of your Kubernetes cluster, you lose everything: every running application, every configuration, every network rule. That brain is called etcd, and knowing how to back it up and restore it is not optional for passing the CKA exam; it is a survival skill. This chapter will show you exactly what etcd is and how to save and recover it, step by step.
Jump to a section
A simple way to picture etcd Backup and Restore
Your office is buzzing with activity. The central filing cabinet in the corner holds the master list of every employee, every department, their roles, and who reports to whom. Every time a new person is hired, a promotion happens, or a team restructures, a single trusted administrator updates that one cabinet, and they keep a duplicate key locked in their drawer.
One Monday morning, you walk in and find the filing cabinet has been knocked over. Papers are everywhere, some are torn, and the drawer handle is broken. Panic sets in because without that central list, nobody knows who works where, projects stall, and payroll is a mess. But you remember something vital: last Friday, before leaving, you took a perfect photocopy of every single document in that cabinet and locked it in the fireproof safe in the basement.
You head down, retrieve that photocopy, take it to the administrator, and say, "Restore from this." Within the hour, the cabinet is reassembled, every crumpled document is replaced, and the office is running again. The team lost only the few minutes it took to retrieve the backup. That photocopy is your etcd snapshot. The fireproof safe is your secure storage. The administrator is the restore command. Without that backup, you would have spent days reconstructing the entire organisation from memory, and you would probably get it wrong.
Let us start with the absolute basics. A Kubernetes cluster is a group of computers (called nodes) that work together to run your applications. One computer is the control plane, which is the brain of the operation. Inside that brain lives a database called etcd.
Etcd is a key-value store. This means it holds information as pairs: a key (like a label) and a value (like the data attached to that label). Think of it as a giant dictionary where the word is the key and the definition is the value. In Kubernetes, the keys are things like "pod name" or "service name", and the values are the details about that pod or service, such as its IP address, its configuration, and which containers it runs.
Why does Kubernetes use a key-value store instead of a normal database like Microsoft SQL or MySQL? Because etcd is designed for a very specific job: storing configuration data that changes frequently and needs to be accessed incredibly quickly. It is also built for high availability, meaning it can survive the failure of one or more of its copies (called replicas) without losing data. Normal databases are often heavier and slower for this kind of work.
Etcd stores everything that defines your cluster state. This includes:
All the pods that are running and their specifications.
All the services and their network endpoints.
All the configuration maps (ConfigMaps) and secrets (like passwords).
All the rules for which users can do what (RBAC - Role-Based Access Control).
All the nodes that are part of the cluster.
If etcd gets corrupted or deleted, your cluster becomes a zombie. It might still be running, but it no longer knows what it is supposed to be doing. You cannot access it, and you cannot fix it by simply restarting the control plane services. You have to restore that etcd data from a backup.
There are two main ways to back up etcd:
Snapshot backup: This takes a point-in-time copy of the entire etcd database. It is the most common and recommended method. You run a command like ETCDCTL_API=3 etcdctl snapshot save /path/to/backup.db. This creates a single file (the snapshot) that contains everything.
Data directory backup: This involves copying the folder where etcd stores its data (usually /var/lib/etcd). This is less reliable because the data might be in an inconsistent state if the copy happens while etcd is writing, and it requires you to stop the etcd service first.
Restoring a snapshot is equally straightforward but requires careful steps. You use ETCDCTL_API=3 etcdctl snapshot restore /path/to/backup.db to create a new data directory. Then, you configure etcd to use that new directory and restart the etcd service. The key detail is that the restore command creates a new cluster member with a fresh cluster ID, so you must update the etcd configuration file to point to this new data directory.
One crucial detail for the CKA exam: you must set the ETCDCTL_API environment variable to version 3 before running any commands. Etcd has two API versions (v2 and v3), and Kubernetes uses v3. If you forget this, the commands will fail or use the wrong API, and you will lose marks.
Another important point is that the etcd endpoints and certificates are not optional. When you run backup or restore commands, you often need to specify the endpoint address (e.g., --endpoints=https://127.0.0.1:2379) and the paths to the CA certificate, the server certificate, and the server key (e.g., --cacert, --cert, --key). These are stored in the etcd manifest file or the static pod definition file, usually located at /etc/kubernetes/manifests/etcd.yaml. If you do not provide these correctly, the connection to etcd will fail because it requires authentication.
In summary, etcd is the source of truth for your entire Kubernetes cluster. Backing it up regularly is like taking out an insurance policy. Restoring it is the emergency procedure you use when the worst happens.
Prepare the environment
Stop the kubelet to prevent it from restarting the etcd container while you work. Use `systemctl stop kubelet`. Also, set the ETCDCTL_API environment variable to 3 by running `export ETCDCTL_API=3`.
Create the snapshot
Run the ectdctl snapshot save command with the required flags: --endpoints, --cacert, --cert, and --key. Example: `sudo ETCDCTL_API=3 etcdctl snapshot save /tmp/backup.db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key`. This creates a point-in-time backup of the entire etcd database.
Verify the snapshot
Use `sudo ETCDCTL_API=3 etcdctl snapshot status /tmp/backup.db` to check that the snapshot file is not corrupted and contains valid metadata. Look for a hash value and a revision number to confirm integrity.
Restore from the snapshot
Run the ectdctl snapshot restore command, specifying a new data directory. Example: `sudo ETCDCTL_API=3 etcdctl snapshot restore /tmp/backup.db --data-dir=/var/lib/etcd-restore`. This creates a fresh data directory with the restored data and a new cluster ID.
Update the etcd manifest
Edit /etc/kubernetes/manifests/etcd.yaml. Find the volume definition for the hostPath that points to the old data directory (usually /var/lib/etcd). Change it to point to the new directory: /var/lib/etcd-restore. Save the file.
Start the kubelet and verify
Use `systemctl start kubelet` to restart the kubelet, which will read the updated etcd manifest and start the etcd pod with the restored data. Wait a moment, then run `kubectl get nodes` and `kubectl get pods --all-namespaces` to confirm the cluster is functional and all resources are back.
Imagine you work for a medium-sized e-commerce company called ShopFast. The company runs all its services on a single Kubernetes cluster. One Friday afternoon, a junior administrator accidentally runs a kubectl delete deploy --all command on the wrong context, deleting every deployment in the production namespace. The website goes down. Customers cannot place orders. The panic button is pressed.
Your boss runs over to your desk. You are the one who set up the automated etcd snapshot job every night at 2 AM. There is no time to rebuild every deployment manually. That would take days. Instead, you decide to restore the cluster state from last night's etcd snapshot.
Here is the step-by-step process you follow in that real-world crisis:
1. You ssh into the control plane node where etcd is running. You have already identified the correct node because the static pod manifest is on that machine.
2. You stop the kubelet service on that node to prevent it from restarting the etcd container while you work. You use systemctl stop kubelet. This is a critical step because the kubelet will keep trying to restart the etcd container if it is part of a static pod.
3. You locate the most recent snapshot file. It is stored on a network file share for safety, but you have a local copy on the control plane node as well. The file is named etcd-snapshot-2025-03-14.db.
4. You run the restore command:
sudo ETCDCTL_API=3 etcdctl snapshot restore /backups/etcd-snapshot-2025-03-14.db --data-dir /var/lib/etcd-restore
This creates a new data directory at /var/lib/etcd-restore with a fresh cluster.
5. You edit the etcd static pod manifest file /etc/kubernetes/manifests/etcd.yaml and change the volume mount and volume host path to point to the new directory: from /var/lib/etcd to /var/lib/etcd-restore. This is the restore step that matters — if you leave the old path, etcd will start with the broken data.
6. You start the kubelet again: systemctl start kubelet. The kubelet rediscovers the etcd manifest and starts the etcd container with the restored data.
7. After a minute, you run kubectl get pods --all-namespaces and see that all the deployments are back. The website is up again. You just saved the company from a weekend-long outage.
In this scenario, having a recent etcd snapshot was the difference between a 30-minute recovery and a multi-day rebuild. Companies that do not back up etcd are gambling with their entire operation. For the CKA exam, you are tested on the exact commands and flags used in this scenario.
The CKA exam will test your ability to perform etcd backup and restore tasks directly on the command line. This is one of the few performance-based questions where you must type commands and get them right. The exam environment gives you a terminal and a cluster that is partially broken, and you have to fix it by restoring etcd.
Here is what they specifically test:
Snapshot creation: You will be asked to take a snapshot of the etcd database and save it to a specific path. They will provide the endpoint, CA certificate, client certificate, and client key. You must construct the correct etcdctl snapshot save command. The traps include:
Forgetting to set the ETCDCTL_API=3 environment variable. The exam environment defaults to v2, so your command fails.
Using the wrong endpoint. It is usually https://127.0.0.1:2379, but they could give you a different one.
Omitting the certificate flags. Without --cacert, --cert, and --key, the connection is rejected.
Snapshot restore: You will have a snapshot file already saved (or you are asked to restore from one). You must restore it to a new directory and then reconfigure the etcd manifest to use that directory. The traps include:
Not updating the --data-dir flag in the restore command to match what the etcd manifest expects.
Forgetting to modify the hostPath volume in the etcd manifest YAML file after the restore.
Attempting to restore the snapshot to the original data directory without changing it, which fails because the original directory already exists.
Identifying the etcd member: They might ask you to check the health of the etcd cluster or list its members before and after a restore. Commands like etcdctl member list and etcdctl endpoint health are used. They will test if you know to use --cluster flag with listing members.
The static pod path: You must know that the etcd configuration file is located at /etc/kubernetes/manifests/etcd.yaml. This is where the control plane components are defined as static pods. You edit this file directly.
Stopping and starting the kubelet: You must demonstrate that you know to stop the kubelet before modifying etcd files and start it again afterwards. The exam environment may not require this if they provide a non-static setup, but it is tested.
The --initial-cluster and --initial-cluster-token flags: During restore, you use flags to define the new cluster. They love to test that you understand these are required for multi-node etcd clusters. For a single-node cluster, it is simpler, but for a multi-node setup, you must specify the initial cluster state.
Key definitions to memorise for the exam:
- snapshot save: creates a backup file.
- snapshot status: checks the integrity and metadata of a snapshot.
- snapshot restore: creates a new data directory from a snapshot.
- --data-dir: the target directory for the restored data.
- --initial-cluster-token: a unique string that identifies a new cluster, needed when restoring a multi-node etcd.
Exam trap pattern: They will give you a broken cluster but not tell you exactly what is wrong. You must check the logs of the etcd pod to discover the issue. Then, you run the backup and restore procedure. The pass condition is that after your changes, kubectl get nodes returns the node as Ready.
Etcd is the only source of truth for the entire Kubernetes cluster state, storing everything from pods to secrets to RBAC rules.
The correct command to back up etcd is `etcdctl snapshot save` with the `ETCDCTL_API=3` environment variable set.
After restoring an etcd snapshot, you must update the etcd static pod manifest to use the new data directory created by the restore command.
You must provide the correct endpoint, CA certificate, client certificate, and client key when connecting to etcd for backup and restore operations.
A snapshot restore creates a new etcd cluster member with a new member ID, which is why you must adjust the `--initial-cluster` and `--initial-cluster-token` flags for multi-node setups.
Always stop the kubelet (using `systemctl stop kubelet`) before modifying the etcd data directory or its manifest to prevent race conditions.
The etcd static pod manifest is located at `/etc/kubernetes/manifests/etcd.yaml` and is the file you edit to change the data directory path after a restore.
The `snapshot status` command checks the integrity of an existing snapshot file, which is useful for verifying your backup is not corrupted before attempting a restore.
These come up on the exam all the time. Here's how to tell them apart.
Snapshot Backup
Creates a consistent point-in-time copy of the entire etcd database
Can be performed while etcd is running without stopping the service
Results in a single portable file (.db) that is easy to store and transfer
Data Directory Backup
Copies the raw data directory, which may be inconsistent if etcd is running
Requires stopping the etcd service to avoid data corruption
Involves copying an entire directory structure, which is bulkier and less portable
Single-Node etcd
Simpler restore: only one member to recover
No need for cluster token or initial cluster flags during restore
Suitable for test/development environments or low-availability needs
Multi-Node etcd
Requires restoring each member with a unique initial cluster token
Needs careful handling of member IDs and cluster configuration to avoid duplicates
Provides high availability by tolerating the failure of one or two nodes
Static Pod Manifest (etcd)
etcd is run as a static pod managed by the kubelet
Configuration file is /etc/kubernetes/manifests/etcd.yaml
Editing the manifest file causes the kubelet to restart the pod automatically
Systemd Service (etcd)
etcd is run as a traditional systemd service
Configuration is in /etc/etcd/etcd.conf or similar
Requires systemctl restart etcd to apply changes, not automatic
Mistake
Restoring an etcd snapshot will only bring back the applications, not the cluster configuration.
Correct
Restoring an etcd snapshot brings back everything: applications, configurations, secrets, RBAC rules, and node definitions. It is a full state restore.
Beginners often think applications and cluster configuration are stored separately. But etcd stores the entire state of the cluster, including application objects like deployments and pods.
Mistake
I can just copy the /var/lib/etcd directory while the cluster is running and that counts as a valid backup.
Correct
Copying the data directory while etcd is running is unreliable because the data can be in an inconsistent state, leading to corruption. You must either stop etcd or use the snapshot command, which takes a consistent point-in-time backup.
Users are used to copying folders for backup in other contexts (like documents). They do not realise that databases, especially distributed ones like etcd, require special handling to ensure consistency.
Mistake
If I restore a snapshot, I can simply restart etcd and everything will work without modifying any configuration files.
Correct
After restoring, you must edit the etcd manifest file (etcd.yaml) to point to the new data directory created by the restore command. If you do not, etcd will start with the old, broken data.
This mistake comes from thinking the restore command automatically updates the live configuration. In reality, the restore creates a new directory elsewhere, and you have to manually configure the system to use it.
Mistake
The ETCDCTL_API environment variable is optional and only matters for advanced features.
Correct
The ETCDCTL_API variable is mandatory. Kubernetes uses etcd API v3. Without setting it to 3, etcdctl defaults to v2 and the snapshot commands will fail or behave incorrectly.
Beginners see version numbers as optional flags. They do not realise that the API version changes the behaviour of basic commands like snapshot save.
Mistake
Backing up a single node in a multi-node etcd cluster is enough to restore the entire cluster.
Correct
In a multi-node cluster, a restore from a single node's snapshot is more complex. You must generally restore each member separately using specific flags like --initial-cluster-token to prevent member identity conflicts.
People think of etcd like a standard database where one backup fits all. They miss that etcd has a consensus protocol (Raft) and each member has a unique ID and cluster relationship.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
The command is `ETCDCTL_API=3 etcdctl snapshot save <path> --endpoints=<URL> --cacert=<ca.crt> --cert=<server.crt> --key=<server.key>`. You must set the API version to 3 and provide the certificate paths.
It is strongly recommended to stop the kubelet first to prevent it from restarting the etcd container while you modify the manifest and data directory. Skipping this step can cause race conditions and lead to restoration failure.
The etcdctl tool will default to API v2. In v2, the snapshot save command does not exist in the same way, and the command will fail with an error like 'unknown command'. You must always set the API version to 3 for Kubernetes.
They are typically stored in /etc/kubernetes/pki/etcd/. The files are ca.crt, server.crt, and server.key. The exact paths are listed in the etcd static pod manifest file at /etc/kubernetes/manifests/etcd.yaml.
No, it is not safe. The original directory (e.g., /var/lib/etcd) already exists and contains the corrupted data. The restore command expects an empty or non-existent directory. You must restore to a new directory and then update the manifest.
Use the `etcdctl snapshot status <snapshot-file>` command. It displays the hash, revision, total keys, and size. If it returns valid data without errors, the snapshot is likely intact.
You've finished etcd Backup and Restore. Continue through the CKA study guide to build a complete picture of the exam.
Done with this chapter?