# etcd Backup and Restore

> Chapter 5 of the Courseiva CNCF-CKA curriculum — https://courseiva.com/learn/cncf-cka/etcd-backup-and-restore

**Official objective:** Cluster Architecture, Installation & Configuration - 25% — Backup and restore etcd cluster data.

## Introduction

If you lose the central brain of your Kubernetes cluster, you lose everything: every running application, every configuration, every network rule. That brain is called etcd, and knowing how to back it up and restore it is not optional for passing the CKA exam; it is a survival skill. This chapter will show you exactly what etcd is and how to save and recover it, step by step.

## The Office Filing Cabinet Analogy

Your office is buzzing with activity. The central filing cabinet in the corner holds the master list of every employee, every department, their roles, and who reports to whom. Every time a new person is hired, a promotion happens, or a team restructures, a single trusted administrator updates that one cabinet, and they keep a duplicate key locked in their drawer. 

One Monday morning, you walk in and find the filing cabinet has been knocked over. Papers are everywhere, some are torn, and the drawer handle is broken. Panic sets in because without that central list, nobody knows who works where, projects stall, and payroll is a mess. But you remember something vital: last Friday, before leaving, you took a perfect photocopy of every single document in that cabinet and locked it in the fireproof safe in the basement. 

You head down, retrieve that photocopy, take it to the administrator, and say, "Restore from this." Within the hour, the cabinet is reassembled, every crumpled document is replaced, and the office is running again. The team lost only the few minutes it took to retrieve the backup. That photocopy is your etcd snapshot. The fireproof safe is your secure storage. The administrator is the restore command. Without that backup, you would have spent days reconstructing the entire organisation from memory, and you would probably get it wrong.

## Core explanation

Let us start with the absolute basics. A Kubernetes cluster is a group of computers (called nodes) that work together to run your applications. One computer is the control plane, which is the brain of the operation. Inside that brain lives a database called etcd. 

Etcd is a key-value store. This means it holds information as pairs: a key (like a label) and a value (like the data attached to that label). Think of it as a giant dictionary where the word is the key and the definition is the value. In Kubernetes, the keys are things like "pod name" or "service name", and the values are the details about that pod or service, such as its IP address, its configuration, and which containers it runs. 

Why does Kubernetes use a key-value store instead of a normal database like Microsoft SQL or MySQL? Because etcd is designed for a very specific job: storing configuration data that changes frequently and needs to be accessed incredibly quickly. It is also built for high availability, meaning it can survive the failure of one or more of its copies (called replicas) without losing data. Normal databases are often heavier and slower for this kind of work. 

Etcd stores everything that defines your cluster state. This includes:
- All the pods that are running and their specifications.
- All the services and their network endpoints.
- All the configuration maps (ConfigMaps) and secrets (like passwords).
- All the rules for which users can do what (RBAC - Role-Based Access Control).
- All the nodes that are part of the cluster.

If etcd gets corrupted or deleted, your cluster becomes a zombie. It might still be running, but it no longer knows what it is supposed to be doing. You cannot access it, and you cannot fix it by simply restarting the control plane services. You have to restore that etcd data from a backup. 

There are two main ways to back up etcd: 
- Snapshot backup: This takes a point-in-time copy of the entire etcd database. It is the most common and recommended method. You run a command like `ETCDCTL_API=3 etcdctl snapshot save /path/to/backup.db`. This creates a single file (the snapshot) that contains everything. 
- Data directory backup: This involves copying the folder where etcd stores its data (usually `/var/lib/etcd`). This is less reliable because the data might be in an inconsistent state if the copy happens while etcd is writing, and it requires you to stop the etcd service first. 

Restoring a snapshot is equally straightforward but requires careful steps. You use `ETCDCTL_API=3 etcdctl snapshot restore /path/to/backup.db` to create a new data directory. Then, you configure etcd to use that new directory and restart the etcd service. The key detail is that the restore command creates a new cluster member with a fresh cluster ID, so you must update the etcd configuration file to point to this new data directory. 

One crucial detail for the CKA exam: you must set the `ETCDCTL_API` environment variable to version 3 before running any commands. Etcd has two API versions (v2 and v3), and Kubernetes uses v3. If you forget this, the commands will fail or use the wrong API, and you will lose marks.

Another important point is that the etcd endpoints and certificates are not optional. When you run backup or restore commands, you often need to specify the endpoint address (e.g., `--endpoints=https://127.0.0.1:2379`) and the paths to the CA certificate, the server certificate, and the server key (e.g., `--cacert`, `--cert`, `--key`). These are stored in the etcd manifest file or the static pod definition file, usually located at `/etc/kubernetes/manifests/etcd.yaml`. If you do not provide these correctly, the connection to etcd will fail because it requires authentication. 

In summary, etcd is the source of truth for your entire Kubernetes cluster. Backing it up regularly is like taking out an insurance policy. Restoring it is the emergency procedure you use when the worst happens.

## Real-world context

Imagine you work for a medium-sized e-commerce company called ShopFast. The company runs all its services on a single Kubernetes cluster. One Friday afternoon, a junior administrator accidentally runs a `kubectl delete deploy --all` command on the wrong context, deleting every deployment in the production namespace. The website goes down. Customers cannot place orders. The panic button is pressed. 

Your boss runs over to your desk. You are the one who set up the automated etcd snapshot job every night at 2 AM. There is no time to rebuild every deployment manually. That would take days. Instead, you decide to restore the cluster state from last night's etcd snapshot. 

Here is the step-by-step process you follow in that real-world crisis:

1. You ssh into the control plane node where etcd is running. You have already identified the correct node because the static pod manifest is on that machine. 
2. You stop the kubelet service on that node to prevent it from restarting the etcd container while you work. You use `systemctl stop kubelet`. This is a critical step because the kubelet will keep trying to restart the etcd container if it is part of a static pod. 
3. You locate the most recent snapshot file. It is stored on a network file share for safety, but you have a local copy on the control plane node as well. The file is named `etcd-snapshot-2025-03-14.db`. 
4. You run the restore command:
   `sudo ETCDCTL_API=3 etcdctl snapshot restore /backups/etcd-snapshot-2025-03-14.db --data-dir /var/lib/etcd-restore`
   This creates a new data directory at `/var/lib/etcd-restore` with a fresh cluster. 
5. You edit the etcd static pod manifest file `/etc/kubernetes/manifests/etcd.yaml` and change the volume mount and volume host path to point to the new directory: from `/var/lib/etcd` to `/var/lib/etcd-restore`. This is the restore step that matters — if you leave the old path, etcd will start with the broken data. 
6. You start the kubelet again: `systemctl start kubelet`. The kubelet rediscovers the etcd manifest and starts the etcd container with the restored data. 
7. After a minute, you run `kubectl get pods --all-namespaces` and see that all the deployments are back. The website is up again. You just saved the company from a weekend-long outage. 

In this scenario, having a recent etcd snapshot was the difference between a 30-minute recovery and a multi-day rebuild. Companies that do not back up etcd are gambling with their entire operation. For the CKA exam, you are tested on the exact commands and flags used in this scenario.

## Exam focus

The CKA exam will test your ability to perform etcd backup and restore tasks directly on the command line. This is one of the few performance-based questions where you must type commands and get them right. The exam environment gives you a terminal and a cluster that is partially broken, and you have to fix it by restoring etcd. 

Here is what they specifically test:

- Snapshot creation: You will be asked to take a snapshot of the etcd database and save it to a specific path. They will provide the endpoint, CA certificate, client certificate, and client key. You must construct the correct `etcdctl snapshot save` command. The traps include:
  - Forgetting to set the `ETCDCTL_API=3` environment variable. The exam environment defaults to v2, so your command fails.
  - Using the wrong endpoint. It is usually `https://127.0.0.1:2379`, but they could give you a different one.
  - Omitting the certificate flags. Without `--cacert`, `--cert`, and `--key`, the connection is rejected.

- Snapshot restore: You will have a snapshot file already saved (or you are asked to restore from one). You must restore it to a new directory and then reconfigure the etcd manifest to use that directory. The traps include:
  - Not updating the `--data-dir` flag in the restore command to match what the etcd manifest expects.
  - Forgetting to modify the `hostPath` volume in the etcd manifest YAML file after the restore.
  - Attempting to restore the snapshot to the original data directory without changing it, which fails because the original directory already exists.

- Identifying the etcd member: They might ask you to check the health of the etcd cluster or list its members before and after a restore. Commands like `etcdctl member list` and `etcdctl endpoint health` are used. They will test if you know to use `--cluster` flag with listing members.

- The static pod path: You must know that the etcd configuration file is located at `/etc/kubernetes/manifests/etcd.yaml`. This is where the control plane components are defined as static pods. You edit this file directly. 

- Stopping and starting the kubelet: You must demonstrate that you know to stop the kubelet before modifying etcd files and start it again afterwards. The exam environment may not require this if they provide a non-static setup, but it is tested. 

- The `--initial-cluster` and `--initial-cluster-token` flags: During restore, you use flags to define the new cluster. They love to test that you understand these are required for multi-node etcd clusters. For a single-node cluster, it is simpler, but for a multi-node setup, you must specify the initial cluster state. 

Key definitions to memorise for the exam:
- `snapshot save`: creates a backup file.
- `snapshot status`: checks the integrity and metadata of a snapshot.
- `snapshot restore`: creates a new data directory from a snapshot.
- `--data-dir`: the target directory for the restored data.
- `--initial-cluster-token`: a unique string that identifies a new cluster, needed when restoring a multi-node etcd. 

Exam trap pattern: They will give you a broken cluster but not tell you exactly what is wrong. You must check the logs of the etcd pod to discover the issue. Then, you run the backup and restore procedure. The pass condition is that after your changes, `kubectl get nodes` returns the node as Ready.

## Step by step

1. **Prepare the environment** — Stop the kubelet to prevent it from restarting the etcd container while you work. Use `systemctl stop kubelet`. Also, set the ETCDCTL_API environment variable to 3 by running `export ETCDCTL_API=3`.
2. **Create the snapshot** — Run the ectdctl snapshot save command with the required flags: --endpoints, --cacert, --cert, and --key. Example: `sudo ETCDCTL_API=3 etcdctl snapshot save /tmp/backup.db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key`. This creates a point-in-time backup of the entire etcd database.
3. **Verify the snapshot** — Use `sudo ETCDCTL_API=3 etcdctl snapshot status /tmp/backup.db` to check that the snapshot file is not corrupted and contains valid metadata. Look for a hash value and a revision number to confirm integrity.
4. **Restore from the snapshot** — Run the ectdctl snapshot restore command, specifying a new data directory. Example: `sudo ETCDCTL_API=3 etcdctl snapshot restore /tmp/backup.db --data-dir=/var/lib/etcd-restore`. This creates a fresh data directory with the restored data and a new cluster ID.
5. **Update the etcd manifest** — Edit /etc/kubernetes/manifests/etcd.yaml. Find the volume definition for the hostPath that points to the old data directory (usually /var/lib/etcd). Change it to point to the new directory: /var/lib/etcd-restore. Save the file.
6. **Start the kubelet and verify** — Use `systemctl start kubelet` to restart the kubelet, which will read the updated etcd manifest and start the etcd pod with the restored data. Wait a moment, then run `kubectl get nodes` and `kubectl get pods --all-namespaces` to confirm the cluster is functional and all resources are back.

## Comparisons

### Snapshot Backup vs Data Directory Backup

**Snapshot Backup:**
- Creates a consistent point-in-time copy of the entire etcd database
- Can be performed while etcd is running without stopping the service
- Results in a single portable file (.db) that is easy to store and transfer

**Data Directory Backup:**
- Copies the raw data directory, which may be inconsistent if etcd is running
- Requires stopping the etcd service to avoid data corruption
- Involves copying an entire directory structure, which is bulkier and less portable

### Single-Node etcd vs Multi-Node etcd

**Single-Node etcd:**
- Simpler restore: only one member to recover
- No need for cluster token or initial cluster flags during restore
- Suitable for test/development environments or low-availability needs

**Multi-Node etcd:**
- Requires restoring each member with a unique initial cluster token
- Needs careful handling of member IDs and cluster configuration to avoid duplicates
- Provides high availability by tolerating the failure of one or two nodes

### Static Pod Manifest (etcd) vs Systemd Service (etcd)

**Static Pod Manifest (etcd):**
- etcd is run as a static pod managed by the kubelet
- Configuration file is /etc/kubernetes/manifests/etcd.yaml
- Editing the manifest file causes the kubelet to restart the pod automatically

**Systemd Service (etcd):**
- etcd is run as a traditional systemd service
- Configuration is in /etc/etcd/etcd.conf or similar
- Requires systemctl restart etcd to apply changes, not automatic

## Diagram

_Flowchart showing the relationship between the Kubernetes cluster, etcd, and the backup/restore process using snapshot commands._

```mermaid
flowchart TD
    A[Kubernetes Cluster] --> B[Control Plane Node]
    B --> C[etcd Database]
    C --> D[Stores all cluster state]
    D --> E[Pods, Services, Configs, Secrets, RBAC]
    F[etcdctl snapshot save] --> G[Creates .db snapshot file]
    G --> H[Safe storage]
    I[etcdctl snapshot restore] --> J[New data directory]
    J --> K[Edit etcd.yaml manifest]
    K --> L[Restart kubelet]
    L --> M[Cluster restored]
```

## Common misconceptions

- **Misconception:** Restoring an etcd snapshot will only bring back the applications, not the cluster configuration. **Reality:** Restoring an etcd snapshot brings back everything: applications, configurations, secrets, RBAC rules, and node definitions. It is a full state restore. (Beginners often think applications and cluster configuration are stored separately. But etcd stores the entire state of the cluster, including application objects like deployments and pods.)
- **Misconception:** I can just copy the /var/lib/etcd directory while the cluster is running and that counts as a valid backup. **Reality:** Copying the data directory while etcd is running is unreliable because the data can be in an inconsistent state, leading to corruption. You must either stop etcd or use the snapshot command, which takes a consistent point-in-time backup. (Users are used to copying folders for backup in other contexts (like documents). They do not realise that databases, especially distributed ones like etcd, require special handling to ensure consistency.)
- **Misconception:** If I restore a snapshot, I can simply restart etcd and everything will work without modifying any configuration files. **Reality:** After restoring, you must edit the etcd manifest file (etcd.yaml) to point to the new data directory created by the restore command. If you do not, etcd will start with the old, broken data. (This mistake comes from thinking the restore command automatically updates the live configuration. In reality, the restore creates a new directory elsewhere, and you have to manually configure the system to use it.)
- **Misconception:** The ETCDCTL_API environment variable is optional and only matters for advanced features. **Reality:** The ETCDCTL_API variable is mandatory. Kubernetes uses etcd API v3. Without setting it to 3, etcdctl defaults to v2 and the snapshot commands will fail or behave incorrectly. (Beginners see version numbers as optional flags. They do not realise that the API version changes the behaviour of basic commands like snapshot save.)
- **Misconception:** Backing up a single node in a multi-node etcd cluster is enough to restore the entire cluster. **Reality:** In a multi-node cluster, a restore from a single node's snapshot is more complex. You must generally restore each member separately using specific flags like --initial-cluster-token to prevent member identity conflicts. (People think of etcd like a standard database where one backup fits all. They miss that etcd has a consensus protocol (Raft) and each member has a unique ID and cluster relationship.)

## Key takeaways

- Etcd is the only source of truth for the entire Kubernetes cluster state, storing everything from pods to secrets to RBAC rules.
- The correct command to back up etcd is `etcdctl snapshot save` with the `ETCDCTL_API=3` environment variable set.
- After restoring an etcd snapshot, you must update the etcd static pod manifest to use the new data directory created by the restore command.
- You must provide the correct endpoint, CA certificate, client certificate, and client key when connecting to etcd for backup and restore operations.
- A snapshot restore creates a new etcd cluster member with a new member ID, which is why you must adjust the `--initial-cluster` and `--initial-cluster-token` flags for multi-node setups.
- Always stop the kubelet (using `systemctl stop kubelet`) before modifying the etcd data directory or its manifest to prevent race conditions.
- The etcd static pod manifest is located at `/etc/kubernetes/manifests/etcd.yaml` and is the file you edit to change the data directory path after a restore.
- The `snapshot status` command checks the integrity of an existing snapshot file, which is useful for verifying your backup is not corrupted before attempting a restore.

## FAQ

**What is the exact command to take an etcd snapshot in Kubernetes?**

The command is `ETCDCTL_API=3 etcdctl snapshot save <path> --endpoints=<URL> --cacert=<ca.crt> --cert=<server.crt> --key=<server.key>`. You must set the API version to 3 and provide the certificate paths.

**Can I restore etcd without stopping the kubelet?**

It is strongly recommended to stop the kubelet first to prevent it from restarting the etcd container while you modify the manifest and data directory. Skipping this step can cause race conditions and lead to restoration failure.

**What happens if I forget to set ETCDCTL_API=3?**

The etcdctl tool will default to API v2. In v2, the snapshot save command does not exist in the same way, and the command will fail with an error like 'unknown command'. You must always set the API version to 3 for Kubernetes.

**Where are the etcd certificates stored on a Kubernetes control plane node?**

They are typically stored in /etc/kubernetes/pki/etcd/. The files are ca.crt, server.crt, and server.key. The exact paths are listed in the etcd static pod manifest file at /etc/kubernetes/manifests/etcd.yaml.

**Is it safe to use the `--data-dir` flag in the restore command to point to the original etcd directory?**

No, it is not safe. The original directory (e.g., /var/lib/etcd) already exists and contains the corrupted data. The restore command expects an empty or non-existent directory. You must restore to a new directory and then update the manifest.

**How do I verify that my etcd snapshot is not corrupted before restoring?**

Use the `etcdctl snapshot status <snapshot-file>` command. It displays the hash, revision, total keys, and size. If it returns valid data without errors, the snapshot is likely intact.

## Check your understanding

1. **What environment variable must be set before running etcdctl snapshot commands, and to what value?**

   Answer: You must set ETCDCTL_API=3 because Kubernetes uses etcd API version 3, without which the snapshot commands will fail.

2. **After restoring an etcd snapshot to a new directory, which file must you edit to point to the new directory?**

   Answer: You must edit /etc/kubernetes/manifests/etcd.yaml, specifically the hostPath in the volume definition for the data directory.

3. **What is the difference between etcdctl snapshot save and etcdctl snapshot restore?**

   Answer: Snapshot save creates a backup file of the current etcd state, while snapshot restore creates a new etcd data directory from a backup file.

4. **Why is it important to stop the kubelet before modifying the etcd data directory?**

   Answer: Stopping the kubelet prevents it from automatically restarting the etcd container while you are modifying its data or configuration, which could corrupt the restoration process.

5. **What does the 'snapshot status' command do?**

   Answer: It checks the integrity of an existing snapshot file and displays metadata such as the hash, revision number, and total number of keys to verify the snapshot is valid.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/cncf-cka/etcd-backup-and-restore