# Supply Chain Security: Secrets Management and Encryption

> Chapter 7 of the Courseiva CNCF-CKS curriculum — https://courseiva.com/learn/cncf-cks/supply-chain-secrets-management

**Official objective:** 2.3 — Manage secrets securely at rest and in transit

## Introduction

How do you keep a database password, an API key, or a TLS certificate safe from hackers when thousands of lines of code and dozens of servers can see it? In a Kubernetes cluster, secrets are the most valuable targets an attacker looks for, and the entire CKS exam is built on the assumption that you will secure them properly — because one leaked secret means full cluster compromise, no matter how strong your firewalls are.

## The Restaurant Kitchen Safe Analogy

A restaurant's kitchen safe is the single most guarded location in the entire building. This safe holds the chef's private recipe book, which contains the exact ingredient proportions, the supplier contact information, and the secret sauce formula that makes the restaurant famous. The sous chef has a key, the head chef has a combination, and the owner has a biometric fingerprint scanner — everyone accesses the safe with different credentials, and the safe keeps a log of every single opening.

In a busy kitchen, waiters need to see parts of recipes to explain allergens to customers, but they never get the full book. The delivery driver needs the supplier phone number but never sees the recipes. This system of limited, tracked, and encrypted information access is exactly how Kubernetes secrets management works. The safe is the encryption layer, the different keys are different user permissions, and the log of openings is the audit trail. If a waiter loses their access card, the head chef doesn't need to change the entire safe combination — they just revoke that one card and issue a new one. This mirrors how secrets rotation works in a Kubernetes cluster: you never change the entire encryption configuration; you rotate just the compromised credential.

The restaurant's success depends on this safe staying secure and the access rules being followed by every single employee, from the dishwasher to the owner. One unlocked safe and the entire business's competitive advantage disappears.

## Core explanation

Kubernetes stores secrets as small objects that contain sensitive data like passwords, OAuth tokens, and SSH keys. A secret object looks like a YAML or JSON file, but instead of being stored in plain text, its data field is base64 encoded. Base64 is not encryption — it is simple encoding that transforms bytes into text that can travel over networks safely. Anyone with access to the Kubernetes API server can decode a base64 string with a single command. This is a critical point: base64 is not security; it is just formatting.

When you create a secret, you give it a name and set the data field. For example, you might create a secret called 'db-password' with a key called 'password' and a value of 'MyS3cret!'. Kubernetes stores this secret inside a database called etcd, which is the cluster's brain — it holds all configuration data. By default, Kubernetes stores secrets inside etcd in base64 form, which means anyone who gains access to the etcd database can read every single secret in the cluster. That is why the CKS exam tests you on encrypting secrets at rest.

Encryption at rest means the data stored on disk inside etcd is encrypted using a cryptographic algorithm, typically AES (Advanced Encryption Standard) with 256-bit keys (AES-256). To enable this, you create an EncryptionConfiguration resource — a YAML file that tells Kubernetes which encryption provider to use. The providers are:

- identity: No encryption (the default, which stores secrets in plain base64).
- aescbc: Uses AES-CBC (Cipher Block Chaining) with a provided encryption key. This is the minimum acceptable option for CKS.
- secretbox: Uses XSalsa20-Poly1305, a modern and faster algorithm.
- kms: Uses a Key Management Service, often from a cloud provider like AWS KMS or Azure Key Vault. This option allows you to rotate keys without restarting the API server.

The EncryptionConfiguration is passed to the kube-apiserver component via a flag called --encryption-provider-config. Only after you restart the API server does the encryption take effect. But it is crucial to understand that existing secrets in etcd are not automatically re-encrypted. Kubernetes provides a command — kubectl get secrets --all-namespaces -o json | kubectl replace -f - — that forces a re-read and re-write of all secrets, which triggers encryption on the new write.

Secrets in transit (moving across the network) are encrypted by default because the kube-apiserver uses TLS (Transport Layer Security) to serve the HTTPS API. The TLS certificate on the API server ensures that when a kubectl command sends a secret creation request, the data is encrypted during transmission. This is why you never see secrets in plain text when you use kubectl get secret with a secure connection. However, the CKS exam will test you on the weakness: if TLS is misconfigured (for example, using a self-signed certificate without verification, or disabling TLS entirely), secrets can be intercepted on the wire.

Role-Based Access Control (RBAC) is the other half of secret security. RBAC defines who can access which secrets. A user or a service account (a non-human identity used by applications) must have explicit permission via a Role or ClusterRole to get, list, watch, create, update, patch, or delete secrets. The principle of least privilege applies: give only the minimum permissions necessary. For example, a pod that needs to read the database password should have a service account with a Role that only allows get on that specific secret, not list all secrets in the namespace.

Secrets should also be encrypted in etcd backups. If you back up etcd (which you do for disaster recovery), the backup file contains all secrets in whatever format they are stored in etcd. If encryption is enabled, the backup will contain encrypted secrets. But if you back up etcd before enabling encryption, the backup contains plain base64 secrets — a massive security risk.

The CKS exam expects you to know how to create an EncryptionConfiguration, apply it to the API server, and verify that secrets are encrypted. You also need to understand how to rotate encryption keys: create a new key, add it to the EncryptionConfiguration as the first entry (the new encryption key will be used for new writes), keep the old key listed second (for decrypting existing data), and then re-encrypt all secrets. This is called key rotation and is a common exam scenario.

Finally, there is the concept of secrets stored in external secret stores — solutions like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. In advanced setups, pods do not directly read Kubernetes secrets; instead, they use a sidecar container or a CSI (Container Storage Interface) driver to fetch the secret from the external vault at runtime. This means the secret never sits inside etcd at all. The CKS exam touches on this principle, though the focus remains on native Kubernetes secrets and encryption.

## Real-world context

Consider a financial technology company that processes credit card transactions. The company runs a Kubernetes cluster with 200 microservices. One of those services — the payment gateway — needs to authenticate to a third-party payment processor using an API key. That API key is a 64-character string of random letters and numbers. If that key leaks, an attacker could initiate fraudulent transactions on behalf of the company, leading to millions of dollars in liability.

An IT professional responsible for securing this cluster does the following:

1. First, they create the secret in a dedicated namespace called 'payment-system'. They do not put secrets in the 'default' namespace — that would be like leaving the safe key in the front door lock. They use a YAML manifest with the secret type set to 'Opaque' (the default type for arbitrary key-value pairs).

2. They enable encryption at rest by writing an EncryptionConfiguration file. They use the 'aescbc' provider with a 32-byte key (256-bit). The key is generated using: openssl rand -base64 32. They place this configuration file on every control plane node where the kube-apiserver runs.

3. They edit the kube-apiserver manifest file (typically at /etc/kubernetes/manifests/kube-apiserver.yaml) to add the --encryption-provider-config flag pointing to the EncryptionConfiguration file. They also add a hostPath volume mount so the API server pod can read the file. The API server automatically restarts as a static pod.

4. After the API server restarts, they run the command to re-encrypt all existing secrets: kubectl get secrets --all-namespaces -o json | kubectl replace -f -. They verify encryption by checking the raw etcd data with: ETCDCTL_API=3 etcdctl get /registry/secrets/payment-system/[secret-name] --endpoints=127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C. They look for 'k8s:enc:aescbc:v1' in the output, which confirms encryption is active.

5. They then implement RBAC. They create a ServiceAccount named 'payment-gateway-sa' in the 'payment-system' namespace. Next, they create a Role named 'secret-reader' that only allows 'get' on secrets in that namespace. They bind the Role to the ServiceAccount using a RoleBinding. The payment gateway pod is configured with 'serviceAccountName: payment-gateway-sa' so it can only read that one secret and no others.

6. Finally, they set up automated key rotation. Every 90 days, a script generates a new encryption key, appends it to the EncryptionConfiguration as the first provider, restarts the API server, re-encrypts all secrets, and then removes the old key from the configuration file. The script runs during a maintenance window with minimal traffic.

This step-by-step process ensures that even if an attacker gains access to the etcd database or the host filesystem, the secrets remain unreadable. If an attacker compromises a single pod, RBAC restricts them to only the secrets that pod is authorised to read. The entire system is designed on the principle of defence in depth: multiple layers of security so that no single failure leads to catastrophic data exposure.

## Exam focus

The CKS exam tests Secrets Management and Encryption very directly. Expect to see two main question types: configuration tasks and troubleshooting scenarios.

In configuration tasks, you are given a Kubernetes cluster that has no encryption at rest enabled. The exam asks you to create an EncryptionConfiguration, apply it, and verify the secrets are encrypted. The trap is that candidates often forget to re-encrypt existing secrets. The correct answer pattern is: create the EncryptionConfiguration YAML with proper indentation, mount it into the API server manifest, restart the API server (or wait for the static pod restart), and then run the kubectl replace command to re-encrypt all secrets in all namespaces. If you forget the replace step, the psecret will still be readable in etcd even though encryption is enabled for new writes.

Another common task is rotating encryption keys. The exam might present a scenario where a key has been compromised. You are expected to add a new key as the first entry in the providers list, restart the API server, re-encrypt secrets, and then remove the old key. The trap here: removing the old key before re-encrypting will make existing secrets permanently unreadable because they were encrypted with the old key. The correct order is: add new key first, restart, re-encrypt, remove old key second.

In troubleshooting scenarios, the exam presents a cluster where secrets are not being encrypted despite an EncryptionConfiguration being present. Common reasons tested:

- The EncryptionConfiguration file is mounted in the wrong path inside the API server container.
- The --encryption-provider-config flag points to a non-existent file.
- The API server manifest has a typo in the flag name (e.g., --encryption-provider-config vs --encryption-provider-configuration).
- The encryption key was generated with the wrong base64 encoding and is therefore invalid.
- The aescbc provider requires the key to be exactly 32 bytes base64 encoded; any other length fails silently.

The exam also tests RBAC for secrets. You might be asked to create a Role that allows a service account to read a specific secret but not list all secrets. The trap: if you create a Role that allows 'list' and 'get' on all secrets, it violates least privilege. The correct answer is a Role that only allows 'get' on the specific secret resource name, e.g., 'resourceNames: ["my-secret"]'.

Key definitions to memorise:

- etcd: The key-value store where Kubernetes stores all cluster data, including secrets. Needs to be encrypted at rest.
- EncryptionConfiguration: A YAML resource that defines how Kubernetes encrypts secrets at rest using specified providers.
- aescbc: A provider using AES-CBC with a 32-byte key — the minimum acceptable encryption for CKS.
- kms: A provider that delegates encryption to an external Key Management Service like AWS KMS or Azure Key Vault.
- identity: The default provider which applies no encryption — this is the insecure default that the exam expects you to replace.
- Secret: A Kubernetes object holding sensitive data, stored base64-encoded by default, not encrypted.
- RBAC (Role-Based Access Control): The system that controls access to secrets by users and service accounts.
- ServiceAccount: A non-human identity used by pods to authenticate to the Kubernetes API.
- TLS: Transport Layer Security — encrypts data in transit between clients and the API server.

The exam loves to ask: 'What is the difference between secret encryption at rest and secret encoding in base64?' The correct answer is that base64 is not encryption — it is simple encoding reversible by anyone with access to the API. Encryption uses a cryptographic key to make data unreadable without the key.

Another frequent trap: the exam says 'the secret is encrypted in transit' — this is true by default only if TLS is correctly configured. If the question states that TLS is disabled or the certificate is self-signed and verification is disabled, then secrets in transit are NOT encrypted. Always check the TLS configuration description in the scenario.

## Step by step

1. **Generate encryption key** — Use the command: openssl rand -base64 32. This creates a 32-byte (256-bit) key encoded in base64. This key will be used by the AES-CBC provider to encrypt and decrypt secrets. Store this key securely — if lost, all encrypted secrets become permanently unreadable.
2. **Create the EncryptionConfiguration YAML** — Write a YAML file (e.g., encryption-config.yaml) that defines the encryption provider as 'aescbc' and includes the generated key. The structure includes an 'apiVersion: apiserver.config.k8s.io/v1', 'kind: EncryptionConfiguration', 'resources:' block specifying 'secrets', and 'providers:' list with the aescbc provider containing the key. This file tells Kubernetes how to encrypt secrets at rest.
3. **Mount the configuration into the API server** — Edit the kube-apiserver manifest file at /etc/kubernetes/manifests/kube-apiserver.yaml. Add a hostPath volume that points to the directory containing the encryption-config.yaml. Mount this volume into the API server container at the same path. Also add the flag '--encryption-provider-config=/path/to/encryption-config.yaml' to the API server command. The API server restarts automatically as a static pod.
4. **Re-encrypt all existing secrets** — Run: kubectl get secrets --all-namespaces -o json | kubectl replace -f -. This command reads all secrets across all namespaces in JSON format and sends them back to the API server as update requests. Since encryption is now enabled, each update triggers encryption on write. Without this step, old secrets remain in etcd in their original unencrypted state.
5. **Verify encryption is working** — Use etcdctl (the etcd command-line tool) to read a secret directly from etcd: etcdctl get /registry/secrets/<namespace>/<secret-name> --endpoints=127.0.0.1:2379 with appropriate TLS certificates. The output should contain the prefix 'k8s:enc:aescbc:v1:', which confirms that encryption is active. If the output shows 'k8s:enc:aescbc:v1:' is missing, the encryption is not applied correctly.

## Comparisons

### Base64 Encoding vs Encryption at Rest

**Base64 Encoding:**
- Reversible without any key using a simple command.
- Used for formatting binary data as text in YAML files.
- Provides no confidentiality; anyone with API access can decode it.

**Encryption at Rest:**
- Requires a cryptographic key to decrypt data.
- Protects data stored on disk (in etcd) from unauthorized access.
- Uses algorithms like AES-256 that are computationally expensive to break.

### Secret in Environment Variable vs Secret Mounted as Volume

**Secret in Environment Variable:**
- Visible to every process in the container via /proc/1/environ.
- Can leak into logs if the application prints environment variables.
- Harder to rotate without restarting the pod.

**Secret Mounted as Volume:**
- Only accessible at the specific mount path inside the container.
- Not visible in process listings or default logs.
- Can be updated dynamically without restarting the pod if using subPath or projected volumes.

### aescbc Provider vs kms Provider

**aescbc Provider:**
- Encryption key is stored locally on the API server host.
- Requires manual key rotation by editing the EncryptionConfiguration.
- No dependency on external services; works offline.

**kms Provider:**
- Encryption key is stored externally in a cloud Key Management Service.
- Key rotation is managed by the KMS, often automated via IAM policies.
- Requires network connectivity to the KMS endpoint; adds latency.

### Etcd Backup Without Encryption vs Etcd Backup With Encryption

**Etcd Backup Without Encryption:**
- Contains secrets in plain base64 (or encrypted if at-rest encryption was enabled).
- If encryption was never enabled, the backup exposes all secrets in readable form.
- Requires manual encryption of the backup file after creation.

**Etcd Backup With Encryption:**
- Contains secrets encrypted with the cluster's encryption key.
- Even if the backup file is stolen, secrets remain unreadable without the encryption key.
- Backup encryption is automatic if at-rest encryption is enabled before backup.

## Diagram

_Flowchart showing the lifecycle of a Kubernetes secret from creation to consumption, highlighting the two key decision points: encryption at rest and RBAC access control._

```mermaid
flowchart TD
    A[User creates Secret] --> B[kubectl sends to API Server]
    B --> C{Is EncryptionConfiguration present?}
    C -- Yes --> D[Encrypt secret using AES-CBC]
    C -- No --> E[Store secret as base64 only]
    D --> F[Store encrypted secret in etcd]
    E --> F
    F --> G[Pod requests secret via ServiceAccount]
    G --> H{RBAC allows access?}
    H -- Yes --> I[API server reads from etcd]
    H -- No --> J[Access denied]
    I --> K{Is secret encrypted?}
    K -- Yes --> L[Decrypt secret and return to pod]
    K -- No --> M[Return base64 secret as is]
```

## Common misconceptions

- **Misconception:** Base64 encoding is a form of encryption that makes secrets safe in etcd. **Reality:** Base64 is not encryption; it is just encoding that anyone can reverse with a single command (echo <base64string> | base64 --decode). Secrets must be encrypted using a proper cipher like AES. (Many beginners see the scrambled base64 string and assume it's secure because it looks unreadable, not realising that base64 is designed for safe data transmission, not confidentiality.)
- **Misconception:** If you enable encryption at rest, all existing secrets are automatically encrypted immediately. **Reality:** Enabling encryption at rest only encrypts new or updated secrets. Existing secrets remain in their original unencrypted form until they are explicitly rewritten using kubectl get secrets --all-namespaces -o json | kubectl replace -f -. (This misunderstanding stems from expecting encryption to be applied retroactively, similar to how a disk encryption tool encrypts the whole disk — but Kubernetes does not re-write data already stored.)
- **Misconception:** You can rotate encryption keys by just replacing the old key with a new one in the EncryptionConfiguration file. **Reality:** When rotating encryption keys, you must add the new key as the first entry in the providers list (for new writes), restart the API server, re-encrypt all secrets (which uses the first key for encryption), and only then remove the old key. Removing the old key first makes existing secrets permanently unreadable. (People think of key rotation like changing a password — you delete the old one and create a new one. But encryption keys are used for decryption of existing data, so you need both keys during the transition.)
- **Misconception:** All secrets in a Kubernetes cluster are automatically encrypted in transit by default. **Reality:** Secrets are encrypted in transit only if the API server's TLS is correctly configured. If TLS is disabled, or if the client does not verify the server certificate (e.g., using --insecure-skip-tls-verify), secrets can be intercepted over the network in plain text. (Kubernetes documentation defaults to TLS, but misconfigurations are common in lab environments. Beginners assume TLS is always on, but the exam tests scenarios where it is not.)

## Key takeaways

- Base64 encoding is not encryption; always enable encryption at rest using aescbc or kms provider in etcd.
- Existing secrets are not automatically re-encrypted — you must run kubectl get secrets --all-namespaces -o json | kubectl replace -f - after enabling encryption.
- When rotating encryption keys, add the new key first in the EncryptionConfiguration list, restart the API server, re-encrypt, then remove the old key.
- RBAC for secrets must follow least privilege — use resourceNames in a Role to restrict access to specific secrets, not all secrets in a namespace.
- Secrets in transit are encrypted by default only if the API server TLS is correctly configured and client verification is enabled.
- Always encrypt etcd backups because a backup file contains all secrets in whatever format they are stored in etcd (plain or encrypted).

## FAQ

**Does enabling encryption at rest slow down my Kubernetes cluster?**

Yes, there is a small performance overhead because every read and write to etcd now involves encryption and decryption. The impact is minimal for most workloads (single-digit percentage slowdown), but in high-throughput clusters, using a KMS provider can reduce the load because the encryption key is not stored locally.

**Can I use the same encryption key for multiple clusters?**

Technically yes, but it is a bad security practise. Each cluster should have its own unique encryption key. If one cluster is compromised, the key from that cluster cannot be used to decrypt secrets in another cluster. Treat keys like passwords — never reuse them.

**What happens if I delete the encryption key while secrets are still encrypted with it?**

All secrets encrypted with that key become permanently unreadable and cannot be decrypted. The only recovery option is to restore from a backup that was taken before the key was deleted. This is why key management is critical — always back up encryption keys separately from the cluster data.

**Is it safe to store secrets in environment variables in a Pod?**

No, because environment variables are visible to any process inside the container and can be logged or leaked through debugging tools. The CKS exam teaches you to mount secrets as volumes instead, which are only accessible to the specific mount path and not exposed in process listings.

**Why does Kubernetes use base64 for secrets if it is not secure?**

Base64 is used purely for formatting — it allows binary data (like SSH keys or certificate files) to be represented as text in YAML. It is not intended as a security measure. The actual security comes from encryption at rest and RBAC. Base64 is simply a transport format, like how you might write a password on a sticky note — the note itself is not secure.

**Do I need to encrypt secrets at rest if I already have full disk encryption on the server?**

Yes, because full disk encryption protects data only when the server is turned off. When the server is running, the operating system decrypts the disk, and anyone with access to the server's RAM or running processes can read etcd. Kubernetes-level encryption ensures that even a running process cannot read the raw secret data from etcd unless it has the encryption key.

## Check your understanding

1. **What command do you use to re-encrypt all existing secrets after enabling encryption at rest?**

   Answer: kubectl get secrets --all-namespaces -o json | kubectl replace -f -

2. **What is the default encryption provider in Kubernetes, and what does it do?**

   Answer: The default provider is 'identity', which does not encrypt secrets at all; it stores them in base64-encoded form.

3. **When rotating encryption keys, why must you add the new key before removing the old one?**

   Answer: The old key is needed to decrypt all existing secrets that were encrypted with it. If you remove the old key first, those secrets become permanently unreadable until re-encrypted with the new key.

4. **How can you verify that a secret is actually encrypted in etcd?**

   Answer: Use etcdctl to read the secret from etcd and look for the prefix 'k8s:enc:aescbc:v1:' in the raw data, which indicates it was encrypted with the aescbc provider.

5. **What is the difference between a Role that grants 'get' on all secrets and one that uses 'resourceNames' to limit access to a single secret?**

   Answer: A Role without 'resourceNames' allows reading any secret in the namespace, violating least privilege. Using 'resourceNames' restricts access to only the named secret, which is the correct secure practice.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/cncf-cks/supply-chain-secrets-management