# TLS Certificates and API Server Security

> Chapter 3 of the Courseiva CNCF-CKS curriculum — https://courseiva.com/learn/cncf-cks/tls-certificates-api-server-security

**Official objective:** 1.3 — Secure Kubernetes API server with TLS certificates and authentication

## Introduction

Exam objective 1.3 asks you to secure the Kubernetes API Server with TLS certificates and authentication. This is the single most important security layer because the API server is the front door to your entire cluster, and TLS certificates are the cryptographic IDs that prove who is talking to whom. For the CKS exam, you must understand not just how to generate these certificates, but when to use them and how to protect the keys that underpin the cluster's trust.

## The Bouncer Checking IDs at a VIP Club Analogy

The Head of Security at a very exclusive VIP club has a critical job before anyone can enter.

First, the club has a master list of everyone allowed in, like a guest list. The bouncer doesn’t just let anyone in who says they are a member. They must check identification. This ID is a government-issued passport, which is like a TLS certificate. The passport has a photo, a name, and a hologram that proves it was issued by a trusted authority like the passport office, not printed at home. The bouncer verifies the hologram and the photo matches the person, which is the TLS handshake.

The club also has different areas: the main floor, the VIP balcony, and the backstage green room. The bouncer checks wristbands. A purple wristband means you can go to the main floor only. A silver wristband means you can go to the VIP balcony. A gold wristband with a special chip means you can go backstage. These wristbands are like the permissions in a TLS certificate for users or machines.

Now, the club manager doesn’t walk around handing out wristbands. The manager sits in a secure office and issues wristbands only after the bouncer sends a written request, signed by the guest, for a specific wristband. This secure office is the Certificate Authority (CA). If a guest loses their wristband, they must go back to the manager’s office to get a new one. They cannot just tell the bouncer their name and be let in.

Finally, the bouncer has a radio to call the manager if there is a problem. The manager never leaves the office. The bouncer trusts the manager’s decision because the manager is the source of truth. This trust relationship is the chain of trust in TLS, where the Kubernetes API Server trusts a CA, and the CA signs certificates for users and other components like kubelets.

## Core explanation

To understand TLS certificates and API Server security, you must first know what the API Server is. The Kubernetes API Server is the central control plane component that all other components and users talk to. Think of it as a government office window. Every request to do anything in Kubernetes must go through this window: to create a pod, delete a service, or check logs.

Now, how does the API Server know who is knocking? This is where TLS, or Transport Layer Security, comes in. TLS is a cryptographic protocol that provides three things: encryption, authentication, and integrity. Encryption scrambles the data so no one can read it in transit, like a sealed envelope. Authentication proves the identity of the person or machine on the other end, like checking a passport. Integrity ensures the message hasn't been tampered with, like a wax seal on a letter.

TLS uses digital certificates to perform authentication. A digital certificate is a small data file that binds a public key to an identity. A public key is like a lock that anyone can see. A private key is like a secret key to that lock, kept only by the owner. If you want to send a secure message to someone, you lock it with their public key. Only they can unlock it with their private key.

The certificate is signed by a Certificate Authority (CA). A CA is a trusted entity that verifies the identity of the certificate owner and then signs the certificate with its own private key. This is like a passport office verifying your photo and then stamping your passport. The API Server has its own certificate, often called the apiserver certificate. When you run the command 'kubectl get pods', your kubectl client requests a TLS connection to the API Server. The API Server presents its certificate. Your client checks if that certificate is signed by a CA that your client trusts. If yes, the communication is encrypted and the API Server's identity is verified.

But the API Server also needs to verify the identity of the client, which is you or a service account. This is called mutual TLS or mTLS. In mTLS, both sides present certificates. The client (kubectl) presents its own client certificate to the API Server. The API Server checks if this client certificate is signed by a trusted CA. If it is, the API Server knows exactly who the client is based on the Common Name (CN) field in the certificate. The CN is like the name on a passport.

Why does this matter for the CKS exam? You must know how to generate these certificates using tools like OpenSSL or kubeadm. You must know what the default filenames are, like 'ca.crt', 'apiserver.crt', 'apiserver.key', 'admin.crt', and 'admin.key'. You must know that these files live in the directory '/etc/kubernetes/pki/' by default on a kubeadm cluster. You must understand the certificate expiration and know how to renew them. If a certificate expires, all communication stops. No one can login.

The concept of certificate rotation is crucial. In production, certificates expire after a set period, often one year. You renew them before they expire. kubeadm can do this automatically for cluster components, but you must know how to do it manually if it fails. The exam will test your ability to check certificate expiry using the 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout' command to see the 'Not After' date.

Another key point is that the CKS exam tests your ability to audit the API Server configuration. You will need to check what flags are passed to the kube-apiserver binary. For example, you can pass the flag '--client-ca-file' to specify the CA certificate used to validate client certificates. You can pass '--tls-cert-file' and '--tls-private-key-file' to specify the server certificate and key. If these flags are missing or misconfigured, the API Server may fall back to insecure modes, which is a massive security risk.

The exam also tests authentication mechanisms layered on top of TLS. TLS gives you a secure channel and a verified identity. That identity must be mapped to a user or group in Kubernetes. This is done through the Common Name (CN) and Organization (O) fields in the client certificate. Kubernetes maps the CN to a username and the O to a group. These are used with Role-Based Access Control (RBAC) to decide what that user can do.

Finally, you must understand that not all components in a cluster use certificates. For example, kubelets, the agents on worker nodes, also have their own certificates. They use TLS to communicate with the API Server. The exam will ask you to troubleshoot why a kubelet cannot join the cluster because its certificate is invalid or expired. You must know how to regenerate the kubelet client certificate and how to approve the Certificate Signing Request (CSR) in Kubernetes.

In summary, TLS certificates are the foundation of trust in a Kubernetes cluster. Without them, anyone could impersonate the API Server or a user, and all data would be sent in plaintext. The CKS exam expects you to be able to generate, inspect, configure, and troubleshoot these certificates with the same ease as a mechanic knows how to check oil in a car.

## Real-world context

An IT professional, let's call them Sam, is the platform engineer for a mid-sized fintech company running Kubernetes in production. The company handles sensitive financial data, so security is paramount. Sam's first task every Monday morning is to check the certificate expiry dates for all cluster components.

Sam connects to the control plane node via SSH and runs 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout | grep -i after'. The output shows the certificate expires in 14 days. Sam notes this and schedules a certificate renewal window for next Tuesday at 2 AM, during the lowest traffic period.

During the renewal window, Sam performs a static pod rollout of the API Server. Sam first backs up the existing certificates by copying the entire /etc/kubernetes/pki/ directory to a secure backup location. Then, Sam uses kubeadm to renew the certificates. The command is 'kubeadm certs renew apiserver'. This generates a new certificate signed by the cluster's CA, which is still valid. Sam verifies the new certificate with the same openssl command.

Now, Sam's colleague, a developer named Alex, is trying to access the cluster from a new laptop. Alex needs a client certificate to authenticate to the API Server. Sam needs to generate one. Sam generates a private key for Alex using 'openssl genrsa -out alex.key 2048'. Then, Sam creates a Certificate Signing Request (CSR) using 'openssl req -new -key alex.key -out alex.csr -subj "/CN=alex/O=developers"'. Note: the CN is 'alex', which will be the username, and the Organization is 'developers', which will become the Kubernetes group.

Sam submits this CSR to Kubernetes as a CertificateSigningRequest object. Sam then approves it using 'kubectl certificate approve alex-csr'. The Kubernetes controller manager, which includes the CSR approver, signs the CSR with the cluster's CA key and creates a certificate. Sam retrieves this certificate as 'alex.crt' from the Kubernetes resource. Sam then configures Alex's kubeconfig file to include the new 'client-certificate' and 'client-key' fields. Alex can now run 'kubectl get pods'. But Alex gets a Forbidden error. This is expected because Alex has no RBAC permissions yet. Sam must now create a Role and a RoleBinding for Alex's group 'developers' to allow read access to pods.

Another common task is troubleshooting a broken cluster. A junior admin reports that no one can connect to the cluster. Sam first checks if the API Server is running. It is. Sam then checks the API Server logs with 'kubectl logs -n kube-system kube-apiserver-control-plane'. The logs show an error: 'x509: certificate has expired or is not yet valid'. Sam immediately checks the apiserver certificate and finds it expired yesterday. Sam uses 'kubeadm certs renew apiserver' and restarts the static pod by moving the manifest file. The cluster is back online.

Sam also sets up automation. Sam writes a cron job that runs weekly to check certificate expiry and sends a Slack alert if any certificate is within 30 days of expiry. This prevents future outages. Sam documents the entire certificate lifecycle: how to generate, renew, approve CSRs, and revoke them if a key is compromised.

Finally, Sam audits the API Server configuration. Sam reads the static pod manifest at '/etc/kubernetes/manifests/kube-apiserver.yaml'. Sam checks the command arguments. Sam looks for '--client-ca-file'. It is set to '/etc/kubernetes/pki/ca.crt'. Sam checks '--tls-cert-file' and '--tls-private-key-file'. They point to the correct files. Sam also checks if '--anonymous-auth' is set to 'false'. If it were 'true', any unauthenticated user could access the cluster, which is a security risk. Sam ensures '--authorization-mode' includes 'Node,RBAC', not 'AlwaysAllow'. This all ensures the TLS and authentication layer is as secure as possible.

## Exam focus

The CKS exam tests TLS certificates and API Server security in several specific ways. You must understand these questions and traps.

First, the exam loves to test your ability to read and interpret certificate information from the command line. You will be given a scenario where a kubelet cannot join the cluster. You must SSH into the node and run 'openssl x509 -in /path/to/cert.crt -text'. You must identify if the certificate is expired, if the CN matches the hostname, or if it was signed by the wrong CA.

Second, they test the certificate generation workflow. A typical question will ask you to create a new client certificate for a user named 'jack'. You must know the steps:

- Generate a private key with openssl genrsa.
- Create a CSR with a specific CN and O.
- Create a CertificateSigningRequest object in Kubernetes.
- Approve the CSR.
- Extract the certificate and update the kubeconfig.

Third, they test API Server configuration. You will see a kube-apiserver manifest file with missing or wrong arguments. They will ask you to fix it. For example, they might have '--client-ca-file' pointing to a non-existent file. You must correct the path.

Fourth, they test certificate rotation. Questions will ask you what to do when a certificate expires. The correct answer is to renew it using kubeadm certs renew or manually by generating a new CSR and signing it with the CA key. They will also ask about the '--csr-approver' flag and the 'CertificateSigningRequest' API resource.

Fifth, they test the concept of the 'Certificate Authorities (CA)' and the 'Chain of Trust'. They might ask you to explain what happens if the CA private key is compromised. The answer is that all certificates signed by that CA are now untrusted and must be regenerated. They will ask you to identify which files must be kept secure. The answer is 'ca.key' the CA private key, which must be stored in a secure location with restricted permissions.

Key trap patterns to watch out for:

- They will give you a certificate that looks valid but has the wrong 'CN' field. For example, a kubelet certificate must have the node's hostname as the CN. If the CN is wrong, the kubelet will fail to authenticate.
- They will set '--anonymous-auth=true' in the API Server manifest. This allows unauthenticated access. You must set it to 'false'.
- They will set '--authorization-mode=AlwaysAllow'. This bypasses all RBAC checks. You must change it to 'Node,RBAC'.
- They will put certificates in the wrong directory or give them incorrect permissions. For example, 'apiserver.key' must be readable only by root (0600), not world-readable.
- They will ask about the default certificate paths. Memorise '/etc/kubernetes/pki/' for control plane components and '/var/lib/kubelet/pki/' for kubelet certificates.

Key definitions to memorise:

- TLS: Transport Layer Security, a protocol for secure communication.
- CA: Certificate Authority, the trusted entity that signs certificates.
- CSR: Certificate Signing Request, a request sent to a CA to get a certificate signed.
- CN: Common Name, the primary identifier in a certificate, mapped to a Kubernetes username.
- O: Organisation, the field in a certificate mapped to a Kubernetes group.
- kubelet: the agent running on each worker node that communicates with the API Server.
- kubeadm: a tool to bootstrap a Kubernetes cluster, includes commands to manage certificates.

They also love to ask about the 'kubeconfig' file. You must know how to add a new cluster, user, and context, and how to set the 'certificate-authority-data', 'client-certificate-data', and 'client-key-data' fields. The exam environment will not have a browser, so you must be able to edit YAML files from the command line using vim or nano. You must know how to base64 decode certificate data for inspection.

Finally, the exam will test the concept of 'mTLS' or mutual TLS. They might ask you why the kube-apiserver requires a client certificate. The answer is for authentication of the client. They might ask what happens if you do not provide a client certificate. The answer is the request is either denied or treated as anonymous, depending on the '--anonymous-auth' flag.

## Step by step

1. **Generate a Private Key** — The first step for any certificate is to create a private key. This is a long, random string of numbers that only the owner should know. You generate it with a tool like OpenSSL. For example, 'openssl genrsa -out myuser.key 2048' creates a 2048-bit RSA key. The number 2048 represents the key length in bits. A longer key is more secure but slower. This key is the secret that proves your identity. You never share this key.
2. **Create a Certificate Signing Request (CSR)** — You take your private key and use it to create a Certificate Signing Request. This is a formal request to a Certificate Authority (CA) asking them to issue you a certificate. The CSR contains your public key (automatically derived from the private key) and your identity details, such as the Common Name (CN) which is your username. For example, 'openssl req -new -key myuser.key -out myuser.csr -subj "/CN=john/O=developers"'. The 'O' stands for Organisation and maps to a Kubernetes group.
3. **Submit the CSR to the Kubernetes API Server** — You submit the CSR as a Kubernetes resource. You create a YAML file that includes the base64-encoded CSR from the previous step. You use 'kubectl create -f csr.yaml'. The Kubernetes API Server receives this and stores it. This makes the CSR visible to an administrator who can approve it. The CSR resource includes the raw base64 request, the signer name (e.g., kubernetes.io/kube-apiserver-client), and optional groups.
4. **Approve the CSR** — An administrator (or an automated process) approves the CSR using 'kubectl certificate approve <csr-name>'. The Kubernetes controller manager has a built-in signing controller. When it sees an approved CSR, it signs the CSR using the cluster's CA private key (ca.key). This creates a digital signature that guarantees the certificate's authenticity. The CSS's status changes from 'Pending' to 'Approved' then 'Issued'.
5. **Retrieve the Signed Certificate and Update kubeconfig** — Once the CSR is issued, you retrieve the signed certificate from the CSR resource using 'kubectl get csr <csr-name> -o jsonpath="{.status.certificate}"'. This outputs a base64-encoded string. You decode it with 'base64 --decode' and save it as a .crt file. Then, you update your kubeconfig file to use this certificate and your private key. You set 'client-certificate: myuser.crt' and 'client-key: myuser.key' in the user section of your kubeconfig. Now, when you run kubectl, it presents this certificate to the API Server for authentication.

## Comparisons

### Server Certificate (apiserver.crt) vs Client Certificate (admin.crt)

**Server Certificate (apiserver.crt):**
- Presented by the API Server to the client during TLS handshake
- CN must match the DNS name or IP of the API Server
- Used to prove the server's identity to the client

**Client Certificate (admin.crt):**
- Presented by the client (e.g., kubectl) to the API Server
- CN must match the username of the client
- Used to prove the client's identity to the server

### kubeadm certs renew vs Manual openssl renewal

**kubeadm certs renew:**
- Automated process built into kubeadm
- Renewal does not require re-creating the cluster
- Uses the existing CA key on the control plane

**Manual openssl renewal:**
- Requires generating a new CSR, signing it with the CA key manually
- More control over certificate parameters (e.g., extended key usage)
- Needed if the cluster was not built with kubeadm

### Certificate Signing Request (CSR) Object in Kubernetes vs External CA (e.g., Let's Encrypt)

**Certificate Signing Request (CSR) Object in Kubernetes:**
- Managed entirely within the Kubernetes API
- Signed by the cluster's internal CA
- Approved via kubectl certificate approve

**External CA (e.g., Let's Encrypt):**
- Managed by an external service outside Kubernetes
- Signed by a public CA trusted by browsers
- Requires manual setup of the external CA and integration

### Anonymous Authentication (--anonymous-auth=true) vs Client Certificate Authentication

**Anonymous Authentication (--anonymous-auth=true):**
- Allows any unauthenticated request to proceed
- Mapped to the system:anonymous user
- Extremely insecure for production

**Client Certificate Authentication:**
- Requires a valid signed certificate for every request
- Identifies the user via the CN field
- Provides strong, verifiable identity

## Diagram

_A flowchart showing how a root CA certificate is used to sign server and client certificates, which are then presented during TLS handshake for mutual trust verification._

```mermaid
flowchart TD
    A[CA Private Key (kept secure)] --> B[CA Certificate (ca.crt)]
    B --> C[API Server Certificate (apiserver.crt)]
    B --> D[Kubelet Certificate (kubelet.crt)]
    B --> E[User Certificate (user.crt)]
    C --> F[API Server presents certificate to kubectl]
    D --> G[Kubelet presents certificate to API Server]
    E --> H[User presents certificate to API Server]
    F --> I[Client trusts CA?]
    G --> J[API Server trusts CA?]
    H --> J
    I -->|Trusted| K[Encrypted session established]
    I -->|Not Trusted| L[Connection rejected]
    J -->|Trusted| K
    J -->|Not Trusted| L
```

## Common misconceptions

- **Misconception:** If the API Server's TLS certificate is self-signed, it is just as secure as one signed by a trusted CA. **Reality:** A self-signed certificate means the API Server is its own CA. Every client must be configured to trust that specific self-signed CA, which is less scalable and harder to manage. A certificate signed by a well-known or organisational CA is more secure because the trust chain is standard. (Beginners often assume a certificate is a certificate and don't understand the trust chain. They see a padlock and think it is secure, not realising that the padlock only works if the client trusts the key maker.)
- **Misconception:** The CA private key (ca.key) must be kept on every control plane node for the cluster to function. **Reality:** The CA private key is a highly sensitive secret. It is only needed when signing new certificates. It should be stored offline or in a dedicated secure location, not on every control plane node. The cluster can run with just the CA public certificate (ca.crt). (This error comes from the fact that kubeadm by default places the ca.key in /etc/kubernetes/pki/. Novices assume it must be there permanently, not realising it can and should be moved to a secure vault after initial generation.)
- **Misconception:** As long as the certificate is not expired, the connection is always secure. **Reality:** A non-expired certificate does not guarantee security. The certificate could be revoked (e.g., if the private key was compromised), the CN could be wrong for the expected identity, or the certificate could be signed by a CA that is no longer trusted. The check must include validity, purpose, and chain of trust. (People naturally focus on the one check they understand: expiry date. They do not think about revocation lists, certificate purpose extensions, or CA trust changes.)
- **Misconception:** When you create a CSR in Kubernetes, you must manually sign it with the CA private key using the openssl command. **Reality:** You submit the CSR as a Kubernetes API resource. The Kubernetes controller manager has a built-in CSR approving and signing controller. You approve the CSR using 'kubectl certificate approve', and the controller signs it automatically if configured correctly. Manual openssl signing is an older or alternative method, not the standard for cluster-internal certificates. (Many tutorials show the openssl command to sign a CSR to explain the theory. Beginners then think that is the only way to do it in Kubernetes, missing the automated CSR workflow.)
- **Misconception:** TLS only secures the API Server port (6443). The internal cluster communication (e.g., between kubelet and API Server) does not need TLS because it is within the private network. **Reality:** Kubernetes requires all communication between components to be encrypted using TLS, even within the cluster. Internal traffic can be intercepted. kubelet-to-API-Server, API-Server-to-etcd, and scheduler-to-API-Server all use TLS. (People come from a background where internal networks are considered safe, such as in a small office. They do not realise that in cloud environments, the network is shared and logs may be examined, so encryption is mandatory.)

## Key takeaways

- The Kubernetes API Server uses TLS certificates for both server authentication to clients and client authentication via mutual TLS.
- The certificate authority (CA) private key (ca.key) is the most sensitive secret in the cluster and must be stored offline or in a secure secret store after initial certificate generation.
- Every kubelet on a worker node has its own client certificate used to authenticate to the API Server; the CN of the kubelet certificate must match the node's hostname.
- To renew an expired API Server certificate on a kubeadm cluster, use the command 'kubeadm certs renew apiserver' and then restart the static pod.
- The 'kubectl certificate approve' command is used to approve Certificate Signing Requests (CSRs) for new client certificates within the cluster.
- Inspect certificate details from the command line with 'openssl x509 -in /path/to/cert.crt -text -noout' to check its validity, CN, and issuer.
- The API Server flags '--client-ca-file', '--tls-cert-file', and '--tls-private-key-file' must be correctly configured in the static pod manifest for TLS to work.
- Anonymous authentication to the API Server is controlled by the '--anonymous-auth' flag and should typically be set to 'false' in production.
- Certificate revocation is not automatic; you must revoke a certificate by updating a Certificate Revocation List or by removing the trusted CA if the key is compromised.
- A Certificate Signing Request in Kubernetes contains a 'spec.request' field with a base64 encoded CSR, and the 'spec.groups' field defines the user's group membership.

## FAQ

**What is a TLS certificate in simple terms?**

A TLS certificate is a digital ID card. It contains the name of the person or machine it belongs to, their public key (like a lock), the name of the authority that issued it, and an expiry date. It proves that you are who you say you are when connecting to a server.

**How do I check if my Kubernetes API Server certificate is expired?**

SSH into the control plane node and run 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout | grep -i after'. This will show the 'Not After' date, which is the expiry date. If that date is in the past, the certificate is expired.

**What happens if the CA key is compromised?**

If the CA private key is stolen, anyone can create fake certificates that your cluster will trust. You must generate a new CA key and reissue all certificates (for the API Server, kubelets, users, etc.) immediately. The old CA must be rotated out completely.

**Do I need to restart the API Server after renewing its certificate?**

Yes, the API Server must be restarted to pick up the new certificate. On a kubeadm cluster, this is done by moving or removing the static pod manifest file (/etc/kubernetes/manifests/kube-apiserver.yaml) and then replacing it after a few seconds. The kubelet then recreates the pod with the new certificate.

**What is the difference between 'apiserver.crt' and 'apiserver.key'?**

'apiserver.crt' is the public certificate that the API Server presents to clients. It can be seen by anyone. 'apiserver.key' is the private key that corresponds to the certificate. Only the API Server should have access to this key. It is used to decrypt incoming connections.

**Can a user authenticate to the API Server without a client certificate?**

Yes, if the API Server is configured with an '--anonymous-auth' flag set to 'true'. In that case, any request without a certificate is treated as the 'system:anonymous' user. This is extremely insecure and should be disabled in production. Other authentication methods like bearer tokens or OIDC can also be used instead of certificates.

## Check your understanding

1. **What command do you use to check the expiry date of the API server certificate on a control plane node?**

   Answer: Open a shell on the node and run 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout | grep -i after'.

2. **You have just generated a new private key and CSR for a user named 'sarah'. What is the next step to get her a signed certificate using Kubernetes?**

   Answer: Create a CertificateSigningRequest Kubernetes resource, submit it with kubectl, and then approve it with 'kubectl certificate approve <csr-name>'.

3. **A kubelet on worker node 'node-1' cannot join the cluster. You suspect a TLS issue. Which two fields on the kubelet client certificate must you check?**

   Answer: You must check the Common Name (CN) field is set to 'node-1' and the certificate is not expired.

4. **In the API server static pod manifest, which flag specifies the file containing the Certificate Authority to validate client certificates?**

   Answer: The flag is '--client-ca-file' and it should point to the CA certificate file (e.g., /etc/kubernetes/pki/ca.crt).

5. **If a user's private key is compromised, what must you do to prevent the attacker from using it?**

   Answer: You must revoke the user's certificate by either deleting the corresponding CSR or by rotating the CA if the compromise was widespread, and then generate a new key and certificate for the user.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/cncf-cks/tls-certificates-api-server-security