Troubleshooting Cluster Security

Let's start with the foundation: Kubernetes is a system for running containerised applications. To interact with it, you need to authenticate (prove who you are) and then be authorised (be allowed to do what you're trying to do). Two major security components handle this: RBAC (Role-Based Access Control) and certificates.

What is RBAC?

RBAC is a method of restricting system access to authorised users. Think of it like a hotel key card system. A guest might have a key that opens their room door but not the staff-only maintenance closet. A cleaner has a key that opens the maintenance closet but not guest rooms. In Kubernetes, roles are like sets of permissions (e.g., 'can list pods', 'can create deployments'), and role bindings attach those roles to users, groups, or service accounts.

There are two types of roles: Role and ClusterRole. A Role applies permissions within a single namespace (a virtual cluster within the cluster). A ClusterRole applies permissions across the entire cluster (e.g., ability to view nodes, which exist globally). Similarly, RoleBinding binds a Role to a subject within a namespace, and ClusterRoleBinding binds a ClusterRole cluster-wide.

Why RBAC misconfigurations cause trouble

A very common mistake is giving too few permissions (e.g., a developer cannot view logs of their own pod) or too many (e.g., a CI/CD pipeline has cluster-admin access when it only needs to deploy to one namespace). Another is forgetting to create a RoleBinding after creating a Role — the role exists on paper but no one is actually assigned to it. When a user or service account tries an action they aren't permitted to do, Kubernetes returns a 'Forbidden' error. Troubleshooting RBAC means checking: what role exists? What subjects (users, groups, service accounts) are bound to it? Does the role contain the needed verbs ('get', 'list', 'create') on the right resources ('pods', 'deployments', 'services')?

What are certificates?

Certificates are digital ID cards used to prove identity and establish encrypted communication. In Kubernetes, the API server (the brain of the cluster) uses a certificate to prove its identity to clients. Every component (kubelet, scheduler, controller-manager) also uses client certificates to prove their identity to the API server. If any of these certificates are expired, misconfigured, or signed by an untrusted Certificate Authority (CA — the entity that issues certificates), the connection fails.

How certificate problems manifest

A classic sign: 'certificate has expired or is not yet valid'. This happens when the system clock is wrong (certificates have a validity window), or the certificate was not renewed before expiry. Another sign: 'x509: certificate signed by unknown authority'. This means the client doesn't trust the CA that signed the server's certificate — either the CA certificate is missing from the client's trust store, or the server certificate was self-signed and the client doesn't have the self-signed root.

Where certificates live in Kubernetes

The main certificates are stored on the control plane node (the master node that runs the API server). The kubeconfig file (a configuration file used to connect to the cluster) contains the client certificate and key, and the CA certificate used to verify the server. The kubelet (an agent running on each worker node) also has its own certificate.

How they interact in practice

An RBAC configuration often depends on the identity proven by a certificate. For example, a kubelet presents a certificate that identifies it as 'system:node:worker-1'. The API server checks that the certificate is valid, extracts the username, and then checks if that username (or its groups) has RBAC permissions to perform the requested action (e.g., report node status). If the certificate is invalid, the authentication step fails before RBAC is even checked. If the certificate is valid but RBAC denies the action, the user gets a 'Forbidden' error.

Troubleshooting cluster security, therefore, involves a two-step process: first verify that the identity (certificate) is trusted and not expired, then verify that the identity (username/group) has the correct RBAC permissions. Both layers must work for the cluster to function correctly.

A medium-sized e-commerce company runs its production and staging environments on the same Kubernetes cluster, separated by namespaces (production-ns and staging-ns). A new DevOps engineer, Priya, is tasked with setting up a CI/CD pipeline that deploys application updates to the staging namespace only.

Priya creates a service account (a non-human identity used by tools) called 'cicd-bot' in the staging namespace. She then creates a Role called 'deployer' that grants permissions to 'get', 'list', 'create', 'update', and 'delete' on deployments, services, and pods. She creates a RoleBinding binding 'cicd-bot' to the 'deployer' role. Everything seems fine.

However, when the pipeline runs, it fails with a 'Forbidden' error when trying to list pods. Priya checks the YAML files. She sees that the Role correctly lists 'pods' in the resources field, but she forgot to include the verb 'list' — she included 'get' but not 'list'. The pipeline needs to 'list' pods to check the status. She updates the Role to include 'list', applies it, and the pipeline succeeds. This is a classic RBAC troubleshooting scenario.

Later, the company's IT department rotates the cluster certificates as part of a security policy. They generate new certificates for the API server and the kubelets, but the kubeconfig files on the developer laptops still contain the old CA certificate. When developers try to use 'kubectl' commands, they get 'x509: certificate signed by unknown authority'. The fix is to update the kubeconfig files with the new CA certificate. The IT team pushes a script that does this automatically, and the connection is restored.

In another real-world case, a node goes offline because the kubelet certificate expires. The kubelet has a built-in mechanism to auto-renew its certificate, but this relies on the cluster being configured with a controller that handles certificate signing requests (CSRs). If that controller is not running or misconfigured, the kubelet's certificate expires, and the node loses connection to the API server. The fix is to check the CSR state, approve any pending CSRs, and ensure the kubelet controller manager is healthy.

Tools used: - 'kubectl auth can-i' — quickly check whether a user or service account can perform a specific action. - 'kubectl describe role/rolebinding' — inspect the exact permissions and bindings. - 'openssl x509 -in certificate.crt -text -noout' — inspect certificate details (issuer, expiry date). - 'kubectl get csr' — view pending certificate signing requests.