Troubleshooting Storage Persistence

In Kubernetes, containers are temporary. When a container restarts, any data written inside its own filesystem disappears – unless that data is stored in a volume. A volume is like a USB drive you plug into a container. The data on the USB drive stays even if the computer reboots.

Kubernetes manages storage through three main objects: PersistentVolume (PV), PersistentVolumeClaim (PVC), and StorageClass. A PersistentVolume is a piece of storage in the cluster – think of it as a 100 GB hard drive that an administrator has set up. A PersistentVolumeClaim is a request for storage by a user – like saying “I need 50 GB of that hard drive”. The system matches the claim to an available volume that meets the request’s size and access modes (e.g., ReadWriteOnce – only one pod can write at a time; ReadOnlyMany – many pods can read; ReadWriteMany – many pods can write). TROUBLESHOOTING comes into play when the PVC stays in a Pending state. This happens when no PV matches the PVC’s requirements. Common reasons: the requested size is larger than any PV; the access mode does not match; the StorageClass specified in the PVC does not exist or is misconfigured.

StorageClass is the third piece. It tells Kubernetes how to provision a new PV dynamically. For example, if you define a StorageClass named “fast” that references a specific cloud provider’s SSD provisioner, then when a PVC asks for “fast” storage, Kubernetes automatically creates a PV backed by an SSD. Without a StorageClass, you must create PVs manually (static provisioning). Dynamic provisioning is more common in real clusters.

When troubleshooting, the first command is usually kubectl get pvc to see if the claim is Bound or Pending. Then kubectl describe pvc <name> shows events and details. Common error: “Failed to provision volume with StorageClass” – this means the provisioner (e.g., a cloud driver) is not installed correctly or the cloud credentials are wrong. Another error: “waiting for a volume to be created, either by external provisioner or by manual creation” – this means dynamic provisioning is failing or there is no matching PV.

A pod that uses a PVC must have the volume mounted correctly. If the pod starts but cannot write to the volume, check the access modes. If the pod crashes with “permission denied”, the filesystem permissions on the volume might be wrong. Using a StatefulSet (a controller that gives each pod a unique stable identity and persistent storage) often uses a VolumeClaimTemplate – if one pod fails, its replacement gets the same PVC, preserving data.

For raw block volumes (a volume presented as a block device without a filesystem), the pod must be configured to use the volume as a block device, not as a mount path. Troubleshooting those requires checking device paths with lsblk inside the node.

Another critical point: reclaim policy. PVs have three reclaim policies: Retain (volume kept after PVC deletion, but needs manual cleanup), Recycle (deprecated, wipes data), and Delete (volume deleted along with PVC). If a volume is lost after deleting a PVC, check the reclaim policy – a Delete policy will destroy the underlying storage.

In summary, the mental model is: PV = storage resource; PVC = request ticket; StorageClass = how to make new storage. Troubleshooting is matching these three pieces correctly.

Imagine you work as a junior platform engineer at a company running a customer-facing e-commerce site on Kubernetes. The site uses a PostgreSQL database running in a pod. The database stores its data on a PersistentVolume backed by a cloud SSD. One Monday morning, you get an alert: the database pod is CrashLoopBackOff. You check the pod logs – they say “cannot write to directory /var/lib/postgresql/data”. You start troubleshooting.

First, you run kubectl get pvc and see the PVC is Bound. Good. Then kubectl describe pvc postgres-pvc – the volume name, access modes, and status all look fine. Next, you check the underlying PV with kubectl describe pv <pv-name>. You notice the PV’s capacity is 100 Gi, but the pod only requested 50 Gi – that’s fine. Then you see the access mode is ReadWriteOnce, which matches the pod’s single replica. Still no obvious issue.

You SSH into the worker node where the pod is running. You run ls -la /var/lib/kubelet/pods/<pod-uid>/volumes/ to find the volume mount path. Using df -h, you see the volume is mounted. But sudo -u postgres touch /path/to/test.txt fails with “Permission denied”. You check the ownership: ls -n /path/ shows uid 999 owns the directory, but the pod runs as uid 70 (the Postgres image default). That’s the bug! The PV retained permissions from a previous backup restore. You fix it by running a one-off pod that chowns the directory to the correct uid.

Another real scenario: a developer tells you their new application cannot start because the PVC is never bound. You check the PVC and see the status is Pending. Using kubectl describe pvc, the events say “no persistent volumes available for this claim”. You look at existing PVs – all 10 Gi, but the claim asks for 200 Gi. You either create a new PV of 200 Gi or resize the existing PV (if your storage backend supports expansion). You also check if the PVC specifies a StorageClass that does not exist – the provisioner name might be misspelled. You correct the StorageClass name in the PVC definition and reapply.

A third scenario: a batch job finishes but the output data is gone. You check the pod’s volumeMounts – it used an emptyDir volume, which is ephemeral. The solution is to change the volume to a PVC that persists after the pod completes. You advise the team to use a PVC with a mount path that writes to a shared filesystem like NFS.

These real events happen every day in production clusters. The tools are: kubectl describe, kubectl get events, kubectl logs, and direct node inspection via SSH or kubectl debug. The mindset is: storage is a separate resource; failures often come from mismatched sizes, access modes, permissions, or missing provisioners.