Storage practice questions

Use a network filesystem (NFS) server running as a single pod with a PersistentVolume backed by a regional Persistent Disk

Why it fails: Running an NFS server as a single pod introduces a critical single point of failure for the entire storage solution. While the underlying regional Persistent Disk offers data redundancy, the NFS server pod itself is not highly available; if it fails, all applications relying on it lose access to storage until it recovers. Furthermore, network file systems inherently add latency compared to block storage directly attached to a node.

Create a StorageClass with WaitForFirstConsumer binding and volumeBindingMode: WaitForFirstConsumer

Why it fails: Configuring a StorageClass with volumeBindingMode: WaitForFirstConsumer ensures that a PersistentVolume is provisioned in the same zone where the consuming pod is scheduled. While this helps with zone affinity and prevents premature volume binding in an unsuitable zone, it does not inherently provide zone resilience for the data itself or guarantee high performance. The underlying storage type, not the binding mode, determines data replication and performance characteristics.

Deploy a StatefulSet with a local SSD on each node and use a DaemonSet to manage replication

Why it fails: Utilizing local SSDs on each node with a DaemonSet for replication is problematic because local SSDs are ephemeral and tied directly to a specific node; if that node fails, the data is lost unless the custom replication mechanism has successfully synchronized it. Managing data replication across nodes and zones with a DaemonSet is extremely complex, error-prone, and typically less reliable or performant than platform-managed regional storage solutions. This approach lacks inherent zone resilience and adds significant operational overhead.

A PVC can be bound to a PV that has been released and is pending reclamation

Why it fails: When a PVC is deleted, its bound PV transitions to the Released phase, meaning it still contains data from the previous claim. To protect data integrity, Kubernetes prevents new PVCs from binding to a Released volume until the administrator manually reclaims it or the configured ReclaimPolicy (such as Recycle or Delete) completes its execution.

A PV can be bound to multiple PVCs simultaneously if it has ReadWriteMany access mode

Why it fails: Although a ReadWriteMany access mode allows multiple Pods to mount the same volume concurrently, the binding relationship between a PV and a PVC is strictly one-to-one. Once a PersistentVolume is bound to a specific PersistentVolumeClaim, it is marked as Bound and becomes unavailable to satisfy any other claims, regardless of its underlying access capabilities.

A PV can only be bound to a PVC that requests exactly the same amount of storage

Why it fails: Incorrect. The PersistentVolume must provide at least as much storage as requested by the PVC, but the amounts do not need to be exactly equal. The PVC can request less storage than the PV provides.

AWS Elastic Block Store (EBS) directly

Why it fails: Kubernetes pods cannot directly reference or provision cloud provider-specific storage like an AWS EBS volume using its raw ID within a pod specification. Kubernetes abstracts underlying infrastructure details through its PersistentVolume and PersistentVolumeClaim API objects. To utilize EBS, it must first be provisioned as a PersistentVolume or dynamically provisioned via a CSI driver and then requested by a pod through a PersistentVolumeClaim.

Docker volume

Why it fails: While Kubernetes uses container runtimes like Docker, it does not directly expose or utilize Docker's internal volume management mechanisms within its pod specifications. Kubernetes employs its own robust volume API, which includes types like emptyDir, hostPath, and PersistentVolumeClaim, to manage storage for pods. Therefore, attempting to specify a 'Docker volume' directly in a Kubernetes pod definition is not a valid or recognized volume type.

Increase the memory limit of the CSI controller pod to reduce OOM risk

Why it fails: The 'context deadline exceeded' error specifically indicates a timeout issue, typically related to network latency, API server responsiveness, or gRPC communication failures, rather than memory exhaustion. Increasing the memory limit of the CSI controller pod would not resolve these timing-related problems, as Out-Of-Memory (OOM) errors manifest as pod restarts due to memory pressure, not as connection timeouts. Therefore, this action misidentifies the root cause.

Restart the CSI controller pod to clear its memory and reset connections

Why it fails: While restarting the CSI controller pod might temporarily alleviate issues by clearing transient states or re-establishing gRPC connections, it does not address the fundamental architectural limitations causing the 'context deadline exceeded' errors. The underlying problem of insufficient capacity for concurrent operations or overly long timeouts will persist, leading to a recurrence of the same issues under sustained multi-tenant load. This is a symptomatic fix, not a resolution to the root cause.

Add a network policy to isolate tenant namespaces to reduce load on the CSI driver

Why it fails: Network policies primarily control ingress and egress IP traffic between pods and network endpoints within the Kubernetes cluster, acting as a firewall. Storage provisioning requests, however, are typically handled via API calls to the Kubernetes API server, which then interacts with the CSI driver's controller service, often using gRPC. Implementing network policies would not reduce the volume of these API-driven storage requests or impact the CSI driver's internal processing capacity or gRPC connection pool issues.

Create a PersistentVolumeClaim with access mode ReadOnlyMany and mount it in each pod.

Why it fails: ReadOnlyMany (ROX) access mode allows multiple nodes to mount a volume concurrently, but only in read-only mode. Since a stateful application typically requires the ability to write and persist its data, using ROX would prevent the application from functioning correctly by denying necessary write operations. This access mode is suitable for distributing static content, not for applications needing to modify shared state.

Create a PersistentVolumeClaim with access mode ReadWriteOnce and mount it in each pod.

Why it fails: Creating a PersistentVolumeClaim with ReadWriteOnce (RWO) access mode is incorrect because RWO permits the volume to be mounted as read-write by only a single node, preventing multiple pods on different nodes from sharing it concurrently. This directly contradicts the requirement for multiple pods to share the same volume, implying potential access from various nodes. However, RWO is a common and appropriate choice for stateful applications where a single pod on a single node requires exclusive read-write access to its persistent storage, such as a database instance.

Use a hostPath volume configured on each node where pods are scheduled.

Why it fails: Using a hostPath volume is unsuitable for a stateful application requiring shared access across multiple pods, especially if those pods are scheduled on different nodes. A hostPath volume binds data directly to a specific node's filesystem, making it non-portable and inaccessible to pods on other nodes. Furthermore, hostPath volumes lack the robust lifecycle management, dynamic provisioning, and abstraction provided by PersistentVolumes, making them generally inappropriate for production-grade shared storage.

The PV's reclaim policy is 'Retain' and the PV must be manually recycled.

Why it fails: When a PersistentVolumeClaim (PVC) is deleted, if its associated PersistentVolume (PV) has a 'Retain' reclaim policy, the PV transitions to a 'Released' state. In this state, the PV is no longer bound to a PVC but still exists, preserving its data. An administrator must manually delete the PV or change its reclaim policy to 'Recycle' or 'Delete' to make it available for new claims, effectively 'recycling' it.

The PV has a faulty disk and needs replacement.

Why it fails: A faulty disk would typically manifest as the PersistentVolume (PV) entering a 'Failed' or 'Error' state, or the pod experiencing I/O errors, rather than a 'Released' status. The 'Released' status specifically indicates that the PV was previously bound to a PersistentVolumeClaim (PVC) that has since been deleted, not a hardware malfunction. There is no information in the problem statement to suggest a physical disk issue.

The StorageClass has an incorrect provisioner.

Why it fails: An incorrect provisioner in a StorageClass would prevent the initial dynamic provisioning and binding of a PersistentVolume (PV) to a PersistentVolumeClaim (PVC) from ever succeeding. Since the problem states a pod *was using* a PV, it implies that the provisioning and binding process completed successfully at some point. Therefore, the issue is not with the StorageClass's provisioner but rather with the PV's state after its usage.

The PV is not bound to any PVC.

Why it fails: While a PersistentVolume (PV) in a 'Released' state is indeed no longer actively bound to a PersistentVolumeClaim (PVC), this statement is an incomplete diagnosis. The 'Released' status specifically indicates that the PV *was* previously bound, and its associated PVC was deleted, triggering the reclaim policy. The critical detail is *why* it's no longer bound, which points to the reclaim policy, not just a general unbound state.

CSI only works with dynamic provisioning.

Why it fails: Although CSI is most often used with dynamic provisioning through StorageClasses, the interface fully supports static provisioning. An administrator can pre-created a volume on the storage backend and manually define a PersistentVolume using `csi.volumeAttributes` and the `volumeHandle` that identifies the existing backend volume, and the CSI `ControllerPublishVolume` and `NodePublishVolume` RPCs will still attach and mount it when a pod schedules. Thus, CSI does not mandate dynamic provisioning; it simply provides the same standard RPCs for both pre-provisioned and dynamically created volumes.

CSI drivers must be compiled into the Kubernetes binary.

Why it fails: CSI drivers are intentionally out-of-tree and are not compiled into any Kubernetes binary. Instead, they run as separate pod containers (usually a DaemonSet on every node plus a controller sidecar) that expose a gRPC endpoint via a shared hostPath socket. The kubelet and the external-provisioner act as CSI clients that speak over that socket, making the driver a completely pluggable component. This architecture is the foundational reason CSI exists: to decouple storage plugins from the Kubernetes core so drivers can be updated, debugged, and distributed independently.

Scale down the StatefulSet to 0 replicas, then scale back to 3.

Why it fails: Scaling a StatefulSet down to 0 replicas will terminate all associated pods, including app-db-0. However, the PersistentVolumeClaims (PVCs) created by the StatefulSet, which are bound to the PersistentVolumes (PVs) on the unrecoverable node1, will persist. When scaled back up, the StatefulSet controller will attempt to reuse these existing PVCs, which are still bound to PVs on the failed node, preventing new pods from starting successfully on healthy nodes.

Manually repair node1 and rejoin it to the cluster.

Why it fails: The question explicitly states that node1 is unrecoverable, meaning it cannot be repaired or brought back online. Therefore, any solution relying on the recovery or rejoining of node1 to the cluster is fundamentally infeasible and does not address the problem under the given constraints. This option is a non-starter based on the problem statement.

Delete only the pod app-db-0, and let the StatefulSet recreate it.

Why it fails: Deleting only the app-db-0 pod will cause the StatefulSet controller to immediately recreate it, maintaining its stable identity. However, the recreated pod will still reference its existing PersistentVolumeClaim (PVC), app-db-pvc-app-db-0, which remains bound to the PersistentVolume (PV) provisioned on the unrecoverable node1. Consequently, the new pod will fail to mount its volume, as the underlying storage is inaccessible.

Use a StorageClass with volumeBindingMode: Immediate

Why it fails: Setting volumeBindingMode: Immediate forces the PersistentVolumeClaim to bind to a PersistentVolume immediately upon creation, before the Pod is scheduled. This prevents the scheduler from evaluating the Pod's node constraints, often resulting in the PV being bound to a node that cannot satisfy the Pod's scheduling requirements.

Use a podAffinity rule to schedule the pod near the storage node

Why it fails: podAffinity is designed to co-locate Pods relative to other running Pods based on labels, rather than binding a Pod to a specific physical node hosting a local storage volume. It cannot target node-specific hardware or local storage resources directly, making it unsuitable for ensuring local volume availability.

Use a PersistentVolume with nodeAffinity in the PV spec

Why it fails: Configuring nodeAffinity within the PersistentVolume specification merely restricts which nodes the volume itself can be attached to. It does not directly dictate the Pod's scheduling behavior or guarantee that the scheduler will place the Pod on the correct node, especially if the volume binding mode is not configured to delay binding.

Recycle policy: the PV is deleted and data is scrubbed.

Why it fails: Recycle has never deleted the PV; when implemented, it runs a scrubbing operation (historically an `rm -rf` script) on the volume content and then marks the PV as Available again. It is deprecated as of Kubernetes 1.20 and lacks support in modern CSI drivers, so stating that it deletes the PV conflates it with the Delete policy.

Delete policy: the PV remains but the underlying storage is deleted.

Why it fails: The Delete policy removes both the PV API object and the underlying storage in a single operation; the PV cannot survive after its backing storage is deleted because the PV's definition is exactly a handle to that external storage. Leaving the PV behind after destroying the storage would leave a dangling reference and is contrary to how the volume controller implements deletion.

Retain policy: the PV is automatically deleted after 30 days.

Why it fails: Retain imposes no time-based lifecycle; a Released PV persists indefinitely until a human intervenes. There is no built-in '30-day' or any other automatic deletion timer in the Retain policy—only the Delete policy removes resources, and it does so immediately when invoked by the volume controller, not after a delay.

A PersistentVolume can only be created by a cluster administrator using static provisioning.

Why it fails: Static provisioning by a cluster administrator is only one way to create PersistentVolumes; dynamic provisioning automatically generates a PV when a PVC requests a StorageClass that has a provisioner configured. The dynamic provisioner (e.g., AWS EBS, GCE PD, or a CSI driver) creates the underlying storage and a matching PV object on demand without administrator intervention. Therefore, the statement is false because it excludes the dynamic provisioning path, which is a common and fundamental mechanism in Kubernetes.

A PVC is a cluster-scoped resource, similar to a PV.

Why it fails: PersistentVolumeClaims are namespace-scoped resources, unlike PersistentVolumes, which are cluster-scoped. A PVC is created inside a specific Kubernetes namespace and can only be referenced by pods in that same namespace, while a PV exists at the cluster level and can be bound by a PVC from any namespace. This distinction affects RBAC permissions and resource management: users granted PVC access in one namespace cannot affect PVCs in another, but a PV's cluster-wide nature requires cluster-level authorization to modify.

A PVC can be deleted while bound to a PV without affecting the PV.

Why it fails: Deleting a PVC that is bound to a PV does not have a universal no-effect outcome because the PV's reclaim policy determines the result. If the reclaim policy is Delete, the PV and its associated underlying storage will be removed when the PVC is deleted. If the policy is Retain, the PV remains in the Released state and can be manually reclaimed, but it is no longer available for binding; in either case, the PV is definitely affected by the deletion.

Using a ConfigMap volume

Why it fails: A ConfigMap volume is an API object that injects configuration data into the container as either files or environment variables. When mounted, it is read-only (unless using subPath with individual file mounts) and represents key-value pairs as a filesystem, but it is not designed to store application data or serve as general-purpose storage. ConfigMaps have a 1MB size limit and are meant for non-sensitive configuration, so they are not a valid choice for persistent, durable, or even scratch storage in a pod.

Using a Secret volume

Why it fails: A Secret volume is an API object intended to deliver sensitive data such as passwords, CA certificates, or API keys to containers. When mounted, it exposes the secret's key-value pairs as files in a tmpfs RAM-backed volume, which is neither persistent nor designed for general application data. Secrets are stored with a 1MB size limit and are base64-encoded, but they are configuration-like and cannot be treated as a general storage volume for application logs, databases, or cache files.

Define NetworkPolicy YAML with podSelector and ingress from specific pods, ensure namespace exists, apply using kubectl, test ingress connectivity

Why it fails: This order is incorrect because while defining the YAML can be done anytime, you cannot apply the policy before the namespace exists; but the namespace must be ensured before applying, not after defining. However, the sequence is still plausible if one defines first and then creates the namespace, but then the apply would fail if the namespace didn't exist at define time? Actually, the define step is just writing the file, so it doesn't require a namespace. The main issue is that applying without ensuring namespace existence would fail. So this order works if the namespace already exists, but it's not the recommended order because you should verify existence first. But it's a common oversight.

Ensure namespace exists, apply using kubectl, define NetworkPolicy YAML with podSelector and ingress from specific pods, test ingress connectivity

Why it fails: This order is logically flawed because it attempts to execute the apply command before the NetworkPolicy manifest has actually been defined. Without a valid YAML file containing the podSelector and ingress specifications, the kubectl deployment command will fail immediately due to a missing file or empty input. Defining the resource must always precede its application to the cluster.

Ensure namespace exists, define NetworkPolicy YAML with podSelector and ingress from specific pods, test ingress connectivity, apply using kubectl

Why it fails: This sequence is incorrect because it schedules connectivity testing before the NetworkPolicy has been applied to the Kubernetes API server. Testing at this stage is premature and will only reflect the default non-isolated state of the namespace, rather than the intended ingress restrictions. The policy must be actively enforced by the container network interface before any validation tests can yield meaningful results.

Apply YAML, then Update YAML, then Monitor status, then Check history, then Rollback if needed

Why it fails: Applying a YAML file before updating it is logically impossible: kubectl apply reads the file as-is and submits that spec to the API server, so the first apply would re-apply the current (unchanged) configuration, producing no rolling update. The file must be edited first because apply uses a three-way merge against the last-applied-configuration annotation; if the file still holds the old image or environment, the operation patched in is a no-op. Only after editing the file can apply trigger the desired reconciliation, and then monitoring, history, and rollback become meaningful. This sequence inverts the declarative workflow and would require a second apply later, making the order incorrect.

Update YAML, then Monitor status, then Apply YAML, then Check history, then Rollback if needed

Why it fails: Monitoring rollout status before applying the manifest is impossible because no change has been submitted to the API server yet; kubectl rollout status would either report the existing Deployment as fully available or block waiting for a new rollout that does not exist. The apply step is what creates the new ReplicaSet and starts the rolling update, so it must come before any status check. Updating the YAML only modifies the file locally, and without apply the control plane remains unaware of the intended change. Thus, this order incorrectly places observation ahead of the action that actually triggers the rollout.

Update YAML, then Apply YAML, then Check history, then Monitor status, then Rollback if needed

Why it fails: Placing kubectl rollout history before kubectl rollout status skips the critical verification that the new ReplicaSet has become available; immediately after apply, history typically shows the old revision and a new revision that is still in progress or may fail. You need to monitor the rollout first, because if it crashes, status reveals the problem and tells you whether to continue with history and rollback. Checking history too early can mislead you into analyzing a revision that has not been reconciled, and it deprives you of the failure signal that monitoring provides. The correct sequence is to let the rollout settle, review its health, inspect revision history for context, and then rollback only if the monitored status or post-deployment checks indicate a problem.

snapshot: Reclaims storage space by freeing unused pages.

Why it fails: This description is incorrect because reclaiming storage space by freeing unused pages describes the `etcdctl defrag` operation, not `snapshot`. A snapshot creates a backup file that captures the full state of the key-value store at a specific revision, but it does not modify the live database or affect storage usage. Confusing these two operations would lead you to expect a snapshot to shrink disk usage when it actually only produces a backup artifact.

compact: Creates a point-in-time backup of etcd data.

Why it fails: This description is incorrect because creating a point-in-time backup is the function of `etcdctl snapshot`, not `compact`. Compact removes old revisions from the MVCC store to control database growth and is a destructive operation that permanently deletes historical data. A compact never generates a file that can be used for recovery, so equating it with snapshot would mistakenly downplay the irreversibility of compaction and its operational risk.

Upgrade kubeadm: Upgrade the node agent (kubelet) on each node

Why it fails: This is incorrect because kubeadm is not a node agent; it is a command-line administration tool used to bootstrap and upgrade clusters. The node agent that runs on every node and manages Pod creation and container execution is the kubelet, which is deployed as a standalone service. Upgrading kubeadm on a node only updates the tool used for initial setup and future upgrades; it does not change the kubelet version or its running configuration. A separate `kubeadm upgrade node` command is required to apply the new kubelet and kube-proxy components on worker nodes.

Upgrade control plane nodes: Upgrade workloads running on the cluster

Why it fails: This is incorrect because upgrading the control plane targets only the cluster's master components—the API server, scheduler, controller manager, and etcd—and leaves user deployments untouched. User workloads are managed by higher-level controllers like Deployments and StatefulSets, which are updated by changing container image tags, performing rolling updates, or using CI/CD delivery pipelines. The control-plane upgrade process does not touch Pod replicas or application containers; it only replaces the binaries and static pods that constitute the management plane, ensuring cluster-wide services remain compatible with the new version.

Upgrade kubelet: Update the cluster administration tool (kubeadm)

Why it fails: This mismatch of terminology conflates the kubelet with the cluster administration tool. The kubelet is the node-level agent that communicates with the API server to ensure containers in Pod specs are healthy; it is not an admin CLI and cannot be used to manage cluster lifecycle actions like `kubeadm upgrade`. Upgrading kubelet affects only the compute node where it runs—refreshing the daemon that creates and monitors Pods—whereas kubeadm is the tool that requires updating before running upgrade planning and rollout commands. These are two distinct binaries and serve separate purposes in a cluster.

Expanding PVCs is not supported by StatefulSets

Why it fails: The statement that PVC expansion is not supported by StatefulSets is incorrect. Kubernetes itself supports PersistentVolumeClaim expansion, provided the underlying StorageClass allows it and the volume plugin supports it. While a StatefulSet's volumeClaimTemplates do not automatically expand existing PVCs, individual PVCs associated with StatefulSet pods can be manually expanded by modifying their `spec.resources.requests.storage` field, demonstrating that the capability exists within Kubernetes.

The StorageClass must have allowVolumeExpansion set to true

Why it fails: While it is true that the StorageClass must have `allowVolumeExpansion: true` for any PersistentVolumeClaim (PVC) expansion to occur, this option incorrectly identifies the primary reason for the issue. Even if the StorageClass permits expansion, the StatefulSet controller does not automatically detect changes in its `volumeClaimTemplates` and apply them to existing, already bound PVCs. The problem lies with the StatefulSet's update mechanism for existing PVCs, not solely with the StorageClass's capability.

The StatefulSet must be deleted and recreated to apply changes

Why it fails: Deleting and recreating a StatefulSet is an unnecessary and potentially disruptive action to apply changes, especially for PVC expansion. While recreating might provision new PVCs if the template changed, it does not modify existing ones and often leads to data loss if not handled carefully. StatefulSets support rolling updates for pod templates, but changes to `volumeClaimTemplates` for existing PVCs are not automatically propagated through this mechanism, nor does deletion solve the problem of updating existing PVCs.

StorageClass

Why it fails: While a StorageClass defines the provisioner (which points to the CSI driver) and parameters for dynamic volume provisioning, it does not represent or register the driver itself within the cluster. It merely acts as a blueprint for creating PersistentVolumes using an already registered CSI driver.

CSINode

Why it fails: CSINode is a node-specific resource that maps a Kubernetes node to the CSI driver's node ID and advertises topology constraints. It is automatically created by the node-driver-registrar during driver installation, but it does not serve as the primary cluster-wide registration object for the driver itself.

VolumeAttachment

Why it fails: VolumeAttachment is an API object used by the attach/detach controller to track the attachment state of a specific PersistentVolume to a particular node. It is a transactional resource created during volume attachment operations, rather than a registration mechanism for the underlying CSI driver.

The pod using the PVC must be deleted and recreated

Why it fails: Deleting the pod is an unnecessarily destructive action. The PVC's capacity increase is already recorded by the API server; what remains is for the node-side volume manager to resize the filesystem. A simple pod restart (or in many cases automatic detection) triggers that resize, so a full delete/recreate cycle only adds avoidable downtime and is not required.

No further action is needed; the PVC size will update immediately

Why it fails: Although the PVC's spec is updated instantly, the filesystem inside the volume does not automatically expand on every storage driver. The kubelet has to execute the resize operation, and depending on the CSI driver and filesystem, that may require a remount of the volume. So 'no further action needed' wrongly assumes the API update alone expands the filesystem, ignoring the node-side step.

The CSI driver must be updated to support expansion

Why it fails: The storage class already has allowVolumeExpansion: true, which means the CSI driver or other provisioner advertises expansion support; there is no driver update required. The remaining work is operational, not installation-related: the kubelet must resize the filesystem and possibly remount the volume. Upgrading the driver would not help because the driver's capability is already present.

The cluster has no nodes with the required storage driver.

Why it fails: If the cluster truly lacked a node with the required storage driver, the dynamic provisioner specified by the StorageClass would fail to create the underlying PersistentVolume. This failure would typically manifest as specific error messages in the PVC's events, detailing the inability to provision the volume, rather than the PVC merely remaining in a silent 'Pending' state without any diagnostic information.

The PVC requests a capacity that exceeds the available quota.

Why it fails: When a PersistentVolumeClaim's requested capacity exceeds a configured ResourceQuota for storage, the Kubernetes API server typically rejects the PVC creation outright or marks it with a distinct error message in its events. This error would explicitly state that the quota limit has been exceeded, providing clear feedback, rather than the PVC being stuck in 'Pending' without such an indication.

The PVC's storage class name is misspelled.

Why it fails: A misspelled `storageClassName` in the PersistentVolumeClaim definition would prevent Kubernetes from finding a matching StorageClass object. While this would result in the PVC entering a `Pending` state, its events would clearly indicate an error such as "no StorageClass with name 'misspelled-name' found," rather than silently waiting for a volume to be provisioned without a specific error message.