Troubleshooting Scenario Questions

kubectl run -it --rm debug --image=busybox

Why it fails: The kubectl run command is designed to create and run a new Pod, not to enter an existing one. Even though -it grants an interactive terminal and --rm deletes the ephemeral Pod afterward, the debug Pod is completely separate from the affected container and has its own IP and filesystem. It does not share the process or network namespace of the failing container, so it cannot directly inspect or interact with that container's runtime state.

kubectl exec <pod> -- /bin/bash

Why it fails: Without the -i and -t flags, kubectl exec does not provide a TTY or keep standard input open, so the /bin/bash process either fails to start because no terminal is available or immediately exits due to no input. In practice, this results in a non-interactive one-shot command that cannot be used for ongoing troubleshooting. You might see an error like 'the input device is not a TTY' because docker exec requires a terminal for an interactive shell.

kubectl debug -it <pod> --image=busybox --target=<container>

Why it fails: kubectl debug creates an ephemeral container inside the same Pod, which can be useful when the target container is missing a shell or its binary is broken. However, this approach adds unnecessary complexity for a normal container that is simply 'not responding' but still has a shell, because you can exec directly into that existing container. Ephemeral containers also have limitations (e.g., they do not appear in the Pod spec and may not be supported by all clusters), making kubectl exec the simpler and more portable first step.

The pod 'frontend' is not in the same namespace as the service 'backend'.

Why it fails: The service's Endpoints are populated only by pods that live in the same namespace as the service and match its label selector. A client pod such as 'frontend' being in a different namespace has no effect on the service's endpoint creation; it would simply need to use the fully qualified service DNS name (backend.<namespace>.svc.cluster.local). Thus this condition could break resolution or connectivity, but it does not explain why the endpoint list is empty.

The pod's container port is different from the service port.

Why it fails: The service port and the container port are deliberately decoupled: spec.ports[].port is the virtual ClusterIP port, while spec.ports[].targetPort points to the container port. As long as targetPort matches the port the container actually listens on, the endpoint list is populated with that targetPort address. A mismatch between service.port and containerPort is normal and does not affect endpoint generation; it would merely cause traffic to be sent to the wrong port if targetPort were also misconfigured.

The kube-proxy is misconfigured and not updating iptables rules.

Why it fails: kube-proxy is responsible for translating Services into iptables, IPVS, or userspace rules, and it consumes Endpoints to build those forwarding rules. If kube-proxy is misconfigured, the rules may be missing or stale, causing traffic to the ClusterIP to fail even when the Endpoints list is correctly populated. Endpoints themselves are generated by the endpoint controller, not kube-proxy, so a kube-proxy fault explains a connectivity problem but never an empty endpoint list.

The Pod's service account does not have permission to create Pods.

Why it fails: A missing RBAC role binding for the pod's service account causes the API server to reject the create request during authorization, long before admission controllers evaluate ResourceQuota. The error would say `pods is forbidden: User "system:serviceaccount:default:mysa" cannot create resource "pods" in API group ""`, not that a quota has been exceeded. Since ResourceQuota is a validating admission step that runs after authorization, an RBAC failure and a quota failure produce different messages and are diagnosed differently.

The Pod's resource requests exceed the quota.

Why it fails: A ResourceQuota that limits compute resources, such as `requests.cpu` or `requests.memory`, returns an error naming the resource that was exceeded, for example `requested: requests.cpu=1, used: requests.cpu=4, limited: requests.cpu=5`. The error described in the question explicitly mentions `pods=1` and `pods=5`, which indicates a count quota (`count/pods: 5`) rather than a resource-request quota. Furthermore, a pod with zero resource requests can still be rejected by a pod-count quota, so the two failures are independent.

The Pod is trying to mount a Secret that does not exist.

Why it fails: A Pod that references a non-existent Secret is still admitted by the API server; the missing Secret is not discovered until the kubelet tries to set up the container's volume mounts or environment variables. This manifests as a `CreateContainerConfigError` pod phase and events saying `Secret "foo" not found`, not as an admission-time rejection. Because the Pod object itself is created successfully, such a failure would not generate an 'exceeded quota' message from the API server.

kubectl describe pod client

Why it fails: kubectl describe pod client inspects the source pod's metadata, status, and recent events, including its own IP and container states. It does not query the service's selector or the Endpoints object, so it cannot reveal which server pods are ready and receiving traffic. This command helps with issues like a client failing to start or an unresolved hostname, but it gives no visibility into whether the backing pods are registered as endpoints.

kubectl get pods -l app=server

Why it fails: kubectl get pods -l app=server lists all pods carrying that label regardless of readiness or lifecycle; a terminating, failed, or unscheduled pod may appear while contributing nothing to the load-balanced pool. The service's Endpoints resource is populated only with healthy pods that pass both the selector and the readiness probe. Without inspecting the Endpoints object, you cannot tell whether these labeled pods are actually wired into the service.

kubectl get svc server

Why it fails: kubectl get svc server displays the service's ClusterIP, port mapping, and selector, but the service object is declarative intent rather than runtime state. The control plane resolves that selector into concrete pod addresses and stores them in a separate Endpoints resource. Even if the service definitively selects pods, a missing or stale Endpoints entry is often the root failure, so this command alone cannot diagnose that layer.

kubectl rollout resume deployment my-deployment

Why it fails: kubectl rollout resume is intended to continue a deployment that was explicitly paused with kubectl rollout pause. It does not stop or freeze a stuck rollout; instead it tells the controller to proceed with the current ReplicaSet scaling process, which can make a bad rollout worse. Since the goal is to halt progress and investigate, this command acts in the opposite direction and is therefore incorrect.

kubectl rollout status deployment my-deployment

Why it fails: kubectl rollout status only queries and reports the current rollout state, such as waiting for ReplicaSet availability or the progress deadline being exceeded. It passively observes the deployment but does not alter its desired state or prevent the controller from continuing to scale ReplicaSets. While useful for diagnosis after pausing, it cannot stop or isolate the stuck rollout, so it is not the correct command here.

kubectl rollout stop deployment my-deployment

Why it fails: kubectl rollout stop is not a valid kubectl rollout subcommand; the valid subcommands are history, pause, restart, resume, status, and undo. Running this command would fail with a usage error before any action is taken. To pause the rollout, you must use the exact, supported command kubectl rollout pause.

The liveness probe uses a TCP socket check instead of an HTTP GET.

Why it fails: Changing the liveness probe from an HTTP GET to a TCP socket check alters the mechanism of the probe but not its frequency, failure threshold, or restart consequences. A TCP check merely attempts to open a connection to a port, while an HTTP GET additionally verifies a path and response code, so at most it changes the resource footprint of each probe. It does not trigger endpoint changes or iptables updates by itself, and therefore cannot explain the observed iptables churn.

The deployment sets resource requests and limits that are too low, causing pod throttling.

Why it fails: Setting resource requests and limits too low causes the container's CPU to be throttled by the kubelet, which can make the application slower and possibly cause probe failures. However, CPU throttling does not directly modify service endpoints or iptables rules; those rules are updated only when the set of Ready pods changes. If throttling indirectly leads to restarts, the root cause of churn is still the probe configuration and restart policy, not the resource limits, so this option is irrelevant to the iptables issue.

The deployment uses a large number of environment variables, causing slow container startup.

Why it fails: A large number of environment variables increases the container's environment block, which can slow down the initial container startup and delay readiness. This is a one-time cost per pod and does not produce repeated changes to Service endpoints or iptables tables. Even during the extended startup, the pod is not yet an endpoint, so no iptables updates occur because of the environment variables themselves; they simply postpone readiness without causing the ongoing churn described.

The container does not have the CAP_SYS_ADMIN capability

Why it fails: CAP_SYS_ADMIN is a Linux capability that grants administrative operations such as mounting filesystems or performing system administration tasks. It is unrelated to the runAsNonRoot security context setting, which only validates the numeric user ID under which the container will run. The CreateContainerConfigError in this scenario indicates a user ID mismatch, not a missing capability, so this option is incorrect.

The container's filesystem is read-only

Why it fails: Setting readOnlyRootFilesystem: true mounts the container's root filesystem as read-only, which prevents writes to the filesystem but has no effect on user ID validation. The runAsNonRoot check is performed by the kubelet before the container starts, based on the image's user metadata, not on mount options. Therefore a read-only filesystem would not produce this error and this option is wrong.

The runAsUser field is missing, so the pod uses a random UID

Why it fails: If runAsUser is omitted, the container runs as the user defined in the image's configuration, not a randomly assigned UID. When runAsNonRoot: true is set, Kubernetes only requires that the resulting effective UID be non-zero; if the image's default user is already non-root, the pod will start successfully. The error message specifically states the image will run as root, so the issue is the image's default user, not a missing runAsUser field.