Application Observability and Maintenance practice questions

Create a new pod with a higher memory limit and delete the old pods manually.

Why wrong: Manually creating a separate pod and deleting Deployment pods fights the controller: the Deployment's ReplicaSet immediately recreates deleted pods to match its current template, so the manual replacement is cosmetic and short-lived. A standalone pod is outside the Deployment's management, meaning it receives no rolling updates, self-healing, or replica reconciliation. This approach causes unnecessary downtime and does not persist the increased memory limit in the desired state.

Set the memory limit to unlimited by removing the limit section and restart the pod.

Why wrong: Removing the memory limit entirely removes the safety boundary that prevents a single container from exhausting node memory; the kubelet or OOM killer may then terminate the container or evict neighboring pods when the workload spikes. Restarting the pod manually only applies to that ephemeral instance—the Deployment controller will enforce the original spec on the next reconciliation, so the change is lost. Unbounded pods are a cluster-risk anti-pattern, not a fix for memory pressure.

Use 'kubectl set resources' to increase the limit on the running pod dynamically.

Why wrong: 'kubectl set resources' is a subcommand for mutating workload controllers such as Deployments, StatefulSets, or ReplicaSets—it cannot mutate an individual Pod because a Pod's resource requests and limits are immutable after creation. Running it against a Pod either fails or changes the controller's template, which then triggers a rollout of new pods, not a dynamic in-place adjustment. The running pod keeps its old limit until it is terminated and replaced, so the phrase 'increase the limit on the running pod dynamically' is technically false.

kubectl logs -f my-pod

Why wrong: The -f flag puts kubectl into follow mode, attaching to the container's stdout/stderr stream and blocking the terminal with continuous output. This is counterproductive for a crash-looping pod because you need a static snapshot of the most recent failure, not an interactive stream that may terminate or wrap around as the container restarts.

kubectl logs my-pod

Why wrong: Without a --tail limit, this command dumps the entire log history from the current container instance. In a CrashLoopBackOff scenario, that output is polluted with repeated stack traces and error messages from numerous previous restarts, making it cumbersome to isolate the precise cause of the latest failure. The sheer volume can also hide the crucial final lines.

kubectl get events --field-selector involvedObject.name=my-pod

Why wrong: This command queries the Kubernetes API for events associated with the pod object, such as schedule failures, image pull errors, or probe failures. It does not retrieve the application's own stdout/stderr log output, which is where the actual crash error is recorded. Events might hint at why the pod is being killed (e.g., OOMKilled or failed liveness probe), but they will never show the application-specific stack trace or exception messages required to fix the crash loop.

The pod 'frontend' is not in the same namespace as the service 'backend'.

Why wrong: The service's Endpoints are populated only by pods that live in the same namespace as the service and match its label selector. A client pod such as 'frontend' being in a different namespace has no effect on the service's endpoint creation; it would simply need to use the fully qualified service DNS name (backend.<namespace>.svc.cluster.local). Thus this condition could break resolution or connectivity, but it does not explain why the endpoint list is empty.

The pod's container port is different from the service port.

Why wrong: The service port and the container port are deliberately decoupled: spec.ports[].port is the virtual ClusterIP port, while spec.ports[].targetPort points to the container port. As long as targetPort matches the port the container actually listens on, the endpoint list is populated with that targetPort address. A mismatch between service.port and containerPort is normal and does not affect endpoint generation; it would merely cause traffic to be sent to the wrong port if targetPort were also misconfigured.

The kube-proxy is misconfigured and not updating iptables rules.

Why wrong: kube-proxy is responsible for translating Services into iptables, IPVS, or userspace rules, and it consumes Endpoints to build those forwarding rules. If kube-proxy is misconfigured, the rules may be missing or stale, causing traffic to the ClusterIP to fail even when the Endpoints list is correctly populated. Endpoints themselves are generated by the endpoint controller, not kube-proxy, so a kube-proxy fault explains a connectivity problem but never an empty endpoint list.

Check the liveness probe events via 'kubectl describe pod' to see the exact probe responses.

Why wrong: kubectl describe pod shows probe events like 'Liveness probe failed' with the HTTP status code or command exit code, but these events are generated by the kubelet and do not contain the application's internal error message, stack trace, or logging output. The event will tell you that the probe failed and possibly the raw response (e.g., 500), but not why your application returned that response. This makes describe a useful first step for confirming the failure, but insufficient for root-cause diagnosis.

Run 'kubectl exec' to curl the endpoint from another pod to test network connectivity.

Why wrong: Executing curl from another pod tests network reachability to the pod's IP/port from the cluster network, but the kubelet runs liveness probes directly on the node, so a network partition between pods is rarely the cause of a liveness probe failure. Even if the endpoint is reachable, the application can still be internally unhealthy—for example, deadlocked or stuck in a bad state—and return a 500 or fail to respond to the probe specifically. This action adds extra load and could even change timing, making it a poor diagnostic for application-level failures.

Review the liveness probe parameters in the deployment YAML and increase the failureThreshold.

Why wrong: Increasing failureThreshold or adjusting other probe parameters only tells Kubernetes to tolerate more consecutive failures before restarting the container; it does nothing to address the underlying application defect. During the extended threshold window, the pod remains unhealthy and may be removed from Service endpoints, meaning user traffic continues failing while the container is kept alive artificially. This is a config tuning approach that masks the symptom, not a diagnostic step, and should only be considered after the root cause is identified and the threshold is aligned with actual startup and response behavior.

Increase the CPU limits on the pod to give it more resources.

Why wrong: Increasing CPU limits does not affect the scheduler's pending decision because limits are only applied as cgroup quotas after the pod is placed on a node. An unschedulable pod is pending because its requested CPU exceeds the allocatable CPU on every node, and limits are never inspected during scheduling. Raising limits without lowering requests merely increases the maximum CPU the container can burst to, which can lead to throttling or policy violations while leaving the pod permanently unschedulable.

Add another node to the cluster to increase overall CPU capacity.

Why wrong: Manually adding a node is an infrastructure-level operation that may be impossible, costly, or outside your permissions, and it does not repair the actual mismatch between the pod's requested CPU and the cluster's existing allocatable capacity. If the same request is applied, the pod could land on the new node only if that node has sufficient free CPU, but the original unschedulability will persist on all existing nodes. In many real clusters with Cluster Autoscaler, adding nodes is triggered automatically only when pending pods exist, so manually doing so is redundant and slower than reducing the request.

Add a nodeSelector to the pod to target a specific node.

Why wrong: A nodeSelector only restricts scheduling to nodes carrying specific labels; it tells the scheduler to ignore all other nodes, but it does nothing to change CPU requests or the amount of allocatable CPU on any node. If every node in the cluster is already underallocated for the pod's request, the selector simply limits the candidate set to a subset that still cannot satisfy the request, leaving the pod pending. It can also introduce an additional failure mode if the named node does not exist or does not have the label.

Set the log level to ERROR only to reduce verbosity and avoid logging sensitive data.

Why wrong: Setting the log level to ERROR only reduces verbosity but does not guarantee that sensitive data is omitted, as error messages often contain contextual details like passwords or tokens. Moreover, this approach degrades observability by suppressing WARN, INFO, and DEBUG messages, making it harder to diagnose issues. It also fails to address sensitive data already being logged at ERROR level, so it is neither a complete nor a reliable solution.

Configure a LogCollector DaemonSet to filter logs at the node level.

Why wrong: Configuring a LogCollector DaemonSet to filter logs at the node level is impractical because DaemonSets like Fluentd or Filebeat typically collect raw logs from all containers and forwarded them as-is; node-level filtering would require intercepting streams per container before aggregation, which adds significant complexity and overhead. Additionally, redaction logic often depends on application-specific log formats, which the node-level collector cannot reliably interpret. This approach also fails to prevent sensitive data from being written to the node's local logs or transmitted before filtering, leaving exposure windows.

Modify the application code to stop logging sensitive data before redeploying.

Why wrong: Modifying the application code to stop logging sensitive data is the ideal long-term fix, but it is not a recommendation for an immediate requirement. Code changes require a full CI/CD cycle, testing, and redeployment, which can take hours or days, and does nothing for logs already produced or for running instances. A time-sensitive remediation should use a runtime solution like a sidecar redaction container, while code changes can be planned in parallel.

Create an Ingress resource that exposes the /metrics endpoint externally.

Why wrong: Creating an Ingress resource would expose the /metrics endpoint to the outside world via HTTP routing rules, but Prometheus scrapes inside the cluster by reaching pod IPs directly through service discovery. Ingress is an API object for external L7 load balancing, not for registering internal scrape targets, and it doesn't contribute to Prometheus's dynamic service discovery, which uses Kubernetes Services or Pods as sources.

Create a ConfigMap with the Prometheus scrape configuration and mount it into the Prometheus pod.

Why wrong: Mounting a ConfigMap into the Prometheus pod only provides static configuration files; it does not register the target pod’s /metrics endpoint for dynamic service discovery. This option is tempting because ConfigMaps are the standard way to supply Prometheus with its own configuration, such as scrape intervals or static targets. However, in this scenario, the application pod must be discovered and scraped automatically, which requires a ServiceMonitor or PodMonitor custom resource definition that Prometheus Operator uses to generate the actual scrape configuration.

Add a PrometheusRule resource that defines the scrape target.

Why wrong: A PrometheusRule custom resource defines alerting or recording rules that evaluate PromQL expressions on scraped metrics; it does not configure where to collect metrics from. Scrape targets are instead defined by ServiceMonitor or PodMonitor objects that specify selector labels, ports, and intervals, which the Prometheus Operator processes to build the actual scrape_config file. Thus, adding a PrometheusRule would only affect alerting behavior, not register the application's /metrics endpoint as a scrape target.

The readiness probe is failing, causing the pod to be considered not ready and restarted.

Why wrong: The readiness probe controls whether the pod receives traffic from a Service; when it fails, the kubelet marks the pod as NotReady and removes its endpoints, but the container continues running its main process. Restart decisions are made exclusively by the kubelet based on the liveness probe and restartPolicy, so a failed readiness probe alone would not cause a container to be killed or restarted. The exhibit shows restart activity, which is not a characteristic of readiness probe handling.

The container is running out of memory (OOM).

Why wrong: An out-of-memory termination is recorded with a distinct reason of OOMKilled in the container status, and the exit code is typically 137 (SIGKILL), not a generic failure. While OOM events do trigger a restart when restartPolicy is Always or OnFailure, the exhibit would show the container currently in a CrashLoopBackOff or Waiting state with OOMKilled identified, rather than simply an increasing restart count that accompanies liveness probe failures.

The container image is being pulled repeatedly.

Why wrong: Repeated image pulls manifest as ImagePullBackOff or ErrImagePull, which occur during the container creation phase before the container can start. The exhibit depicts a container that was running and then killed (as indicated by an increasing restart count), meaning the image had already been successfully pulled and the image layers were present on the node. Once a container is running, image pulls are no longer involved; therefore, repeated pulling cannot be the reason for the observed termination.

Run 'kubectl describe pod backend' to see the pod's status.

Why wrong: kubectl describe pod backend aggregates pod metadata, status conditions, and recent events, but because you have already inspected events and seen the CrashLoopBackOff state, it will likely repeat the same backoff/restart information without exposing the process-level failure. The describe output does not include the container's application stdout/stderr, so it cannot reveal why the process inside the container exits non-zero. Logs are the direct source for the crash reason.

Run 'kubectl get events' with a different filter to capture more details.

Why wrong: kubectl get events with a different filter, such as --field-selector involvedObject.name=backend or --watch, can surface additional Event objects, yet the existing event stream already identified the symptom as BackOff/restarting, not the underlying cause. Event objects are generated by the kubelet and contain only high-level warnings such as failed health checks or image pull failures; they do not capture the application's own error output. Thus re-filtering events merely changes the view, not the diagnostic depth.

Check if the image tag '1.0' exists in the registry.

Why wrong: Checking the registry for whether the image tag '1.0' exists assumes a pull problem, but the exhibit indicates the image is already present on the node and the pod is stuck in a restart loop, not ImagePullBackOff. Since the kubelet has already pulled the image successfully, verifying the tag's existence in the registry would not explain an application crash after container start. The relevant failure occurs in the container's runtime process, not during image resolution.

Reduce the number of replicas to 2 to force the HPA to react more aggressively.

Why wrong: Manually reducing the replica count to 2 will spike CPU utilization per pod, but the HPA's scaling decision still compares current CPU usage to the existing target, such as 70%. If per-pod CPU remains below that target, the HPA will not trigger a scale-out; it could even compute a desired replica count of 1, shrinking the deployment further. This approach does not alter the scaling metric or its threshold, and it deliberately removes capacity, increasing the risk of overload and service disruption.

Increase the CPU target utilization to 90% so the HPA triggers earlier.

Why wrong: Raising the CPU target utilization to 90% actually raises the bar that must be crossed before the HPA adds replicas. For example, if the current average CPU usage is 80% and the target is 70%, the HPA scales out; with a 90% target, that same 80% usage results in no scaling action. Since the problem is that scaling is not happening soon enough, increasing the threshold makes the autoscaler even less sensitive and delays the very reaction you need, worsening response times under load.

Optimize the database queries to reduce response time.

Why wrong: Optimizing database queries can lower per-request CPU and latency, but it does not change the HPA's configuration or its decision-making process. If the true issue is that the HPA ignores memory pressure and therefore keeps too few replicas, query optimization may temporarily mask the symptom but leaves the scaling mechanism blind to the real bottleneck. Furthermore, such optimizations are application-level fixes, whereas the HPA operates at the infrastructure layer; they address a different cause and fail to ensure automatic scaling under future memory spikes.

The liveness probe uses a TCP socket check instead of an HTTP GET.

Why wrong: Changing the liveness probe from an HTTP GET to a TCP socket check alters the mechanism of the probe but not its frequency, failure threshold, or restart consequences. A TCP check merely attempts to open a connection to a port, while an HTTP GET additionally verifies a path and response code, so at most it changes the resource footprint of each probe. It does not trigger endpoint changes or iptables updates by itself, and therefore cannot explain the observed iptables churn.

The deployment sets resource requests and limits that are too low, causing pod throttling.

Why wrong: Setting resource requests and limits too low causes the container's CPU to be throttled by the kubelet, which can make the application slower and possibly cause probe failures. However, CPU throttling does not directly modify service endpoints or iptables rules; those rules are updated only when the set of Ready pods changes. If throttling indirectly leads to restarts, the root cause of churn is still the probe configuration and restart policy, not the resource limits, so this option is irrelevant to the iptables issue.

The deployment uses a large number of environment variables, causing slow container startup.

Why wrong: A large number of environment variables increases the container's environment block, which can slow down the initial container startup and delay readiness. This is a one-time cost per pod and does not produce repeated changes to Service endpoints or iptables tables. Even during the extended startup, the pod is not yet an endpoint, so no iptables updates occur because of the environment variables themselves; they simply postpone readiness without causing the ongoing churn described.

Increase the restart policy to Always

Why wrong: Setting restartPolicy to Always is ineffective because CrashLoopBackOff already indicates the kubelet is in a wait/retry loop after the container exits nonzero. The restartPolicy is a Pod-level spec field, not a diagnostic tool, and switching it to Always merely increases automatic restarts without addressing the process's failing startup. This action can also hide the problem and make the broken container appear normal.

Run kubectl describe pod

Why wrong: kubectl describe pod aggregates object metadata, labels, node scheduling, container images, and recent events, but it deliberately omits the captured stdout/stderr from the application. Although describe shows exit codes and restart counts, that only tells you the container failed, not why it failed. The actual error message, such as a missing database connection string or malformed config file, is almost always in the logs, which describe will not surface.

Check the pod's events with kubectl get events

Why wrong: kubectl get events surfaces cluster-scoped incidents like failed image pulls, liveness probe rejections, or OOM kills, but it does not grab application output. An 'unhealthy' or 'Back-off restarting failed container' event merely indicates the restart mechanism fired, whereas the app error could be an internal exception invisible to the event system. Because events are also aggregated and pruned by TTL, they are less reliable and less direct than kubectl logs for diagnosing a pure application crash.

Create an Ingress with the metrics path

Why wrong: Ingress routes traffic, not for scraping.

Add annotations 'prometheus.io/scrape: "true"' to the Deployment

Why wrong: Deployment is not scraped; pods are.

Store metrics in a ConfigMap and mount it to Prometheus

Why wrong: ConfigMap is static config, not live metrics.

The Pod is not ready yet.

Why wrong: The Pod is not ready yet. This is incorrect because readiness probes only affect Service endpoints and Pod readiness status, not container restarts. A Pod stuck in CrashLoopBackOff has a container that has already exited multiple times, as indicated by the restart count and the CrashLoopBackOff reason itself. A not-ready Pod would remain Running with no restarts, not enter a crash-loop state.

The liveness probe is failing.

Why wrong: The liveness probe is failing. This is incorrect in this specific scenario because a failing liveness probe would kill a running container, and the application would typically have logged some output before the probe timeout occurred. Since `kubectl logs` shows no output at all, the container is crashing before the probe even gets a chance to run, or before the application emits logs. Liveness probe failures can cause CrashLoopBackOff, but they usually leave behind logs from the failed attempt.

The container never started due to a missing image.

Why wrong: The container never started due to a missing image. This is incorrect because a missing or invalid image would result in ImagePullBackOff, not CrashLoopBackOff. In CrashLoopBackOff, the container has successfully started and then exited with a non-zero exit code, whereas ImagePullBackOff means the container was never created because the image could not be pulled. The `kubectl describe pod` output would show image-pull errors, not repeated container exits.

The container is being throttled due to CPU limits.

Why wrong: CPU throttling, whether caused by the container's CPU limit or the node's cpu manager, only limits CPU time and does not prevent the process from binding to a socket or accepting connections at the TCP layer. Even when a container is heavily throttled, the kernel keeps the port open, so a liveness probe that merely opens a TCP connection to port 8080 will still succeed — the probe will connect, even though actual request processing is slow or stalled. Therefore, throttling cannot explain a pod that is 'running but not responding to requests' when the liveness probe is a TCP check; it would only cause application-level timeouts if the probe were HTTP-based.

The probe's initial delay is too short.

Why wrong: A liveness probe's initialDelaySeconds controls how long Kubernetes waits after the container starts before the probe is performed. Setting this value too short will cause the kubelet to probe while the application is still initializing, and if the probe fails, the container will be restarted — leading to a crash loop, not a stable 'running' state. The symptom described — a pod that stays running but doesn't respond — is inconsistent with an overly short initial delay, because that would trigger frequent restarts and likely show a CrashLoopBackOff status, rather than a persistent but unresponsive container.

The liveness probe is misconfigured and should be an HTTP GET.

Why wrong: A TCP liveness probe is a valid and common Kubernetes probe type; it is not misconfigured merely by being TCP. The probe checks whether a connection can be established to the container's port, which is often sufficient for simple services, but it does not make any assessment of the application's status after handshake. Although an HTTP GET probe would be a more effective way to detect this particular failure (by expecting a 200 response within a timeout), the use of a TCP probe is a design choice — not a misconfiguration — and the correct description is that the probe is 'insufficient' rather than 'wrongly set.'

kubectl get events --sort-by=.lastTimestamp

Why wrong: Cluster events track orchestration-level transitions such as BackOff or Killing, but they rarely capture the process's own stderr/stdout, which is where the fatal exception or panic is actually written. Sorting by .lastTimestamp only orders those lifecycle events chronologically and does not expose the application-level error that caused the container to exit. Thus, events can confirm the crash loop, but they do not provide the direct cause.

kubectl exec -it <pod-name> -- sh

Why wrong: kubectl exec requires a running container with a shell or binary available, but a pod in CrashLoopBackOff has a container that is continuously terminating and restarting, so there is no stable live process to attach to. Even during the brief running window, exec often fails or immediately exits if the application shuts down, and the shell may not exist in a distroless image. This command is therefore ineffective for debugging the crash itself and cannot replace log inspection.

kubectl describe pod <pod-name>

Why wrong: kubectl describe pod provides the pod's status, restart count, and recent events, which confirm the CrashLoopBackOff condition and the BackOff reason, but it does not fetch the container logs or application error output. The Last State field may show an exit code, yet the actual error message, like a panic or a fatal config issue, remains visible only in kubectl logs. So describe is useful for cluster-level context, not for the first, root-cause command.

The init container failed to complete.

Why wrong: The init container failed to complete. Init containers run to completion sequentially before any main container starts, and their logs appear under a separate init container log, not the main container's. A failure in an init container would place the pod in an Init:Error or Init:CrashLoopBackOff state, but the log here shows a runtime bind() error from the main application process, which is unrelated to init container execution.

The readiness probe is misconfigured.

Why wrong: The readiness probe is misconfigured. Readiness probe failures only remove the pod from Service endpoints; the container itself continues running and does not crash. An 'address already in use' error occurs during the main process's socket bind at startup, which is a fundamental network conflict, not a misconfiguration of HTTP or TCP probe settings. Probes do not interact with port binding and cannot produce this kind of kernel-level error.

The liveness probe is failing and restarting the container.

Why wrong: The liveness probe is failing and restarting the container. A liveness probe failure causes the kubelet to kill the container and restart it, but the resulting logs show probe timeout, connection refused, or HTTP non-2xx responses, not an 'address already in use' bind error. This error happens during the initial socket creation, before the liveness probe ever runs, and it is a direct consequence of a port conflict, not a health check failure. Restarting due to a liveness probe would not produce this specific system call error.

The liveness probe is misconfigured and should use a TCP check instead.

Why wrong: A liveness probe only determines whether to restart the container based on HTTP response; it does not prevent the kernel from killing a process that exceeds its cgroup memory limit. Even switching to a TCP check would only alter how liveness is evaluated, leaving the memory constraint unchanged. The observed restarts are OOM kills, not probe failures, so the probe type is irrelevant to this issue.

The readiness probe is failing because the application is not ready, but the liveness probe keeps it alive.

Why wrong: A failing readiness probe removes the pod from service load balancers and returns 503s, but it never triggers a container restart; kubelet only restarts on liveness failure or OOM kill. The notion that liveness keeps it alive is also off: a successful liveness probe doesn't override the kernel's OOM killer. Since the restart count matches memory-limit termination, readiness probe behavior is not the cause.

The CPU limit is too low, causing the container to be throttled and timeout.

Why wrong: CPU limits apply CFS throttling, which restricts how much CPU time the container gets but does not terminate the process. If the application needs more CPU, it simply waits in runnable state, causing latency or HTTP timeouts, yet the container remains alive. Restarts require a terminal event like OOMKill, and memory pressure—not CPU throttling—is what produces the observed container resets.

Increase the replica count for each deployment to reduce latency.

Why wrong: Increasing replicas does not improve observability.

Configure readiness probes to ensure only healthy pods receive traffic.

Why wrong: Readiness probes control traffic routing, not observability.

Add liveness probes to all containers to detect when they are unresponsive.

Why wrong: Liveness probes only detect dead containers, not performance issues.

The container has exceeded its memory limit and is being OOMKilled.

Why wrong: When a container exceeds its memory limit, the kernel OOM killer terminates it and the kubelet records lastState.reason as OOMKilled, not CrashLoopBackOff. Although restartPolicy: Always can cause the container to be restarted repeatedly, the termination reason displayed in pod status remains OOMKilled, making it clearly distinguishable from a generic application crash. Use kubectl describe pod to inspect the last termination reason.

The node running the pod is down, preventing the container from starting.

Why wrong: If the node were down, the kubelet would be unavailable to report container status, so the pod would remain in Pending (if it could not be scheduled) or transition to Unknown/NodeLost, not CrashLoopBackOff. CrashLoopBackOff is an active state maintained by a running kubelet that detects container exits and calculates restart backoff. A ReplicaSet controller would reschedule the pod to a healthy node instead of leaving it in this crash loop state.

A network policy is blocking the container from accessing required services.

Why wrong: NetworkPolicies are enforced by the CNI plugin at the pod's network interfaces and only filter traffic; they do not send signals to or terminate container processes. A container blocked by a policy continues running normally but may fail to connect to required services, resulting in timeouts or connection errors—not a CrashLoopBackOff exit loop. This symptom is an operational/connectivity issue, not a process lifecycle issue.