CCNA Troubleshooting Questions

The kubelet is the primary node agent that communicates with the Kubernetes control plane to report node status and heartbeat. Running systemctl status kubelet immediately reveals whether the service is active, inactive, or failing, which is the most direct and efficient first step when troubleshooting a NotReady node via SSH.

Kubernetes only includes pods in a service's endpoints if they are in the Ready state. If the readiness probe fails, the pod is not considered ready, and its IP is not added to the endpoints. Even if the pods are running and have the correct label, they will not receive traffic. Checking the pod status with 'kubectl get pods -n prod -l app=api' and describing them to see readiness probe failures is the next step.

kubectl logs web-pod --previous fetches the log stream from the last terminated container instance, which is exactly where the application's crash output is preserved. The --previous flag reads the container's previous log file that survives restarts, allowing you to see the exception, error, or stack trace that triggered the CrashLoopBackOff rather than an empty or fresh current log.

kubectl describe pod is the correct diagnostic because it displays the pod's status, conditions, container states, and a list of recent Events aggregated for that pod. These Events will contain specific errors such as FailedCreatePodSandBox, ErrImagePull, FailedMount, or CreateContainerError, along with the underlying message from the kubelet or container runtime. This gives you exactly the information needed to resolve ContainerCreating.

kubectl top nodes queries the Metrics API (typically provided by metrics-server) to display the current CPU and memory usage for every node in the cluster. It reports actual consumption values, such as 82% memory, making it the correct command for checking real-time memory usage.

On clusters where control plane components are managed by systemd (e.g., packaged Kubernetes distributions or those set up with kubeadm before static pods became the default), `systemctl status kube-apiserver` queries the service manager for the exact process state, reporting whether the service is active, running, or failed. It also surfaces recent journal logs, making it a direct and authoritative check of the kube-apiserver service. However, this command is only valid when the component actually runs as a systemd unit; on clusters using static pods, it would return an error, so it is environment-specific but nonetheless a correct approach for systemd-based clusters.

When a container's memory usage exceeds its cgroup memory limit, the kernel's OOM killer terminates the container process, and kubelet records the reason as OOMKilled. Even with no explicit request, the limit is enforced as a hard cap, so the container is killed precisely for trying to allocate more memory than that cap. This is the only condition that sets the OOMKilled termination reason.

kubectl describe pod pod-name aggregates the pod's current and recent states under the Status section, including the 'Last State' fields where the container's terminated reason and exit code are explicitly shown. This is the canonical way to inspect why a container crashed, because it presents the last container's exit code (e.g., 137 for SIGKILL or 1 for a general error) along with the reason, message, and timestamps. It also includes recent event history, making it the most direct command to answer the question.

Logs are the first place to look for process exit reasons. `kubectl logs pod-name --previous` retrieves the captured stdout/stderr from the last crashed container instance, whereas the default `kubectl logs` shows the current (possibly non-existent) container. This lets you see the application's actual error, panic, or stack trace that occurred before the crash, even if the current container is in a CrashLoopBackOff state. This makes it a valid and often essential troubleshooting step.

The kube-controller-manager is a client of the Kubernetes API server; every control loop (deployment, replicaset, namespace, service account) lists objects through the API. A repeated 'failed to list' or connection-refused error means the API server is either down, unresponsive, or unreachable from the controller node. This is the only condition that explains the manager's own client logs while the manager process itself remains running.

The error message for a missing tag in a registry is typically 'manifest unknown' or 'not found' (HTTP 404). The question states that the error clearly indicates the manifest is not found for the tag 'myimage:v2', which means that specific tag does not exist in the registry. This is a definitive registry-side response: the registry is reachable, the repository may exist, but the requested tag has no associated manifest.

kubectl top pods and kubectl top nodes are the correct commands because they query the Metrics API (backed by metrics-server) and display real-time CPU and memory utilization for pods and nodes. kubectl top provides immediate, current usage values, which is exactly what 'resource usage' means in a Kubernetes context.

The error "manifest not found" specifically indicates that the container registry was successfully contacted, but it does not contain a schema or manifest matching the requested repository and tag combination. This occurs when the deployment specifies an image tag, such as "latest", that was never pushed to the registry or has been deleted.

kubectl top nodes queries the Metrics API, which is backed by metrics-server, to report each node's current CPU and memory usage as a percentage of allocatable capacity. This is the standard command for quickly assessing real-time node utilization in a cluster. It requires the metrics-server (or a compatible metrics API) to be installed and producing node metrics.

ImagePullBackOff is the kubelet's status when it fails to fetch a container image, and the most frequent cause is a typo in the image name or an invalid/unavailable tag (for example, `nginx:lateest` or `myapp:v2` that does not exist in the registry). The kubelet retries the pull with an exponential backoff, and the underlying error — such as `manifest unknown` or `not found` — is visible in the pod's Events via `kubectl describe pod`. Because the image reference itself is unresolvable, the container never starts and the pod remains in this state until the image reference is corrected.

kubectl top nodes relies on the metrics-server to collect resource usage from each kubelet and expose them through the metrics.k8s.io API. If metrics-server is not installed, the API endpoint returns 404 or no data, and kubectl top prints an error such as "error: metrics not available yet". If metrics-server is installed but not running (CrashLoopBackOff, pending), the same failure occurs because the API aggregator has no healthy backing service. Therefore, verifying the metrics-server deployment and its API service is the first step.

When a container enters CrashLoopBackOff, the kubelet has successfully started the process and the runtime is restarting it because of a non-zero exit. If the application dies immediately—for example, due to a missing executable, an invalid ENTRYPOINT, a panic before logging setup, or an exec format error—there is no stdout/stderr capture for kubectl logs to return. The restart count increments and the container's state becomes Waiting with reason CrashLoopBackOff, but the logs remain empty.

`journalctl -u kubelet` is the correct approach because kubelet is registered as a systemd service under the unit name `kubelet.service` on essentially all managed Kubernetes distributions. This command displays all journal entries associated with that unit, preserving timestamps, priorities, and source metadata. It is the standard first step for diagnosing kubelet failures, and can be combined with flags like `-f`, `-n 100`, or `--since` for targeted live troubleshooting.

`kubectl describe pod <pod-name>` is the correct command because it aggregates the container's current status (Waiting with reason ImagePullBackOff) and the recent pod Events, including the detailed error message from the container runtime, such as "Failed to pull image" with the specific registry response (e.g., "not found" or "unauthorized"). This single command shows the image name, the node, and a timestamped history of pull attempts, making it the fastest path to diagnose why the image pull failed.

`kubectl top pods` is the correct command to view current resource usage. It queries the Metrics API, typically served by metrics-server, which aggregates per-container CPU and memory data from kubelet/cAdvisor. The output shows values like CPU in cores or millicores and memory in bytes or mebibytes, giving an accurate snapshot of live consumption. However, it depends on metrics-server being deployed in the cluster.

kubectl logs pod-name --previous retrieves the log output from the previous instantiation of the container in the same pod. When a container crashes and restarts, the kubelet replaces its log file, so the current logs only contain output from the new, possibly healthy instance. The --previous flag accesses the log file of the terminated container, making it the correct way to see the crash-related output.

A container whose startup command includes invalid arguments (for example, a misspelled flag, undefined positional parameter, or a file path that does not exist) causes the runtime to launch the process only for it to exit immediately with a non-zero exit code. Because the process fails before it can do meaningful work, the kubelet sees a completed/error container and applies the restartPolicy, which rapidly restarts it and transitions to CrashLoopBackOff. The backoff timer doubles on each restart, but the underlying command flaw remains, so the crash loop persists until the arguments are corrected.

The `podAntiAffinity` rule explicitly prevents a pod from being scheduled on a node that already hosts another pod matching specific labels within a defined topology domain. If the `kubectl describe po` output indicates an anti-affinity conflict on the third node, either relaxing the `podAntiAffinity` rule in the pod's specification or removing the existing, conflicting pod from that node would allow the pending pod to be scheduled. This directly resolves the constraint preventing the pod from finding a suitable node.

The events described are exactly what kubectl describe pod would show when the kube-scheduler cannot find a feasible node: a combination of 'Insufficient memory' (the pod's resource requests exceed allocatable memory on every node) and 'node(s) had untolerated taint' (the pod lacks a matching toleration for a taint applied to candidate nodes). Since the scheduler must satisfy both the resource request/limit check and the taint/toleration rule before binding, the pod is never assigned and remains Pending. This is a scheduling failure, not a runtime issue.

The application's logs clearly show an attempt to establish a database connection to the hardcoded IP address "10.0.0.1". In a Kubernetes environment, applications should typically connect to services using their DNS-resolvable service names (e.g., 'mysql' or 'mysql.default.svc.cluster.local') rather than static cluster IPs, which are ephemeral and subject to change. This misconfiguration prevents the application from correctly resolving and connecting to the intended 'mysql' service, leading to the CrashLoopBackOff.

The kubectl logs command uses the --container (or shorthand -c) flag to select a specific container in a multi-container pod. Here, web-pod is the pod name and app-container is the container name, which is the exact syntax required. This command correctly returns only the logs from app-container, making it the right choice.

kubectl top pods --all-namespaces is the correct command because it retrieves current CPU and memory utilization for every pod running in every namespace. It sources these figures from the metrics API (backed by metrics-server) and displays them per pod along with the pod's namespace and node. Because the question asks for resource usage of all pods cluster-wide, this command is the only one that directly provides that data without omitting any namespace.

This is the correct answer because if the CoreDNS pods are not running or are crashing, the Kubernetes `kube-dns` service will have no healthy endpoints. The `kube-proxy` component, responsible for managing service IPs, will be unable to forward DNS queries from pods to any functional CoreDNS instance. Consequently, any DNS lookup attempt from a pod will result in a connection timeout as the query never reaches an active DNS server to be processed.

The --previous flag retrieves log output from the previous terminated container instance. After a pod crashes and the container restarts, the newly started container starts with a fresh log file, so only --previous exposes the output from the failed run that triggered the restart. This is the standard way to diagnose why a container exited.

kubectl top pods is the direct client for the metrics.k8s.io API, which is normally served by metrics-server after it scrapes each node's Summary API from kubelet/cAdvisor. The command shows per-pod CPU usage in cores or millicores and memory usage in MiB, making it the standard way to view live resource consumption from the command line. This is precisely the resource-usage view the user is asking for.

Running 'systemctl start kubelet' is the correct first recovery action because Node NotReady status almost universally means the kubelet is not actively running or has failed to report its status. Starting the kubelet makes it contact the API server, renew its heartbeat, and re-register the node with its capacity and conditions, which allows the control plane to mark it Ready again. Before starting, you should verify the service status and check logs with 'journalctl -u kubelet', but the start command is the minimal and precise fix for a stopped kubelet daemon.

This is the correct command because `kubectl get` supports the `--sort-by` flag, which takes a JSONPath expression evaluated against each returned object. Here `.metadata.creationTimestamp` is the standard field that records when each event object was created, and the client sorts all events by that timestamp before printing them, giving chronological order. Combining the `events` resource with this sort key is the direct, supported way to list all events sorted by time.

journalctl -u kubelet reads the systemd journal for the kubelet service, the daemon that registers the node and reports its status to the control plane. When a node is NotReady, kubelet logs will contain the underlying error, such as a failed heartbeat to the API server, CNI interface issues, or a misconfigured kubeconfig. This is the authoritative source because kubelet runs as a systemd unit and its stderr/stdout are captured in the journal.

kubectl exec first-pod -- ping <second-pod-ip> is the correct diagnostic because it enters the network namespace of the first pod and sends ICMP echo requests directly to the second pod's IP address. This actively tests layer 3 connectivity along the exact path the failing application would use, rather than relying on any proxy, load balancer, or DNS resolution. A successful reply confirms the network route and firewall rules permit traffic, while a failure localizes the problem to the pod-to-pod networking layer.

The --previous flag instructs kubectl to fetch logs from the container's prior terminated instance rather than the current one. In CrashLoopBackOff the running container has just restarted, so only the previous instance holds the crash output needed for diagnosis.

If a node has a taint and the pod has a matching toleration, the Kubernetes scheduler can successfully assign the pod to that node. This scheduling mechanism has no bearing on the container runtime's ability to pull images. If scheduling succeeds, the pod will not enter ImagePullBackOff unless there is an unrelated image retrieval issue.

The 'context deadline exceeded' error indicates that the kubelet's or CRI runtime's operation to create or start the container exceeded its deadline before completing successfully. This is typical when the container runtime hangs during image pull, storage setup, runc init, or other pre-start steps, and after repeated failures the kubelet marks the pod as CrashLoopBackOff. It is a runtime-level timeout rather than an application or command issue, so inspecting the container runtime service and underlying node resources is the appropriate debugging path.

CoreDNS is the cluster's authoritative DNS service, and its Service IP is what kubelet writes into every pod's /etc/resolv.conf as the nameserver. If the CoreDNS Deployment has no available replicas, the Service has no endpoints, or the Corefile is misconfigured (e.g., invalid forward upstreams), queries will time out or return SERVFAIL. This directly explains why a running pod with a valid resolv.conf still cannot resolve any DNS names.

The kubelet is the primary agent responsible for registering the node with the Kubernetes API server and continuously reporting its health, resource utilization, and the status of pods running on it. If the kubelet service is stopped or inactive, it ceases to send heartbeats and status updates to the control plane. This complete lack of communication directly causes the API server to mark the node as `NotReady`, as it can no longer ascertain the node's operational state or manage its workloads.

This is the standard systemd command used to query the active state, process ID, and recent log outputs of the kube-apiserver service when it is run as a system daemon. It directly queries the systemd init manager to verify if the process is running, active, or failing. This is crucial for troubleshooting control plane bootstrap issues on master nodes where components are managed as systemd services.

`kubectl describe node <node-name>` is the correct first command because it displays the node's Conditions section, where the kubelet reports DiskPressure with a True/False status, the last heartbeat time, and a message explaining the eviction signal. It also aggregates related events, capacity, allocatable resources, and configured kubelet pressure thresholds. This directly and authoritatively reveals whether the node is under disk pressure and gives actionable diagnostic context.

The livenessProbe is explicitly configured to send an HTTP request to a specific containerPort, so if that port does not match the port the application is actually listening on, kubelet will receive a connection refused error (or non-2xx response). Kubernetes treats any non-2xx response or connection error as a probe failure; after the failureThreshold is reached, kubelet kills and restarts the container. This produces exactly the observed symptom: the container starts, becomes ready for a moment, then repeatedly restarts while the application itself is healthy on its real port.

The --previous flag tells kubectl to fetch logs from the last terminated instance of a container in the pod, which is exactly what you need when the current container has already crashed and restarted. Since a container that has exited no longer has accessible live logs, this flag retrieves the previously captured log output from the terminated container, allowing you to diagnose the cause of the crash. This is the standard approach for debugging CrashLoopBackOff or a one-time termination before the current container starts. Without the flag, you would only get logs from the current, possibly already-restarted container, missing the crash evidence.

systemctl start kubelet directly addresses the most common cause of a NotReady node: the kubelet service being stopped or inactive. When started, the kubelet initializes, connects to the API server, and begins sending its Node status heartbeat, which the node controller uses to mark the node Ready once all conditions (such as memory, disk, and runtime) are satisfied. This is the immediate, targeted remediation for a node stuck in NotReady due to a down kubelet, and it is the correct first step after confirming the service is not running.

The 'kubelet' runs as a 'systemd' service on each Kubernetes node, responsible for registering the node with the cluster and managing pods. To diagnose issues where a node is not ready, checking the 'kubelet' service status directly on the node using 'systemctl status kubelet' is the most effective first step. This command provides real-time information about whether the service is active, stopped, or failed, along with recent log entries that often pinpoint the root cause of any operational problems.

This is the correct command because it queries the Kubernetes API server for the detailed state, conditions, and event history of the specified Pod. The "Events" section at the bottom of the output will explicitly reveal why the Pod is pending, such as insufficient CPU/memory on nodes, taints and tolerations mismatches, or unbound PersistentVolumeClaims.

A Service's spec.selector uses exact key-value matching to choose backing pods. If the selector is app: payment and the pods are labeled app: payment-service, the values are different, so the Endpoints controller does not add any pod IP to the Service's backend. Label matching is exact, not substring-based, so the Service will have no endpoints and in-cluster clients cannot connect to it.

This command retrieves detailed lifecycle information directly from the Kubernetes API server, including the pod's current state, container statuses, and historical events. Crucially, it displays the "Last State" field, which reveals the exit code and termination message of the previous failed container run. This makes it an essential first-line troubleshooting tool for identifying configuration or startup failures.

The kubelet continuously monitors filesystem usage against eviction thresholds, and DiskPressure is cleared only when usage falls below the configured soft-eviction threshold. Adding physical storage increases available capacity, while cleaning up unused container images, stopped container writable layers, and journal/application logs directly reclaims space. After space is freed, kubelet automatically updates the condition and scheduling on the node resumes.

The event log explicitly reports a taint toleration mismatch, meaning the node carries taints the pod does not tolerate, and simultaneously reports a pod anti-affinity conflict, meaning the pod's scheduling constraints prevent it from co-locating with pods that match its anti-affinity selector. Both of these conditions together are present in the pod's unschedulable event. This answer correctly identifies the two distinct scheduling constraints that block the pod, so it is the right choice.

Connection refused means the TCP connection attempt was actively rejected, which typically happens when no process is listening on the specified port. It could be that the application is listening on a different port, or it is bound only to 127.0.0.1 inside the container, but 'localhost' should resolve to 127.0.0.1, so binding to 127.0.0.1 would still be reachable. However, if the application is listening on a different port or not at all, you get connection refused. Checking the application logs and the actual listening ports inside the container is necessary.

journalctl -u kubelet is the correct first place to look because the kubelet is the component that actually reports NodeReady status to the API server and runs the node's static pods and system components. Kubelet logs will show concrete errors such as failed to GET /healthz, admission rejections, CNI plugin failures, or inability to connect to the API server, making this the most direct source of truth for diagnosing a NotReady node.

nslookup is a dedicated network administration tool specifically designed to query Domain Name System servers to obtain mapping or domain name resource records. In a Kubernetes environment, it allows administrators to directly query the CoreDNS service IP for specific internal records, such as Service A-records or Headless Service SRV records. This direct querying capability makes it the standard utility for isolating and verifying DNS resolution issues within a cluster.

When a container's memory consumption, specifically its Resident Set Size (RSS), exceeds the `memory.limit_in_bytes` enforced by its cgroup, the Linux kernel's Out-Of-Memory (OOM) killer is invoked. This mechanism is designed to protect the host system from memory exhaustion by terminating processes that have surpassed their allocated memory resources. In this specific scenario, the container was terminated precisely because its application attempted to allocate or utilize memory beyond the 128Mi limit, triggering the OOM killer to reclaim system resources.

kubectl describe pod surfaces the container's last terminated state, exit code and events, which reveal why the process exited and triggered the restart loop. Logs show application output but not the termination reason, so describe is the direct diagnostic for CrashLoopBackOff.