# Debugging and Troubleshooting Applications in Kubernetes

> Chapter 9 of the Courseiva CNCF-CKAD curriculum — https://courseiva.com/learn/cncf-ckad/debugging-troubleshooting-applications

**Official objective:** 4.3 — Troubleshoot application failures using kubectl and other tools

## Introduction

Debugging and Troubleshooting Applications in Kubernetes. When you deploy software into a cluster, things will break—it's not a matter of 'if', but 'when'. For the CKAD exam, you must prove you can act like a detective, not just a builder, using 'kubectl' commands to find out why a pod is crashing, why a service isn't reachable, or why an app is slow, and then fix it under time pressure.

## The Chef in a Chaotic Restaurant Kitchen Analogy

A professional chef is the central figure in a busy restaurant kitchen. They don't just cook—they orchestrate. Every ticket that comes in, every ingredient that runs out, every oven that makes a strange noise, and every plate that comes back uneaten is a problem they have to diagnose and fix, all while the dinner rush is happening.

When a waiter brings a plate back with a complaint ('The steak is overcooked'), the chef doesn't just cook a new one. They check the grill temperature, ask the station cook what went wrong, inspect the meat quality, and look at the ticket to see if the order was misread. This is debugging. The chef uses their senses—sight, smell, touch, and taste—just like a Kubernetes admin uses 'kubectl describe', 'kubectl logs', and 'kubectl get events'.

If the kitchen printer keeps jamming (the application crashes), the chef sends the sous-chef to check if the paper is loaded and the power is on (checking the pod's status). If the walk-in freezer is beeping (alerts), they check the temperature gauge (resource metrics). The chef doesn't panic; they follow a mental checklist: what is the symptom, where could the fault be, and what is the quickest way to get food out again? This is exactly the mindset you need when an application in Kubernetes goes down—calm, methodical, and ruthless about finding the root cause instead of just restarting everything.

## Core explanation

In Kubernetes, an 'application' is rarely a single thing. It is usually a collection of 'pods' (the smallest deployable units that run your software 'containers'), 'services' (stable network endpoints that let you reach pods), 'deployments' (which manage the desired state of your pods, like how many copies should be running), and 'ingress' (a way to get external traffic into your cluster). When something goes wrong, you need to figure out which of these parts is failing.

Let us start with the most common tool: 'kubectl'. This is your command-line interface to your Kubernetes cluster. Think of it as a remote control. You will become very familiar with four key actions: 'get', 'describe', 'logs', and 'exec'.

'kubectl get' is your first pass. It shows you the current 'status' of resources. For example, 'kubectl get pods' will list all pods in your current 'namespace' (a virtual cluster within the physical cluster, used to organise resources). You will see a column called 'STATUS'. A healthy pod shows 'Running'. If you see 'CrashLoopBackOff', it means your container inside the pod keeps starting, crashing, and being restarted by Kubernetes. If you see 'Pending', your pod hasn't been placed on a 'node' (a worker machine in the cluster) yet, often because there are not enough resources (CPU or memory) or the node is unhealthy. If you see 'ImagePullBackOff', Kubernetes cannot download the container image from the 'registry' (the storage place for your software images, like Docker Hub).

'kubectl describe' is your magnifying glass. It gives you detailed information about a specific resource. If a pod is 'Pending', running 'kubectl describe pod <pod-name>' will show you 'Events' at the bottom of the output. The events might say 'FailedScheduling' with a reason like '0/2 nodes are available: 1 node(s) had taint and the pod didn't tolerate the taint, 1 node(s) insufficient memory.' This tells you exactly why it is stuck. Always check the 'Events' section first—it is the gossip column of the cluster.

'kubectl logs' shows you the output from your application inside the container. If your pod is 'Running' but your web page is blank, the logs might show a database connection error or a missing configuration file. This is your primary tool for debugging application-level code issues, like a Python traceback or a Java stack trace.

'kubectl exec' lets you run a command inside a running container. This is for deeper investigation. For example, if your app is not connecting to a database, you might 'exec' into the pod and try to curl the database service URL from inside the container. This bypasses network policies and helps you check if the issue is with the service name resolution or a firewall rule.

Beyond pods, you need to check 'Services'. A service provides a stable IP and DNS name for a set of pods. Use 'kubectl describe svc <service-name>' to check its 'Endpoints'. If the endpoints list is empty, it means the service's 'selector' (the labels it uses to find matching pods) does not match any running pods. This is a classic 'why can't I reach my app?' problem.

For external access, check 'Ingress'. Use 'kubectl describe ingress <ingress-name>' to see the rules (e.g., which hostname maps to which service) and whether the ingress 'controller' (the software that implements those rules, like NGINX) has assigned an address.

Finally, remember 'Events' are cluster-wide. Run 'kubectl get events --sort-by='.metadata.creationTimestamp' to see a chronological log of everything that happened recently. This can surface issues like nodes going offline or persistent volumes failing to mount.

The purpose of all this is to replace guesswork with systematic investigation. Before Kubernetes, you might ssh into a server and poke around manually. Now, you use a standardised set of commands to look inside a distributed system without needing to remember each individual server's IP address. It makes troubleshooting scalable and repeatable.

## Real-world context

Imagine you work for an e-commerce company that runs its online store on Kubernetes. It is a Monday morning, and the sales team calls frantically: customers are seeing '500 Internal Server Error' when they try to view product pages. You are on-call.

Your first step is to check if the application is actually running. You open your terminal and run 'kubectl get pods -n production'. You see several pods with status 'CrashLoopBackOff'. That is bad. A crash loop means the container inside the pod is failing immediately after starting.

Next, you get more details on one of the failing pods: 'kubectl describe pod product-page-xyz -n production'. Scrolling to the 'Events' section, you see 'Back-off restarting failed container'. Below that, you see 'Readiness probe failed: HTTP probe failed with status code 500'. This tells you the problem is application-level—the container starts, but a health check (the 'readiness probe') gets a 500 error from the application code itself.

Now you need the application logs. You run 'kubectl logs product-page-xyz -n production --previous'. The '--previous' flag is crucial because it shows the logs from the last crashed instance of the container, not the current one that might have just started and done nothing. The logs reveal a 'java.sql.SQLNonTransientConnectionException: Could not create connection to database server'. The database connection is failing.

You check the database service: 'kubectl get svc -n production'. You see a service called 'db-service'. You describe it: 'kubectl describe svc db-service -n production'. The endpoints are empty. This means the service is configured but there are no pods backing it. You check the database deployment: 'kubectl get pods -n production | grep db'. You see a single pod in 'Pending' state. You describe it and see '0/1 nodes are available: 1 Insufficient memory'. The database pod needs more memory than any node (worker machine) has available.

You contact the infrastructure team to add a bigger node to the cluster. While waiting, you verify that the application pods are not just crashing—they are also not able to run because the database is down. Once the node is added and the database pod starts, you check the endpoints again: they are populated. You check the application pods again: they are now 'Running'. You check the site—it loads. The problem is resolved.

In a real-world scenario, you would also check recent changes. You might run 'kubectl rollout history deployment/product-page -n production' to see if a bad code version was deployed. You might check recent events with 'kubectl get events -n production --sort-by='.lastTimestamp'' to see if a node was cordoned (taken offline for maintenance). The systematic approach saves time and prevents you from restarting things blindly, which would only hide the problem temporarily.

## Exam focus

The CKAD exam tests your ability to diagnose and fix problems in a live Kubernetes cluster under time pressure. You will not be asked abstract theory questions. Instead, you will be given a terminal with a pre-configured 'kubectl' context (your target cluster) and a broken application. Your job is to identify the issue and fix it.

Here are the exact types of questions you can expect:

- A pod is in 'CrashLoopBackOff'. You must look at its logs (using 'kubectl logs --previous') to find the application error, then edit the deployment or configmap to fix the problem.
- A service has no endpoints. You must check the selector labels on the service and the labels on the pods and fix the mismatch.
- An ingress is not routing traffic. You must describe the ingress, find the correct hostname and service backend, and ensure the service exists.
- Pods cannot start because of a resource issue (CPU/memory limits too tight for the node). You must modify the deployment's resource requests.
- A pod is stuck in 'Pending' because of a persistent volume claim (PVC) that cannot be bound. You must check the PVC and storage class.

Common traps the exam sets:

- The '--previous' flag is required when checking logs for a pod that has restarted. If you run 'kubectl logs' without it, you get the current (empty or partial) log, leading you to think nothing is wrong.
- The 'selector' field on a service is a label match. Beginners look for the service name to match something on the pod, but it is actually the 'labels' on the pod that must match the service's 'selector'.
- The 'Events' section at the bottom of 'kubectl describe' output—beginners ignore this section, but it often contains the direct reason for failure (e.g., 'FailedMount' for volumes, 'FailedScheduling' for resources).
- Namespace scope. If you run 'kubectl get pods' without the '-n' flag, you only see pods in the 'default' namespace. The broken pod might be in a different namespace. The exam expects you to know to use '--all-namespaces' or check the correct one.

Key commands to memorise:
- 'kubectl describe pod <name>' (the primary diagnostic tool)
- 'kubectl logs --previous <name>' (for crash loops)
- 'kubectl get events --sort-by='.metadata.creationTimestamp'' (to see cluster-wide history)
- 'kubectl get endpoints <service-name>' (to check if service has backends)
- 'kubectl get node -o wide' (to see node health)
- 'kubectl exec -it <pod-name> -- /bin/sh' (to get a shell inside a container for testing)

The exam stress is real. Practise these commands until they are automatic. Time wasted hunting for the right command is time you do not have.

## Step by step

1. **Identify the scope of the failure** — Run 'kubectl get pods --all-namespaces' to see all pods in the cluster. Look for non-'Running' statuses like 'CrashLoopBackOff', 'Pending', 'ImagePullBackOff', or 'Error'. This gives you a bird's-eye view of which parts of the application are broken.
2. **Deep-dive into a failing pod with describe** — Run 'kubectl describe pod <pod-name> -n <namespace>'. Read the 'Status' section to see the restart count and condition states. Then scroll to the 'Events' section at the bottom. This is often the quickest path to the root cause, e.g., 'FailedScheduling' or 'Readiness probe failed'.
3. **Examine application logs** — If the pod is running but not working correctly, use 'kubectl logs <pod-name> -n <namespace>'. If the pod has restarted, add the '--previous' flag to see the logs from the last terminated container. This reveals application-level errors like missing database connections or code exceptions.
4. **Check service connectivity** — Run 'kubectl describe svc <service-name> -n <namespace>' and check the 'Endpoints' field. If it shows '<none>', the service has no matching pods. Verify the service's 'Selector' labels match the labels on your 'Running' pods using 'kubectl get pods --show-labels'.
5. **Verify ingress and external access** — If the issue is external traffic not reaching the app, run 'kubectl describe ingress <ingress-name> -n <namespace>'. Check that the 'Backend' service name and port are correct. Also verify that an ingress controller pod exists in the cluster (e.g., in the 'ingress-nginx' namespace).

## Comparisons

### kubectl logs vs kubectl logs --previous

**kubectl logs:**
- Shows current container output
- Useful for a running, healthy pod to see live logs
- Will be empty or incomplete if the container has restarted

**kubectl logs --previous:**
- Shows logs from the last terminated instance
- Essential for crash loops to see the error that caused the crash
- Provides the logs from before the restart, often containing the root cause

### Pod status 'Running' vs Pod status 'Ready'

**Pod status 'Running':**
- Indicates the container process is alive
- Does not mean the application is responding to requests
- The pod may still fail readiness probes

**Pod status 'Ready':**
- Indicates all probes (readiness) have passed
- Means the pod is capable of receiving traffic
- A pod can be 'Running' but not 'Ready' if probes fail

### Service selector vs Pod labels

**Service selector:**
- Defined in the service YAML under 'selector'
- Consists of key-value pairs that must match pod labels exactly
- If no pods match, the service has empty endpoints

**Pod labels:**
- Defined in the pod metadata or pod template spec
- Used by services to find which pods to send traffic to
- Can have multiple labels; only those matching the selector matter

## Diagram

_A decision tree for diagnosing common Kubernetes pod failures, starting from a user report and ending with a fix._

```mermaid
flowchart TD
    A[User reports app broken] --> B(Run kubectl get pods --all-namespaces)
    B --> C{Is status Running?}
    C -- Yes --> D[Check logs: kubectl logs <pod>]
    C -- No --> E[Run kubectl describe pod <pod>]
    E --> F[Check Events section]
    F --> G{Type of failure?}
    G -- CrashLoopBackOff --> H[Use kubectl logs --previous]
    G -- Pending --> I[Check node resources / PVC]
    G -- ImagePullBackOff --> J[Check image name & registry auth]
    D --> K[Fix application code or config]
    H --> K
    I --> L[Adjust resources or fix volume]
    J --> M[Correct image reference]
    K --> N[Delete pod to restart with fix]
    L --> N
    M --> N
    N --> O[Verify with curl or browser]
```

## Common misconceptions

- **Misconception:** If a pod status is 'Running', the application inside must be working perfectly. **Reality:** A pod can be 'Running' (the container process is alive) but the application inside might be returning errors, not listening on the correct port, or failing health checks. Always check logs and probes. (Beginners equate 'running process' with 'working software'. In Kubernetes, the status only reflects the container lifecycle, not the application health.)
- **Misconception:** If I delete a broken pod, the problem is fixed because a new one will replace it. **Reality:** Deleting a pod will restart it, but if the root cause is a misconfigured environment variable, a bad image, or insufficient resources, the new pod will fail identically. You must diagnose and fix the underlying issue. (This is a 'quick fix' instinct from restarting a home router. In Kubernetes, a Deployment will recreate the pod with the same broken configuration.)
- **Misconception:** The 'kubectl describe' output is just information overload; I should only look at the status field. **Reality:** The 'Events' section at the bottom of 'kubectl describe' is the most valuable part for troubleshooting. It contains the direct scheduler, kubelet, and controller manager messages explaining exactly why something failed. (The output is long, so beginners skim or skip it entirely. They miss the 'smoking gun' event, like 'FailedScheduling' or 'FailedMount'.)
- **Misconception:** A service named 'my-app' will automatically connect to pods named 'my-app'. **Reality:** A service connects to pods based on matching 'labels', not names. The service's 'selector' field defines which labels to match. If the pods have label 'app: my-app', the service selector must be 'app: my-app'. The name of the pod is irrelevant. (The word 'selector' is abstract. Beginners think naming conventions are implicit, but Kubernetes uses explicit key-value pairs.)
- **Misconception:** If my ingress is configured, external traffic will automatically reach the service. **Reality:** An ingress resource only defines routing rules. It requires an 'ingress controller' (like NGINX Ingress Controller) to actually process those rules. If the controller is not deployed or working, the ingress does nothing. (People confuse the 'Ingress' resource (a set of rules) with the actual software that implements routing. The resource is just a config, not a running process.)

## Key takeaways

- Always start debugging by checking the 'Events' section of 'kubectl describe pod' to see the direct reason for failure.
- Use 'kubectl logs --previous' to see logs from a crashed or restarted container, not just the current one.
- A pod in 'CrashLoopBackOff' means your container is repeatedly failing; check logs and probes, do not just delete the pod.
- A service with empty 'Endpoints' indicates a label mismatch between the service's selector and the pod's labels.
- The 'Pending' pod status usually means the scheduler cannot place it due to resource constraints (CPU/memory) or a volume mounting issue.
- Always specify the namespace with '-n' or '--all-namespaces' to avoid debugging the wrong set of resources.

## FAQ

**What does 'CrashLoopBackOff' mean and how do I fix it?**

It means your container inside the pod keeps crashing immediately after starting. Fix it by running 'kubectl logs --previous <pod-name>' to see the crash reason, then fix the application code or config, then delete the pod so a new one is created with the fix.

**How do I get a shell inside a running Kubernetes pod?**

Use 'kubectl exec -it <pod-name> -- /bin/sh' (or '/bin/bash' if available). This gives you an interactive terminal inside the container, useful for testing network connectivity or checking files.

**My pod is 'Pending'. What do I check first?**

Run 'kubectl describe pod <pod-name>' and look at the 'Events' section. Common reasons are insufficient CPU/memory on nodes or a Persistent Volume Claim that cannot be bound. Check node resources with 'kubectl describe node'.

**Why is my service not working even though pods are running?**

Check the service endpoints with 'kubectl describe svc <service-name>'. If endpoints are empty, the service's 'selector' labels do not match the labels on your pods. Use 'kubectl get pods --show-labels' to see pod labels and fix the selector.

**What is the difference between 'kubectl logs' and 'kubectl logs --previous'?**

'kubectl logs' shows the current standard output of the running container. If the container has restarted, you need '--previous' to see the logs from the last terminated instance, which often contains the error that caused the crash.

**How can I see all events in the cluster?**

Run 'kubectl get events --sort-by='.metadata.creationTimestamp' --all-namespaces'. This shows a chronological list of all events (warnings, failures, and normal operations) across all namespaces, helping you spot patterns.

## Check your understanding

1. **A pod is in 'CrashLoopBackOff'. What two commands do you run in order to diagnose the problem?**

   Answer: First, run 'kubectl describe pod <pod-name>' to check events. Second, run 'kubectl logs --previous <pod-name>' to see the error from the crashed container.

2. **If a service has no endpoints, what is the most likely cause?**

   Answer: The service's 'selector' labels do not match the labels on any running pods. Fix the selector or correct the pod labels so they align.

3. **What is the difference between a pod status of 'Pending' and 'CrashLoopBackOff'?**

   Answer: 'Pending' means the pod has not started running yet (usually due to scheduling or volume issues). 'CrashLoopBackOff' means the pod started but the container keeps crashing and being restarted.

4. **You need to check if a pod can reach a database service by DNS. How do you do that without installing tools?**

   Answer: Use 'kubectl exec -it <pod-name> -- nslookup <service-name> or 'kubectl exec -it <pod-name> -- curl <service-name>:<port>' to test connectivity from inside the container.

5. **A read of 'kubectl get events' shows 'FailedScheduling'. What should you check next?**

   Answer: Check node resources with 'kubectl describe node' to see if there is insufficient CPU or memory, or if a node has taints that the pod does not tolerate.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/cncf-ckad/debugging-troubleshooting-applications