Debugging and Troubleshooting Applications in Kubernetes

In Kubernetes, an 'application' is rarely a single thing. It is usually a collection of 'pods' (the smallest deployable units that run your software 'containers'), 'services' (stable network endpoints that let you reach pods), 'deployments' (which manage the desired state of your pods, like how many copies should be running), and 'ingress' (a way to get external traffic into your cluster). When something goes wrong, you need to figure out which of these parts is failing.

Let us start with the most common tool: 'kubectl'. This is your command-line interface to your Kubernetes cluster. Think of it as a remote control. You will become very familiar with four key actions: 'get', 'describe', 'logs', and 'exec'.

'kubectl get' is your first pass. It shows you the current 'status' of resources. For example, 'kubectl get pods' will list all pods in your current 'namespace' (a virtual cluster within the physical cluster, used to organise resources). You will see a column called 'STATUS'. A healthy pod shows 'Running'. If you see 'CrashLoopBackOff', it means your container inside the pod keeps starting, crashing, and being restarted by Kubernetes. If you see 'Pending', your pod hasn't been placed on a 'node' (a worker machine in the cluster) yet, often because there are not enough resources (CPU or memory) or the node is unhealthy. If you see 'ImagePullBackOff', Kubernetes cannot download the container image from the 'registry' (the storage place for your software images, like Docker Hub).

'kubectl describe' is your magnifying glass. It gives you detailed information about a specific resource. If a pod is 'Pending', running 'kubectl describe pod <pod-name>' will show you 'Events' at the bottom of the output. The events might say 'FailedScheduling' with a reason like '0/2 nodes are available: 1 node(s) had taint and the pod didn't tolerate the taint, 1 node(s) insufficient memory.' This tells you exactly why it is stuck. Always check the 'Events' section first—it is the gossip column of the cluster.

'kubectl logs' shows you the output from your application inside the container. If your pod is 'Running' but your web page is blank, the logs might show a database connection error or a missing configuration file. This is your primary tool for debugging application-level code issues, like a Python traceback or a Java stack trace.

'kubectl exec' lets you run a command inside a running container. This is for deeper investigation. For example, if your app is not connecting to a database, you might 'exec' into the pod and try to curl the database service URL from inside the container. This bypasses network policies and helps you check if the issue is with the service name resolution or a firewall rule.

Beyond pods, you need to check 'Services'. A service provides a stable IP and DNS name for a set of pods. Use 'kubectl describe svc <service-name>' to check its 'Endpoints'. If the endpoints list is empty, it means the service's 'selector' (the labels it uses to find matching pods) does not match any running pods. This is a classic 'why can't I reach my app?' problem.

For external access, check 'Ingress'. Use 'kubectl describe ingress <ingress-name>' to see the rules (e.g., which hostname maps to which service) and whether the ingress 'controller' (the software that implements those rules, like NGINX) has assigned an address.

Finally, remember 'Events' are cluster-wide. Run 'kubectl get events --sort-by='.metadata.creationTimestamp' to see a chronological log of everything that happened recently. This can surface issues like nodes going offline or persistent volumes failing to mount.

The purpose of all this is to replace guesswork with systematic investigation. Before Kubernetes, you might ssh into a server and poke around manually. Now, you use a standardised set of commands to look inside a distributed system without needing to remember each individual server's IP address. It makes troubleshooting scalable and repeatable.

Imagine you work for an e-commerce company that runs its online store on Kubernetes. It is a Monday morning, and the sales team calls frantically: customers are seeing '500 Internal Server Error' when they try to view product pages. You are on-call.

Your first step is to check if the application is actually running. You open your terminal and run 'kubectl get pods -n production'. You see several pods with status 'CrashLoopBackOff'. That is bad. A crash loop means the container inside the pod is failing immediately after starting.

Next, you get more details on one of the failing pods: 'kubectl describe pod product-page-xyz -n production'. Scrolling to the 'Events' section, you see 'Back-off restarting failed container'. Below that, you see 'Readiness probe failed: HTTP probe failed with status code 500'. This tells you the problem is application-level—the container starts, but a health check (the 'readiness probe') gets a 500 error from the application code itself.

Now you need the application logs. You run 'kubectl logs product-page-xyz -n production --previous'. The '--previous' flag is crucial because it shows the logs from the last crashed instance of the container, not the current one that might have just started and done nothing. The logs reveal a 'java.sql.SQLNonTransientConnectionException: Could not create connection to database server'. The database connection is failing.

You check the database service: 'kubectl get svc -n production'. You see a service called 'db-service'. You describe it: 'kubectl describe svc db-service -n production'. The endpoints are empty. This means the service is configured but there are no pods backing it. You check the database deployment: 'kubectl get pods -n production | grep db'. You see a single pod in 'Pending' state. You describe it and see '0/1 nodes are available: 1 Insufficient memory'. The database pod needs more memory than any node (worker machine) has available.

You contact the infrastructure team to add a bigger node to the cluster. While waiting, you verify that the application pods are not just crashing—they are also not able to run because the database is down. Once the node is added and the database pod starts, you check the endpoints again: they are populated. You check the application pods again: they are now 'Running'. You check the site—it loads. The problem is resolved.

In a real-world scenario, you would also check recent changes. You might run 'kubectl rollout history deployment/product-page -n production' to see if a bad code version was deployed. You might check recent events with 'kubectl get events -n production --sort-by='.lastTimestamp'' to see if a node was cordoned (taken offline for maintenance). The systematic approach saves time and prevents you from restarting things blindly, which would only hide the problem temporarily.