Application Observability: Logs and Metrics

Application observability is the ability to understand the internal state of a system by examining its outputs. For a Kubernetes application, you cannot look inside the container, so you must rely on the data it emits. This data falls into two main categories: logs and metrics.

Logs are a record of discrete events. Each entry in a log is a text line (or a structured object) that describes something that happened at a specific point in time. For example, a web server might write a log line every time it receives a request: '192.168.1.1 - - [10/Oct/2023:13:55:36] "GET /index.html" 200 2326'. This tells you who requested it, when, what they asked for, and the server's response code. Logs are great for investigating a specific failure because they give you the timeline of events leading up to it.

You view logs in Kubernetes using the kubectl logs command. If you have a pod named my-app-7d4f8, you run kubectl logs my-app-7d4f8 to see the output from the main container. If the pod has multiple containers, you specify the container name with a flag (e.g., kubectl logs my-app-7d4f8 -c sidecar). To keep an eye on live output, you use the -f flag (short for 'follow'), which streams new log lines to your terminal, similar to watching a command output in real-time.

Metrics are numerical measurements collected at intervals. They are not events; they are snapshots of a value over time. Common metrics include CPU usage percentage, memory utilisation, request count per second, and error rate. Metrics are aggregated over time (e.g., the average CPU usage over the last 5 minutes) and are displayed on dashboards. They are excellent for identifying trends and anomalies. For example, if memory usage is steadily climbing over 24 hours, that indicates a memory leak, which you would then investigate using logs.

In Kubernetes, basic node-level and pod-level metrics (CPU and memory) are provided by the Metrics Server. This is a cluster add-on that collects resource usage data and makes it available to the kubectl top command. You use kubectl top pod to see the current CPU and memory consumption of your pods. However, the CKAD exam focuses more on the application-level metrics that your application emits, not just the infrastructure metrics. To monitor custom metrics (like request duration), you typically use a monitoring platform like Prometheus, but the exam expects you to understand how to get the metrics that are built-in to Kubernetes.

The key distinction for a developer is this: logs answer 'what happened?', while metrics answer 'how is the system performing?'. For the CKAD, you are not required to set up a full monitoring stack, but you must be able to use kubectl logs to retrieve application output and kubectl top to view resource usage. Additionally, you need to understand the concept of sidecar containers for logging. A sidecar container is a separate container running in the same pod that processes the logs from the main application container. For example, you could have a sidecar that reads the application logs and forwards them to a central logging service like Elasticsearch. The exam might ask you to edit a pod manifest to add a sidecar container for logging purposes.

Finally, remember that containerised applications should write their logs to stdout (standard output) and stderr (standard error). They should not write to a file inside the container, because if the container crashes, that file is lost. Kubernetes automatically captures stdout and stderr and makes them available via kubectl logs. This pattern is called 'logging to stdout/stderr' and is a best practice for 12-factor applications.

Imagine you are a site reliability engineer for an e-commerce website running on Kubernetes. On a typical Tuesday afternoon, you get a notification from your monitoring dashboard that the checkout service's error rate has suddenly increased from 0.1% to 15%. This is a metric alert. You are now in firefighting mode.

Your first step is to look at the metrics to understand the scope. You open your dashboard (e.g., Grafana) and see a spike in HTTP 500 errors from the checkout-api pods. You also notice that CPU usage for those pods is at 95%, compared to a normal 30%. This tells you the problem is likely resource contention, not a bug in the code. However, you need more detail.

Next, you switch to logs. You run kubectl logs -l app=checkout-api --tail=50 to see the last 50 log lines from all pods with the label app=checkout-api. You see a lot of java.lang.OutOfMemoryError exceptions. Now you know the specific error: the Java heap is full. The logs give you the exact stack trace, which points to a memory leak in a new feature that was deployed this morning.

Now you use metrics to confirm. You look at the pod memory metric (kubectl top pod -l app=checkout-api). One pod is using 500MB of the configured 512MB limit. Another pod is using 504MB. The metrics confirm that the pods are hitting their memory limits.

With this evidence, you take action. You could scale the deployment horizontally (add more replicas) to spread the load, but that would only delay the inevitable. You need to fix the memory leak. You identify the offending pod with the highest memory usage and run kubectl logs checkout-api-6f4d8 –previous. This shows the logs from the previous terminated container (the one that crashed due to OutOfMemory), which might have additional clues.

The standard operating procedure in this scenario involves the following steps:

This real-world process shows that logs and metrics are not separate tools; they are complementary. Metrics give you the 'what' (something is wrong), and logs give you the 'why' (this specific exception occurred). A professional always uses both to diagnose effectively.