KCNA Cloud Native Observability — 15 Questions

Question 1

A DevOps team notices that a microservice is returning 503 errors intermittently. The service runs in Kubernetes and uses a liveness probe. The team wants to understand the root cause without restarting the pod. Which observability approach should they use first?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Query Prometheus for kubelet metrics on probe successes and failures. Prometheus can scrape kubelet metrics that expose liveness probe success and failure counts directly, allowing the team to see if the probe is failing without restarting the pod. This approach provides historical data on probe behavior, which is essential for diagnosing intermittent 503 errors that stem from the kubelet restarting the container when the liveness probe fails. Unlike other options, it does not require modifying the application or restarting the pod, and it directly surfaces the root cause if the probe is the issue.

Answer

Use kubectl describe pod to check recent events

Answer

Increase log verbosity in the application to capture all requests

Answer

Enable distributed tracing across the service mesh

Question 2

A platform team is designing a monitoring strategy for a multi-tenant Kubernetes cluster. Each tenant runs workloads in separate namespaces. The team needs to ensure tenant isolation while providing aggregated cluster-wide dashboards. Which approach best meets these requirements?

Accepted Answer

Deploy a Prometheus instance per tenant and use Thanos to aggregate metrics globally. Deploying a Prometheus instance per tenant enforces strong tenant isolation by preventing cross-tenant metric access or resource contention, while Thanos provides a global view by aggregating metrics from all tenants via sidecar-based or query-frontend federation. This approach satisfies both isolation and aggregated dashboards without compromising security or scalability.

Answer

Deploy a single Prometheus instance with namespace labels on all metrics

Answer

Use a global Prometheus with recording rules to aggregate per-namespace metrics

Answer

Have each tenant deploy their own monitoring stack and view separately

Question 3

A Kubernetes administrator is troubleshooting a pod that is stuck in CrashLoopBackOff. The pod's restart count is increasing. Which initial step should the administrator take to diagnose the issue?

Accepted Answer

Run 'kubectl logs ' to view the application logs. When a pod is in CrashLoopBackOff, the immediate priority is to inspect the application logs to understand why the container is failing. `kubectl logs ` retrieves the stdout/stderr output from the container, which typically contains error messages, stack traces, or configuration issues that caused the crash. This is the most direct and efficient first step before deeper investigation.

Answer

Run 'kubectl describe pod <pod-name>' to check events

Answer

Check the Prometheus metrics for the pod's CPU usage

Answer

Run 'kubectl exec -it <pod-name> -- /bin/sh' to inspect the container

Question 4

An organization uses Prometheus and Grafana for monitoring. They want to alert when the 99th percentile of request latency exceeds 500ms for more than 5 minutes. Which PromQL query should they use in the alert rule?

Accepted Answer

histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5. It uses `histogram_quantile(0.99, rate(...[5m]))` to calculate the 99th percentile request latency over a 5-minute window, matching the requirement to alert when this value exceeds 500ms (0.5 seconds) for more than 5 minutes. The `rate()` function with a 5m range computes the per-second increase of bucket counters over that duration, which is necessary for accurate quantile estimation in Prometheus.

Answer

histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[1m])) > 0.5

Answer

avg(rate(http_request_duration_seconds_bucket[5m])) > 0.5

Answer

max(rate(http_request_duration_seconds_bucket[5m])) > 0.5

Question 5

A company runs a Kubernetes cluster with 50 worker nodes, each hosting multiple microservices. They use Prometheus for metrics collection and Grafana for dashboards. Recently, the Prometheus server has been experiencing out-of-memory (OOM) kills during peak hours, causing gaps in metric collection. The cluster has a dedicated monitoring namespace. The team has already increased the Prometheus pod's memory limits to 8GB, but OOMs still occur. The metrics retention is set to 15 days. The cardinality of certain metrics (e.g., HTTP request labels with user IDs) is very high. The team needs to resolve the OOM issue without losing critical alerting capability for at least the last 7 days of data. Which action should they take first?

Accepted Answer

Implement recording rules to pre-aggregate high-cardinality metrics at a lower granularity. Recording rules allow Prometheus to pre-aggregate high-cardinality metrics (e.g., HTTP request labels with user IDs) at a lower granularity, reducing the number of unique time series stored in memory. This directly addresses the OOM issue caused by cardinality explosion without discarding raw data entirely, preserving the ability to query aggregated metrics for alerting over the required 7-day window.

Answer

Drop high-cardinality metrics like HTTP request labels using relabel_configs

Answer

Reduce metrics retention to 7 days to free memory

Answer

Enable vertical pod autoscaler for the Prometheus pod

Question 6

A company deploys a microservice application on Kubernetes. They notice that one of the services is returning 5xx errors intermittently. Which observability tool should they use to correlate the errors with resource usage across all pods of that service?

Accepted Answer

Prometheus. Prometheus is the correct choice because it is a monitoring and alerting toolkit designed to collect and store time-series metrics, such as CPU, memory, and request error rates. By querying Prometheus with PromQL, you can correlate 5xx error spikes with resource usage across all pods of a service, as it scrapes metrics from each pod's /metrics endpoint. This direct correlation of application-level errors with infrastructure metrics is not natively provided by the other tools listed.

Answer

Grafana

Answer

Fluentd

Answer

Jaeger

Question 7

You are an SRE managing a Kubernetes cluster with 200 nodes and 10,000 pods. The cluster runs a critical payment processing application. Users report that transactions are occasionally failing with a 'timeout' error. You have Prometheus and Grafana set up for monitoring, and you use Fluentd with Elasticsearch for logging. You notice that during peak hours, the CPU usage of the payment service pods spikes to 90%, but memory usage remains stable. The pod restart count is low. You also see that the response time of the payment service increases significantly during these spikes. You need to identify the root cause and propose a fix. Which course of action is most appropriate?

Accepted Answer

Increase the CPU limits for the payment service pods to allow more CPU resources during spikes. The CPU usage spikes to 90% during peak hours, indicating that the payment service pods are CPU-bound. Increasing CPU limits allows the pods to burst and utilize more CPU resources, reducing response times and preventing timeouts. This directly addresses the bottleneck without adding unnecessary replicas or changing memory settings.

Answer

Add more replicas of the payment service to distribute the load

Answer

Increase the memory limits for the payment service pods to improve caching

Answer

Implement a circuit breaker pattern to fail fast and avoid timeouts

Question 8

Match each Kubernetes security concept to its definition.

Accepted Answer

RBAC: Controls access to the Kubernetes API based on roles and permissions.. RBAC and NetworkPolicy are fundamental security mechanisms. ServiceAccount provides identity. The distractors confuse RBAC with secrets management and NetworkPolicy with user access control.

Answer

RBAC: Manages secrets and encryption at rest.

Answer

NetworkPolicy: Controls which users can access the cluster.

Question 9

Which command retrieves logs from a specific container named 'sidecar' in a multi-container pod?

Accepted Answer

kubectl logs pod-name -c sidecar. The -c flag specifies the container name.

Answer

kubectl logs pod-name sidecar

Answer

kubectl logs pod-name --container sidecar

Answer

kubectl logs sidecar pod-name

Question 10

Which TWO are components of a distributed trace? (Select two.)

Accepted Answer

Spans. A distributed trace is composed of spans, which represent individual units of work within the trace. The trace itself is the overarching collection of spans, not a component. Metrics, logs, and alerts are separate observability signals and not components of a trace.

Answer

Metrics

Answer

Traces

Answer

Logs

Answer

Alerts

Question 11

A developer wants to view the logs of a specific container named 'sidecar' in a pod called 'web-app'. Which command should they use?

Accepted Answer

kubectl logs web-app -c sidecar. The -c flag specifies the container name when a pod has multiple containers.

Answer

kubectl logs sidecar --pod web-app

Answer

kubectl logs -c sidecar web-app

Answer

kubectl logs sidecar web-app

Question 12

A company uses Prometheus for monitoring and wants to alert when the average CPU usage over 5 minutes exceeds 80%. Which PromQL query would correctly define this alert rule?

Accepted Answer

avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) > 0.8. The correct query uses rate() over a 5-minute window on the non-idle CPU counter, then averages across all CPU series. node_cpu_seconds_total is a counter, so you must apply rate() to get per-second usage; filtering mode!="idle" isolates busy CPU time, and avg() aggregates across cores and instances. Comparing the resulting fraction to 0.8 expresses 80% utilization.

Answer

avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) > 0.8

Answer

avg(node_cpu_seconds_total{mode!="idle"}[5m]) > 0.8

Answer

sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) > 0.8

Question 13

A team wants to set up alerts when a Kubernetes pod consumes more than 90% of its memory limit for over 5 minutes. They use Prometheus and Alertmanager. Which Prometheus query would trigger an alert for a specific pod named 'web-app' in the 'default' namespace?

Accepted Answer

container_memory_usage_bytes{pod='web-app'} / container_spec_memory_limit_bytes{pod='web-app'} > 0.9. It calculates the ratio of actual memory usage to the configured memory limit for the pod, then checks if that ratio exceeds 0.9 (90%). This directly answers the requirement to alert when a pod consumes more than 90% of its memory limit. The query uses two Prometheus metrics: container_memory_usage_bytes (current usage) and container_spec_memory_limit_bytes (the limit), both filtered by pod name. Dividing them yields a dimensionless fraction that can be compared to 0.9.

Answer

container_memory_usage_bytes{pod='web-app'} > 0.9

Answer

avg(container_memory_usage_bytes{pod='web-app'}) > 0.9

Answer

container_memory_limit_bytes{pod='web-app'} > 0.9

Question 14

Which TWO of the following are common log aggregation tools used in Kubernetes environments? (Select two)

Accepted Answer

Fluentd. Fluentd (B) is correct because it is a widely deployed open-source log aggregator and unified logging layer in Kubernetes, commonly run as a DaemonSet to collect container logs from /var/log/containers and forward them to backends such as Elasticsearch or Loki. Fluent Bit (E) is correct because it is a lightweight, C-based log processor and forwarder designed for resource-constrained environments, frequently used in Kubernetes as a DaemonSet or sidecar to collect and ship container logs. Loki (A) is a log aggregation system, but it is a storage/query backend rather than a log collection and forwarding agent, so it is not the aggregation tool being asked about here. Prometheus (C) is a metrics monitoring and time-series database system, not a log aggregation tool. Jaeger (D) is a distributed tracing system for request tracing, not log aggregation.

Answer

Loki

Answer

Prometheus

Answer

Jaeger

Question 15

What is the primary purpose of structured logging?

Accepted Answer

To enable automated analysis and querying of logs. Structured logging formats log data in a consistent, machine-parseable format (e.g., JSON) with key-value pairs. This enables automated tools like Elasticsearch, Loki, or Splunk to efficiently index, search, filter, and aggregate logs, which is essential for observability at scale. The primary purpose is to facilitate automated analysis and querying, not to replace other telemetry signals or to focus on human readability alone.

Answer

To replace metrics and traces

Answer

To reduce the size of log files

Answer

To make logs human-readable only