KCNA Cloud Native Observability — 20 Questions

Question 1

A DevOps team notices that a microservice is returning 503 errors intermittently. The service runs in Kubernetes and uses a liveness probe. The team wants to understand the root cause without restarting the pod. Which observability approach should they use first?

Accepted Answer

Query Prometheus for kubelet metrics on probe successes and failures. Option B is correct because Prometheus can scrape kubelet metrics that expose liveness probe success and failure counts directly, allowing the team to see if the probe is failing without restarting the pod. This approach provides historical data on probe behavior, which is essential for diagnosing intermittent 503 errors that stem from the kubelet restarting the container when the liveness probe fails. Unlike other options, it does not require modifying the application or restarting the pod, and it directly surfaces the root cause if the probe is the issue.

Answer

Use kubectl describe pod to check recent events

Answer

Increase log verbosity in the application to capture all requests

Answer

Enable distributed tracing across the service mesh

Question 2

A platform team is designing a monitoring strategy for a multi-tenant Kubernetes cluster. Each tenant runs workloads in separate namespaces. The team needs to ensure tenant isolation while providing aggregated cluster-wide dashboards. Which approach best meets these requirements?

Accepted Answer

Deploy a Prometheus instance per tenant and use Thanos to aggregate metrics globally. Option D is correct because deploying a Prometheus instance per tenant enforces strong tenant isolation by preventing cross-tenant metric access or resource contention, while Thanos provides a global view by aggregating metrics from all tenants via sidecar-based or query-frontend federation. This approach satisfies both isolation and aggregated dashboards without compromising security or scalability.

Answer

Deploy a single Prometheus instance with namespace labels on all metrics

Answer

Use a global Prometheus with recording rules to aggregate per-namespace metrics

Answer

Have each tenant deploy their own monitoring stack and view separately

Question 3

A Kubernetes administrator is troubleshooting a pod that is stuck in CrashLoopBackOff. The pod's restart count is increasing. Which initial step should the administrator take to diagnose the issue?

Accepted Answer

Run 'kubectl logs ' to view the application logs. Option D is correct because when a pod is in CrashLoopBackOff, the immediate priority is to inspect the application logs to understand why the container is failing. `kubectl logs ` retrieves the stdout/stderr output from the container, which typically contains error messages, stack traces, or configuration issues that caused the crash. This is the most direct and efficient first step before deeper investigation.

Answer

Run 'kubectl describe pod <pod-name>' to check events

Answer

Check the Prometheus metrics for the pod's CPU usage

Answer

Run 'kubectl exec -it <pod-name> -- /bin/sh' to inspect the container

Question 4

An organization uses Prometheus and Grafana for monitoring. They want to alert when the 99th percentile of request latency exceeds 500ms for more than 5 minutes. Which PromQL query should they use in the alert rule?

Accepted Answer

histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5. Option B is correct because it uses `histogram_quantile(0.99, rate(...[5m]))` to calculate the 99th percentile request latency over a 5-minute window, matching the requirement to alert when this value exceeds 500ms (0.5 seconds) for more than 5 minutes. The `rate()` function with a 5m range computes the per-second increase of bucket counters over that duration, which is necessary for accurate quantile estimation in Prometheus.

Answer

histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[1m])) > 0.5

Answer

avg(rate(http_request_duration_seconds_bucket[5m])) > 0.5

Answer

max(rate(http_request_duration_seconds_bucket[5m])) > 0.5

Question 5

Which TWO of the following are best practices for structuring log output in cloud-native applications to maximize observability?

Accepted Answer

Output logs in structured format such as JSON. Option C is correct because structured logging (e.g., JSON) enables automated parsing, filtering, and querying by log aggregation tools like Fluentd, Logstash, or cloud-native observability backends (e.g., Elasticsearch, Loki). This format ensures each log entry has consistent key-value pairs, making it machine-readable and facilitating correlation across distributed services without manual text parsing.

Answer

Include verbose debug-level information in every log line

Answer

Use multi-line log entries for detailed error information

Answer

Avoid timestamps to reduce log size

Question 6

Which THREE of the following are valid use cases for distributed tracing in a microservices architecture?

Accepted Answer

Understanding the dependency graph between microservices. Distributed tracing is designed to track the flow of a single request across multiple microservices, recording timing and causality. Option B is correct because tracing systems like Jaeger or Zipkin automatically build a dependency graph by analyzing the parent-child relationships between spans, which reveals how services interact. This is a core use case for understanding service topology and identifying bottlenecks in a distributed system.

Answer

Monitoring CPU and memory usage of each service instance

Answer

Capturing detailed error messages and stack traces

Question 7

A company runs a Kubernetes cluster with 50 worker nodes, each hosting multiple microservices. They use Prometheus for metrics collection and Grafana for dashboards. Recently, the Prometheus server has been experiencing out-of-memory (OOM) kills during peak hours, causing gaps in metric collection. The cluster has a dedicated monitoring namespace. The team has already increased the Prometheus pod's memory limits to 8GB, but OOMs still occur. The metrics retention is set to 15 days. The cardinality of certain metrics (e.g., HTTP request labels with user IDs) is very high. The team needs to resolve the OOM issue without losing critical alerting capability for at least the last 7 days of data. Which action should they take first?

Accepted Answer

Implement recording rules to pre-aggregate high-cardinality metrics at a lower granularity. Option A is correct because recording rules allow Prometheus to pre-aggregate high-cardinality metrics (e.g., HTTP request labels with user IDs) at a lower granularity, reducing the number of unique time series stored in memory. This directly addresses the OOM issue caused by cardinality explosion without discarding raw data entirely, preserving the ability to query aggregated metrics for alerting over the required 7-day window.

Answer

Drop high-cardinality metrics like HTTP request labels using relabel_configs

Answer

Reduce metrics retention to 7 days to free memory

Answer

Enable vertical pod autoscaler for the Prometheus pod

Question 8

A company deploys a microservice application on Kubernetes. They notice that one of the services is returning 5xx errors intermittently. Which observability tool should they use to correlate the errors with resource usage across all pods of that service?

Accepted Answer

Prometheus. Prometheus is the correct choice because it is a monitoring and alerting toolkit designed to collect and store time-series metrics, such as CPU, memory, and request error rates. By querying Prometheus with PromQL, you can correlate 5xx error spikes with resource usage across all pods of a service, as it scrapes metrics from each pod's /metrics endpoint. This direct correlation of application-level errors with infrastructure metrics is not natively provided by the other tools listed.

Answer

Grafana

Answer

Fluentd

Answer

Jaeger

Question 9

A platform team wants to implement observability for a Kubernetes cluster running 500+ microservices. They need to reduce the cost of storing logs while retaining the ability to search for specific error patterns. Which strategy best achieves this?

Accepted Answer

Use structured logging and sample debug logs, retaining error logs fully. Option D is correct because structured logging (e.g., JSON format) enables efficient indexing and querying of logs, while sampling debug logs and retaining error logs fully reduces storage costs without losing critical error patterns. This approach balances observability needs with cost optimization, a key principle in cloud-native environments.

Answer

Increase log retention to one year for compliance

Answer

Store all logs in a centralized Elasticsearch cluster with high retention

Answer

Aggregate logs into a single pod for easier indexing

Question 10

A developer wants to monitor the health of a Kubernetes deployment by checking if the number of ready replicas matches the desired replicas. Which metric from kube-state-metrics should they query?

Accepted Answer

kube_deployment_status_replicas_ready. Option A is correct because `kube_deployment_status_replicas_ready` directly exposes the number of ready replicas for a Deployment, which can be compared against `kube_deployment_spec_replicas` to determine if the desired state matches the actual healthy state. This metric is emitted by kube-state-metrics, which generates Prometheus-compatible metrics from Kubernetes API objects, making it the standard choice for monitoring Deployment health.

Answer

kube_deployment_spec_replicas

Answer

kube_node_status_condition

Answer

kube_pod_container_status_running

Question 11

Which TWO of the following are best practices for implementing observability in a cloud-native environment?

Accepted Answer

Add unique request IDs to logs for end-to-end tracing correlation. Option C is correct because adding unique request IDs (e.g., via OpenTelemetry trace IDs or custom correlation IDs) to logs enables end-to-end tracing across microservices. This allows operators to correlate a single user request as it traverses multiple services, which is essential for debugging distributed systems in a cloud-native environment.

Answer

Store all raw observability data indefinitely for forensic analysis

Answer

Use only metrics and avoid logs to reduce complexity

Answer

Randomly sample all traces and logs to reduce storage

Question 12

You are an SRE managing a Kubernetes cluster with 200 nodes and 10,000 pods. The cluster runs a critical payment processing application. Users report that transactions are occasionally failing with a 'timeout' error. You have Prometheus and Grafana set up for monitoring, and you use Fluentd with Elasticsearch for logging. You notice that during peak hours, the CPU usage of the payment service pods spikes to 90%, but memory usage remains stable. The pod restart count is low. You also see that the response time of the payment service increases significantly during these spikes. You need to identify the root cause and propose a fix. Which course of action is most appropriate?

Accepted Answer

Increase the CPU limits for the payment service pods to allow more CPU resources during spikes. Option D is correct because the CPU usage spikes to 90% during peak hours, indicating that the payment service pods are CPU-bound. Increasing CPU limits allows the pods to burst and utilize more CPU resources, reducing response times and preventing timeouts. This directly addresses the bottleneck without adding unnecessary replicas or changing memory settings.

Answer

Add more replicas of the payment service to distribute the load

Answer

Increase the memory limits for the payment service pods to improve caching

Answer

Implement a circuit breaker pattern to fail fast and avoid timeouts

Question 13

A company is running a microservices application on a Kubernetes cluster. They have noticed that one of the services, 'payment-api', is experiencing intermittent high latency. The team wants to identify the root cause without modifying the application code. Which approach should they take?

Accepted Answer

Implement distributed tracing using tools like Jaeger or Zipkin to trace requests across services.. Option C is correct because distributed tracing with tools like Jaeger or Zipkin allows you to follow a single request as it traverses multiple microservices, identifying exactly which service or call introduces latency. This approach does not require code changes (if the service mesh or sidecar proxy handles instrumentation) and is specifically designed to pinpoint performance bottlenecks in distributed systems, unlike CPU/memory metrics or log analysis which cannot trace a request's end-to-end path.

Answer

Monitor CPU and memory metrics from kube-state-metrics and correlate with latency.

Answer

Increase log verbosity for all services and search for error messages.

Answer

Check node-level metrics using Prometheus Node Exporter.

Question 14

Which TWO of the following are recommended practices for achieving observability in a Kubernetes cluster?

Accepted Answer

Include correlation IDs in structured logs to enable tracing across services.. Option C is correct because including correlation IDs in structured logs is a key observability practice that enables distributed tracing across microservices. In Kubernetes, where requests often traverse multiple pods and services, correlation IDs allow you to link logs from different components into a single transaction flow, which is essential for debugging and understanding system behavior.

Answer

Use a single centralized logging solution to aggregate logs from all components.

Answer

Store all debug logs for a minimum of 90 days for compliance.

Answer

Disable leader election for monitoring components to reduce complexity.

Question 15

Drag and drop the steps to perform a backup of etcd in a Kubernetes cluster into the correct order.

Question 16

Match each Kubernetes security concept to its definition.

Question 17

Which of the following is a core component of the three pillars of observability?

Accepted Answer

Logs. The three pillars of observability are logs, metrics, and traces. Alerting is derived from metrics, not a pillar itself.

Answer

Alerting

Answer

SLIs

Answer

Dashboards

Question 18

What is the primary purpose of Prometheus in cloud native observability?

Accepted Answer

Collect and store metrics. Prometheus is a metric system that collects and stores numeric time-series data.

Answer

Provide distributed tracing

Answer

Visualize data

Answer

Collect and store logs

Question 19

Which command retrieves logs from a specific container named 'sidecar' in a multi-container pod?

Accepted Answer

kubectl logs pod-name -c sidecar. The -c flag specifies the container name.

Answer

kubectl logs pod-name sidecar

Answer

kubectl logs pod-name --container sidecar

Answer

kubectl logs sidecar pod-name

Question 20

In OpenTelemetry, what is the purpose of the Collector component?

Accepted Answer

Receive, process, and export telemetry data. The OpenTelemetry Collector is a vendor-agnostic agent or gateway that receives telemetry data (traces, metrics, logs) from instrumented applications, processes it (e.g., batching, filtering, sampling), and exports it to one or more backends (e.g., Jaeger, Prometheus, or any OTLP-compatible system). It decouples data generation from data export, enabling flexible pipeline management without modifying application code.

Answer

Instrument code automatically

Answer

Visualize traces and metrics

Answer

Aggregate logs from multiple sources