PCDOE Implementing service monitoring strategies — 10 Questions

Question 1

A team is monitoring a production service on Google Kubernetes Engine (GKE) and notices that a deployment is occasionally returning HTTP 503 errors. The team has set up a ServiceMonitor in Prometheus to scrape metrics from the pods. What is the most likely cause of the intermittent 503 errors?

Accepted Answer

The readiness probes are failing, causing the pods to be removed from the service endpoints.. Intermittent HTTP 503 errors in a GKE deployment typically indicate that the service's endpoints are temporarily unavailable. When a readiness probe fails, Kubernetes removes the pod from the Service's endpoints, causing traffic to be routed to remaining healthy pods. If multiple pods fail their readiness probes simultaneously or in quick succession, the Service may have no available endpoints, resulting in 503 errors for incoming requests.

Answer

The pods are crashing and restarting frequently.

Answer

The Prometheus scrape interval is too long, causing missed metrics.

Answer

The container resource limits are set too low, causing out-of-memory errors.

Question 2

A cloud operations team is implementing monitoring for a microservices application deployed on Compute Engine. They want to create a custom dashboard in Cloud Monitoring that shows the 99th percentile latency of a specific service over the last hour. Which combination of Cloud Monitoring features should they use?

Accepted Answer

Use a distribution metric with the 99th percentile alignment function in a Metrics Explorer chart.. Option B is correct because Cloud Monitoring's distribution metrics inherently store a histogram of values, allowing percentile calculations like the 99th percentile. By selecting the 99th percentile alignment function in a Metrics Explorer chart, the dashboard directly computes and displays the desired latency threshold from the distribution data over the specified time window.

Answer

Use a gauge metric with the max alignment function in a Metrics Explorer chart.

Answer

Use an uptime check metric and configure the latency percentile in the chart.

Answer

Create a logs-based metric from application logs and use the count alignment.

Question 3

An e-commerce platform is using Cloud Load Balancing with a backend service that has a custom health check. The health check is failing intermittently, causing traffic to be routed away from healthy instances. The team has enabled Cloud Logging and wants to diagnose the issue. Which log view should they examine to see the health check probe results?

Accepted Answer

Load balancer logs (type: 'loadbalancing.googleapis.com'). Load balancer logs (type: 'loadbalancing.googleapis.com') contain detailed records of health check probes, including the probe source IP, target instance, response code, and latency. This is the exact log view that captures health check probe results, enabling the team to identify intermittent failures by correlating probe timestamps with instance health status changes.

Answer

VPC flow logs

Answer

Cloud Audit Logs (Admin Activity)

Answer

Instance serial port output logs

Question 4

A DevOps engineer is setting up alerting policies for a critical API service. They want to receive an alert if the error rate exceeds 5% for at least 5 minutes, but only during business hours (9 AM to 5 PM). Which approach should they use?

Accepted Answer

Create an alerting policy with a condition that triggers when the error rate is above 5% for 5 minutes, and configure the notification channel to only send notifications during business hours using a webhook receiver that checks time.. Option B is correct because it uses a single alerting policy with a condition that triggers when the error rate exceeds 5% for 5 minutes, and then controls notification delivery via a webhook receiver that checks the current time. This approach ensures the alert is evaluated continuously (so the 5-minute window is respected) but only notifications are suppressed outside business hours, which is the most reliable way to meet the requirement without missing alert evaluations or relying on external scheduling.

Answer

Create a log-based metric for errors and use a condition with a threshold, then set the alert policy to only run during business hours using the 'condition' schedule.

Answer

Create two separate alert policies, one for business hours and one for off-hours, each with different thresholds.

Answer

Use Cloud Scheduler to enable and disable the alerting policy at the start and end of business hours.

Question 5

A company is running a stateful workload on Compute Engine and has configured a TCP health check on port 8080. The health check is failing, but the application is running and responding on port 8080 when tested manually from within the instance. What is the most likely cause of the health check failure?

Accepted Answer

The firewall rules are not allowing traffic from the health check probe IP ranges.. The health check probes originate from Google's health check systems, which use specific IP ranges (e.g., 35.191.0.0/16, 130.211.0.0/22). If firewall rules on the instance or VPC do not explicitly allow inbound traffic from these probe IP ranges on port 8080, the health check will fail even though the application is running and responding to manual tests from within the instance. This is the most common cause of health check failures when the application itself is healthy.

Answer

The health check is configured to use port 80 instead of port 8080.

Answer

The instance's DNS resolution is failing, causing the health check to use the wrong IP.

Answer

The health check response timeout is set too low (e.g., 1 second).

Question 6

Which TWO of the following are best practices for implementing service monitoring in Google Cloud? (Choose 2)

Accepted Answer

Use Cloud Monitoring uptime checks to verify that services are reachable from external locations.. Option B is correct because Cloud Monitoring uptime checks verify that a service is reachable from external locations by sending HTTP, HTTPS, or TCP requests from Google Cloud's global vantage points. This validates external availability and helps detect regional outages or DNS resolution issues, which is a core best practice for service monitoring.

Answer

Set static alert thresholds without considering historical baselines.

Answer

Use the USE method (Utilization, Saturation, Errors) for service-level monitoring.

Answer

Alert on cause-based metrics (e.g., CPU utilization) rather than symptom-based metrics (e.g., latency).

Question 7

Which THREE of the following are valid approaches to monitor a custom application metric in Cloud Monitoring? (Choose 3)

Accepted Answer

Use the Cloud Monitoring API to write time series data directly.. Option B is correct because the Cloud Monitoring API allows you to write custom metric data directly via the `projects.timeSeries.create` endpoint. This is the most direct programmatic approach, supporting arbitrary metric descriptors and time series data without requiring any agent or intermediary.

Answer

Install the Stackdriver Monitoring agent on a Windows VM and configure custom metric collection in the agent configuration file.

Answer

Use the built-in JMX plugin in the Cloud Monitoring agent to collect Java application metrics.

Question 8

A DevOps engineer runs the command above and gets the output shown. What does this output indicate?

Accepted Answer

An application running on the instance encountered a connection timeout to a backend service.. The output shows a 'Connection timed out' error when attempting to reach a backend service. This indicates that the application on the instance is unable to establish a TCP connection to the specified IP and port, typically due to network issues, firewall rules, or the backend service being down. The error is specific to application-level connectivity, not disk space or authentication.

Answer

The instance's disk is full, causing write errors.

Answer

The instance failed to authenticate with the metadata server.

Answer

A health check probe failed to reach the instance.

Question 9

A team has deployed a Prometheus server on GKE using the configuration above. They expect Prometheus to scrape metrics from pods with the label 'app: my-app' and the annotation 'prometheus.io/scrape: true' on port 8080. However, no metrics are being collected. What is the most likely cause?

Accepted Answer

The relabel_config for port incorrectly constructs the target address; it should use the annotation value directly without appending ':8080'.. Option D is correct because the relabel_config is incorrectly constructing the target address. The configuration likely uses `__meta_kubernetes_annotation_prometheus_io_port` to get the port, but then appends ':8080' statically, overriding the annotation value. If the annotation specifies a different port or is missing, this hardcoded port causes Prometheus to scrape the wrong endpoint or fail entirely. The correct approach is to use the annotation value directly without appending a fixed port, or to use a default port only when the annotation is absent.

Answer

The kubernetes_sd_configs role is set to 'pod' but should be 'endpoints'.

Answer

Prometheus needs to be configured to listen on port 9090 for scraping.

Answer

The keep action for the label 'my-app' is filtering out all pods.

Question 10

You are monitoring a microservices application deployed on Google Kubernetes Engine (GKE) that uses Cloud Monitoring for observability. You notice that the error rate for a critical service has increased, but the CPU and memory usage remain normal. The service uses gRPC and logs are structured. Which Cloud Monitoring tool should you use first to diagnose the root cause of the increased error rate?

Accepted Answer

Logs Explorer to filter logs by error status codes. Option A is correct because Logs Explorer allows you to directly query structured gRPC logs by filtering on error status codes (e.g., gRPC status codes like `UNAVAILABLE`, `INTERNAL`, or `DEADLINE_EXCEEDED`). Since the service uses structured logging, you can quickly isolate the exact error messages and stack traces without needing to pre-configure dashboards or wait for automated grouping. This is the fastest first step to identify the root cause of an increased error rate when CPU and memory are normal, as it points to application-level or dependency issues.

Answer

Service Monitoring to create a custom dashboard

Answer

Error Reporting to automatically group error occurrences

Answer

Metrics Explorer to view error rate and latency charts