PCDOE Implementing service monitoring strategies — 40 Questions

Question 1

A team is monitoring a production service on Google Kubernetes Engine (GKE) and notices that a deployment is occasionally returning HTTP 503 errors. The team has set up a ServiceMonitor in Prometheus to scrape metrics from the pods. What is the most likely cause of the intermittent 503 errors?

Accepted Answer

The readiness probes are failing, causing the pods to be removed from the service endpoints.. Intermittent HTTP 503 errors in a GKE deployment typically indicate that the service's endpoints are temporarily unavailable. When a readiness probe fails, Kubernetes removes the pod from the Service's endpoints, causing traffic to be routed to remaining healthy pods. If multiple pods fail their readiness probes simultaneously or in quick succession, the Service may have no available endpoints, resulting in 503 errors for incoming requests.

Answer

The pods are crashing and restarting frequently.

Answer

The Prometheus scrape interval is too long, causing missed metrics.

Answer

The container resource limits are set too low, causing out-of-memory errors.

Question 2

A cloud operations team is implementing monitoring for a microservices application deployed on Compute Engine. They want to create a custom dashboard in Cloud Monitoring that shows the 99th percentile latency of a specific service over the last hour. Which combination of Cloud Monitoring features should they use?

Accepted Answer

Use a distribution metric with the 99th percentile alignment function in a Metrics Explorer chart.. Option B is correct because Cloud Monitoring's distribution metrics inherently store a histogram of values, allowing percentile calculations like the 99th percentile. By selecting the 99th percentile alignment function in a Metrics Explorer chart, the dashboard directly computes and displays the desired latency threshold from the distribution data over the specified time window.

Answer

Use a gauge metric with the max alignment function in a Metrics Explorer chart.

Answer

Use an uptime check metric and configure the latency percentile in the chart.

Answer

Create a logs-based metric from application logs and use the count alignment.

Question 3

An e-commerce platform is using Cloud Load Balancing with a backend service that has a custom health check. The health check is failing intermittently, causing traffic to be routed away from healthy instances. The team has enabled Cloud Logging and wants to diagnose the issue. Which log view should they examine to see the health check probe results?

Accepted Answer

Load balancer logs (type: 'loadbalancing.googleapis.com'). Load balancer logs (type: 'loadbalancing.googleapis.com') contain detailed records of health check probes, including the probe source IP, target instance, response code, and latency. This is the exact log view that captures health check probe results, enabling the team to identify intermittent failures by correlating probe timestamps with instance health status changes.

Answer

VPC flow logs

Answer

Cloud Audit Logs (Admin Activity)

Answer

Instance serial port output logs

Question 4

A DevOps engineer is setting up alerting policies for a critical API service. They want to receive an alert if the error rate exceeds 5% for at least 5 minutes, but only during business hours (9 AM to 5 PM). Which approach should they use?

Accepted Answer

Create an alerting policy with a condition that triggers when the error rate is above 5% for 5 minutes, and configure the notification channel to only send notifications during business hours using a webhook receiver that checks time.. Option B is correct because it uses a single alerting policy with a condition that triggers when the error rate exceeds 5% for 5 minutes, and then controls notification delivery via a webhook receiver that checks the current time. This approach ensures the alert is evaluated continuously (so the 5-minute window is respected) but only notifications are suppressed outside business hours, which is the most reliable way to meet the requirement without missing alert evaluations or relying on external scheduling.

Answer

Create a log-based metric for errors and use a condition with a threshold, then set the alert policy to only run during business hours using the 'condition' schedule.

Answer

Create two separate alert policies, one for business hours and one for off-hours, each with different thresholds.

Answer

Use Cloud Scheduler to enable and disable the alerting policy at the start and end of business hours.

Question 5

A company is running a stateful workload on Compute Engine and has configured a TCP health check on port 8080. The health check is failing, but the application is running and responding on port 8080 when tested manually from within the instance. What is the most likely cause of the health check failure?

Accepted Answer

The firewall rules are not allowing traffic from the health check probe IP ranges.. The health check probes originate from Google's health check systems, which use specific IP ranges (e.g., 35.191.0.0/16, 130.211.0.0/22). If firewall rules on the instance or VPC do not explicitly allow inbound traffic from these probe IP ranges on port 8080, the health check will fail even though the application is running and responding to manual tests from within the instance. This is the most common cause of health check failures when the application itself is healthy.

Answer

The health check is configured to use port 80 instead of port 8080.

Answer

The instance's DNS resolution is failing, causing the health check to use the wrong IP.

Answer

The health check response timeout is set too low (e.g., 1 second).

Question 6

Which TWO of the following are best practices for implementing service monitoring in Google Cloud? (Choose 2)

Accepted Answer

Use Cloud Monitoring uptime checks to verify that services are reachable from external locations.. Option B is correct because Cloud Monitoring uptime checks verify that a service is reachable from external locations by sending HTTP, HTTPS, or TCP requests from Google Cloud's global vantage points. This validates external availability and helps detect regional outages or DNS resolution issues, which is a core best practice for service monitoring.

Answer

Set static alert thresholds without considering historical baselines.

Answer

Use the USE method (Utilization, Saturation, Errors) for service-level monitoring.

Answer

Alert on cause-based metrics (e.g., CPU utilization) rather than symptom-based metrics (e.g., latency).

Question 7

Which THREE of the following are valid approaches to monitor a custom application metric in Cloud Monitoring? (Choose 3)

Accepted Answer

Use the Cloud Monitoring API to write time series data directly.. Option B is correct because the Cloud Monitoring API allows you to write custom metric data directly via the `projects.timeSeries.create` endpoint. This is the most direct programmatic approach, supporting arbitrary metric descriptors and time series data without requiring any agent or intermediary.

Answer

Install the Stackdriver Monitoring agent on a Windows VM and configure custom metric collection in the agent configuration file.

Answer

Use the built-in JMX plugin in the Cloud Monitoring agent to collect Java application metrics.

Question 8

A DevOps engineer runs the command above and gets the output shown. What does this output indicate?

Accepted Answer

An application running on the instance encountered a connection timeout to a backend service.. The output shows a 'Connection timed out' error when attempting to reach a backend service. This indicates that the application on the instance is unable to establish a TCP connection to the specified IP and port, typically due to network issues, firewall rules, or the backend service being down. The error is specific to application-level connectivity, not disk space or authentication.

Answer

The instance's disk is full, causing write errors.

Answer

The instance failed to authenticate with the metadata server.

Answer

A health check probe failed to reach the instance.

Question 9

A team has deployed a Prometheus server on GKE using the configuration above. They expect Prometheus to scrape metrics from pods with the label 'app: my-app' and the annotation 'prometheus.io/scrape: true' on port 8080. However, no metrics are being collected. What is the most likely cause?

Accepted Answer

The relabel_config for port incorrectly constructs the target address; it should use the annotation value directly without appending ':8080'.. Option D is correct because the relabel_config is incorrectly constructing the target address. The configuration likely uses `__meta_kubernetes_annotation_prometheus_io_port` to get the port, but then appends ':8080' statically, overriding the annotation value. If the annotation specifies a different port or is missing, this hardcoded port causes Prometheus to scrape the wrong endpoint or fail entirely. The correct approach is to use the annotation value directly without appending a fixed port, or to use a default port only when the annotation is absent.

Answer

The kubernetes_sd_configs role is set to 'pod' but should be 'endpoints'.

Answer

Prometheus needs to be configured to listen on port 9090 for scraping.

Answer

The keep action for the label 'my-app' is filtering out all pods.

Question 10

You are monitoring a microservices application deployed on Google Kubernetes Engine (GKE) that uses Cloud Monitoring for observability. You notice that the error rate for a critical service has increased, but the CPU and memory usage remain normal. The service uses gRPC and logs are structured. Which Cloud Monitoring tool should you use first to diagnose the root cause of the increased error rate?

Accepted Answer

Logs Explorer to filter logs by error status codes. Option A is correct because Logs Explorer allows you to directly query structured gRPC logs by filtering on error status codes (e.g., gRPC status codes like `UNAVAILABLE`, `INTERNAL`, or `DEADLINE_EXCEEDED`). Since the service uses structured logging, you can quickly isolate the exact error messages and stack traces without needing to pre-configure dashboards or wait for automated grouping. This is the fastest first step to identify the root cause of an increased error rate when CPU and memory are normal, as it points to application-level or dependency issues.

Answer

Service Monitoring to create a custom dashboard

Answer

Error Reporting to automatically group error occurrences

Answer

Metrics Explorer to view error rate and latency charts

Question 11

A company uses Cloud Monitoring to track latency for a multi-region web application. The SLO is 99.9% of requests under 500ms over a 30-day rolling window. The error budget has been rapidly depleting over the last week. The operations team wants to understand the impact of recent deployments. Which approach should they use to correlate deployment changes with latency spikes?

Accepted Answer

Create a custom dashboard in Cloud Monitoring that includes latency charts and use annotation markers to indicate deployment times. Option C is correct because Cloud Monitoring supports custom dashboards with annotation markers that can be programmatically or manually added to indicate deployment events. By overlaying these markers on latency charts, the operations team can visually correlate deployment times with latency spikes, enabling direct root-cause analysis without manual log searching or separate tools.

Answer

Use Cloud Logging to search for deployment logs and manually compare with latency metrics

Answer

Use Cloud Trace to analyze latency distributions for each deployment version

Answer

Configure Error Reporting to alert on latency threshold breaches

Question 12

You are setting up alerting for a batch processing job that runs daily on Compute Engine. The job must complete within 2 hours. Which metric and alert condition should you use to ensure you are notified if the job is still running after 90 minutes?

Accepted Answer

Create a custom metric that emits 1 when the job starts and 0 when it finishes; alert if the metric is 1 for more than 90 minutes. Option B is correct because it directly monitors the job's running state using a custom metric that emits 1 at job start and 0 at completion. By alerting when the metric remains at 1 for more than 90 minutes, you are notified if the job exceeds the 90-minute threshold, ensuring you catch failures before the 2-hour deadline. This approach is precise and avoids false positives from indirect signals like CPU or logs.

Answer

Alert on CPU utilization greater than 80% for the instance running the job

Answer

Use a heartbeat metric that reports every 5 minutes; alert if no heartbeat for 90 minutes

Answer

Set up a log-based metric that counts job completion log entries; alert if the count is zero after 90 minutes

Question 13

Which TWO metrics should be included in a comprehensive monitoring strategy for a production Kubernetes workload to detect performance degradation and capacity issues?

Accepted Answer

Container CPU utilization. Container CPU utilization (Option B) is a direct indicator of resource pressure and potential performance degradation in a Kubernetes workload. High CPU utilization can lead to throttling, increased request latency, and pod evictions, making it essential for detecting capacity issues. Request latency percentiles (Option E) are the gold standard for measuring user-facing performance degradation, as they reflect the actual experience of end users and can reveal subtle slowdowns before resource metrics show saturation.

Answer

Disk read IOPS per pod

Answer

Number of nodes in the cluster

Answer

Network bytes received per second

Question 14

Your organization runs a critical e-commerce platform on Google Kubernetes Engine (GKE). The platform uses Cloud Service Mesh (Anthos Service Mesh) for traffic management and Cloud Monitoring for observability. Recently, after a new release, you observe that the p99 latency of the checkout service has increased from 200ms to 2s. The service's CPU and memory metrics appear normal, and there are no error logs. The release included a change to the Istio VirtualService configuration that added a retry policy: 3 retries with a 500ms timeout per retry. You suspect that the retries are contributing to the latency increase. You want to use Cloud Monitoring to confirm this hypothesis. Which approach should you take?

Accepted Answer

Use Metrics Explorer to query the istio.io/service/server/request_count metric, filtered by response_code_class and destination_service, and include the istio.io/service/server/request_retries metric to see retry counts alongside latency. Option C is correct because it directly correlates retry attempts with latency by querying the `istio.io/service/server/request_retries` metric alongside the `istio.io/service/server/request_count` metric in Metrics Explorer. This allows you to visualize the retry count per destination service (checkout) and compare it with the p99 latency increase, confirming whether the retry policy is causing the observed latency spike. The retry policy (3 retries with 500ms timeout) can add up to 1.5s of additional latency per request, which aligns with the increase from 200ms to 2s.

Answer

Use Cloud Trace to analyze distributed traces for the checkout service and look for retry spans

Answer

Check the 'Services' dashboard in Cloud Monitoring, which shows a pre-built latency chart for all services

Answer

Use Logs Explorer to search for logs containing 'retry' in the checkout service namespace

Question 15

You are a DevOps engineer for a SaaS company that provides a REST API. The API is deployed on Google Cloud Run. You have configured Cloud Monitoring alerts for 5xx errors. Recently, you received an alert that the error rate exceeded 5% for 5 minutes. You investigated and found that the errors were HTTP 503 (Service Unavailable) from a specific endpoint. The endpoint calls an internal Cloud SQL database. The database CPU utilization was at 90% during that period. You suspect the database is the bottleneck. Which action should you take to reduce the error rate without over-provisioning?

Accepted Answer

Implement connection pooling and retry logic with exponential backoff in the API service. Option A is correct because implementing connection pooling and retry logic with exponential backoff directly addresses the database bottleneck without over-provisioning. Connection pooling reduces the number of concurrent connections to Cloud SQL, lowering CPU contention, while exponential backoff prevents thundering herd retries that could further overwhelm the database. This approach optimizes existing resources rather than scaling infrastructure.

Answer

Increase the max instances per revision in Cloud Run to handle more concurrent requests

Answer

Reduce the min instances of Cloud Run to decrease load on the database

Answer

Add a Cloud SQL read replica and route read queries to it

Question 16

A company uses Cloud Run for a critical service and needs to set up alerting for 5xx errors. They want to receive a notification within 1 minute of the error rate exceeding 1% for any 1-minute window. Which alerting approach should they use?

Accepted Answer

Create a Cloud Monitoring alerting policy using the 'Request count' metric with a condition that compares the ratio of 5xx responses to total requests over a 1-minute window.. Option D is correct because Cloud Monitoring's alerting policies can directly compute the ratio of 5xx responses to total requests using the 'Request count' metric with a custom ratio condition over a 1-minute window. This approach meets the 1-minute notification latency requirement without additional infrastructure, as Cloud Monitoring evaluates the metric every 60 seconds and triggers alerts based on the sliding window.

Answer

Set up a log-based metric for 5xx responses and create an alert on the metric.

Answer

Create a Cloud Logging sink to a Pub/Sub topic and trigger a Cloud Function that sends notifications.

Answer

Use Cloud Monitoring's log-based alerting to trigger on every 5xx log entry.

Question 17

Which TWO are best practices for implementing service monitoring strategies in Google Cloud?

Accepted Answer

Monitor the four golden signals (latency, traffic, errors, saturation) for every service.. A is correct because the four golden signals (latency, traffic, errors, saturation) are the foundational metrics recommended by Google SRE practices for monitoring any service. Monitoring these signals provides a comprehensive view of service health and user experience, enabling rapid detection of issues like high latency or resource exhaustion.

Answer

Rely solely on synthetic monitoring to measure user experience.

Answer

Use multiple monitoring tools to cover all aspects of the system.

Answer

Manually analyze logs and metrics to identify issues.

Question 18

A team has set up the alerting policies shown in the exhibit. They receive an alert for High Memory but not for High CPU. What is the most likely reason?

Accepted Answer

The Cloud Monitoring agent is not installed or not reporting on the instance, so the memory metric is missing.. Option A is correct because the High Memory alert fires while the High CPU alert does not, indicating that the memory metric is available but the CPU metric is missing. This typically happens when the Cloud Monitoring agent is installed but not properly reporting CPU metrics, or when the agent is missing entirely and only the memory metric is being collected via a different mechanism (e.g., guest-attributes). Without the agent, standard CPU utilization metrics are not exposed to Cloud Monitoring, while memory metrics may still be available through other means, causing the memory alert to trigger but not the CPU alert.

Answer

The CPU alert's duration of 300 seconds prevents it from firing before the memory alert.

Answer

The memory alert has a higher threshold value, making it easier to trigger.

Answer

The CPU metric is not available because the instance does not have the Cloud Monitoring agent installed.

Question 19

Order the steps to configure a VPC Network Peering between two projects.

Question 20

Match each Google Cloud tool to its function in incident management.