PCDOE Implementing service monitoring strategies — 20 Questions

Question 1

An e-commerce platform is using Cloud Load Balancing with a backend service that has a custom health check. The health check is failing intermittently, causing traffic to be routed away from healthy instances. The team has enabled Cloud Logging and wants to diagnose the issue. Which log view should they examine to see the health check probe results?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Load balancer logs (type: 'loadbalancing.googleapis.com'). Load balancer logs (type: 'loadbalancing.googleapis.com') contain detailed records of health check probes, including the probe source IP, target instance, response code, and latency. This is the exact log view that captures health check probe results, enabling the team to identify intermittent failures by correlating probe timestamps with instance health status changes.

Answer

VPC flow logs

Answer

Cloud Audit Logs (Admin Activity)

Answer

Instance serial port output logs

Question 2

A DevOps engineer is setting up alerting policies for a critical API service. They want to receive an alert if the error rate exceeds 5% for at least 5 minutes, but only during business hours (9 AM to 5 PM). Which approach should they use?

Accepted Answer

Create an alerting policy with a condition that triggers when the error rate is above 5% for 5 minutes, and configure the notification channel to only send notifications during business hours using a webhook receiver that checks time.. It uses a single alerting policy with a condition that triggers when the error rate exceeds 5% for 5 minutes, and then controls notification delivery via a webhook receiver that checks the current time. This approach ensures the alert is evaluated continuously (so the 5-minute window is respected) but only notifications are suppressed outside business hours, which is the most reliable way to meet the requirement without missing alert evaluations or relying on external scheduling.

Answer

Create a log-based metric for errors and use a condition with a threshold, then set the alert policy to only run during business hours using the 'condition' schedule.

Answer

Create two separate alert policies, one for business hours and one for off-hours, each with different thresholds.

Answer

Use Cloud Scheduler to enable and disable the alerting policy at the start and end of business hours.

Question 3

Which FOUR of the following are valid approaches to monitor a custom application metric in Cloud Monitoring? (Choose 4)

Accepted Answer

Install the Stackdriver Monitoring agent on a Windows VM and configure custom metric collection in the agent configuration file.. Option A is correct because the legacy Stackdriver/Cloud Monitoring agent supports custom metrics on Windows VMs via its configuration file (e.g., defining custom metric collection blocks), allowing the agent to collect and send application-defined metrics to Cloud Monitoring. Option B is correct because the Cloud Monitoring API (monitoring.googleapis.com, e.g., timeSeries.create) lets you write time series data directly, which is the canonical programmatic way to publish custom metrics. Option C is correct because a logs-based metric extracts numeric values from log entries (using a metric filter and optional value extractor) and turns them into a Cloud Monitoring metric, a valid approach when the application emits the metric in its logs. Option E is correct because the OpenTelemetry Collector's Google Cloud Monitoring exporter (googlecloud exporter) exports OTLP metrics to Cloud Monitoring, a supported and recommended path for custom application metrics. Option D is not correct because the Cloud Monitoring agent does not include a built-in JMX plugin for collecting Java application metrics; JMX-based collection typically requires the OpenTelemetry Java instrumentation/collector or the legacy Stackdriver JMX integration, not a built-in agent plugin.

Answer

Use the built-in JMX plugin in the Cloud Monitoring agent to collect Java application metrics.

Question 4

A DevOps engineer runs the command above and gets the output shown. What does this output indicate?

Accepted Answer

An application running on the instance encountered a connection timeout to a backend service.. The output shows a 'Connection timed out' error when attempting to reach a backend service. This indicates that the application on the instance is unable to establish a TCP connection to the specified IP and port, typically due to network issues, firewall rules, or the backend service being down. The error is specific to application-level connectivity, not disk space or authentication.

Answer

The instance's disk is full, causing write errors.

Answer

The instance failed to authenticate with the metadata server.

Answer

A health check probe failed to reach the instance.

Question 5

A team has deployed a Prometheus server on GKE using the configuration above. They expect Prometheus to scrape metrics from pods with the label 'app: my-app' and the annotation 'prometheus.io/scrape: true' on port 8080. However, no metrics are being collected. What is the most likely cause?

Accepted Answer

The relabel_config for port incorrectly constructs the target address; it should use the annotation value directly without appending ':8080'.. The relabel_config is incorrectly constructing the target address. The configuration likely uses `__meta_kubernetes_annotation_prometheus_io_port` to get the port, but then appends ':8080' statically, overriding the annotation value. If the annotation specifies a different port or is missing, this hardcoded port causes Prometheus to scrape the wrong endpoint or fail entirely. The correct approach is to use the annotation value directly without appending a fixed port, or to use a default port only when the annotation is absent.

Answer

The kubernetes_sd_configs role is set to 'pod' but should be 'endpoints'.

Answer

Prometheus needs to be configured to listen on port 9090 for scraping.

Answer

The keep action for the label 'my-app' is filtering out all pods.

Question 6

You are monitoring a microservices application deployed on Google Kubernetes Engine (GKE) that uses Cloud Monitoring for observability. You notice that the error rate for a critical service has increased, but the CPU and memory usage remain normal. The service uses gRPC and logs are structured. Which Cloud Monitoring tool should you use first to diagnose the root cause of the increased error rate?

Accepted Answer

Logs Explorer to filter logs by error status codes. Logs Explorer allows you to directly query structured gRPC logs by filtering on error status codes (e.g., gRPC status codes like `UNAVAILABLE`, `INTERNAL`, or `DEADLINE_EXCEEDED`). Since the service uses structured logging, you can quickly isolate the exact error messages and stack traces without needing to pre-configure dashboards or wait for automated grouping. This is the fastest first step to identify the root cause of an increased error rate when CPU and memory are normal, as it points to application-level or dependency issues.

Answer

Service Monitoring to create a custom dashboard

Answer

Error Reporting to automatically group error occurrences

Answer

Metrics Explorer to view error rate and latency charts

Question 7

You are setting up alerting for a batch processing job that runs daily on Compute Engine. The job must complete within 2 hours. Which metric and alert condition should you use to ensure you are notified if the job is still running after 90 minutes?

Accepted Answer

Create a custom metric that emits 1 when the job starts and 0 when it finishes; alert if the metric is 1 for more than 90 minutes. It directly monitors the job's running state using a custom metric that emits 1 at job start and 0 at completion. By alerting when the metric remains at 1 for more than 90 minutes, you are notified if the job exceeds the 90-minute threshold, ensuring you catch failures before the 2-hour deadline. This approach is precise and avoids false positives from indirect signals like CPU or logs.

Answer

Alert on CPU utilization greater than 80% for the instance running the job

Answer

Use a heartbeat metric that reports every 5 minutes; alert if no heartbeat for 90 minutes

Answer

Set up a log-based metric that counts job completion log entries; alert if the count is zero after 90 minutes

Question 8

Which TWO metrics should be included in a comprehensive monitoring strategy for a production Kubernetes workload to detect performance degradation and capacity issues?

Accepted Answer

Container CPU utilization. Container CPU utilization (B) is a core resource metric that reveals whether pods are approaching their CPU requests/limits, causing throttling (container_cpu_cfs_throttled_seconds_total) and performance degradation, and it directly informs capacity planning for scaling. Request latency percentiles such as p99 (E) are golden-signal metrics that expose actual user-facing performance degradation even when resource usage looks normal, since averages hide tail latency problems that indicate saturation or downstream bottlenecks. Together these two cover both the resource-consumption cause and the service-level symptom of degradation and capacity pressure. Disk read IOPS per pod (A) is only relevant for storage-intensive workloads and is not a general indicator, the number of nodes (C) is a coarse cluster-inventory value that does not reflect per-workload performance or saturation, and network bytes received per second (D) is a throughput counter that alone does not indicate degradation or capacity limits.

Answer

Disk read IOPS per pod

Answer

Number of nodes in the cluster

Answer

Network bytes received per second

Question 9

Your organization runs a critical e-commerce platform on Google Kubernetes Engine (GKE). The platform uses Cloud Service Mesh (Anthos Service Mesh) for traffic management and Cloud Monitoring for observability. Recently, after a new release, you observe that the p99 latency of the checkout service has increased from 200ms to 2s. The service's CPU and memory metrics appear normal, and there are no error logs. The release included a change to the Istio VirtualService configuration that added a retry policy: 3 retries with a 500ms timeout per retry. You suspect that the retries are contributing to the latency increase. You want to use Cloud Monitoring to confirm this hypothesis. Which approach should you take?

Accepted Answer

Use Metrics Explorer to query the istio.io/service/server/request_count metric, filtered by response_code_class and destination_service, and include the istio.io/service/server/request_retries metric to see retry counts alongside latency. It directly correlates retry attempts with latency by querying the `istio.io/service/server/request_retries` metric alongside the `istio.io/service/server/request_count` metric in Metrics Explorer. This allows you to visualize the retry count per destination service (checkout) and compare it with the p99 latency increase, confirming whether the retry policy is causing the observed latency spike. The retry policy (3 retries with 500ms timeout) can add up to 1.5s of additional latency per request, which aligns with the increase from 200ms to 2s.

Answer

Use Cloud Trace to analyze distributed traces for the checkout service and look for retry spans

Answer

Check the 'Services' dashboard in Cloud Monitoring, which shows a pre-built latency chart for all services

Answer

Use Logs Explorer to search for logs containing 'retry' in the checkout service namespace

Question 10

You are a DevOps engineer for a SaaS company that provides a REST API. The API is deployed on Google Cloud Run. You have configured Cloud Monitoring alerts for 5xx errors. Recently, you received an alert that the error rate exceeded 5% for 5 minutes. You investigated and found that the errors were HTTP 503 (Service Unavailable) from a specific endpoint. The endpoint calls an internal Cloud SQL database. The database CPU utilization was at 90% during that period. You suspect the database is the bottleneck. Which action should you take to reduce the error rate without over-provisioning?

Accepted Answer

Implement connection pooling and retry logic with exponential backoff in the API service. Implementing connection pooling and retry logic with exponential backoff directly addresses the database bottleneck without over-provisioning. Connection pooling reduces the number of concurrent connections to Cloud SQL, lowering CPU contention, while exponential backoff prevents thundering herd retries that could further overwhelm the database. This approach optimizes existing resources rather than scaling infrastructure.

Answer

Increase the max instances per revision in Cloud Run to handle more concurrent requests

Answer

Reduce the min instances of Cloud Run to decrease load on the database

Answer

Add a Cloud SQL read replica and route read queries to it

Question 11

A company uses Cloud Run for a critical service and needs to set up alerting for 5xx errors. They want to receive a notification within 1 minute of the error rate exceeding 1% for any 1-minute window. Which alerting approach should they use?

Accepted Answer

Create a Cloud Monitoring alerting policy using the 'Request count' metric with a condition that compares the ratio of 5xx responses to total requests over a 1-minute window.. Cloud Monitoring's alerting policies can directly compute the ratio of 5xx responses to total requests using the 'Request count' metric with a custom ratio condition over a 1-minute window. This approach meets the 1-minute notification latency requirement without additional infrastructure, as Cloud Monitoring evaluates the metric every 60 seconds and triggers alerts based on the sliding window.

Answer

Set up a log-based metric for 5xx responses and create an alert on the metric.

Answer

Create a Cloud Logging sink to a Pub/Sub topic and trigger a Cloud Function that sends notifications.

Answer

Use Cloud Monitoring's log-based alerting to trigger on every 5xx log entry.

Question 12

A team has set up the alerting policies shown in the exhibit. They receive an alert for High Memory but not for High CPU. What is the most likely reason?

Accepted Answer

The CPU alert's duration of 300 seconds prevents it from firing before the memory alert.. The most likely reason the CPU alert has not fired while the memory alert has is that the CPU alert's duration is set to 300 seconds, requiring sustained high CPU for that period before firing. The memory alert likely has a shorter duration and its condition was met sooner, triggering the alert. Option A is incorrect because if the agent were missing, both memory and CPU metrics would be unavailable; the fact that memory alert fired indicates the agent is functioning. Option B correctly identifies the duration difference.

Answer

The Cloud Monitoring agent is not installed or not reporting on the instance, so the memory metric is missing.

Answer

The memory alert has a higher threshold value, making it easier to trigger.

Answer

The CPU metric is not available because the instance does not have the Cloud Monitoring agent installed.

Question 13

Order the steps to configure a VPC Network Peering between two projects.

Accepted Answer

Create VPC networks in both projects, then configure firewall rules to allow traffic between the VPCs, then create a peering connection from Project A to Project B, then create a peering connection from Project B to Project A, then verify the peering connection.. The correct sequence for configuring VPC Network Peering is to first create the VPC networks in both projects, then configure firewall rules to allow traffic, then create the peering connection in both directions (from Project A to Project B and from Project B to Project A), and finally verify connectivity. Common mistakes include performing steps out of order or omitting the reverse peering.

Answer

Create VPC networks in both projects, then create peering from A to B, then create peering from B to A, then configure firewall rules, then verify.

Answer

Create VPC networks in both projects, then create peering from A to B, then configure firewall rules, then verify.

Answer

Configure firewall rules, then create VPC networks in both projects, then create peering from A to B, then create peering from B to A, then verify.

Question 14

Match each Google Cloud tool to its function in incident management.

Accepted Answer

Cloud Logging: Collect and store log data from applications and services.. In incident management, Cloud Logging collects logs, Cloud Monitoring provides metrics and alerts, Error Reporting aggregates errors, and Cloud Trace traces requests. Common confusions include mixing Cloud Logging with monitoring or debugger functions.

Answer

Cloud Logging: Provide visibility into performance and health.

Answer

Cloud Debugger: Store and analyze logs.

Question 15

A DevOps team is defining an SLO for a web application that runs on Compute Engine behind an HTTP Load Balancer. They need to measure the proportion of requests that complete within 300ms. Which Cloud Monitoring metric is most appropriate as the SLI?

Accepted Answer

loadbalancing.googleapis.com/https/total_latencies. The SLI must measure the proportion of requests completing within 300ms, which is a latency distribution metric. The `total_latencies` metric from the HTTP Load Balancer provides a histogram of request latencies, allowing you to compute the percentage of requests below a threshold (e.g., 300ms). This directly supports the SLO definition.

Answer

loadbalancing.googleapis.com/https/backend_request_bytes

Answer

loadbalancing.googleapis.com/https/frontend_tcp_rtt

Answer

loadbalancing.googleapis.com/https/request_count

Question 16

You are designing a monitoring strategy for a microservices application running on Google Kubernetes Engine (GKE). You need to create a custom metric that counts the number of failed login attempts from the application logs. The logs are in JSON format and contain a field 'status' with value 'FAILED'. Which approach should you use?

Accepted Answer

Configure a logs-based metric in Cloud Logging that filters for the condition and counts.. A logs-based metric in Cloud Logging directly counts occurrences of a specific log entry pattern (e.g., 'status' = 'FAILED') without requiring additional agents or data exports. This approach is purpose-built for deriving metrics from log data and integrates seamlessly with Cloud Monitoring for alerting and dashboards.

Answer

Use Cloud Monitoring's Metrics Explorer to create a metric from logs using a filter.

Answer

Install the Ops Agent on GKE nodes to collect application metrics directly.

Answer

Export logs to BigQuery and then create a custom metric from the exported data.

Question 17

You are creating a Cloud Monitoring dashboard to display the 99th percentile latency of your HTTP Load Balancer over the last 6 hours. Which MQL query should you use?

Accepted Answer

fetch loadbalancing.googleapis.com/https/total_latencies | align percentile(99). It uses the correct metric type (`total_latencies`) and the proper MQL function `percentile(99)` to compute the 99th percentile latency. The `fetch` statement targets the exact Cloud Monitoring metric for HTTPS load balancer latencies, and `align percentile(99)` aggregates the raw latency distribution data over the specified time window (last 6 hours) to produce the desired percentile value.

Answer

fetch https_lb_rule :: latency | align 99p

Answer

fetch loadbalancing.googleapis.com/https/request_count | align | ratio

Answer

fetch loadbalancing.googleapis.com/https/total_latencies | align 99 | with latency

Question 18

A company has on-premises servers running Linux and GKE clusters. They want to monitor all infrastructure using Cloud Monitoring. Which solution is most scalable and aligned with Google's best practices?

Accepted Answer

Use Google's managed service for Prometheus on GKE and a Prometheus federation for on-prem.. It leverages Google's managed service for Prometheus on GKE, which is fully integrated with Cloud Monitoring and eliminates the operational overhead of self-managing Prometheus. For on-premises servers, Prometheus federation allows scraping metrics from the on-prem Prometheus instance and forwarding them to the managed service, providing a unified, scalable monitoring solution that aligns with Google's best practices for hybrid environments.

Answer

Use collectd on on-prem servers to send to Cloud Monitoring via the Stackdriver agent configuration.

Answer

Deploy Prometheus on both environments and use the PromQL adapter for Cloud Monitoring.

Answer

Install the Ops Agent on all on-prem servers and use Google's default GKE monitoring.

Question 19

An application running on App Engine is throwing exceptions. The DevOps team wants to be notified when a new type of exception appears. Which Cloud Monitoring feature should they use?

Accepted Answer

Error Reporting. Error Reporting is the correct choice because it is a Cloud Monitoring feature specifically designed to count, analyze, and increase the severity of exceptions in application logs. It automatically groups exceptions by type and can trigger notifications when a new type of exception (one not seen before) appears, which directly meets the DevOps team's requirement.

Answer

Uptime Checks

Answer

Custom alerts

Answer

Logs-based metrics

Question 20

An alerting policy for high CPU utilization on a VM is firing even when CPU is not high. The team suspects a misconfiguration. Which two possible issues should they check? (Choose two.)

Accepted Answer

The alert condition is using the average aggregation with a short alignment period.. Using a short alignment period with average aggregation can cause the alert to fire on brief spikes in CPU utilization that do not represent sustained high usage. If the alignment period is too short (e.g., 1 minute), the alerting policy may trigger on transient bursts, even when the overall CPU load is low. This is a common misconfiguration in Google Cloud Monitoring (formerly Stackdriver) where the alignment period and aggregator settings must match the expected workload pattern.

Answer

The threshold is set too low compared to actual CPU usage.

Answer

The alerting policy was created in a different project and not imported.

Answer

The VM is reporting metrics from a custom namespace instead of the standard agent.