PCDOE Managing service incidents Practice Test 4 — 15 Questions

Question 1

After a recent deployment, the mean latency of a user-facing service increased from 200ms to 500ms. The engineer uses Cloud Trace to analyze traces. Which trace characteristic should the engineer focus on to identify the bottleneck?

Accepted Answer

Distribution of span latencies across services.. The engineer should focus on the distribution of span latencies across services (Option B) because Cloud Trace captures the latency of each span in a distributed trace. By examining the histogram or distribution of span latencies, the engineer can identify which specific service or component is contributing the most to the overall increase from 200ms to 500ms, pinpointing the bottleneck. This approach aligns with the principle of distributed tracing, where the critical path is determined by the slowest span in the trace.

Answer

Timestamps of the trace ID.

Answer

Error count per span.

Answer

Total number of spans in the trace.

Question 2

A team defines an SLO of 99.9% availability over a 30-day window. They use a multi-window, multi-burn-rate alerting approach. Which alerting condition should trigger a page based on fast burn rate?

Accepted Answer

2% error budget consumed in 10 minutes.. Option C is correct because a multi-window, multi-burn-rate alerting approach uses a short window (e.g., 10 minutes) to detect fast burn rates that could rapidly exhaust the error budget. Consuming 2% of the error budget in 10 minutes corresponds to an annualized burn rate of over 1000x, which is dangerously fast and requires immediate paging to prevent a breach of the 99.9% SLO over the 30-day window.

Answer

1% error budget consumed in 1 hour.

Answer

10% error budget consumed in 1 hour.

Answer

5% error budget consumed in 6 hours.

Question 3

During a canary deployment of a new version of a microservice, the engineer notices increased error rates in the canary instances. What is the best immediate action?

Accepted Answer

Perform a rollback of the canary to the previous version.. When error rates increase in a canary, the safest immediate action is to roll back the canary to prevent further impact on users. Pausing and investigating is reasonable but allows continued errors. Scaling up the canary would worsen the issue. Continuing the rollout would be irresponsible.

Answer

Continue the rollout to see if errors stabilize.

Answer

Scale up the canary instances to handle load.

Answer

Pause the rollout and investigate the errors.

Question 4

An engineer receives an alert that a service's error rate has exceeded the threshold. To investigate, which log-based metric should the engineer query in Cloud Logging to identify the root cause?

Accepted Answer

Error log count grouped by service name.. Option A is correct because error log count grouped by service name directly surfaces which specific service is generating the elevated error rate. In Cloud Logging, log-based metrics are user-defined counters extracted from log entries; querying the error log count per service isolates the offending component, enabling root cause identification without mixing in unrelated performance data.

Answer

Request latency histogram.

Answer

CPU utilization of the service instances.

Answer

Network bytes sent per instance.

Question 5

In Google's incident management process, which role is responsible for communication with stakeholders and users during an incident?

Accepted Answer

Communications Lead.. In Google's incident management process, the Communications Lead is explicitly responsible for managing all external and internal communications, including updates to stakeholders and users. This role ensures that accurate, timely information is disseminated while the Incident Commander focuses on coordinating the response. The Communications Lead does not engage in technical troubleshooting or operational tasks, which are handled by other roles.

Answer

Incident Commander.

Answer

Technical Lead.

Answer

Operations Lead.

Question 6

An engineer wants to ensure that an alert is escalated if not acknowledged within 5 minutes. Which feature of Cloud Monitoring can achieve this?

Accepted Answer

Notification channel with escalation configuration.. Notification channels in Google Cloud Monitoring can be configured with an escalation policy that defines a sequence of recipients or notification methods to be contacted if an alert is not acknowledged within a specified time. By setting the escalation duration to 5 minutes, the system automatically escalates the alert to the next tier or channel if no acknowledgment is received, directly meeting the engineer's requirement.

Answer

Incident management tool like PagerDuty.

Answer

Using a webhook notification channel.

Answer

Alerting policy with a condition that checks for acknowledgment.

Question 7

Which TWO practices help reduce Mean Time to Resolve (MTTR) for production incidents?

Accepted Answer

Implement runbooks for common incident types.. B is correct because runbooks provide step-by-step, pre-approved procedures for common incident types, enabling engineers to follow a consistent, repeatable process without needing to diagnose from scratch. This reduces the time spent on investigation and decision-making, directly lowering Mean Time to Resolve (MTTR) by standardizing the response for known issues.

Answer

Conduct postmortems only after major incidents.

Answer

Use a shared on-call rotating schedule.

Answer

Increase logging verbosity for all services.

Question 8

Which THREE steps are typically part of a formal incident postmortem according to Google SRE best practices?

Accepted Answer

Assign action items with deadlines.. Option B is correct because Google SRE postmortems emphasize creating actionable follow-ups to prevent recurrence. Assigning action items with deadlines ensures that identified issues are systematically addressed, which is a core principle of blameless postmortems focused on process improvement rather than punishment.

Answer

Identify the person responsible for the incident.

Answer

Implement a solution immediately.

Question 9

Which TWO tools should be used for real-time incident collaboration and communication?

Accepted Answer

Cloud Monitoring Incident Response (beta).. Cloud Monitoring Incident Response (beta) provides a dedicated interface for real-time incident management, including automated notifications, escalation policies, and a centralized timeline for collaboration. Google Chat with a dedicated incident room enables real-time communication and coordination among incident responders, allowing them to share updates, logs, and runbooks in a structured channel. Together, these tools fulfill the need for both incident orchestration and synchronous collaboration during active incidents.

Answer

Google Meet only.

Answer

Jira.

Answer

Cloud Trace.

Question 10

Refer to the exhibit. If the error rate spikes to 2% for only 2 minutes, why does the alert not fire?

Accepted Answer

The duration of 300s requires the condition to be met for 5 minutes.. Option B is correct because the alert condition requires the error rate to exceed 1% for a duration of 300 seconds (5 minutes). A spike lasting only 2 minutes does not meet the minimum duration requirement, so the alert remains in a 'pending' state and never fires. In Cisco PCDOE, the duration parameter defines how long the condition must be continuously true before the alert transitions from pending to firing.

Answer

The notification rate limit prevents the alert from firing.

Answer

The threshold value of 1% is too high.

Answer

The alignment period of 60s is too short.

Question 11

You are the DevOps engineer for a social media platform. After a recent code rollout, you receive multiple user complaints about failed logins. The service logs show a sharp increase in 5xx errors from the authentication service. However, the existing alerting policy for the authentication service did not fire. The policy is configured to trigger if the error rate exceeds 5% for 5 minutes. Upon checking Cloud Monitoring, you see that the error rate spiked to 15% for 3 minutes, then dropped back to normal. What is the most likely reason the alert did not fire?

Accepted Answer

The duration condition of 5 minutes was not satisfied.. The alert did not fire because the policy requires the error rate to exceed 5% for a continuous duration of 5 minutes. The spike only lasted 3 minutes, which is shorter than the configured duration condition, so the alerting policy's condition was never fully met. In Google Cloud Monitoring, alerting policies evaluate both the threshold and the duration window before transitioning to a firing state.

Answer

The error rate threshold of 5% was too low, causing the alert to be suppressed.

Answer

The alignment period for the metric was set to 5 minutes, hiding the spike.

Answer

The notification channel was incorrectly configured.

Question 12

Your company runs an e-commerce application on Google Kubernetes Engine (GKE) with a microservice architecture. During a Black Friday sale, the orders service experiences a sudden increase in latency and errors. You notice that the database connection pool in the orders service is exhausted, leading to timeouts. The service is written in Java and uses HikariCP connection pool. You need to mitigate the incident quickly. Which action should you take first?

Accepted Answer

Increase the number of replicas of the orders service.. Increasing the number of replicas of the orders service is the fastest way to mitigate the incident because it horizontally scales the application tier, distributing incoming requests across more pods. This reduces the load per pod, which in turn reduces the number of concurrent database connections each pod attempts to acquire from its HikariCP pool, alleviating pool exhaustion and timeouts without requiring a database or code change.

Answer

Add more database instances.

Answer

Enable connection pooling at the database side.

Answer

Temporarily reduce the maximum connection pool size.

Question 13

You manage a production environment with a web service deployed on Compute Engine instances behind a HTTP(S) Load Balancer. The service has a health check configured on the load balancer, probing a health endpoint every 10 seconds. After a recent configuration change, you observe that all instances are marked as unhealthy and traffic is failing. The health check response is 200 OK from the instances, but the load balancer still marks them unhealthy. The health check configuration: protocol: HTTP, port: 80, request path: /health, interval: 10s, timeout: 5s, unhealthy threshold: 2. The instances are running a custom web server. What is the most likely cause?

Accepted Answer

The health check response is coming from a different server (e.g., a reverse proxy) that returns 200 but does not represent the actual service health.. Option C is correct because the health check is receiving a 200 OK response from a reverse proxy or intermediary that is not the actual web server, so the load balancer's health check does not reflect the true health of the application. Even though the response is successful, the load balancer marks instances unhealthy because the health check is likely failing to reach the intended health endpoint on the custom web server, or the response is not coming from the service itself. This scenario often occurs when a reverse proxy (e.g., nginx) returns a static 200 for /health without forwarding the request to the backend service, causing the load balancer to see a healthy response but the service to be actually unhealthy.

Answer

The load balancer is misconfigured with an incorrect protocol (HTTPS instead of HTTP).

Answer

The health check port is incorrectly configured as 80 but the service is listening on 8080.

Answer

The health check timeout is greater than the interval, causing overlapping probes.

Question 14

You are the SRE for a financial services application running on Google Cloud. Users report that certain transactions are taking over 10 seconds, while most complete in under 200ms. You use Cloud Profiler and Cloud Trace. Upon reviewing the profiler data, you see a hotspot in a method that calls a Cloud SQL database with a slow query. You identify the query and create an index to speed it up. However, you cannot deploy the index change immediately due to change management processes. The incident response team needs to mitigate the impact now. Which temporary measure should you take?

Accepted Answer

Implement caching of query results using Cloud Memorystore.. Option C is correct because caching the results of the slow query in Cloud Memorystore (Redis) immediately reduces the load on the Cloud SQL database and eliminates the need to execute the slow query repeatedly. This provides a temporary performance improvement without requiring any database schema changes or deployments, bypassing the change management delay. The hotspot in the profiler indicates the query itself is the bottleneck, and caching avoids that bottleneck entirely for repeated reads.

Answer

Increase the database connection pool size in the application.

Answer

Add a database read replica to offload read queries.

Answer

Scale up the Cloud SQL instance to more vCPUs.

Question 15

Your company runs a microservices application on a private GKE cluster with Workload Identity enabled. Services communicate via gRPC and HTTP. After a recent update to the payment service, users report intermittent 503 errors and 2-second latency spikes during peak hours (10 AM - 12 PM). Cloud Monitoring shows the payment service's CPU utilization averages 60%, but memory spikes to 90% during errors. The existing alert on HTTP 503 responses fires only after 5 consecutive errors over 5 minutes, but the errors are sporadic. You need to diagnose and resolve the issue. What should you do?

Accepted Answer

Enable detailed logging and metrics for the payment service in Cloud Logging and Cloud Monitoring. Analyze logs around the error timestamps to identify memory consumption patterns, and lower the alert threshold for 503 errors to trigger on 2 errors within 1 minute. Also, set up a custom alert on memory usage exceeding 85%.. Option C is correct because the issue requires a two-pronged approach: first, enable detailed logging and metrics to diagnose the root cause (memory spikes during peak hours), and second, adjust alerting thresholds to catch sporadic 503 errors earlier (2 errors in 1 minute) and add a proactive memory usage alert at 85%. This aligns with the PCDOE incident management process of first gathering data before making changes, and ensures you can correlate memory pressure with gRPC/HTTP errors in a GKE environment with Workload Identity.

Answer

Increase the memory limits for the payment service containers and set a memory request equal to the limit. Then restart the pods to clear any memory leaks.

Answer

Switch the payment service's HTTP/2 protocol to HTTP/1.1 to reduce overhead, and increase the memory request limit to avoid out-of-memory errors.

Answer

Scale the payment service horizontally by increasing the minimum number of replicas to handle peak load, and adjust the HPA to scale faster.