PCDOE Managing service incidents — Timed 10 Questions

Question 1

A team uses Google Kubernetes Engine (GKE) with cluster telemetry enabled. During an incident, they notice that a deployment's pods are repeatedly crashing with Exit Code 137. The team wants to investigate the root cause. Which two Google Cloud services should they use together to correlate resource usage and logs?

Accepted Answer

Cloud Monitoring and Cloud Logging. Exit Code 137 indicates that a container was killed by SIGKILL (signal 9), typically due to an out-of-memory (OOM) condition. Cloud Monitoring provides metrics such as memory usage and OOM kill counts, while Cloud Logging captures the container's termination logs and system events. By correlating these two services, the team can identify when memory usage spiked and confirm that the pod was OOM-killed, enabling root cause analysis.

Answer

Security Command Center and Cloud Logging

Answer

Cloud Error Reporting and Cloud Logging

Question 2

A DevOps engineer receives an alert that the error budget for a critical service has been exhausted. The service runs on Compute Engine behind an HTTP(S) load balancer. The team wants to reduce the impact on users while investigating. What should the engineer do first?

Accepted Answer

Roll back the most recent deployment. Rolling back the most recent deployment is the correct first action because it immediately restores the service to a known stable state, stopping further consumption of the error budget. This aligns with the incident management principle of 'mitigate first, investigate later' — reducing user impact takes priority over root cause analysis. The HTTP(S) load balancer will automatically route traffic to the previous healthy version once the rollback is complete.

Answer

Begin a detailed postmortem analysis

Answer

Disable the alerting policy to reduce noise

Answer

Increase the number of instances in the managed instance group

Question 3

A company uses Cloud Run for a stateless API service with concurrency set to 80. During a traffic spike, some requests return HTTP 500 errors and latency spikes. Cloud Monitoring shows container CPU utilization at 100% and memory usage at 70%. What is the most likely cause and the best first step?

Accepted Answer

Concurrency per container is too high; reduce concurrency to 10. The correct answer is A because with CPU at 100% and memory at only 70%, the bottleneck is CPU, not memory. Cloud Run containers handle requests concurrently; setting concurrency to 80 means each container processes up to 80 requests simultaneously. When CPU is saturated, requests queue up, causing latency spikes and eventual HTTP 500 errors as the container becomes unresponsive. Reducing concurrency to 10 lowers the per-container request load, allowing each request to complete before CPU saturation occurs.

Answer

Maximum instances limit is too low; increase from 10 to 100

Answer

Min idle instances is too low; set min idle to 5 to reduce cold starts

Answer

Memory limit is too low; increase memory from 256 MiB to 512 MiB

Question 4

A team uses Cloud SQL for PostgreSQL. They receive an alert that the database's CPU utilization is above 95% for the past 30 minutes. Queries are taking longer than usual. They want to investigate without causing further impact. What should they do first?

Accepted Answer

Use Cloud SQL Query Insights to find the most time-consuming queries. Cloud SQL Query Insights is a managed monitoring tool that automatically captures and analyzes query performance metrics, including CPU consumption, latency, and execution plans. In this scenario, it allows the team to identify the specific queries causing high CPU utilization without making any changes to the instance, thus avoiding further impact. This is the first and safest diagnostic step before any remediation.

Answer

Increase the number of vCPUs of the Cloud SQL instance

Answer

Restart the Cloud SQL instance to clear the cache

Answer

Migrate the database to Cloud Spanner

Question 5

A company's SRE team is designing an incident management process. They want to ensure that alerts are actionable and that on-call engineers are not overwhelmed by false positives. Which approach should they take?

Accepted Answer

Define SLOs and set alert thresholds based on historical error budget consumption. Option D is correct because defining SLOs and setting alert thresholds based on historical error budget consumption ensures alerts are directly tied to user-facing reliability. This approach prevents false positives by only triggering when the error budget is being consumed faster than expected, making alerts actionable and reducing noise for on-call engineers.

Answer

Use only critical severity alerts and rely on manual dashboard review for lower severity

Answer

Create alerting policies for every available metric to ensure nothing is missed

Answer

Set all alert thresholds to 50% above the average value to avoid false positives

Question 6

An incident is declared for a production service running on GKE. The on-call engineer suspects a recent code change may have introduced a memory leak. Which THREE actions should the engineer take to investigate and mitigate?

Accepted Answer

Increase the memory limit for the container as a temporary mitigation. Option A is correct because increasing the memory limit for the container provides a temporary mitigation to prevent the service from being killed by the Out of Memory (OOM) killer while the root cause is investigated. In GKE, the container's memory limit is defined in the pod spec under `resources.limits.memory`, and raising it gives the application more headroom to continue serving requests without immediate termination. This is a standard incident response practice to buy time for deeper analysis, such as reviewing logs and metrics, before applying a permanent fix.

Answer

Scale down the number of replicas to reduce memory pressure

Answer

Roll back the deployment immediately without further investigation

Question 7

A service experiences increased latency and HTTP 503 errors. The engineer finds that the backend managed instance group (MIG) is at max instances and CPU utilization is 90%. Which TWO actions should the engineer take to restore the service quickly?

Accepted Answer

Enable autoscaling based on HTTP load balancing utilization. Option A is correct because enabling autoscaling based on HTTP load balancing utilization allows the MIG to scale out based on the actual request load, which directly addresses the 503 errors caused by the backend being at max capacity. This metric is more responsive to traffic spikes than CPU utilization alone, as it reflects the frontend load balancer's view of backend capacity.

Answer

Increase the autoscaling target CPU utilization to 95%

Answer

Reduce the autoscaling target CPU utilization to 50%

Answer

Reduce the number of instances to avoid resource contention

Question 8

Refer to the exhibit. A GKE pod is repeatedly crashing with the error shown. The deployment has resource requests of 512 MiB memory and limits of 1 GiB. What is the most likely cause and the best remediation?

Accepted Answer

The Java heap size exceeds the container memory limit; reduce the JVM heap size or increase the container memory limit. The error indicates an OutOfMemoryError (OOM) in the Java application, which occurs when the JVM heap size exceeds the container's memory limit. Since the deployment has a memory limit of 1 GiB, if the JVM is configured with a heap size larger than this limit (or if the heap plus other memory usage exceeds it), the container will be killed by Kubernetes. Reducing the JVM heap size or increasing the container memory limit directly resolves the mismatch.

Answer

The node is under memory pressure; add more nodes to the cluster

Answer

The container needs more CPU; increase CPU request and limit

Answer

The application has a memory leak; refactor the DataProcessor class

Question 9

Refer to the exhibit. A Cloud Function (2nd gen) is timing out. The function's timeout is set to 60 seconds. The function queries a Cloud SQL database. What is the most likely cause and the best action?

Accepted Answer

Add indexes to the database tables queried by the function. The most likely cause of the timeout is that the database queries are slow due to missing indexes, causing the function to wait longer than its 60-second timeout for results. Adding indexes to the queried columns reduces query execution time, resolving the timeout without changing the function's configuration. This aligns with best practices for optimizing Cloud SQL queries in serverless environments.

Answer

Reduce the function's allocated memory to decrease cold start time

Answer

Increase the function timeout to 120 seconds

Answer

Increase the Cloud SQL max connections setting

Question 10

You are a Site Reliability Engineer (SRE) for an e-commerce platform running on Google Kubernetes Engine (GKE) with a microservices architecture. Your team uses Cloud Monitoring for alerting and Cloud Logging for centralized logs. Recently, during a flash sale event, you observed intermittent latency spikes in the checkout service, causing checkout failures and abandoned carts. The latency spikes last 1-2 seconds and occur roughly every 5-10 minutes during peak traffic. The checkout service runs as a Deployment with 10 replicas, each with resource requests of 500m CPU and 512Mi memory. The service has a Service Level Objective (SLO) of 99.9% of requests completing in under 1 second (p99 latency < 1s). Current p99 latency is 2.1s during peak. You reviewed the Cloud Monitoring dashboard and noticed that CPU utilization across pods is around 60%, memory around 50%, and there are no OOM kills. The logs show occasional 'connection reset by peer' errors in the checkout service logs, but no consistent pattern. You suspect the issue might be related to the database (Cloud SQL) or a downstream dependency. After checking the database, you find that query latency is normal. You also notice that the checkout service makes a synchronous HTTP call to a payment validation service that runs as a separate Deployment with 3 replicas. The payment service's p99 latency is 500ms, but its error rate is below 1%. Your task is to identify the most likely cause of the intermittent latency spikes and propose a remediation. Which action should you take first?

Accepted Answer

Check the garbage collection logs of the checkout service pods to identify if long GC pauses coincide with the latency spikes.. The intermittent latency spikes every 5-10 minutes with no CPU/memory pressure or database issues strongly suggest a periodic process like garbage collection. Java-based services (common in microservices) can experience stop-the-world GC pauses that cause latency spikes of 1-2 seconds, matching the observed pattern. Checking GC logs is the fastest way to confirm this before making architectural changes.

Answer

Increase the number of replicas of the payment validation service to 10 to handle peak load.

Answer

Enable connection pooling and retries with exponential backoff in the checkout service for the HTTP call to the payment service.

Answer

Investigate the checkout service pod restarts due to liveness probe failures, as 'connection reset by peer' indicates pod instability.