PCDOE Applying site reliability engineering practices — 20 Questions

Question 1

Refer to the exhibit. If the error rate spikes to 2% for only 2 minutes, why does the alert not fire?

Accepted Answer

The duration of 300s requires the condition to be met for 5 minutes.. The alert condition requires the error rate to exceed 1% for a duration of 300 seconds (5 minutes). A spike lasting only 2 minutes does not meet the minimum duration requirement, so the alert remains in a 'pending' state and never fires. In Google Cloud Monitoring, the duration parameter defines how long the condition must be continuously true before the alert transitions from pending to firing.

Answer

The notification rate limit prevents the alert from firing.

Answer

The threshold value of 1% is too high.

Answer

The alignment period of 60s is too short.

Question 2

A team uses Google Kubernetes Engine (GKE) with cluster telemetry enabled. During an incident, they notice that a deployment's pods are repeatedly crashing with Exit Code 137. The team wants to investigate the root cause. Which two Google Cloud services should they use together to correlate resource usage and logs?

Accepted Answer

Cloud Monitoring and Cloud Logging. Exit Code 137 indicates a container was killed by SIGKILL, typically due to an out-of-memory (OOM) condition. To correlate resource usage and logs, Cloud Monitoring provides resource usage metrics (e.g., memory, CPU), while Cloud Logging captures container termination logs and system events. Together, they allow correlation of spikes in resource usage with log entries to identify OOM kills. Cloud Trace is not primarily a logging service; it provides distributed tracing for latency analysis, but does not directly correlate logs with resource usage. Therefore, only Cloud Monitoring and Cloud Logging together (option A) satisfy the requirement.

Answer

Security Command Center and Cloud Logging

Answer

Cloud Trace and Cloud Monitoring

Answer

Cloud Error Reporting and Cloud Logging

Question 3

A DevOps engineer receives an alert that the error budget for a critical service has been exhausted. The service runs on Compute Engine behind an HTTP(S) load balancer. The team wants to reduce the impact on users while investigating. What should the engineer do first?

Accepted Answer

Roll back the most recent deployment. Rolling back the most recent deployment is the correct first action because it immediately restores the service to a known stable state, stopping further consumption of the error budget. This aligns with the incident management principle of 'mitigate first, investigate later' — reducing user impact takes priority over root cause analysis. The HTTP(S) load balancer will automatically route traffic to the previous healthy version once the rollback is complete.

Answer

Begin a detailed postmortem analysis

Answer

Disable the alerting policy to reduce noise

Answer

Increase the number of instances in the managed instance group

Question 4

A company uses Cloud Run for a stateless API service with concurrency set to 80. During a traffic spike, some requests return HTTP 500 errors and latency spikes. Cloud Monitoring shows container CPU utilization at 100% and memory usage at 70%. What is the most likely cause and the best first step?

Accepted Answer

Concurrency per container is too high; reduce concurrency to 10. With CPU at 100% and memory at only 70%, the bottleneck is CPU, not memory. Cloud Run containers handle requests concurrently; setting concurrency to 80 means each container processes up to 80 requests simultaneously. When CPU is saturated, requests queue up, causing latency spikes and eventual HTTP 500 errors as the container becomes unresponsive. Reducing concurrency to 10 lowers the per-container request load, allowing each request to complete before CPU saturation occurs.

Answer

Maximum instances limit is too low; increase from 10 to 100

Answer

Min idle instances is too low; set min idle to 5 to reduce cold starts

Answer

Memory limit is too low; increase memory from 256 MiB to 512 MiB

Question 5

A team uses Cloud SQL for PostgreSQL. They receive an alert that the database's CPU utilization is above 95% for the past 30 minutes. Queries are taking longer than usual. They want to investigate without causing further impact. What should they do first?

Accepted Answer

Use Cloud SQL Query Insights to find the most time-consuming queries. Cloud SQL Query Insights is a managed monitoring tool that automatically captures and analyzes query performance metrics, including CPU consumption, latency, and execution plans. In this scenario, it allows the team to identify the specific queries causing high CPU utilization without making any changes to the instance, thus avoiding further impact. This is the first and safest diagnostic step before any remediation.

Answer

Increase the number of vCPUs of the Cloud SQL instance

Answer

Restart the Cloud SQL instance to clear the cache

Answer

Migrate the database to Cloud Spanner

Question 6

A company's SRE team is designing an incident management process. They want to ensure that alerts are actionable and that on-call engineers are not overwhelmed by false positives. Which approach should they take?

Accepted Answer

Define SLOs and set alert thresholds based on historical error budget consumption. Defining SLOs and setting alert thresholds based on historical error budget consumption ensures alerts are directly tied to user-facing reliability. This approach prevents false positives by only triggering when the error budget is being consumed faster than expected, making alerts actionable and reducing noise for on-call engineers.

Answer

Use only critical severity alerts and rely on manual dashboard review for lower severity

Answer

Create alerting policies for every available metric to ensure nothing is missed

Answer

Set all alert thresholds to 50% above the average value to avoid false positives

Question 7

An incident is declared for a production service running on GKE. The on-call engineer suspects a recent code change may have introduced a memory leak. Which THREE actions should the engineer take to investigate and mitigate?

Accepted Answer

Increase the memory limit for the container as a temporary mitigation. Increasing the memory limit for the container provides a temporary mitigation to prevent the service from being killed by the Out of Memory (OOM) killer while the root cause is investigated. In GKE, the container's memory limit is defined in the pod spec under `resources.limits.memory`, and raising it gives the application more headroom to continue serving requests without immediate termination. This is a standard incident response practice to buy time for deeper analysis, such as reviewing logs and metrics, before applying a permanent fix.

Answer

Scale down the number of replicas to reduce memory pressure

Answer

Roll back the deployment immediately without further investigation

Question 8

A service experiences increased latency and HTTP 503 errors. The engineer finds that the backend managed instance group (MIG) is at max instances and CPU utilization is 90%. Which TWO actions should the engineer take to restore the service quickly?

Accepted Answer

Enable autoscaling based on HTTP load balancing utilization. Enabling autoscaling based on HTTP load balancing utilization allows the MIG to scale out based on the actual request load, which directly addresses the 503 errors caused by the backend being at max capacity. This metric is more responsive to traffic spikes than CPU utilization alone, as it reflects the frontend load balancer's view of backend capacity.

Answer

Increase the autoscaling target CPU utilization to 95%

Answer

Reduce the autoscaling target CPU utilization to 50%

Answer

Reduce the number of instances to avoid resource contention

Question 9

Refer to the exhibit. A GKE pod is repeatedly crashing with the error shown. The deployment has resource requests of 512 MiB memory and limits of 1 GiB. What is the most likely cause and the best remediation?

Accepted Answer

The Java heap size exceeds the container memory limit; reduce the JVM heap size or increase the container memory limit. The error indicates an OutOfMemoryError (OOM) in the Java application, which occurs when the JVM heap size exceeds the container's memory limit. Since the deployment has a memory limit of 1 GiB, if the JVM is configured with a heap size larger than this limit (or if the heap plus other memory usage exceeds it), the container will be killed by Kubernetes. Reducing the JVM heap size or increasing the container memory limit directly resolves the mismatch.

Answer

The node is under memory pressure; add more nodes to the cluster

Answer

The container needs more CPU; increase CPU request and limit

Answer

The application has a memory leak; refactor the DataProcessor class

Question 10

Refer to the exhibit. A Cloud Function (2nd gen) is timing out. The function's timeout is set to 60 seconds. The function queries a Cloud SQL database. What is the most likely cause and the best action?

Accepted Answer

Add indexes to the database tables queried by the function. The most likely cause of the timeout is that the database queries are slow due to missing indexes, causing the function to wait longer than its 60-second timeout for results. Adding indexes to the queried columns reduces query execution time, resolving the timeout without changing the function's configuration. This aligns with best practices for optimizing Cloud SQL queries in serverless environments.

Answer

Reduce the function's allocated memory to decrease cold start time

Answer

Increase the function timeout to 120 seconds

Answer

Increase the Cloud SQL max connections setting

Question 11

You are a Site Reliability Engineer (SRE) for an e-commerce platform running on Google Kubernetes Engine (GKE) with a microservices architecture. Your team uses Cloud Monitoring for alerting and Cloud Logging for centralized logs. Recently, during a flash sale event, you observed intermittent latency spikes in the checkout service, causing checkout failures and abandoned carts. The latency spikes last 1-2 seconds and occur roughly every 5-10 minutes during peak traffic. The checkout service runs as a Deployment with 10 replicas, each with resource requests of 500m CPU and 512Mi memory. The service has a Service Level Objective (SLO) of 99.9% of requests completing in under 1 second (p99 latency < 1s). Current p99 latency is 2.1s during peak. You reviewed the Cloud Monitoring dashboard and noticed that CPU utilization across pods is around 60%, memory around 50%, and there are no OOM kills. The logs show occasional 'connection reset by peer' errors in the checkout service logs, but no consistent pattern. You suspect the issue might be related to the database (Cloud SQL) or a downstream dependency. After checking the database, you find that query latency is normal. You also notice that the checkout service makes a synchronous HTTP call to a payment validation service that runs as a separate Deployment with 3 replicas. The payment service's p99 latency is 500ms, but its error rate is below 1%. Your task is to identify the most likely cause of the intermittent latency spikes and propose a remediation. Which action should you take first?

Accepted Answer

Check the garbage collection logs of the checkout service pods to identify if long GC pauses coincide with the latency spikes.. The intermittent latency spikes every 5-10 minutes with no CPU/memory pressure or database issues strongly suggest a periodic process like garbage collection. Java-based services (common in microservices) can experience stop-the-world GC pauses that cause latency spikes of 1-2 seconds, matching the observed pattern. Checking GC logs is the fastest way to confirm this before making architectural changes.

Answer

Increase the number of replicas of the payment validation service to 10 to handle peak load.

Answer

Enable connection pooling and retries with exponential backoff in the checkout service for the HTTP call to the payment service.

Answer

Investigate the checkout service pod restarts due to liveness probe failures, as 'connection reset by peer' indicates pod instability.

Question 12

Your team is using Cloud Monitoring to track the health of a distributed microservices application. You notice that the error rate for the checkout service has increased significantly, but no alerts are firing. The SLO for checkout is 99.9% availability over a 28-day rolling window. You inspect the alerting policy and find it uses a time series aggregation with a 1-minute alignment period and a condition that triggers when the ratio of errors to total requests exceeds 0.001 for 5 consecutive minutes. What is the most likely reason the alert is not firing?

Accepted Answer

The alert condition requires 5 consecutive minutes of breach, but the error rate spikes are intermittent and not sustained.. The alert condition requires the error ratio to exceed 0.001 for 5 consecutive 1-minute alignment periods. If the error rate spikes are intermittent—lasting only a minute or two before returning to normal—the condition of 5 consecutive minutes of breach is never met, so the alert remains silent. This is a classic case where the alerting policy's duration setting is too long relative to the bursty nature of the errors.

Answer

The error budget has been exhausted, so the alert is suppressed.

Answer

The SLO window is too long, and the alert condition uses a different measurement period.

Answer

The ratio threshold is too high because the total request count is low.

Question 13

You are the DevOps engineer for a large e-commerce platform running on Google Kubernetes Engine (GKE). During a flash sale, you observe that the payments service is experiencing high latency and intermittent errors. The service is deployed with HorizontalPodAutoscaler (HPA) based on CPU utilization. You need to quickly diagnose and mitigate the issue. Which TWO actions should you take?

Accepted Answer

Use Cloud Monitoring to examine the payments service's request latency and error rate metrics, and create a custom dashboard for real-time monitoring.. Cloud Monitoring provides the necessary observability to diagnose the root cause of high latency and intermittent errors by examining request latency and error rate metrics. Creating a custom dashboard enables real-time monitoring, allowing you to correlate performance degradation with traffic spikes during the flash sale. This is the first step in incident management: observe before acting.

Answer

Check the GKE node's network performance using VPC Flow Logs and increase the node pool size.

Answer

Modify the HPA to use memory utilization instead of CPU, as memory is more indicative of the service's performance.

Answer

Configure a custom metric in Cloud Monitoring for the payments service's request queue depth and use it for HPA.

Question 14

Refer to the exhibit. You are investigating a performance issue where the api-server container is using excessive CPU. You run a Cloud Monitoring API query and receive the JSON configuration shown. However, the query returns no data points. What is the most likely cause?

Accepted Answer

The filter is missing required resource labels such as project_id, location, cluster_name, and namespace_name, causing no time series to match.. The Cloud Monitoring API query for the 'kubernetes.io/container/cpu/core_usage_time' metric type requires mandatory resource labels—specifically 'project_id', 'location', 'cluster_name', and 'namespace_name'—to uniquely identify the time series for a GKE container. Without these labels in the filter, the query cannot match any time series, resulting in no data points returned, even if the metric is actively emitting data.

Answer

The time interval specified is too short and falls outside the data retention period.

Answer

The metric type 'kubernetes.io/container/cpu/core_usage_time' is deprecated and no longer available.

Answer

The aggregation perSeriesAligner 'ALIGN_MEAN' is incompatible with the metric type, which requires 'ALIGN_RATE'.

Question 15

Order the steps to deploy a new version of a microservice to Google Kubernetes Engine using a rolling update.

Accepted Answer

Update the manifest, then apply the manifest using kubectl apply, then monitor the rollout status, then verify the deployment, then roll back if needed.. The correct sequence for a rolling update in GKE is: update the manifest, apply it with kubectl apply, monitor the rollout status, verify the deployment, and roll back if needed. This ensures the new version is properly deployed and validated before considering it successful, with a safety net for issues.

Answer

Update the manifest, then apply the manifest using kubectl apply, then verify the deployment, then monitor the rollout status, then roll back if needed.

Answer

Apply the manifest using kubectl apply, then update the manifest, then monitor the rollout status, then verify the deployment, then roll back if needed.

Answer

Update the manifest, then monitor the rollout status, then apply the manifest using kubectl apply, then verify the deployment, then roll back if needed.

Question 16

Arrange the steps to migrate a monolithic application to microservices on Google Kubernetes Engine.

Accepted Answer

Identify contexts, then containerize, then deploy, then set up communication between microservices, then redirect traffic.. Identify contexts, containerize, deploy, set up communication, redirect traffic.

Answer

Containerize, then identify contexts, then deploy, then set up communication between microservices, then redirect traffic.

Answer

Identify contexts, then deploy, then containerize, then set up communication between microservices, then redirect traffic.

Answer

Identify contexts, then containerize, then set up communication between microservices, then deploy, then redirect traffic.

Question 17

Match each CI/CD concept to its definition.

Accepted Answer

Continuous Integration: The practice of frequently merging code changes into a shared repository, verified by automated builds and tests.. Continuous Integration focuses on frequent merging and automated testing; Continuous Delivery extends to staging deployment with manual production release; Continuous Deployment automates production releases. Common confusions arise from mixing these definitions.

Answer

Continuous Integration: The practice of automatically deploying code to production after each change.

Answer

Continuous Delivery: The practice of merging code changes frequently into a shared repository.

Question 18

Match each Google Cloud DevOps capability to its benefit.

Accepted Answer

Cloud Build: Automates build, test, and deployment. Cloud Build automates CI/CD pipelines, and Artifact Registry stores build artifacts. Common confusions involve mixing capabilities like monitoring (Cloud Monitoring) or source code management (Cloud Source Repositories) with other tools.

Answer

Cloud Build: Monitors application performance

Answer

Cloud Deploy: Manages source code repositories

Answer

Cloud Monitoring: Provides continuous delivery pipelines

Question 19

Your team receives an alert that the Error Reporting count for a critical service has increased tenfold in the last 10 minutes. You suspect a recent code deployment is the cause. What is the first action you should take?

Accepted Answer

Roll back the deployment to the previous version.. A tenfold increase in error reporting within 10 minutes strongly indicates a recent code deployment introduced a critical bug. The immediate priority is to restore service stability by rolling back the deployment to the previous known-good version, as this directly mitigates the root cause. Delaying the rollback risks further degradation of the service and potential data loss or corruption.

Answer

Disable the alert to reduce noise.

Answer

Increase the instance count to handle the load.

Answer

Open a post-mortem to document the incident.

Question 20

You are investigating a slow increase in latency for a service running on Compute Engine. You have Cloud Monitoring and Cloud Logging set up. Which tool would best help you identify the cause of the latency?

Accepted Answer

Cloud Trace. Cloud Trace is designed to capture latency data from distributed applications by tracing requests as they propagate through services. It provides detailed latency distributions and per-span breakdowns, allowing you to pinpoint which component or operation is causing the slowdown. For a Compute Engine service with Cloud Monitoring and Logging already in place, Cloud Trace is the most direct tool to analyze request-level latency.

Answer

Error Reporting

Answer

Cloud Profiler

Answer

Cloud Debugger