PCDOE Applying site reliability engineering practices — Timed 25 Questions

Question 1

A team uses Google Kubernetes Engine (GKE) with cluster telemetry enabled. During an incident, they notice that a deployment's pods are repeatedly crashing with Exit Code 137. The team wants to investigate the root cause. Which two Google Cloud services should they use together to correlate resource usage and logs?

25 minute time limit — choose an answer to begin.

Accepted Answer

Cloud Monitoring and Cloud Logging. Exit Code 137 indicates a container was killed by SIGKILL, typically due to an out-of-memory (OOM) condition. To correlate resource usage and logs, Cloud Monitoring provides resource usage metrics (e.g., memory, CPU), while Cloud Logging captures container termination logs and system events. Together, they allow correlation of spikes in resource usage with log entries to identify OOM kills. Cloud Trace is not primarily a logging service; it provides distributed tracing for latency analysis, but does not directly correlate logs with resource usage. Therefore, only Cloud Monitoring and Cloud Logging together (option A) satisfy the requirement.

Answer

Security Command Center and Cloud Logging

Answer

Cloud Trace and Cloud Monitoring

Answer

Cloud Error Reporting and Cloud Logging

Question 2

Refer to the exhibit. A GKE pod is repeatedly crashing with the error shown. The deployment has resource requests of 512 MiB memory and limits of 1 GiB. What is the most likely cause and the best remediation?

Accepted Answer

The Java heap size exceeds the container memory limit; reduce the JVM heap size or increase the container memory limit. The error indicates an OutOfMemoryError (OOM) in the Java application, which occurs when the JVM heap size exceeds the container's memory limit. Since the deployment has a memory limit of 1 GiB, if the JVM is configured with a heap size larger than this limit (or if the heap plus other memory usage exceeds it), the container will be killed by Kubernetes. Reducing the JVM heap size or increasing the container memory limit directly resolves the mismatch.

Answer

The node is under memory pressure; add more nodes to the cluster

Answer

The container needs more CPU; increase CPU request and limit

Answer

The application has a memory leak; refactor the DataProcessor class

Question 3

You are a Site Reliability Engineer (SRE) for an e-commerce platform running on Google Kubernetes Engine (GKE) with a microservices architecture. Your team uses Cloud Monitoring for alerting and Cloud Logging for centralized logs. Recently, during a flash sale event, you observed intermittent latency spikes in the checkout service, causing checkout failures and abandoned carts. The latency spikes last 1-2 seconds and occur roughly every 5-10 minutes during peak traffic. The checkout service runs as a Deployment with 10 replicas, each with resource requests of 500m CPU and 512Mi memory. The service has a Service Level Objective (SLO) of 99.9% of requests completing in under 1 second (p99 latency < 1s). Current p99 latency is 2.1s during peak. You reviewed the Cloud Monitoring dashboard and noticed that CPU utilization across pods is around 60%, memory around 50%, and there are no OOM kills. The logs show occasional 'connection reset by peer' errors in the checkout service logs, but no consistent pattern. You suspect the issue might be related to the database (Cloud SQL) or a downstream dependency. After checking the database, you find that query latency is normal. You also notice that the checkout service makes a synchronous HTTP call to a payment validation service that runs as a separate Deployment with 3 replicas. The payment service's p99 latency is 500ms, but its error rate is below 1%. Your task is to identify the most likely cause of the intermittent latency spikes and propose a remediation. Which action should you take first?

Accepted Answer

Check the garbage collection logs of the checkout service pods to identify if long GC pauses coincide with the latency spikes.. The intermittent latency spikes every 5-10 minutes with no CPU/memory pressure or database issues strongly suggest a periodic process like garbage collection. Java-based services (common in microservices) can experience stop-the-world GC pauses that cause latency spikes of 1-2 seconds, matching the observed pattern. Checking GC logs is the fastest way to confirm this before making architectural changes.

Answer

Increase the number of replicas of the payment validation service to 10 to handle peak load.

Answer

Enable connection pooling and retries with exponential backoff in the checkout service for the HTTP call to the payment service.

Answer

Investigate the checkout service pod restarts due to liveness probe failures, as 'connection reset by peer' indicates pod instability.

Question 4

Refer to the exhibit. You are investigating a performance issue where the api-server container is using excessive CPU. You run a Cloud Monitoring API query and receive the JSON configuration shown. However, the query returns no data points. What is the most likely cause?

Accepted Answer

The filter is missing required resource labels such as project_id, location, cluster_name, and namespace_name, causing no time series to match.. The Cloud Monitoring API query for the 'kubernetes.io/container/cpu/core_usage_time' metric type requires mandatory resource labels—specifically 'project_id', 'location', 'cluster_name', and 'namespace_name'—to uniquely identify the time series for a GKE container. Without these labels in the filter, the query cannot match any time series, resulting in no data points returned, even if the metric is actively emitting data.

Answer

The time interval specified is too short and falls outside the data retention period.

Answer

The metric type 'kubernetes.io/container/cpu/core_usage_time' is deprecated and no longer available.

Answer

The aggregation perSeriesAligner 'ALIGN_MEAN' is incompatible with the metric type, which requires 'ALIGN_RATE'.

Question 5

Match each CI/CD concept to its definition.

Accepted Answer

Continuous Integration: The practice of frequently merging code changes into a shared repository, verified by automated builds and tests.. Continuous Integration focuses on frequent merging and automated testing; Continuous Delivery extends to staging deployment with manual production release; Continuous Deployment automates production releases. Common confusions arise from mixing these definitions.

Answer

Continuous Integration: The practice of automatically deploying code to production after each change.

Answer

Continuous Delivery: The practice of merging code changes frequently into a shared repository.

Question 6

Match each Google Cloud DevOps capability to its benefit.

Accepted Answer

Cloud Build: Automates build, test, and deployment. Cloud Build automates CI/CD pipelines, and Artifact Registry stores build artifacts. Common confusions involve mixing capabilities like monitoring (Cloud Monitoring) or source code management (Cloud Source Repositories) with other tools.

Answer

Cloud Build: Monitors application performance

Answer

Cloud Deploy: Manages source code repositories

Answer

Cloud Monitoring: Provides continuous delivery pipelines

Question 7

Your SLO for availability is 99.9% over a 30-day window. You want an alert that fires when the error budget burn rate is high, leaving less than 5% of the error budget remaining in the next 6 hours. What type of alerting policy should you configure?

Accepted Answer

A burn rate alert based on the forecasted consumption.. A burn rate alert based on forecasted consumption directly monitors the rate at which the error budget is being consumed and triggers when the projected remaining budget falls below 5% within the next 6 hours. This aligns with Google SRE best practices for proactive error budget management, using a multi-window, multi-burn-rate approach to detect rapid consumption before the budget is exhausted.

Answer

A custom log-based metric.

Answer

A static threshold alert based on the error rate.

Answer

An exponential decay alert.

Question 8

During a post-mortem, you identify that an incident was caused by a configuration change that was not reviewed. Which of the following is the most effective preventive action?

Accepted Answer

Implement a change management process with mandatory peer review.. A change management process with mandatory peer review directly addresses the root cause: a configuration change was made without oversight. By requiring at least one additional engineer to review and approve changes before implementation, the process catches misconfigurations, policy violations, or unintended side effects before they reach production. This is a preventive control, not a detective or corrective one, and aligns with ITIL best practices for change management.

Answer

Add more monitoring alerts.

Answer

Schedule weekly meetings to review changes.

Answer

Use a configuration management database (CMDB).

Question 9

Your incident response team uses a follow-the-sun model. An incident occurs during the Asia-Pacific shift, but the escalation path requires sign-off from the US-based team lead. This causes delays. What change should you recommend?

Accepted Answer

Implement a global incident commander role with delegated authority.. A global incident commander with delegated authority can make escalation decisions without waiting for a specific time-zone-based team lead. This role operates across shifts, ensuring that critical incident response actions are not delayed by geographic handoffs. In a follow-the-sun model, this role provides continuous, authoritative decision-making, aligning with ITIL incident management best practices for global teams.

Answer

Use a chatbot for automated responses.

Answer

Increase the number of US team members.

Answer

Schedule the US team lead to work overnight.

Question 10

You are debugging a production issue where a Cloud Function occasionally throws a 'memory limit exceeded' error. You want to inspect the memory usage at the time of the error. What should you do?

Accepted Answer

Enable Cloud Profiler and analyze the snapshot.. Cloud Profiler provides continuous, low-overhead profiling of CPU and memory usage, and its snapshot analysis can pinpoint memory allocation patterns at the time of a memory limit exceeded error. Unlike other tools, Profiler captures the call stack and memory consumption per function invocation, enabling you to identify the specific code path causing the spike.

Answer

Check Cloud Logging for memory metrics.

Answer

Use Cloud Trace to trace the invocations.

Answer

Use Cloud Debugger to set a breakpoint.

Question 11

Your application runs in two GCP regions. A regional outage occurs in the primary region. You have a Cloud Load Balancer with a failover backend. However, the failover did not trigger because the health check passed on a stale connection. What is the best solution?

Accepted Answer

Use a global load balancer with HTTP health checks based on application health.. A global load balancer with HTTP health checks can probe the actual application endpoint (e.g., /healthz) to verify end-to-end functionality, not just TCP connectivity. This prevents the stale connection issue where a TCP health check passes on an existing but broken session, ensuring failover triggers only when the application is genuinely unhealthy.

Answer

Use a passive health check.

Answer

Configure a custom health check that checks the database.

Answer

Use TCP health checks with a shorter interval.

Question 12

You are responding to an incident where a new release has caused increased error rates. Which TWO actions should you take immediately?

Accepted Answer

Notify stakeholders.. Option B (Notify stakeholders) is correct because during an active incident, keeping affected teams, customers, and leadership informed ensures transparency, enables coordinated response, and manages expectations while remediation is underway. Option D (Roll back the release) is correct because reverting to the last known good version is the fastest way to restore service and stop the increased error rates caused by the new release, aligning with incident response best practices. Option A (Disable the alert) is wrong because silencing alerts hides the problem and prevents monitoring from confirming recovery. Option C (Push a hotfix without testing) is wrong because deploying untested code during an incident risks introducing additional failures. Option E (Create a post-mortem document) is wrong because post-mortems are conducted after the incident is resolved, not immediately during it.

Answer

Disable the alert.

Answer

Push a hotfix without testing.

Answer

Create a post-mortem document.

Question 13

Refer to the exhibit. You see this log entry from a Cloud Run service. The stack trace shows the error occurs in handler.js at line 50. You want to see the state of variables at that point in the production environment without adding logging or redeploying. What should you do?

Accepted Answer

Use Cloud Debugger to set a snapshot location at line 50 in handler.js.. Cloud Debugger allows you to inspect the state of an application, including local variables and call stack, at a specific line of code in a production environment without modifying or redeploying the application. By setting a snapshot at line 50 in handler.js, you can capture the variable values at the exact point where the error occurs, which directly addresses the need to debug without adding logging or redeploying.

Answer

Use Error Reporting to view similar errors.

Answer

Use Cloud Profiler to capture a heap snapshot.

Answer

Use Cloud Trace to trace the request.

Question 14

During an incident, a DevOps engineer needs to temporarily increase the capacity of a Google Kubernetes Engine (GKE) cluster to handle the traffic surge. Which approach minimizes manual intervention and follows Google best practices?

Accepted Answer

Enable cluster autoscaler and update the horizontal pod autoscaler to scale faster.. Enabling the cluster autoscaler automatically adjusts the number of nodes in the node pool based on resource demands, while updating the horizontal pod autoscaler (HPA) to scale faster (e.g., reducing the stabilization window or increasing the target CPU utilization threshold) allows pods to replicate more quickly. This combination minimizes manual intervention by automating both pod-level and node-level scaling, aligning with Google's best practices for handling traffic surges in GKE.

Answer

Manually add a node pool with larger machines via the Google Cloud Console.

Answer

Create a new node pool and migrate pods using kubectl drain.

Answer

Scale the existing node pool by increasing the maximum node count in the cluster.

Question 15

An organization has a service that must meet a 99.99% SLO. The service runs on GKE and uses Cloud SQL. The team notices that during a major incident, the error budget is consumed rapidly. They want to implement a mechanism to automatically rollback deployments that cause sustained error budget consumption above a threshold. What is the best approach?

Accepted Answer

Set up a deployment pipeline with Cloud Deploy that includes a predeployment validation step that checks the current error budget burn rate and blocks the release if the burn rate exceeds 10% per hour.. Cloud Deploy supports predeployment validation steps that can query the current error budget burn rate and block a release if it exceeds a defined threshold, which directly implements automated rollback prevention of error-budget-destroying deployments. This integrates SLO enforcement into the delivery pipeline rather than reacting after the fact.

Answer

Use Cloud Scheduler to run a script that checks error budget and rolls back if needed.

Answer

Implement a canary deployment strategy with manual approval steps.

Answer

Configure Cloud Build to automatically revert the last commit if error budget is consumed.

Question 16

During a post-incident review, the team discovers that a misconfiguration in Cloud Armor caused legitimate traffic to be blocked, leading to a outage. The misconfiguration was introduced by a junior engineer who had overly permissive IAM roles. What is the best way to prevent similar incidents in the future?

Accepted Answer

Use Organization Policy constraints to restrict allowed IP ranges and rules in Cloud Armor security policies.. Organization Policy constraints in Google Cloud allow administrators to enforce guardrails at the organization, folder, or project level that restrict what configurations are allowed — for example, limiting allowed IP ranges or rule types in Cloud Armor security policies. This prevents a misconfigured policy from ever being applied, regardless of the engineer's IAM permissions, addressing the root cause of overly permissive changes.

Answer

Enforce a mandatory peer review for all Cloud Armor configuration changes.

Answer

Revoke the junior engineer's access to Cloud Armor and grant read-only access.

Answer

Enable Cloud Armor security policy logs and create alerting for blocked traffic spikes.

Question 17

Which THREE of the following are valid techniques for mitigating a denial-of-service (DoS) attack against a Google Cloud HTTP(S) Load Balancer?

Accepted Answer

Enable autoscaling on the backend services to handle increased load.. Option B is correct because enabling autoscaling on the backend services lets the managed instance groups add capacity dynamically as the load balancer's traffic spikes, absorbing volumetric DoS traffic without manual intervention. Option D is correct because rate limiting per client IP via Cloud Armor security policies (or the load balancer's rate-limiting settings) throttles abusive clients so a single source cannot exhaust backend resources. Option E is correct because Cloud Armor security policies can block or deny suspicious source IP addresses and apply preconfigured WAF rules (e.g., OWASP Top 10) at the edge before traffic reaches the HTTP(S) Load Balancer. Option A is not a valid mitigation technique because simply increasing backend instances is a manual, reactive capacity change rather than a DoS mitigation mechanism, and it does not stop the attack traffic itself. Option C is not valid because VPC firewall rules apply to traffic within the VPC and cannot filter or block traffic destined for an external HTTP(S) Load Balancer's frontend IP; edge protection must be done with Cloud Armor instead.

Answer

Increase the number of backend instances to absorb traffic.

Answer

Modify VPC firewall rules to block all traffic from the source IP.

Question 18

A DevOps engineer notices that a critical service is down, but no alert has been received. The engineer checks Cloud Monitoring and sees that the alerting policy appears to be correctly configured. What is the most likely cause?

Accepted Answer

The notification channel is misconfigured.. If the notification channel is misconfigured (e.g., invalid webhook URL, incorrect email address, or missing PagerDuty integration key), Cloud Monitoring will correctly detect the incident but fail to deliver the alert. The engineer sees the alerting policy appears correct because the policy itself is valid, but the channel configuration prevents the notification from reaching the recipient.

Answer

The metrics for the service are not being collected.

Answer

The incident was created but automatically closed.

Answer

The SLO is defined too loosely, so the error budget is not exhausted.

Question 19

A team defines an SLO of 99.9% availability over a 30-day window. They use a multi-window, multi-burn-rate alerting approach. Which alerting condition should trigger a page based on fast burn rate?

Accepted Answer

2% error budget consumed in 10 minutes.. A multi-window, multi-burn-rate alerting approach uses a short window (e.g., 10 minutes) to detect fast burn rates that could rapidly exhaust the error budget. Consuming 2% of the error budget in 10 minutes corresponds to an annualized burn rate of over 1000x, which is dangerously fast and requires immediate paging to prevent a breach of the 99.9% SLO over the 30-day window.

Answer

1% error budget consumed in 1 hour.

Answer

10% error budget consumed in 1 hour.

Answer

5% error budget consumed in 6 hours.

Question 20

During a canary deployment of a new version of a microservice, the engineer notices increased error rates in the canary instances. What is the best immediate action?

Accepted Answer

Perform a rollback of the canary to the previous version.. The best immediate action is to perform a rollback of the canary to the previous version. This stops the increased error rates from affecting users and allows the team to investigate the issue without impacting the production environment. Rollback is a standard canary deployment practice when the canary shows signs of failure.

Answer

Continue the rollout to see if errors stabilize.

Answer

Scale up the canary instances to handle load.

Answer

Pause the rollout and investigate the errors.