Hard Difficulty Questions

Use a global external HTTP(S) load balancer with a single backend service pointing to one regional cluster.

Why wrong: A single backend service does not provide multi-region failover; traffic is sent to one region.

Use Cloud CDN in front of a single regional GKE cluster to cache content.

Why wrong: Cloud CDN does not provide load balancing failover across regions.

Use a single regional GKE cluster with auto-scaling across zones.

Why wrong: A single regional cluster provides zone-level HA but cannot survive a regional failure.

Cloud VPN

Why wrong: Uses public internet, latency not guaranteed.

Cloud NAT

Why wrong: Used for outbound internet access, not on-prem connectivity.

Peering with Google

Why wrong: Public peering, no dedicated bandwidth SLA.

Default Google-managed encryption

Why wrong: Google manages the keys, not the customer, so rotation cannot be controlled.

Cloud HSM

Why wrong: Cloud HSM provides hardware-backed keys but also uses CMEK approach; however, the question specifies managing keys themselves, and CMEK is the general term. Cloud HSM is a type of CMEK.

Customer-Supplied Encryption Keys (CSEK)

Why wrong: CSEK requires customers to provide keys with each API call, which is not suitable for automatic rotation every 90 days.

Use Cloud SQL for reads and Spanner for writes only

Why wrong: This would require application changes and adds complexity, not a simple configuration change.

Switch to a multi-region configuration with witness nodes

Why wrong: Multi-region increases cost, not reduces.

Reduce the number of read replicas in the instance configuration

Why wrong: Spanner does not have separate read replicas; it uses nodes that handle both reads and writes.

Use preemptible VMs to reduce cost

Why wrong: Preemptible VMs are for cost savings, not performance improvement.

Deploy additional instances in us-west1

Why wrong: us-west1 is still not close to Europe or Asia; better to deploy in Europe and Asia.

Use Cloud Armor to block high-latency requests

Why wrong: Cloud Armor does not improve latency; it's a security service.

The cluster autoscaler is using the 'least-waste' expander.

Why wrong: That expander is fine.

The horizontal pod autoscaler (HPA) is misconfigured.

Why wrong: HPA scales pods, not nodes.

The pod disruption budget (PDB) is too restrictive.

Why wrong: PDB prevents voluntary disruptions, not scaling.

Create a service account key and mount it as a secret in the pod

Why wrong: This would use long-lived keys, which Workload Identity is designed to avoid.

Grant the GSA the roles/iam.serviceAccountUser role on the project

Why wrong: This would allow the GSA to be used by users, not bind a KSA to a GSA.

Add an annotation to the GSA referencing the KSA, and grant the KSA the iam.workloadIdentityUser role on the GSA

Why wrong: The annotation is on the KSA, not the GSA.

Only the Shared VPC host project admin can create subnets.

Why wrong: The host project admin can grant subnet creation permissions to service project admins.

The Shared VPC is not enabled for the service project.

Why wrong: If not enabled, the error would be different; permission denied indicates lack of IAM.

Subnets must be created in the service project, not the host project.

Why wrong: Shared VPC subnets are created in the host project.

The Kubernetes service account needs the iam.workloadIdentityUser role

Why wrong: The role is on the Google service account, not the Kubernetes SA.

The pod needs a node pool with Workload Identity enabled

Why wrong: Workload Identity is enabled at the cluster level, not per node pool.

The BigQuery dataset must be in the same project as the cluster

Why wrong: Cross-project access is supported with proper IAM.

Use Cloud Bigtable with replication across regions

Why wrong: Bigtable's replication is eventually consistent, not strongly consistent.

Use Cloud SQL with read replicas in each region and enable synchronous replication

Why wrong: Cloud SQL does not support synchronous replication across regions.

Use Cloud SQL with cross-region replication and Memorystore cache in each region

Why wrong: Cloud SQL cross-region replication is asynchronous, so strong consistency is not guaranteed across regions.

Increase the SLO target to 99.99% to reduce future burn rate.

Why wrong: Increasing the SLO target would make it harder to meet, not easier.

Trigger a rollback of the latest deployment to stabilize the service.

Why wrong: Rolling back might help, but the first step should be to stop any changes that could worsen the situation.

Scale up the service to handle more traffic and reduce error rate.

Why wrong: Scaling may not address the root cause and could be resource-intensive; investigation is needed first.

Create a new MIG with the new template and use a Cloud Load Balancer's traffic splitting

Why wrong: This is a manual method; Cloud Deploy provides automation and verification.

Manually update each instance by SSH'ing and running a script

Why wrong: Manual process is error-prone and not scalable.

Use gcloud compute instance-groups managed rolling-action start-update with a maxSurge of 0

Why wrong: This does a rolling update without canary traffic splitting.

Scale up the Cloud Spanner instance to handle higher peak CPU, as the 80% spikes indicate insufficient capacity.

Why wrong: Spanner CPU at 65% average with spikes to 80% is still within acceptable range; the errors correlate with node CPU, not Spanner.

Increase the CPU request for the frontend pods and set a higher target CPU utilization for the Horizontal Pod Autoscaler.

Why wrong: This might reduce node CPU but would not fix cross-zone routing issues that cause latency and health check failures.

Increase the health check interval and timeout settings to give pods more time to respond before being marked unhealthy.

Why wrong: This would mask the problem but not address the root cause of cross-zone traffic and latency.

BGP MD5 authentication failure.

Why wrong: Authentication failure would show a clear error message, not flapping.

Incorrect local AS number in Cloud Router configuration.

Why wrong: AS number mismatch would prevent the session from establishing at all.

BGP timer misconfiguration between peers.

Why wrong: Timer misconfiguration might cause session timeout but not flapping due to network changes.

Cloud Memorystore

Why wrong: Memorystore (Redis) is great for caching but not a durable database; data loss possible.

Cloud Spanner

Why wrong: Spanner provides strong consistency, which is not required, and is more costly.

Cloud Firestore

Why wrong: Firestore is real-time but may not scale to millions of concurrent writes without sharding.

Create a read replica in a different region.

Why wrong: Read replicas require manual promotion for failover, not automatic.

Export the database daily and import into a new instance if failure occurs.

Why wrong: This is a manual backup strategy, not high availability.

Deploy Cloud SQL across multiple regions using cross-region replication.

Why wrong: Cross-region replication is not available for Cloud SQL MySQL; it's for PostgreSQL and SQL Server.

Set up Cloud Monitoring alerts with email notifications to the on-call engineer.

Why wrong: Alerts still require manual response.

Create detailed runbooks and require the on-call to follow them step by step.

Why wrong: Runbooks are manual processes.

Enable Cloud Logging and set up a custom dashboard for the on-call.

Why wrong: Dashboards provide visibility but do not automate response.

Migrate the backend to Cloud SQL for MySQL with cross-region replication, and keep the frontend on GKE with multi-region ingress.

Why wrong: The backend is a statefulset, not a relational database; migrating to Cloud SQL would require significant application changes and may not be compatible.

Deploy the frontend and backend in a regional GKE cluster and use regional persistent disks for the statefulset, replicating data synchronously across zones.

Why wrong: Regional persistent disks replicate within a region, not across regions; cannot survive a regional outage.

Deploy the frontend and backend in a single zonal cluster in us-central1-a, and use scheduled snapshots of persistent disks to a different region.

Why wrong: A single zonal cluster cannot survive a regional failure; the cluster itself is down.

A single-region deployment for consistency

Why wrong: A single-region deployment is vulnerable to regional outages.

Load testing to validate performance

Why wrong: Load testing is part of performance testing, not core to DR plan.