Ensure solution and operations reliability practice questions

Export the database daily using gsutil and import in the other region using pg_restore.

Why it fails: Daily export and import is a batch restore, so recovery point and time depend on the last dump; 500 GB cannot restore within ten minutes. It suits periodic backup copies, not sub-ten-minute regional failover. Cloud SQL cross-region replicas provide continuous replication instead.

Automated backups with point-in-time recovery to a new instance in the other region.

Why it fails: Automated backups with point-in-time recovery require restoring a backup to a new instance, which can take longer than 10 minutes for a 500 GB database, and RPO depends on backup frequency, typically 1-7 days, so it does not meet the requirements.

Use a Deployment with a PVC that has allowedTopologies restricting to the desired zone.

Why it fails: Deployment pods are not guaranteed to have stable identities, and PVC binding can break on rescheduling.

Use a Deployment with a persistent volume that is manually attached after pod creation.

Why it fails: Deployments do not guarantee persistent pod identity, and manual attachment is error-prone.

Use a StatefulSet with a persistent disk that has access mode ReadOnlyMany.

Why it fails: StatefulSet requires ReadWriteOnce for stateful workloads to allow the pod to write.

Configure DNS-based failover using Cloud DNS with health checks.

Why it fails: DNS failover is slower and less reliable than load balancer-based failover.

Use an internal load balancer to route traffic between regions.

Why it fails: Internal load balancers handle internal traffic and are not suitable for external user traffic.

Use a regional external HTTP load balancer with a multi-region backend.

Why it fails: Regional load balancers only serve traffic within one region; they cannot failover across regions.

The content is compressed with gzip.

Why it fails: Compression does not affect cache staleness.

The CDN is configured with signed URLs.

Why it fails: Signed URLs control access, not cache freshness.

Use a single-zone node pool with multiple replicas.

Why it fails: Single-zone node pools are vulnerable to zone failures and may become unavailable during upgrades in that zone.

Enable cluster autoscaling to add nodes during upgrades.

Why it fails: Autoscaling adds nodes based on resource demand, not to maintain availability during upgrades.

Enable regional clusters for multi-zone control plane.

Why it fails: Regional clusters provide high availability for the control plane, but node pools must also be multi-zone for workload availability.

Increase the maximum number of Cloud Run instances to 20 to handle more requests.

Why it fails: More instances increase the number of database connections, potentially exceeding the limit further.

Upgrade the Cloud SQL instance to db-n1-standard-4 (15 GB memory) to handle more connections.

Why it fails: A larger machine provides more memory and CPU but does not automatically increase max_connections; also cost increases significantly.

Reduce the concurrency setting on Cloud Run from 80 to 40.

Why it fails: Reducing concurrency reduces throughput and may not fully solve the connection limit issue if the number of instances remains high.

Increase the number of CPUs on the primary Cloud SQL instance.

Why it fails: Upgrading the primary instance helps but is less cost-effective than read replicas.

Use Cloud SQL Proxy with connection pooling.

Why it fails: Connection pooling helps with connection management, not read throughput.

Enable automatic storage increase to allow more data.

Why it fails: Storage increase does not improve read performance.

Use a regional persistent disk to replicate data across zones.

Why it fails: Regional disk protects against zone failure but not VM failure; logs are still on disk but not streamed off-instance.

Schedule persistent disk snapshots every 5 minutes.

Why it fails: Snapshots are not real-time and may miss logs generated between snapshots.

Create a script to copy logs to a Cloud Storage bucket every minute.

Why it fails: This is a manual workaround but not as reliable as a managed agent; also, if VM fails, the script stops.

Deploy all nodes in the same zone to simplify networking.

Why it fails: Single zone is a single point of failure.

Prefer using preemptible VMs for cost savings.

Why it fails: Preemptible VMs can be terminated at any time, reducing availability.

Use a single zonal cluster to avoid cross-zone latency.

Why it fails: A single zone is a single point of failure.

Use a managed instance group with a startup script that processes data, and use Cloud Pub/Sub to coordinate.

Why it fails: This adds complexity and requires custom orchestration; not as reliable as managed services.

Increase the VM to a high-CPU machine type with a regional persistent disk for HA.

Why it fails: A single VM is a single point of failure; upgrading machine type may not achieve 4 hours and increases cost.

Deploy the processing logic in Cloud Functions and trigger from Cloud Storage events.

Why it fails: Cloud Functions have execution timeouts and are not designed for long-running batch processing.

Increase the maximum connections setting on the Cloud SQL instance and also increase the instance's tier to handle more concurrent connections.

Why it fails: Only temporarily resolves the symptom but does not scale efficiently; the database may still become a bottleneck.

Migrate the database to Cloud Spanner to provide unlimited scalability and automatic sharding.

Why it fails: An expensive and complex migration that may not be necessary for this use case.

Implement a connection pooling library in the application code to reuse database connections and reduce the number of new connections.

Why it fails: Connection pooling reduces the overhead of opening new database connections by reusing existing ones, but it does not increase the maximum connections limit of the Cloud SQL instance itself. During the flash sale, the MIG scaled up to many instances, each potentially holding a pool of connections; if the pool size per instance is not carefully tuned, the aggregate connections can still exceed the Cloud SQL limit. This option is tempting because connection pooling is a standard technique to improve database efficiency and reduce latency under normal load, and it would be correct if the problem were slow connection establishment rather than hitting a hard cap on concurrent connections.

Use preemptible VMs to reduce costs.

Why it fails: Preemptible VMs are short-lived and increase the risk of interruption, harming reliability.

Place an HTTP(S) load balancer in front of the application.

Why it fails: A load balancer distributes traffic but does not address stateful storage reliability.

Schedule regular snapshots of the persistent disk to Cloud Storage.

Why it fails: Snapshots provide backup but do not ensure high availability during a zone outage.

Cloud Router: A managed service that provides outbound internet access for private instances via NAT.

Why it fails: Wrong. The definition describes Cloud NAT, not Cloud Router. Cloud Router uses BGP to exchange routes between VPC and on-premises.

Cloud NAT: A managed service that dynamically exchanges routes between VPC and on-premises networks using BGP.

Why it fails: Wrong. The definition describes Cloud Router, not Cloud NAT. Cloud NAT provides outbound internet access for private instances.

Increase the ack deadline to the maximum value

Why it fails: This only pushes the problem later, doesn't resolve processing failures.

Set max delivery attempts to 1 to avoid redelivery

Why it fails: This would cause messages to be lost if processing fails.

Use a dead-letter topic to capture failed messages

Why it fails: Dead-letter topic is for messages that repeatedly fail, not for temporary transient errors.

Implement application-level retry with exponential backoff

Why it fails: Retry improves resilience to transient errors but does not address regional failures.

Use a single-region Spanner instance with a standby in a different zone

Why it fails: A single-region instance cannot survive a full regional outage.

Manually failover to a different region using a script

Why it fails: Manual intervention increases recovery time and risk of misconfiguration.

Increase the function timeout to the maximum allowed

Why it fails: This may help but doesn't address transient failures that could be resolved by retrying.

Use Cloud Tasks to decouple and retry synchronously

Why it fails: Cloud Tasks is for asynchronous task queues, not directly for Cloud Functions retry.

Refactor the function to reduce complexity

Why it fails: Refactoring may not address timeout issues if the function already has sufficient timeout.

Number of HTTP 5xx errors

Why it fails: Error count alone doesn't give a ratio of success; need denominator.

Request latency at the 99th percentile

Why it fails: Latency is an SLI for performance, not availability.

CPU utilization of backend instances

Why it fails: CPU utilization is internal, not a customer-facing SLI.

Increase the rate limiting threshold

Why it fails: Static threshold changes may not adapt to patterns and could still block legitimate spikes.

Disable the Web Application Firewall (WAF) rules

Why it fails: This removes protection and increases vulnerability.

Use the JSON Web Token (JWT) authentication to filter requests

Why it fails: JWT checks identity but doesn't prevent DDoS at the edge.

Create a new instance from the backup, then rename and delete the old instance

Why it fails: This is possible but slower and requires additional steps.

Use point-in-time recovery to restore to a time before corruption

Why it fails: Point-in-time recovery requires binary logging enabled, and may not be available for the exact moment before corruption.

Export the backup to Cloud Storage and import into the existing instance

Why it fails: Export/import is for data migration, not quick restore.

Use an internal load balancer with a backend service configured with a health check.

Why it fails: Internal LB is for internal traffic, not external users.

Use a network load balancer with a health check configured on the target pool.

Why it fails: Network LB is for TCP/UDP traffic, less application-aware.

Use an HTTP(S) load balancer with a backend service configured with a health check and enable session affinity.

Why it fails: Session affinity ties clients to specific instances, which can cause traffic to unhealthy instances.

Use a TCP proxy load balancer with a backend service configured with a health check.

Why it fails: TCP proxy works but is not ideal for HTTP applications.