Scaling with Google Cloud operations practice questions

The IT team will need to hire more staff to manage additional cloud infrastructure.

Why it fails: Cloud managed services are designed to reduce, not increase, operational headcount. Rather than hiring more staff to cope with infrastructure complexity, a managed service transfers the burden of physical hardware management, capacity planning, and routine software updates to Google's global infrastructure teams. That reduces toil, lowers 24/7 support demands, and typically allows the same team to support a larger scale of services—so adding staff to 'manage cloud infrastructure' runs counter to the stated goal of offloading maintenance and is not a realistic outcome.

The IT team will still perform the same tasks but remotely via the Cloud Console.

Why it fails: This is a misconception about the nature of managed services: moving to the Cloud Console does not mean the same maintenance tasks are performed remotely. When an organization adopts a managed service, Google runs the storage engines, applies OS security patches, replaces failed disks, and handles failover—the IT team no longer performs those individual tasks at all. The Cloud Console becomes a place for configuring service parameters, monitoring metrics, and managing access policies, not for executing the underlying maintenance that has been abstracted away.

The IT team will be fully automated out of their roles by Google's AI.

Why it fails: Google's AI and automation can replace repetitive operational actions (e.g., health checks, basic provisioning, log analysis), but it does not eliminate the need for IT professionals to make architectural decisions, enforce compliance, design access controls, and balance cost against performance. Human oversight is required for troubleshooting ambiguous incidents, communicating trade-offs to business stakeholders, and prioritizing feature work—tasks that AI currently augments rather than fully owns. The team's roles evolve and become more interesting, but they are not automated 'out of roles'; they are redirected toward judgment-driven activities.

$30,000 (cloud cost increase of $120K minus the $50K DC savings)

Why it fails: This incorrectly treats cloud costs as an addition rather than a replacement for data center costs.

$50,000 (only the data center cost savings count)

Why it fails: The full comparison is $200K previously vs. $120K now. The total savings is $80K, not just the $50K partial DC savings figure.

$120,000 (the entire cloud bill is savings)

Why it fails: The cloud bill is a cost, not a saving. The saving is the difference between what was previously spent ($200K) and what is now spent ($120K).

Basic support

Why it fails: Basic support provides only self-service documentation, community forums, and no access to technical support engineers. It has no guaranteed response times and certainly no dedicated TAM, so it fails both the <1 hour P1 response requirement and the TAM requirement. This tier is only suitable for non-production development and testing workloads.

Standard support

Why it fails: Standard support provides business-hours coverage with a 4-hour P1 response, which is far slower than the required 1-hour threshold. It also does not include a dedicated TAM, so it fails both the response-time requirement and the named-account management requirement. Standard support is best suited for non-critical environments where occasional issues can wait several hours.

Enhanced support

Why it fails: Enhanced support does offer a 1-hour P1 response, but it lacks a dedicated Technical Account Manager (TAM), which is a feature exclusive to the Premium tier. If both a sub-hour response time and a named TAM are mandatory requirements, Enhanced support alone cannot satisfy them. Therefore, Premium is the appropriate tier when both criteria must be met.

Spot VMs — up to 91% savings over on-demand pricing.

Why it fails: Spot VMs provide the deepest discount, up to 91% off on-demand pricing, but they are preemptible with only 30 seconds notice before termination. For a continuous 24/7 workload that cannot tolerate interruption, this makes Spot VMs an unreliable choice, as any preemption would cause downtime and potentially require manual intervention to restart the workload.

On-demand pricing — pay per minute with no commitment.

Why it fails: On-demand pricing charges per second or minute with no upfront commitment and no discount. This is the most expensive option for an always-on workload because it provides zero cost reduction, unlike CUDs which slash costs by 37-55%. For a stable, predictable 24/7 workload, choosing on-demand leaves significant savings unrealized, making it a poor financial decision.

Sustained Use Discounts — automatically applied to all running VMs.

Why it fails: Sustained Use Discounts are automatically applied to VMs that run for more than 25% of the month, offering up to 30% savings without any commitment. However, due to the automatic, per-month nature, they lack the higher discount rates of CUDs, which provide 37% to 55% savings for a known, stable workload. While useful for variable workloads, they are not the optimal financial choice for a guaranteed continuous 24/7 deployment.

Committed Use Discounts (CUDs), by committing to 1 or 3 years of VM usage

Why it fails: CUDs provide 37–55% discounts for committed 1 or 3-year terms. They are appropriate for stable, continuous workloads — not nightly batch jobs that only run a few hours. Spot VMs provide larger discounts with no commitment for interruptible batch workloads.

Sustained Use Discounts (SUDs), which automatically apply when VMs run for more than 25% of a month

Why it fails: Sustained Use Discounts automatically apply to VMs that run for extended periods (25%+ of a month). A nightly batch job running 4–6 hours may accrue some SUD but far less than Spot VM savings, and there's no maximum discount for Spot-sized savings from SUDs alone.

Capacity reservations, which guarantee capacity but do not provide a discounted pricing option for the nightly batch window

Why it fails: Reserved Instances (or Google Cloud Reservations) are a capacity-management tool, not a cost-reduction mechanism by themselves: they guarantee VM resources in a specific zone at on-demand rates, and any discount only materializes when paired with a Committed Use Discount. Purchasing a capacity reservation for a nightly batch window would require paying for the reserved resources 24/7, even during idle hours, which conflicts with the workload's short, interruptible schedule. Since the batch job can tolerate preemption and checkpointing, reserving dedicated capacity adds no value and would increase total cost, making Spot VMs the correct choice.

Manually applying Terraform changes from engineers' local machines and documenting changes in a shared wiki

Why it fails: Manual local Terraform applies mean each engineer's machine holds its own state, which quickly diverges from the shared remote state and from the wiki documentation. There is no automated drift detection or enforcement that the applied infrastructure matches any declared configuration, so the wiki often reflects aspirational rather than actual architecture. Moreover, changes are not peer-reviewed in a systematic way—wiki notes can be edited after the fact—leaving no immutable audit trail and no single source of truth to reconcile against.

Using the Google Cloud Console to make infrastructure changes and exporting the configuration to Git after each change

Why it fails: This approach inverts the GitOps model: Git becomes a post-hoc changelog rather than an enforced source of truth. Console modifications are applied directly to live resources without code review, and they can diverge from the repository until someone remembers to export the configuration. Even if exported, the process does not validate that the exported configuration reproduces the actual state, so drift goes undetected and there is no automated reconciliation to the declared state.

GitOps only applies to application code deployment, not to cloud infrastructure management

Why it fails: This statement confuses GitOps' origin with its scope. GitOps began as a pattern for Kubernetes deployments but has been formalized as a general infrastructure-as-code practice: tools like Terraform, Pulumi, and Config Connector can all store desired state in Git and use operators or CI/CD to apply and reconcile. Cloud infrastructure management benefits even more from GitOps because it requires strict change control, audit trails, and reproducible environments—exactly what a Git-based workflow provides. The same pull-request-and-reconciliation loop that governs application deployments governs infrastructure definitions.

Cloud Monitoring alerting policies that detect and notify when non-compliant VMs are created

Why it fails: Cloud Monitoring alerting policies are purely reactive: they watch metering, logging, and other telemetry signals, then notify you after a non-compliant VM already exists in the project. Resource creation is an API operation that succeeds or fails before any metric or log entry is emitted, so a Monitoring alert cannot intercept that request or block the VM from being provisioned. Even if the alert fires within seconds, the VM remains live and may expose an external IP or miss required labels, leaving a window of vulnerability that a preventive control like Organization Policy would have closed. Thus, Monitoring alerts are useful for detecting policy breaches over time, but they cannot enforce policy at the moment of creation.

Cloud IAM roles that prevent developers from creating VMs without the proper labels

Why it fails: IAM roles grant or deny the ability to perform actions (create a VM) but cannot enforce the content of those actions (what labels or IP configuration the VM has). IAM operates at the action level, not the attribute level.

Cloud Billing budget alerts that detect when VM spending exceeds expected amounts for labeled resources

Why it fails: Cloud Billing budget alerts are designed solely for cost governance: they evaluate aggregated spend against threshold amounts and can notify when forecast or actual costs exceed a budget, but they never inspect resource attributes such as external IPs or label key-value pairs. A VM that violates label or IP constraints may be inexpensive enough that it never triggers a budget alert, and the alert can appear hours later after billing data is processed; in any case, the non-compliant resource already exists. Budget alerts also cannot distinguish between compliant and non-compliant spending because they operate on cost amounts, not on the configuration of individual resources. Therefore, they provide no direct mechanism to block a VM creation that fails required-label or external-IP policies.

burn rate > 1 over 1 hour

Why it fails: A burn rate greater than 1 over a single one-hour window indicates that errors are being consumed faster than the SLO's steady-state allowance, but it does not measure the absolute remaining budget. For example, a short spike can trigger a high burn rate even when the budget is still abundant, and a low remaining budget could exist with a burn rate below 1 if the window is short. This is a velocity-oriented alert, not a budget-exhaustion alert, so it is not the correct trigger for 'error budget nearly exhausted'.

latency > 100ms

Why it fails: Latency above 100 milliseconds is a performance threshold, not a direct indicator of error budget consumption. The error budget tracks the count of bad events, such as requests that are too slow or fail, relative to the total number of requests; a single latency threshold gives no information about the proportion of bad events or the remaining budget. It also confounds response-time expectations with availability targets, making it an indirect and incomplete alerting condition.

SLI < 99.9%

Why it fails: Alerting when the SLI falls below 99.9% is effectively the same as alerting on an SLO breach, which happens only after the entire error budget has been exhausted. Since the goal is to act before the budget runs out, this condition is reactive and offers no lead time for remediation. In contrast, alerting on the remaining budget percentage triggers earlier, allowing the team to address issues before the SLO is violated.

Measuring everything with SLIs, SLOs, and SLAs

Why it fails: SLIs, SLOs, and SLAs are crucial for defining and tracking reliability, but the SRE principle is not to measure everything indiscriminately. Promiscuous instrumentation produces alert fatigue, noise, and a diluted focus on what actually matters to users. Instead, SRE teams should select a small number of meaningful, user-centric SLOs and iterate on them; moreover, SLAs are external commitments, while SLOs are internal targets, and over-measuring can create a false sense of control rather than improving service reliability.

Centralizing all operations in a single team

Why it fails: SRE explicitly rejects a centralized operations silo in favor of shared ownership of reliability between development and operations-oriented engineers. When all operations are consolidated into one team, it creates a bottleneck in communication and separates responsibilities, so developers lack direct feedback on how their code behaves in production. Shared ownership, often implemented via embedded SRE engineers and joint on-call, ensures reliability is everyone's concern: each service team understands its SLOs and is empowered to make trade-offs between feature work and operational health.

Deploying changes only during maintenance windows

Why it fails: Restricting deployments to maintenance windows runs counter to SRE's core philosophy of reducing risk through small, frequent, and reversible changes. Large, infrequent releases concentrate the chance of failure and make rollbacks or root-cause analysis difficult, while also slowing organizational velocity. Instead, SRE employs practices like canary deployments, progressive rollouts, and feature flags to ship low-risk increments continuously. This not only shortens the mean time to detect and fix defects but also creates a culture of experimentation, where a failed change affects only a small subset of users and is quickly reverted.

Decrease container concurrency.

Why it fails: Reducing container concurrency means each Cloud Run instance processes fewer simultaneous requests, so the service must create more instances to handle the same traffic volume. This increases the frequency of instance startup events, which doesn't reduce cold start latency—it actually makes cold starts more likely under load. Cold start latency is governed by container image size, runtime initialization, and instance lifecycle, not by how many requests each instance can accept concurrently.

Increase CPU limit.

Why it fails: Increasing the CPU limit allocates more compute time to each instance after it is already running, which can improve request processing throughput but does nothing to speed up the creation of new instances. Cold start latency is the time from when a new instance is requested until it becomes ready, and that is dominated by container startup and runtime initialization, not by the CPU limit setting. Thus, raising the CPU limit is irrelevant to the cold start latency issue described.

Increase maxScale to 20.

Why it fails: Raising maxScale to 20 only increases the maximum number of instances Cloud Run may create, which raises the scaling ceiling but does not ensure any instances exist before traffic arrives. Without a minimum instance count, when no instances are running, an incoming request still triggers a cold start because there is no warm instance to serve it. Therefore, maxScale alone cannot reduce cold start latency; it merely permits more parallelism once scaling has already occurred.

Disable the cool-down period to react faster to spikes.

Why it fails: Disabling cooldown can cause thrashing and unstable scaling.

Use a single metric (e.g., CPU utilization) for simplicity.

Why it fails: Using multiple metrics provides more stable and responsive scaling.

Scale based on custom metrics only.

Why it fails: Relying solely on custom metrics may miss system-level signals; using both is recommended.

Backend buckets for static content.

Why it fails: Backend buckets are a storage backend that serves static content directly from Cloud Storage, but they are not a global routing or caching layer; without Cloud CDN, requests are sent to the bucket's regional storage location, potentially crossing long network distances. They can be used as an origin behind a load balancer, but by themselves they do not provide anycast IP addressing, global edge caching, or traffic routing, so they are not one of the core global components. In a global design, a backend bucket should be paired with the global HTTP(S) load balancer and Cloud CDN rather than used alone.

Regional internal load balancer.

Why it fails: Regional internal load balancer is designed for traffic inside a VPC network and only distributes traffic among backend instances within a single region, using private IP addresses. It cannot serve external internet users or provide global reach because it does not offer anycast IP or a global forwarding rule. It is a valid component for internal services, but it is not one of the three components that make up a global external HTTP(S) architecture.

The threshold value of 0.8 is too low.

Why it fails: A threshold of 0.8 is not necessarily too low and is unrelated to autoscaler scale-up.

The alignment period of 60 seconds is too short, causing unstable metrics.

Why it fails: A 60-second alignment period is common and not a likely reason an autoscaler fails to scale.

The filter is missing the project ID, so it applies to all projects.

Why it fails: Missing project ID is not a standard cause; resource filters in a Cloud Monitoring alert affect notification scope, not autoscaling.

Implement a caching layer with Cloud CDN.

Why it fails: Cloud CDN only caches content that is static or has cacheable response headers; a Node.js application typically serves dynamic, user-specific responses, so most requests still reach the backend. Even for cacheable requests, CDN reduces origin request volume but does not change how Cloud Run scales instances relative to each incoming request, so it does not fix over-provisioning caused by low concurrency. Additionally, introducing CDN adds configuration, freshness, and invalidation complexity without addressing the core scaling metric.

Move the application to Compute Engine with a smaller machine type.

Why it fails: Migrating from Cloud Run to Compute Engine means you must manage capacity, patching, and autoscaling yourself, and a smaller machine type may easily be overwhelmed during traffic spikes. Compute Engine costs are based on instance uptime, not per-request usage, so even a smaller VM can be more expensive than a properly-configured Cloud Run service with high concurrency. The real issue is that Cloud Run is creating too many instances per request, and moving to a VM does not solve that inefficiency—it only trades it for manual infrastructure management.

Reduce the container memory limit to the minimum required.

Why it fails: Lowering the memory limit will reduce ingress bill if it changes the memory class, but if the limit drops below the application's actual working set, Cloud Run will kill the instance with an OOM error, causing request failures and repeated cold starts. Even if the memory stays within bounds, the instance count remains driven by concurrency and traffic, so reducing memory does not lower how many instances are created per request. The correct cost-reduction action is to adjust concurrency (and CPU allocation if needed), not to force a tight memory limit that harms reliability.

Managed instance groups automatically scale based on CPU utilization only.

Why it fails: Autoscaler supports multiple metrics including load balancer utilization, not just CPU.

To scale based on custom metrics, you must use the autoscaler with a custom metric from Stackdriver.

Why it fails: You can also use standard metrics; custom metrics are an option, not a requirement.

Load balancers can directly trigger scaling actions without autoscaling.

Why it fails: Load balancers distribute traffic; autoscalers decide scaling actions based on metrics.

~4.3 minutes

Why it fails: 4.3 minutes is the monthly downtime budget for 99.99% availability (four nines), not for 99.9% (three nines). A 99.9% SLO allows roughly 10x more downtime than 99.99%, so the correct three-nines budget is 43.2 minutes per 30-day month. Budgeting only 4.3 minutes would over-engineer your architecture beyond the stated requirement.

~7.2 hours

Why it fails: 7.2 hours equals 432 minutes, which is exactly 1% of a 30-day month. This corresponds to 99% availability (two nines), not 99.9%. Since 99.9% downtime is 0.1%, it is an order of magnitude more restrictive — the allowed downtime drops by a factor of 10, from 432 minutes down to 43.2 minutes.

~8.6 hours

Why it fails: 8.6 hours per month corresponds to roughly 99% availability (two nines) for a 30-day month, but it is actually slightly more than the 7.2 hours (432 minutes) that exactly equals 1% downtime. A 99.9% availability target allows only 0.1% downtime — about 43.2 minutes — so 8.6 hours is roughly 12 times larger than the permitted error budget, reflecting a much lower availability goal.

Threshold alerting — alert when error rate exceeds 0.1%.

Why it fails: Threshold alerting fires only after a fixed error rate (here 0.1%) is breached, making it inherently reactive. Because it does not consider the SLO measurement window or how much error budget remains, it cannot predict whether the budget will be exhausted by the end of the period. Moreover, a static threshold may cause alert fatigue from transient spikes or miss sustained lower-level consumption that cumulatively drains the budget.

Uptime check alerting — alert when health checks fail.

Why it fails: Uptime check alerting only verifies that a service is reachable via synthetic probes, returning a binary pass/fail status. It cannot detect gradual degradation, partial outages, or the rate at which errors accumulate, so it provides no visibility into error budget consumption against an SLO. Thus, while useful for availability incidents, it does not measure SLO compliance or enable proactive response to imminent budget exhaustion.

Log-based alerting — alert when specific error messages appear in logs.

Why it fails: Log-based alerting triggers on predefined error messages or patterns in log entries, which are useful for diagnosing known failure modes but not for measuring SLO performance. It lacks aggregation across all requests and does not correlate error occurrence with the total traffic volume or the error budget defined by the SLO. As a result, it produces noise from isolated errors while failing to signal when the overall error rate is consuming budget at an unsustainable pace.

VMs are terminated and restarted automatically on new hardware, causing a few minutes of downtime.

Why it fails: Compute Engine's live migration does not terminate the VM instance and restart it on another host; that would be a cold migration and would interrupt running workloads. Instead, the entire VM runtime state — memory, vCPU state, disk attachments, and network identity — is copied while the instance continues executing. The eventual switch to the destination host occurs with only a brief, often sub-second pause, not the minutes of downtime implied by a full stop/start cycle.

VMs are snapshotted, the snapshot is restored on new hardware, and the VM is restarted.

Why it fails: A snapshot captures the persistent disk state at a specific point in time, so it cannot preserve the contents of RAM, the exact process state, or the continuity of network flows. Restoring a snapshot and booting from it would result in a full system reboot, losing all in-memory data, active sessions, and any I/O that hadn't yet been committed to disk. Live migration is not snapshot-based at all: it transfers the complete runtime state of the instance while it's alive, which is why operating systems and applications never observe a cold start or experience a shutdown.

Customers must subscribe to Google Cloud support to receive advance notice and schedule their own maintenance windows.

Why it fails: No Google Cloud support subscription is required for Compute Engine's standard maintenance event handling; live migration is automatically provided to all customers. Google schedules and executes host maintenance transparently, and the platform performs a live migration or a terminate based on the VM's maintenance policy, without waiting for customer action. While you can control whether a VM is live-migrated or terminated, you do not need to purchase support to get advance notice or to enroll in the feature.

Purchase reserved capacity for peak load and configure it to be active only on weekdays.

Why it fails: Reserved capacity, such as committed use discounts, is intended for stable, predictable baseline usage, not for dynamic peak demand. You cannot schedule reserved capacity to be 'active only on weekdays'; a commitment applies continuously 24/7 once purchased, so paying for peak-level capacity would leave that capacity idle and expensive during nights and weekends. Even if you could purchase enough fixed capacity to cover the morning spike, it remains statically allocated and cannot automatically scale down when demand falls, making it the opposite of an auto-scaling solution.

Deploy additional VMs manually each weekday morning and terminate them at night.

Why it fails: Manually deploying additional VMs each weekday morning and terminating them at night requires ongoing human effort, which violates the explicit 'without manual intervention' requirement. This approach is also inherently fragile—it cannot react to unexpected traffic spikes earlier or later than the scheduled window, and any miscalculation in VM count leads to either over-provisioning or performance degradation. Additionally, manual instance creation takes time to become ready, so users may experience latency during the provisioning window.

Use Cloud Monitoring to send an email alert when CPU exceeds 80% so the team can manually scale.

Why it fails: Using Cloud Monitoring to send an email alert when CPU exceeds 80% still relies on a human to read the alert and manually adjust the infrastructure, which is why it fails the 'no manual intervention' requirement. Email notifications are asynchronous and can be missed, delayed, or acted on inconsistently, meaning the application could experience sustained overload while waiting for someone to respond. This approach also lacks any closed-loop feedback: there is no mechanism to automatically add or remove instances based on the alert, so it is not a scaling mechanism at all.

Deploy identical origin servers in every Google Cloud region globally.

Why it fails: Deploying identical origin servers in every Google Cloud region would mean maintaining full application state, storage, and compute redundancy in dozens of locations, with complex cross-region synchronization and data consistency challenges. This approach is not only operationally expensive but also unnecessary, since a CDN already caches the actual video bytes at edge locations—which are more numerous and closer to viewers than regional origins. For dynamic or personalized content, you would still need a centralized origin or a different strategy; for static video, the CDN gives the same latency benefit without the cost of global server replication.

Use Cloud VPN to route viewer traffic through a direct tunnel to the origin servers.

Why it fails: Cloud VPN establishes an encrypted IPsec tunnel between your VPC and an on-premises or other cloud network; it simply provides secure connectivity between fixed networks. Routing viewer traffic over a VPN to origin servers in us-central1 does not shorten the physical distance between Asian viewers and the origin, so latency remains high. In fact, IPsec encryption and encapsulation add CPU overhead and packet overhead, making streaming performance worse than a direct public internet path, and it provides no caching or edge delivery capability whatsoever.

Increase the origin servers' network bandwidth to handle more simultaneous viewer connections.

Why it fails: Increasing the network bandwidth or egress capacity of the origin servers allows them to handle more simultaneous connections and throughput, but it does nothing to reduce the physical propagation delay between the origin's location (us-central1) and viewers in Asia. Distance-based latency—on the order of hundreds of milliseconds across continents—is a fundamental property of the speed of light in fiber, and additional bandwidth cannot overcome it. The correct solution is to move content closer to viewers via edge caching, not to make a single distant server faster.