Google Professional Data Engineer (PDE) — Questions 451–499

Why this answer

How to eliminate wrong answers

Option A is wrong because Pub/Sub subscriptions already have built-in retry mechanisms (e.g., at-least-once delivery) and the issue is not about message loss from Pub/Sub; the team verified the topic is receiving data correctly. Option B is wrong because enabling Cloud Logging for all pipeline steps would generate excessive logs and is not the most efficient diagnostic approach; Dataflow already provides built-in metrics (e.g., 'pubsub_subscription' and 'bigquery_rows_written') that can directly compare element counts without needing to parse logs. Option C is wrong because the pipeline already uses a global window with a default 10-minute trigger, which inherently captures late data (since the global window never closes); adding a late-data trigger is redundant and does not address the potential data loss between Pub/Sub and BigQuery.

Why this answer

How to eliminate wrong answers

Option A is wrong because deploying multiple versions to the same endpoint with traffic splitting and setting the primary version to 100% traffic does not inherently create separate model resources for each version; it still relies on a single model resource with aliases, which can complicate rollback if the model resource itself is overwritten or corrupted. Option B is wrong because using the same model name for all deployments and overwriting the existing model destroys the previous version's metadata and artifacts, making rollback impossible without manual restoration from backups. Option C is wrong because storing the model in a Cloud Storage bucket with a fixed name and relying on Cloud Build for rollback does not provide native Vertex AI model versioning or endpoint deployment tracking; Cloud Build is a CI/CD tool, not a model registry, and overwriting the bucket contents loses previous versions.

Why this answer

How to eliminate wrong answers

Option B is wrong because Cloud Functions has a 9-minute timeout and is not designed for sustained high-throughput streaming (10 GB/hour), leading to timeouts and data loss. Option C is wrong because Cloud Dataproc (managed Spark/Hadoop) is optimized for batch processing, not near-real-time streaming; it adds latency and complexity compared to Dataflow's native streaming. Option D is wrong because Cloud Storage load jobs are batch-oriented, introducing minutes-to-hours latency and requiring manual orchestration, which fails the near-real-time requirement.

Why this answer

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because default encryption uses Google-managed keys, not customer-managed keys (CMEK), which may not satisfy compliance requirements for controlling encryption keys; Cloud CDN caches content at edge locations but does not reduce data transfer costs from Cloud Storage to the same region (it reduces egress for global distribution, not for frequent access within a region). Option B is wrong because 'Avoid caching to ensure security' contradicts the requirement to minimize data transfer costs for frequently accessed curated datasets; caching with Cloud CDN is secure when properly configured (e.g., signed URLs, IAM), and avoiding it increases costs. Option C is wrong because SSE-C (Server-Side Encryption with Customer-Provided Keys) requires the client to manage keys and is not integrated with Cloud HSM or Cloud KMS; Cloud Load Balancing does not cache data (it distributes traffic), so it does not reduce data transfer costs for frequent access.

Why this answer

How to eliminate wrong answers

Option A is wrong because Cloud Dataflow is a unified stream and batch data processing service that requires setting up and managing pipelines (though serverless, it is not primarily for ad-hoc SQL queries on stored logs). Option B is wrong because Cloud Logging is a real-time log management and analysis service for monitoring and debugging, not designed for complex SQL analytics on large historical log datasets stored in Cloud Storage. Option D is wrong because Cloud Dataproc is a managed Spark and Hadoop service that requires provisioning clusters (even if ephemeral) and is not serverless SQL querying.

Why this answer

How to eliminate wrong answers

Option A is wrong because fixed windows of 1 minute with accumulating panes would create a new window every minute, leading to excessive state and write amplification in Bigtable, increasing cost without benefit for a global sink. Option B is wrong because sliding windows of 1 minute would create overlapping windows, multiplying state and processing overhead even more than fixed windows, which is wasteful for a use case that doesn't require windowed aggregations. Option D is wrong because session windows with a 5-minute gap duration and discarding panes are designed for grouping events by activity sessions, not for a simple streaming pipeline to Bigtable; discarding panes also risks losing late data that arrives within the 10-minute allowed lateness, violating correctness.

Why this answer

How to eliminate wrong answers

Option A is wrong because Cloud Shell is a temporary, per-user environment with limited resources and no persistent storage, making it unsuitable for sharing a development environment with version control across a team. Option C is wrong because sharing notebooks via Cloud Storage is a static file-sharing method that does not provide version control, collaborative editing, or a live execution environment. Option D is wrong because Cloud Source Repositories is a Git repository hosting service for storing code, not a shared interactive development environment; it lacks the compute and runtime capabilities needed for model development.

Why this answer

Why this answer

Why this answer

How to eliminate wrong answers

Option B is wrong because the Data Loss Prevention (DLP) API is used for inspecting and redacting sensitive data (e.g., PII), not for ensuring end-to-end encryption or audit log retention; it does not provide encryption key management or network-level controls. Option C is wrong because while Cloud Audit Logs capture API activity and VPC Service Controls provide a security perimeter, this combination lacks customer-managed encryption keys (CMEK), which are required for the 'encrypted end-to-end' and key control compliance mandate. Option D is wrong because default encryption at rest and in transit uses Google-managed keys, not customer-managed keys, and does not include VPC Service Controls to enforce data exfiltration prevention or audit log retention policies.

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because deploying to a new endpoint and updating the application to call both endpoints adds unnecessary complexity and defeats the purpose of a canary deployment, which should be transparent to the application. Option B is wrong because Cloud Load Balancing operates at the network layer and cannot route traffic based on model version within a single Vertex AI Endpoint; it is designed for distributing traffic across regional endpoints or backends, not for model version canary testing. Option D is wrong because deploying to Cloud Run and using gradual rollout is not the native way to manage model versions in Vertex AI; Vertex AI Endpoints provide built-in traffic splitting for model versions, which is the recommended approach for canary deployments in this context.

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because Vertex AI Vizier is a hyperparameter tuning service that optimizes model performance through black-box optimization, not for monitoring distribution shifts in production. Option C is wrong because Vertex AI Explainable AI provides feature attributions and explanations for individual predictions, but it does not monitor aggregate distribution changes over time. Option D is wrong because Vertex AI Feature Store is a centralized repository for storing, serving, and sharing feature data, but it lacks built-in drift detection or alerting capabilities.

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because reducing maxNumWorkers would further restrict scaling, not trigger more aggressive scaling; the autoscaler already has permission to scale to 10 but chooses not to. Option B is wrong because BigQuery streaming quota limits the rate of inserts, not the number of concurrent workers; quota exhaustion would cause insert errors, not prevent the autoscaler from adding workers. Option C is wrong because Pub/Sub subscriptions have a per-subscriber throughput limit that is very high (typically hundreds of MB/s per subscriber), and the pipeline is not hitting that limit; the limit is on throughput, not on the number of subscribers.

Why this answer

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because Cloud Tasks is a fully managed queue service for asynchronous task execution, not a feature of Cloud Composer; it would introduce unnecessary complexity and is not the native way to handle retries within a Composer DAG. Option C is wrong because high availability (HA) for Cloud Composer ensures the Airflow components are resilient to zone failures, but it does not configure task-level retry behavior for pipeline failures. Option D is wrong because Dataflow retries handle failures at the Dataflow job level (e.g., worker failures), but the question asks for a Composer feature to manage retries of the overall pipeline orchestration, not the underlying data processing job.

Why these pairings

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because switching to batch prediction is for asynchronous, non-real-time inference and does not address the root cause of latency spikes during online serving. Option B is wrong because reducing the number of features may lower model complexity but does not directly resolve infrastructure scaling issues; latency spikes are typically due to insufficient compute resources, not feature count. Option C is wrong because increasing the machine type (e.g., using a larger VM) may improve per-request performance but does not solve the problem of handling concurrent peak traffic; without autoscaling, a single larger machine can still be overwhelmed.

Why these pairings

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because upgrading to n1-standard-8 with local SSDs is overkill and more expensive; the issue is disk space for shuffle data, not CPU or memory, and local SSDs are ephemeral and not cost-effective for persistent storage needs. Option C is wrong because adding more preemptible workers does not increase the persistent disk size per worker; each worker still has only 10 GB, so shuffle data will still exceed the disk quota on those nodes. Option D is wrong because migrating to Cloud Dataflow is a significant architectural change that incurs migration costs and learning curve, and it does not address the immediate disk quota issue in the existing Dataproc workflow; Dataflow also has its own disk management limits.

Why this answer

Why this order

Why this answer

How to eliminate wrong answers

Option A is wrong because the streaming buffer does not have a 24-hour retention; data is flushed automatically within about 90 minutes or when the buffer reaches capacity, not after 24 hours. Option B is wrong because BigQuery time travel allows querying historical data within a 7-day window, but it does not cause data in the streaming buffer to become visible; it only affects how you query already-committed data. Option C is wrong because table expiration settings control when the table is deleted, not when streaming data becomes visible; data visibility is independent of table expiration.

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because running the evaluation step after deployment and then rolling back violates the principle of failing fast; it wastes compute resources and risks serving a bad model to users before rollback. Option B is wrong because retrying the evaluation step on failure does not address the root cause — low model performance — and would simply re-run the same evaluation, potentially masking the failure or delaying the pipeline. Option D is wrong because creating a separate pipeline for manual deployment defeats the purpose of automation and introduces human latency and error, contradicting the goal of an automated orchestrated workflow.

Why this answer

How to eliminate wrong answers

Option A is wrong because Spark dynamic allocation controls how executors are distributed within existing nodes, not how the cluster adds new nodes; even if disabled, the autoscaler would still attempt to add workers if configured correctly. Option B is wrong because the question explicitly states the autoscaler is based on YARN memory utilization, not CPU; a misconfiguration to CPU would cause scaling based on CPU metrics, but the symptom here is no scaling at all, not scaling on the wrong metric. Option C is wrong because the autoscaler is designed to scale up secondary workers when utilization is high; a misconfiguration to scale down would cause premature removal of workers, not a failure to scale up.

Why this answer

Why this answer

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because AI Platform Prediction is a serving endpoint for deploying trained models to make predictions; it does not orchestrate retraining pipelines. Option C is wrong because AI Platform Continuous Evaluation is a service for monitoring model performance and detecting drift, not for orchestrating retraining workflows. Option D is wrong because Cloud Dataflow is a stream and batch data processing service (based on Apache Beam) used for data transformation and ETL, not for orchestrating end-to-end ML pipelines with conditional retraining logic.

Why this answer

How to eliminate wrong answers

Option A is wrong because blindly retraining with more recent data without first confirming data drift may waste resources and could even degrade model performance if the new data is not representative or contains label errors. Option B is wrong because increasing replicas addresses latency, not accuracy decline; the current 200ms latency is well within acceptable bounds and is unrelated to the accuracy problem. Option C is wrong because Cloud Logging captures prediction errors (e.g., runtime exceptions, invalid inputs), but the pipeline runs successfully with no errors, so examining logs for errors will not reveal gradual accuracy degradation caused by data drift.

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because a memory leak in the model container would cause gradual performance degradation or OOM kills, not a sudden 503 error under high traffic; Vertex AI would still attempt to serve requests until the container crashes. Option B is wrong because data drift affects prediction accuracy (e.g., wrong predictions), not the availability or HTTP status of the endpoint; 503 errors are infrastructure-level, not model-level. Option D is wrong because using an unsupported older version would cause deployment or startup failures, not transient 503 errors under load; Vertex AI would reject the deployment or return a different error (e.g., 400 or 404) if the version is incompatible.

Why this answer

How to eliminate wrong answers

Option A is wrong because reducing the window size from 10 minutes to 1 minute increases the number of open windows (from 6 per hour to 60 per hour), which would worsen the 'high uncommitted bytes' issue by creating more in-memory state. Option B is wrong because increasing worker machines does not address the fundamental problem of excessive open windows consuming memory; it may temporarily mask the issue but will not reduce the per-worker uncommitted bytes. Option D is wrong because setting a maximum number of workers and Pub/Sub flow control limits incoming messages but does not reduce the number of open windows or the memory used by uncommitted data; it may cause backpressure and data loss without fixing the window state explosion.

Why this answer

How to eliminate wrong answers

Option A is wrong because removing the enrichment step defeats the purpose of the pipeline and does not address the underlying issue of API call reliability. Option B is wrong because using a global window to accumulate all data before enrichment would introduce unbounded state and memory pressure, and it does not solve API timeout problems; it would also break the streaming nature of the pipeline. Option D is wrong because simply increasing the number of workers does not fix API timeouts; it may even exacerbate the problem by overwhelming the external API with more concurrent requests, leading to more failures.

Why this answer

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because container image size does not prevent startup; Vertex AI supports images up to 10 GB, and a large image would only affect pull time, not the container's ability to start. Option C is wrong because CPU architecture mismatch would cause a runtime crash or 'exec format error' during execution, not a startup failure, and Vertex AI uses x86_64 architecture by default. Option D is wrong because a missing model file would cause a runtime error during prediction (e.g., 404 or model load failure), not a container startup failure, as the container can still start and listen for requests.

Why this answer

Why this answer

How to eliminate wrong answers

Option A is wrong because pre-pulling the image on all nodes is a manual workaround that does not solve the underlying issue of a bloated image; it also adds operational overhead and fails in dynamic clusters where new nodes are added. Option B is wrong because increasing the timeout in the deployment config (e.g., the `imagePullPolicy` or pod-level timeout) only masks the symptom and does not reduce the pull time, potentially leading to other timeouts in the cluster. Option C is wrong because switching to a larger machine type does not affect the network transfer time for pulling the image; it only provides more local resources, which does not address the slow image download.

Why this answer

Why this answer

How to eliminate wrong answers

Option B is wrong because default triggers in Dataflow (which fire on element arrival and after watermark) do not cause duplicate rows; they affect when results are emitted, not whether duplicates occur. Option C is wrong because fixed windows of 10 seconds do not overlap by design; overlapping windows would require a sliding window, not a fixed window. Option D is wrong because using too many Dataflow workers can cause resource inefficiency or shuffle issues, but it does not directly cause duplicate rows in BigQuery output.