PDE Designing Data Processing Systems Practice Test 5 — 15 Questions

Question 1

A media company ingests video playback telemetry from a mobile app into Pub/Sub. The analytics team wants the raw events retained in Cloud Storage for compliance for at least three years at the lowest storage cost, with retrieval taking up to a day acceptable. Only a rare audit needs to read the data. Which Cloud Storage configuration should the data engineer use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Store the objects in the Archive storage class, set a lifecycle rule to transition to Archive after 30 days, and rely on the three-year retention without early deletion.. Archive storage minimizes per-GB cost for long-term retention of data read less than once a year, and a lifecycle rule that respects the 365-day minimum duration avoids early-deletion penalties. Keeping the first 30 days in a warmer class serves any near-term debugging, while retrieval within a day matches the acceptable audit latency. The other classes either cost more or are tuned for more frequent access.

Answer

Store the objects in the Coldline storage class and apply a lifecycle rule to transition them to Archive after 90 days.

Answer

Store the objects in the Standard storage class and enable Object Lifecycle Management to delete objects after three years.

Answer

Store the objects in the Nearline storage class and enable Autoclass to move them to colder classes automatically.

Question 2

A media company needs to process video files uploaded to Cloud Storage. The processing involves transcoding and generating thumbnails, which takes several minutes per file. The pipeline must be triggered automatically when a new file is uploaded, and the processing must be resilient to failures with automatic retries. They want to minimize operational overhead. Which Google Cloud service should they use to orchestrate this workflow?

Accepted Answer

Eventarc with a Cloud Storage trigger that routes to a Cloud Run service, which performs the transcoding.. Eventarc provides a serverless way to trigger Cloud Run services from Cloud Storage events. Cloud Run supports long-running requests and automatic retries, making it suitable for transcoding video files. This solution minimizes operational overhead because both services are fully managed and scale automatically. The other options either have timeout limits, require polling, or add unnecessary complexity.

Answer

Cloud Functions with a Cloud Storage trigger that calls the Cloud Video Intelligence API.

Answer

Cloud Scheduler with a cron job that polls Cloud Storage for new files and publishes to Pub/Sub, which triggers a Dataflow job.

Answer

Cloud Composer with a DAG that triggers on Cloud Storage object changes.

Question 3

A retail company ingests point-of-sale transactions into Cloud Storage as newline-delimited JSON files. Analysts need to query the data with standard SQL in BigQuery, but the schema changes frequently as new product attributes are added, and the team wants to avoid managing table schemas manually. They also want to minimize storage cost for data older than 90 days. Which design should they implement?

Accepted Answer

Load the files into a BigQuery table with autodetect schema enabled and configure a partition expiration of 90 days.. A native BigQuery table loaded with schema autodetect handles evolving JSON fields without manual schema management, while ingestion-time partitioning with a 90-day partition expiration automatically removes older data and reduces storage cost. External tables and Dataproc-based approaches add complexity and do not satisfy the cost requirement as directly.

Answer

Load the files into a BigQuery table using a fixed schema and use a scheduled query to drop columns that are no longer needed.

Answer

Keep the files in Cloud Storage and query them with a Dataproc cluster running Spark SQL, then export results to BigQuery.

Answer

Create an external table over the Cloud Storage files and set the table's expiration to 90 days.

Question 4

You are designing a data processing pipeline that ingests application logs from a Compute Engine instance group. The logs must be stored in Cloud Storage and then loaded into BigQuery for analysis. The pipeline should handle failures gracefully and ensure that no data is lost during a temporary outage of the downstream systems. You want to minimize operational overhead. Which Google Cloud service should you use to orchestrate this batch pipeline?

Accepted Answer

Cloud Composer. The scenario requires a batch pipeline that ingests logs, stores them in Cloud Storage, and loads them into BigQuery with failure handling and minimal operational overhead. Cloud Composer provides managed workflow orchestration, allowing you to define tasks, set dependencies, and implement retries. It integrates with other Google Cloud services and is designed for such batch orchestration, making it the best fit.

Answer

Cloud Scheduler

Answer

Cloud Dataflow

Answer

Cloud Pub/Sub

Question 5

A media company stores raw JSON logs in Cloud Storage and needs to make them queryable in BigQuery with minimal cost. The logs are written once and never modified. Analysts run SQL queries that filter on a few fields and aggregate large time ranges. Which table type should you use?

Accepted Answer

Create an external table over the Cloud Storage bucket using the BigLake connection and rely on automatic schema detection.. An external table over Cloud Storage, especially with a BigLake connection, allows BigQuery to query the raw JSON logs without loading them. This avoids duplication and ingestion costs while still supporting SQL filters and aggregations. Since the logs are immutable and already in Cloud Storage, this is the most cost-effective design.

Answer

Use a Dataproc job to convert the JSON logs to Parquet and then load them into BigQuery with a WRITE_APPEND disposition.

Answer

Create a BigQuery table with a RECORD column and use streaming inserts to load each log entry as it arrives.

Answer

Load the JSON logs into a native BigQuery table using a batch load job with schema autodetect.

Question 6

A media company is designing a data processing system on Google Cloud to analyze video streaming logs. The logs are written as newline-delimited JSON files to a Cloud Storage bucket every hour, and the analytics team needs to run SQL queries on the data with minimal latency. The company wants a serverless solution that requires no cluster management and can automatically scale. Which two Google Cloud services should they use together? (Choose two.)

Accepted Answer

Cloud Dataflow with a pipeline that reads from Cloud Storage and writes to BigQuery.. The combination of Dataflow and BigQuery provides a serverless pipeline: Dataflow reads the JSON files from Cloud Storage, optionally transforms them, and loads them into BigQuery for fast SQL queries. BigQuery external tables can also query the files directly, but for minimal latency and repeated analysis, loading into native BigQuery tables via Dataflow is more efficient. Both services are fully managed and auto-scale.

Answer

Cloud SQL for PostgreSQL with a foreign data wrapper to read from Cloud Storage.

Answer

Cloud Dataproc with a Spark job that reads from Cloud Storage and writes to BigQuery.

Answer

Cloud Pub/Sub to ingest the logs and trigger a Cloud Function to write to BigQuery.

Question 7

A logistics company processes shipment events in a Dataflow pipeline that writes enriched records to both BigQuery and Cloud Storage. During a deployment, the pipeline was updated while a streaming job was running, and the team now sees duplicated shipment records in BigQuery. They want future updates to be safe to apply to a running streaming job without reprocessing already-emitted data. What should they do?

Accepted Answer

Use a transform that supports stateful processing with a stable transform name, and update the job rather than replacing it.. Dataflow update semantics rely on matching transforms by name so that state, such as deduplication buffers and window contents, can be carried over to the new version. Keeping transform names stable and applying an update rather than replacing the job prevents the pipeline from replaying data that was already written.

Answer

Add a deduplication step in BigQuery using a MERGE statement after every pipeline deployment.

Answer

Enable Streaming Engine and use the Update option on the running job instead of stopping and starting a new job.

Answer

Increase the number of worker threads and set the autoscaling algorithm to THROUGHPUT_BASED.

Question 8

A logistics company ingests 4 TB of shipment manifests per day as newline-delimited JSON into Cloud Storage. Analysts run ad-hoc SQL over the entire history (currently 900 TB) but rarely touch records older than 90 days. The team wants to minimize both storage cost and bytes scanned by BigQuery without rewriting the upstream export job. What should they do?

Accepted Answer

Load into a table partitioned by ingestion date (or a date column in the data) so queries can prune partitions, and let BigQuery's automatic long-term storage pricing reduce the cost of untouched older partitions.. Partitioning on a date column or ingestion time lets BigQuery prune partitions that a filter excludes, which directly cuts bytes scanned. Separately, BigQuery applies long-term storage pricing to data that has not been modified for 90 consecutive days, so older partitions cost less to keep online and queryable. This combination meets the storage-cost and scan-cost goals while preserving full history and requiring no change to the JSON export pipeline.

Answer

Load the JSON into a BigQuery table partitioned by ingestion date and set a partition expiration of 90 days on that table.

Answer

Create a BigQuery external table over the Cloud Storage objects and rely on the external table's automatic partition pruning.

Answer

Load into a date-partitioned BigQuery table and set the partition's storage billing model to long-term automatically by leaving the data in place untouched for 90 days.

Question 9

A retail analytics team runs an Apache Hadoop workload on Dataproc that reads and writes shared metadata in a Hive metastore. The team wants the metastore to survive cluster deletion, be shared by multiple ephemeral clusters, and require no HDFS-backed local storage. They also want to keep the metastore reachable by clusters in a different VPC within the same project. Which design should they choose?

Accepted Answer

Create a Dataproc Metastore service and attach it to each ephemeral Dataproc cluster using the dataproc-metastore configuration when creating the cluster.. A managed Dataproc Metastore decouples Hive metadata from cluster lifecycle, letting many short-lived clusters share one catalog without running persistent infrastructure. Because it is serverless and backed by managed storage, it does not depend on HDFS or a long-running cluster. Attaching it at cluster creation is the supported path for shared metadata across ephemeral Dataproc clusters.

Answer

Run the Hive metastore on a dedicated long-lived Dataproc cluster and point every ephemeral cluster at it with the hive.metastore.uris property.

Answer

Configure the Dataproc cluster with a Cloud Bigtable instance as the Hive metastore backend using the hbase-site.xml override.

Answer

Enable the Hive warehouse connector and store all table definitions in a Cloud Storage bucket as JSON manifests.

Question 10

A media company streams video playback events into Pub/Sub and processes them with a Dataflow streaming pipeline that writes enriched records to BigQuery. The pipeline must handle sudden traffic spikes without data loss and must produce exactly-once results in the BigQuery output table. Which two design choices should the engineer make? (Choose two.)

Accepted Answer

Configure the BigQuery sink with exactly-once semantics by using a deterministic insert ID derived from the event.. Durability during spikes depends on retaining unacknowledged Pub/Sub messages long enough for the pipeline to catch up, and exactly-once output in BigQuery depends on deterministic insert IDs with the Storage Write API. Together these choices let Dataflow retry safely without losing or duplicating events.

Answer

Use a BigQuery load job from Cloud Storage instead of streaming inserts to avoid duplicates.

Answer

Increase the number of Pub/Sub topics and have the pipeline read from all of them in parallel.

Answer

Set the Dataflow pipeline to use at-least-once processing and rely on downstream deduplication in a scheduled query.

Question 11

A logistics company runs a Dataproc cluster to process daily shipment manifests with Spark. Jobs must complete within a fixed two-hour window, and the team wants to reduce cost without risking the deadline. They observe that the cluster is idle for most of the day and that shuffle-heavy stages spill to disk. Which two changes should the data engineer make to meet the deadline while lowering cost? (Choose two.)

Accepted Answer

Enable Spark dynamic allocation and configure the shuffle service to use local SSD, and use Dataproc's enhanced flexibility mode so primary and secondary workers scale independently.. Ephemeral clusters eliminate payment for idle hours, and dynamic allocation plus local SSD shuffle storage keeps shuffle-heavy stages within the two-hour window. Enhanced flexibility mode allows secondary workers to scale independently, matching burst demand without over-provisioning primary nodes. Together they lower cost while protecting the deadline, whereas raising the minimum worker count or enlarging the driver leaves both the idle spend and the spill problem in place.

Answer

Enable Dataproc autoscaling on the persistent cluster and set a minimum worker count equal to the peak required for shuffle stages.

Answer

Increase the driver machine type to a high-memory instance and keep the worker count fixed at the current level.

Answer

Move shuffle-heavy stages to a larger number of smaller worker nodes with local SSD attached to increase parallelism.

Question 12

A financial services company needs to process credit card transactions in real-time to detect fraudulent patterns. The transactions are streamed into Pub/Sub. The detection logic requires maintaining state across a sliding window of 5 minutes per credit card number. The results must be written to BigQuery with exactly-once semantics. Which approach should you use with Cloud Dataflow?

Accepted Answer

Use a SlidingWindow with a duration of 5 minutes and a period of 1 minute, keyed by credit card number, and set the pipeline to use the Dataflow Shuffle service for exactly-once processing.. The requirement is to detect patterns across a sliding window of 5 minutes per credit card. A SlidingWindow with duration 5 minutes and period 1 minute creates overlapping windows covering the last 5 minutes for each key. Keying by credit card number ensures state is per card. Dataflow Shuffle service provides exactly-once processing for streaming pipelines when used with appropriate windowing and triggers, ensuring results are written to BigQuery without duplicates.

Answer

Use a FixedWindow with a duration of 5 minutes, keyed by credit card number, and enable the Streaming Engine to achieve exactly-once semantics.

Answer

Use a SessionWindow with a gap duration of 5 minutes, keyed by credit card number, and set the pipeline to use the Dataflow Shuffle service for exactly-once processing.

Answer

Use a GlobalWindow with a trigger that fires every 5 minutes, and enable exactly-once processing by setting the pipeline to use the Streaming Engine.

Question 13

A healthcare company needs to process sensitive patient records in a Dataflow pipeline. They must ensure that data is encrypted at rest and in transit, and that only authorized services can access the data. They also need to audit all access to the data. Which combination of Google Cloud features should they implement?

Accepted Answer

Use Customer-Managed Encryption Keys (CMEK) for Dataflow and Cloud Storage, enable Private Google Access, and enable Cloud Audit Logs for data access.. CMEK provides customer-controlled encryption at rest for both Dataflow and Cloud Storage. Private Google Access ensures traffic between services stays within Google's network, and Data Access audit logs record all access to the data. This trio satisfies encryption, network security, and auditing. Alternatives either lack customer key control or miss data access auditing.

Answer

Use Google-managed encryption keys, enable VPC Service Controls, and rely on Cloud Logging for audit trails.

Answer

Use CMEK for Dataflow and Cloud Storage, enable VPC Service Controls, and enable Cloud Audit Logs for admin activity only.

Answer

Use CMEK for Cloud Storage only, enable Identity-Aware Proxy, and enable Data Access audit logs.

Question 14

A logistics company runs a Dataflow pipeline that processes shipment events and writes to BigQuery. The pipeline currently uses a single global window and writes each event as a separate row. During peak hours, BigQuery rejects requests with 'rateLimitExceeded'. The team wants to reduce API calls without losing data. What should they do?

Accepted Answer

Use BigQueryIO with STORAGE_WRITE_API and apply a GroupIntoBatches transform before writing.. The rateLimitExceeded errors come from too many individual streaming insert requests. Switching to the Storage Write API and batching events with GroupIntoBatches reduces the number of write calls and uses a higher-throughput path. This preserves all shipment events while relieving API pressure, unlike simply adding workers or relying on best-effort deduplication.

Answer

Increase the number of Dataflow workers to spread the write load across more instances.

Answer

Switch the BigQuery sink to use FILE_LOADS with a one-minute triggering frequency.

Answer

Set the BigQuery sink to use insertId for deduplication and retry failed inserts with exponential backoff.

Question 15

A healthcare company is designing a system to process sensitive patient records in BigQuery. They need to ensure that data analysts can only view aggregated, de-identified data, while data engineers can access raw data for pipeline development. They also need to maintain an audit trail of all access. Which two Google Cloud features should they implement? (Choose two.)

Accepted Answer

BigQuery column-level security with policy tags to restrict access to sensitive columns.. Column-level security with policy tags enforces access control at the column level, allowing analysts to see only de-identified columns while engineers access raw data. Cloud Audit Logs provide the necessary audit trail for all access. Together, they meet the access control and auditing requirements. The other options either do not de-identify data, add unnecessary complexity, or address network security rather than data access.

Answer

Cloud Data Loss Prevention (DLP) to de-identify data before it is written to BigQuery.

Answer

BigQuery row-level security with filter expressions to restrict rows based on user attributes.

Answer

VPC Service Controls to create a perimeter around BigQuery and restrict data exfiltration.