PDE Building and operationalizing data processing systems Practice Test 3 — 15 Questions

Question 1

A company uses Cloud Composer to orchestrate data pipelines. They have a DAG that runs hourly and processes files from Cloud Storage. The DAG is triggered by a Pub/Sub message sent from a Cloud Storage bucket notification. Recently, some DAG runs are not starting even though the Pub/Sub messages are published. Which two likely causes should the team investigate? (Choose TWO.)

Accepted Answer

The Cloud Storage bucket notification is not sending messages to the correct Pub/Sub topic, or the subscription's ack deadline is too short.. Option A is correct because if the Cloud Storage bucket notification is misconfigured to send messages to the wrong Pub/Sub topic, the Pub/Sub sensor in the DAG will never receive the trigger message, causing DAG runs to not start. Additionally, if the subscription's ack deadline is too short, the message may be acknowledged before the sensor processes it, leading to message loss and missed triggers. Both issues directly prevent the DAG from being triggered by Pub/Sub messages.

Answer

The DAG's start_date is set in the past and catchup is set to False, so DAG runs are only triggered on schedule.

Answer

The total number of DAGs in the environment exceeds the maximum limit of 100, causing DAG processing to stop.

Answer

The Cloud Composer environment is using a pull subscription instead of a push subscription for the Pub/Sub sensor.

Question 2

A company wants to replicate a Cloud SQL (PostgreSQL) database to BigQuery in near real-time for analytics. The volume is about 10GB per day with frequent updates and deletes. They need to capture changes with low latency and ensure exactly-once delivery to BigQuery. Which approach should they use?

Accepted Answer

Use Cloud Data Fusion with a Debezium streaming source to capture CDC from Cloud SQL and a BigQuery sink with exactly-once mode.. Option C is correct because Cloud Data Fusion with a Debezium streaming source provides native change data capture (CDC) from PostgreSQL, capturing inserts, updates, and deletes with low latency. The BigQuery sink in exactly-once mode ensures no duplicate records, meeting the requirement for near real-time analytics with frequent updates and deletes.

Answer

Export the entire database to Cloud Storage as CSV files every hour and load them into BigQuery using a load job with WRITE_TRUNCATE.

Answer

Use a Dataflow pipeline with JDBCIO to read from Cloud SQL every minute and write changes to BigQuery using upserts.

Answer

Use Cloud SQL's change data capture feature to write changes to a Pub/Sub topic and use a Dataflow pipeline to stream into BigQuery.

Question 3

A company stores raw data files in Cloud Storage in a bucket named 'raw-data'. After processing, the files are moved to a 'processed' bucket. To reduce costs, they want to automatically delete raw data older than 30 days. What should they do?

Accepted Answer

Configure a lifecycle rule on the 'raw-data' bucket to delete objects older than 30 days.. Option B is correct because Cloud Storage lifecycle management allows you to set a rule that automatically deletes objects after a specified number of days from their creation time. By configuring a lifecycle rule on the 'raw-data' bucket to delete objects older than 30 days, the company can achieve cost reduction without manual intervention. This directly addresses the requirement to remove raw data files that have been processed and are no longer needed.

Answer

Enable object versioning on the 'raw-data' bucket and configure a lifecycle rule to delete noncurrent versions.

Answer

Set a retention policy on the 'raw-data' bucket to expire objects after 30 days.

Answer

Use a bucket policy that denies read access to objects older than 30 days.

Question 4

A company needs to grant analysts access to a BigQuery table that contains sensitive PII columns. The analysts should be able to run aggregate queries on the entire dataset but must not see individual PII values. Which approach should the team use?

Accepted Answer

Use BigQuery column-level security with data masking to mask the PII columns for the analysts' role.. Option D is correct because BigQuery column-level security with data masking allows you to define masking policies on specific PII columns (e.g., using `DEFAULT_MASKING_RULE` or custom policies) that automatically transform the data for analysts' roles while still permitting aggregate queries over the entire dataset. This approach ensures analysts never see individual PII values, yet they can run `COUNT`, `SUM`, `AVG`, etc., on the masked columns, meeting both requirements precisely.

Answer

Create a user-defined function (UDF) that aggregates the data and grant analysts permission to call the UDF.

Answer

Use BigQuery row-level security to restrict access to non-PII rows only.

Answer

Create an authorized view that does not include the PII columns and grant analysts access to the view.

Question 5

A Dataflow streaming pipeline processes events from Pub/Sub and writes to BigQuery using a dynamically generated table destination based on the event type. The pipeline is experiencing high latency, and the worker CPU utilization is low. Which action is most likely to reduce latency?

Accepted Answer

Enable Dataflow Streaming Engine to improve throughput and reduce latency.. Option C is correct because Dataflow Streaming Engine moves state and computation from worker VMs to the backend service, reducing per-worker overhead and enabling better resource utilization. This directly addresses the symptom of high latency with low CPU utilization, which indicates workers are bottlenecked on shuffle or state management rather than compute.

Answer

Increase the batch size parameter in the BigQuery sink to write larger batches.

Answer

Reduce the number of workers to increase CPU utilization per worker.

Answer

Increase the worker disk size to reduce I/O wait time.

Question 6

A Dataflow streaming job is processing data from Pub/Sub and writing to BigQuery. The job is stuck with the message 'No progress has been made' for several minutes. Which TWO actions should the team take to troubleshoot and resolve the issue? (Choose TWO.)

Accepted Answer

Examine the worker logs in Cloud Logging for any error messages or exceptions.. Option C is correct because examining worker logs in Cloud Logging is the first step to identify the root cause of a stuck pipeline. Common issues like out-of-memory errors, serialization failures, or worker crashes are logged there, and without inspecting logs, troubleshooting is guesswork.

Answer

Set the updateCompatibility flag to true and restart the pipeline.

Answer

Increase the persistent disk size for all workers to reduce I/O contention.

Answer

Force stop the pipeline and update it with a new version using the --update flag.

Question 7

A company is migrating their on-premises Apache Spark jobs to Google Cloud Dataproc. They want to minimize operational overhead and cost for jobs that run only a few times per day. Which TWO strategies should they adopt? (Choose TWO.)

Accepted Answer

Store all data in Cloud Storage instead of HDFS, and use the Cloud Storage connector to access it.. Option C is correct because storing data in Cloud Storage decouples storage from compute, allowing ephemeral clusters to be spun up and down without data loss. The Cloud Storage connector provides Hadoop-compatible file system access, eliminating the need for HDFS replication and reducing costs by avoiding persistent cluster storage.

Answer

Configure HDFS replication factor to 3 to ensure data durability during cluster restarts.

Answer

Rewrite the Spark jobs as Dataflow pipelines to take advantage of serverless processing.

Answer

Use a small persistent cluster that runs continuously and submit jobs to it.

Question 8

Which THREE Google Cloud services are considered fully managed serverless data processing services? (Choose THREE.)

Accepted Answer

Cloud Functions. Cloud Functions is a fully managed serverless data processing service because it executes code in response to events without requiring any server provisioning or management. It automatically scales from zero to thousands of instances based on incoming requests, and you pay only for compute time used while your code runs. This makes it ideal for lightweight, event-driven data processing tasks such as transforming data in Cloud Storage or reacting to Pub/Sub messages.

Answer

Cloud Dataproc

Answer

Cloud Composer

Question 9

Refer to the exhibit. A BigQuery dataset has the IAM policy shown above. An analyst is trying to run a SELECT query on a table in this dataset but receives an 'Access Denied' error. What is the most likely reason?

Accepted Answer

The analyst only has the roles/bigquery.metadataviewer role, which does not allow reading table data.. The roles/bigquery.metadataviewer role grants permissions to view table and dataset metadata (e.g., table names, schemas) but does not include the bigquery.tables.getData permission required to read table rows. Therefore, when the analyst runs a SELECT query, BigQuery denies access because the role lacks the data-reading privilege. This is the most likely reason for the 'Access Denied' error.

Answer

The analyst does not have permission to list datasets in the project.

Answer

The table is in a different region than the dataset, and the analyst's query is not cross-region compatible.

Answer

The analyst has not been granted the 'bigquery.jobs.create' permission to run queries.

Question 10

A company is ingesting real-time sensor data from thousands of devices into Cloud Pub/Sub. They need to process this data with low latency (seconds) and exactly-once semantics. Which data processing service should they use?

Accepted Answer

Dataflow streaming with exactly-once processing. Dataflow streaming with exactly-once processing is the correct choice because it provides exactly-once semantics for Pub/Sub sources via checkpointing and idempotent sinks, and it meets the low-latency (seconds) requirement through its streaming engine that minimizes per-element overhead. Cloud Dataflow's integration with Pub/Sub ensures that each message is processed exactly once, even in the presence of failures, by using snapshots and consistent state management.

Answer

Cloud Run with Pub/Sub push

Answer

Cloud Functions triggered by Pub/Sub

Answer

Dataproc with Spark Streaming

Question 11

A Dataflow pipeline is processing a high-volume streaming data stream. The job is lagging behind by 30 minutes, and the Dataflow monitoring UI shows high system latency with low CPU utilization. Which action should be taken to improve throughput?

Accepted Answer

Enable Dataflow Shuffle. Option C is correct because high system latency with low CPU utilization indicates a bottleneck in data shuffling, not in processing capacity. Enabling Dataflow Shuffle offloads the shuffle operation to Google-managed resources, reducing disk I/O and network overhead, which directly improves throughput in streaming pipelines.

Answer

Enable Streaming Engine

Answer

Increase the number of workers

Answer

Disable hot key detection

Question 12

A company processes large volumes of GPS sensor data stored in Cloud Storage. Each hour, they run an Apache Spark job that aggregates the data by geohash region. The job must be cost-effective and scale automatically. Currently, they are using a Dataproc cluster with preemptible workers. Which improvement would best reduce costs while maintaining performance?

Accepted Answer

Use Dataproc Serverless Spark. Dataproc Serverless Spark (Option D) eliminates the need to manage a cluster, automatically scaling resources to match job demand and charging only for the resources consumed during execution. This removes the overhead of preemptible worker management and idle cluster costs, directly reducing expenses while maintaining performance for the hourly aggregation job.

Answer

Use a larger Dataproc cluster with standard workers

Answer

Migrate the job to BigQuery scheduled queries

Answer

Switch to Dataflow batch pipeline with Apache Beam

Question 13

You are designing a streaming Dataflow pipeline that reads from Cloud Pub/Sub. Some data may arrive late due to network delays. You need to ensure that late-arriving data is still processed, but after a certain point, it should be discarded to avoid unbounded state. What is the best practice?

Accepted Answer

Set a watermark and allowed lateness. Option D is correct because in streaming Dataflow pipelines, setting a watermark and allowed lateness provides a mechanism to handle late-arriving data from Pub/Sub without unbounded state growth. The watermark defines the point after which data is considered late, and allowed lateness specifies how long to wait for late data before discarding it, balancing completeness and state management.

Answer

Switch to a batch pipeline

Answer

Use fixed windows without allowed lateness

Answer

Discard all late-arriving data

Question 14

A team needs to orchestrate a complex ETL workflow that includes conditional branching (if new data arrives, run transformation A, else run transformation B), error handling, and coordination across multiple services. Which service should they use?

Accepted Answer

Cloud Composer (Apache Airflow). Cloud Composer (Apache Airflow) is the correct choice because it is designed for orchestrating complex, multi-step ETL workflows with conditional branching, error handling, and cross-service coordination. Airflow's directed acyclic graphs (DAGs) natively support conditional logic (e.g., BranchPythonOperator), retries, and dependency management across heterogeneous services, making it ideal for this use case.

Answer

Cloud Functions

Answer

Cloud Workflows

Answer

Cloud Scheduler

Question 15

You are optimizing a BigQuery query that runs on a large table (hundreds of TB). The table is partitioned by date and frequently queried with filters on a specific customer_id column and date range. Queries are slow even after partitioning. Which optimization should you apply?

Accepted Answer

Columnar clustering on customer_id. Clustering on customer_id within the partition improves query performance because BigQuery can prune blocks based on clustered columns. Partitioning alone doesn't help with non-date filters. Materialized views may help pre-aggregated queries but not ad-hoc customer_id filters. Denormalization is not an optimization. Increasing slots is expensive and doesn't address data structure.

Answer

Increase the number of BigQuery slots

Answer

Create materialized views for each customer

Answer

Denormalize the table to reduce joins