GCP-ADP Data Preparation And Ingestion Practice Test 7 — 15 Questions

Question 1

You are planning to ingest log data into Google Cloud. Which THREE of the following services support streaming ingestion?

Accepted Answer

BigQuery Storage Write API. BigQuery Storage Write API (A) is correct because it provides a gRPC-based streaming interface that lets applications write rows directly into BigQuery tables in real time, including exactly-once semantics via the default stream. Cloud Dataflow (D) is correct because it is a fully managed Apache Beam runner that supports streaming pipelines reading from sources such as Pub/Sub and writing to sinks like BigQuery. Cloud Pub/Sub (E) is correct because it is a globally distributed, real-time messaging service designed for streaming ingestion, delivering messages to subscribers with low latency and at-least-once delivery. Dataproc (B) is not a streaming ingestion service; it is a managed Hadoop/Spark cluster platform, and while Spark Streaming can run on it, the service itself is not a streaming ingestion endpoint. Cloud Storage Transfer Service (C) is not correct because it performs batch transfers of data between object stores or from HTTP sources into Cloud Storage, not continuous streaming ingestion.

Answer

Dataproc

Answer

Cloud Storage Transfer Service

Question 2

You need to load a large CSV file from Cloud Storage into BigQuery. The file has a header row and uses comma delimiters, but the data contains commas within quoted fields. Which BigQuery load option should you use to correctly parse the file?

Accepted Answer

Use the default CSV load with the appropriate schema and let BigQuery handle quoted fields. BigQuery's CSV parser natively supports quoted fields, so commas inside double quotes are treated as part of the field value. Providing a schema or using autodetect ensures correct column mapping. No special options are needed for this common CSV scenario.

Answer

Set the field delimiter to a pipe character (|)

Answer

Set the Quote character to a single quote (')

Answer

Enable the AllowQuotedNewlines option

Question 3

Your team runs an Apache Airflow environment on Google Cloud Managed Service for Apache Airflow. A daily ingestion DAG must wait for a partner team's separate DAG to finish loading a landing table before it can begin its own transformation. The partner DAG is in the same Airflow environment but runs on an unpredictable schedule. You want the dependency handled natively by Airflow rather than with a fixed sleep or time sensor. What should you configure?

Accepted Answer

An ExternalTaskSensor that points at the partner DAG and task. The requirement is a true task-level dependency on another DAG's completion, not a time or file heuristic. An ExternalTaskSensor reads the upstream task state from Airflow's metadata database, so the transformation proceeds only after the partner load succeeds. Time sensors, storage sensors, and API triggers either guess at completion or invert the dependency, none of which guarantees the landing table is fully populated.

Answer

A PythonOperator that calls the partner DAG's REST API trigger endpoint

Answer

A Cloud Storage sensor watching the landing bucket for new objects

Answer

A TimeSensor that waits until a predetermined clock time each day

Question 4

You are loading a 4 TB compressed CSV file from Cloud Storage into BigQuery using a load job. The file has a header row and uses pipe delimiters. You need the load to complete as fast as possible and minimize cost. What should you do?

Accepted Answer

Split the file into multiple smaller files (e.g., 100–500 MB each) in Cloud Storage, then run a single load job specifying the pipe delimiter and skipping the header row.. BigQuery load jobs scale by reading multiple files in parallel. A single 4 TB file cannot be parallelized, so splitting it into many smaller files is the key to fast ingestion. Specifying the delimiter and header row ensures correct parsing. This combination minimizes load time and uses BigQuery's load capacity efficiently, keeping costs lower than query-based alternatives.

Answer

Load the file into a BigQuery external table and then run a CREATE TABLE AS SELECT statement to materialize it.

Answer

Use the bq load command with the --max_bad_records flag set to a high value to skip any malformed rows.

Answer

Use the BigQuery web UI to create a load job with autodetect schema and default settings.

Question 5

Your team ingests JSON event files into Cloud Storage, and you must load them into BigQuery daily. The files contain a field named `user_info` that holds a nested JSON object with varying keys. You want BigQuery to store this field as a native STRUCT so analysts can query subfields with dot notation. Which schema type should you define for the `user_info` column when creating the table?

Accepted Answer

RECORD. BigQuery represents nested structures with the RECORD type, which is equivalent to a STRUCT. When you load JSON where a field contains an object, declaring that column as RECORD lets BigQuery preserve the nested keys and types, and analysts can query subfields with dot notation. STRING, BYTES, and GEOGRAPHY do not provide native nested-field access or schema enforcement.

Answer

BYTES

Answer

GEOGRAPHY

Answer

STRING

Question 6

You are designing a data ingestion pipeline that uses Pub/Sub to capture real-time events. You need to ensure that messages are processed exactly once and in order. Which two features should you configure? (Choose two.)

Accepted Answer

Use exactly-once delivery on the subscription. To achieve exactly-once and ordered processing in Pub/Sub, you must enable both message ordering and exactly-once delivery on the subscription. Message ordering ensures sequence per ordering key, while exactly-once delivery prevents duplicates. Other options like dead-letter topics or ack deadlines address different concerns and do not provide these guarantees.

Answer

Increase the number of subscriber clients

Answer

Create a dead-letter topic for failed messages

Answer

Set the acknowledgment deadline to 10 minutes

Question 7

Your team is ingesting streaming clickstream events from Pub/Sub into BigQuery. You need to ensure that the pipeline can handle duplicate message deliveries without creating duplicate rows in the destination table. What should you do?

Accepted Answer

Enable exactly-once delivery on the Pub/Sub subscription and use a Dataflow pipeline with the BigQuery Storage Write API configured for exactly-once semantics.. End-to-end duplicate elimination for Pub/Sub to BigQuery requires exactly-once delivery on the subscription and a sink that supports exactly-once writes. The BigQuery Storage Write API with exactly-once semantics, combined with Pub/Sub exactly-once delivery, ensures each message is processed and written once. Other options either provide best-effort deduplication or do not prevent duplicates.

Answer

Use the BigQuery streaming insert API with insertId populated from a unique event identifier, and rely on BigQuery to deduplicate rows.

Answer

Use Cloud Data Fusion to read from Pub/Sub and write to BigQuery with the deduplicate plugin enabled.

Answer

Configure the Pub/Sub subscription to use a dead-letter topic and set the maximum delivery attempts to 1.

Question 8

You are building a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. Some messages contain malformed JSON that causes the pipeline to fail repeatedly. You want the pipeline to continue processing valid messages and send the bad records to a dead-letter location for later analysis. Which Dataflow feature should you use?

Accepted Answer

A side output with a tagged PCollection for failed records. Side outputs in Apache Beam and Dataflow allow a DoFn to emit elements to additional PCollections using tags. By wrapping the JSON parsing logic in a try/catch and emitting failures to a tagged side output, the pipeline can continue processing valid messages and write the bad records to a separate sink such as Cloud Storage or BigQuery. Pub/Sub dead-letter topics, BigQueryIO retries, and logging sinks do not provide in-pipeline routing of malformed records.

Answer

A dead-letter queue configured in the Pub/Sub subscription

Answer

A Cloud Logging sink that exports error logs to Cloud Storage

Answer

A retry policy on the BigQueryIO write transform

Question 9

Your team needs to ingest data from an on-premises Oracle database into BigQuery with minimal latency and without writing code. The database is behind a firewall and cannot be accessed directly from the internet. You want to use a fully managed Google Cloud service. What should you use?

Accepted Answer

Datastream with a private connectivity configuration. Datastream is a fully managed, serverless change data capture and replication service that can connect to on-premises Oracle databases using private connectivity like VPN or Interconnect. It replicates data to BigQuery with low latency and requires no coding. Other options either lack Oracle support, require code, or are not fully managed for this purpose.

Answer

Cloud SQL for Oracle with federated queries

Answer

BigQuery Data Transfer Service with a VPN connection

Answer

Cloud Data Fusion with the Oracle connector

Question 10

You are using Dataflow to process a stream of JSON messages from Pub/Sub. The messages contain a timestamp field, but the event time is sometimes delayed by up to 30 minutes. You need to handle these late-arriving events correctly. What should you configure in your pipeline?

Accepted Answer

Enable a 30-minute allowed lateness on the window. To handle late-arriving events, you must set an allowed lateness on the window. This tells Dataflow to keep the window state open for a specified duration after the window's end, allowing late events to be processed and included in the correct aggregation. A 30-minute allowed lateness matches the maximum delay described.

Answer

Use a session window with a 30-minute gap

Answer

Set a fixed window of 1 hour

Answer

Set the pipeline's default timeout to 30 minutes

Question 11

You need to load a 5 TB uncompressed CSV file from Cloud Storage into BigQuery. The file is too large to load in a single job. What is the recommended approach?

Accepted Answer

Split the file into multiple smaller files and load them in parallel using multiple load jobs.. For loading very large files into BigQuery, splitting the data into multiple smaller files and using parallel load jobs is the most efficient and reliable method. It avoids per-job size limits and leverages BigQuery's parallel load capabilities. Other methods like streaming or external tables are not optimal for bulk loading.

Answer

Create an external table in BigQuery that points to the CSV file in Cloud Storage.

Answer

Compress the file with gzip and load it as a single file.

Answer

Use the BigQuery Storage Write API to stream the file directly into BigQuery.

Question 12

You are using Cloud Dataflow to process a batch pipeline that reads from Cloud Storage, transforms data, and writes to BigQuery. You notice that the pipeline is running slowly and want to optimize it. Which of the following should you do to improve performance?

Accepted Answer

Increase the number of workers and use a larger machine type.. To optimize a slow Dataflow batch pipeline, increasing the number of workers and using larger machine types can provide more parallelism and resources, directly addressing performance bottlenecks. Other options either change the pipeline type inappropriately or reduce resources, which would worsen performance.

Answer

Switch the pipeline to use streaming mode instead of batch mode.

Answer

Disable autoscaling to keep the worker count fixed.

Answer

Reduce the number of workers to lower contention.

Question 13

A financial analytics team ingests transaction events into Pub/Sub, then uses a Dataflow streaming pipeline with a GroupByKey transform to aggregate per-account totals. The team reports that late-arriving events—sometimes hours behind due to mobile connectivity—are being dropped and never counted in the aggregates. You need the pipeline to include those late events in the correct window. What should you configure?

Accepted Answer

Set a longer allowed lateness on the windowing strategy and emit late panes from the pipeline.. Allowed lateness is the Dataflow setting that governs how long window state persists after the watermark passes the window end. Extending it lets events arriving hours behind be assigned to the correct window and emitted as late panes. Pub/Sub retention, batch mode, and Reshuffle operate at different layers and do not change event-time windowing semantics.

Answer

Increase the Pub/Sub message retention duration to 7 days on the topic.

Answer

Switch the pipeline from streaming mode to batch mode with a longer maximum worker count.

Answer

Add a Reshuffle transform after the GroupByKey to redistribute late records across workers.

Question 14

A colleague must load a 200 GB uncompressed CSV file from a Cloud Storage bucket into a BigQuery table once, with the lowest possible load cost and no transformation. They want the simplest option that BigQuery handles natively. What should they use?

Accepted Answer

bq load with the CSV file specified as a Cloud Storage URI. Loading a file straight from Cloud Storage with the bq load command is BigQuery's native bulk ingest path and does not require a separate compute service, so it is both the simplest and the least expensive option for an untransformed one-time CSV load. Dataflow, Cloud Data Fusion, and Dataproc all add their own compute charges and pipeline complexity that the scenario does not need.

Answer

A Cloud Data Fusion pipeline with a Cloud Storage source and BigQuery sink

Answer

A Dataflow pipeline using the TextIO source and BigQueryIO sink

Answer

A Dataproc job running Spark to read the CSV and write to BigQuery

Question 15

You are building a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must handle late-arriving data up to 1 hour after the event time and still include it in the correct window. Which Dataflow feature should you configure?

Accepted Answer

Set the allowed lateness to 1 hour. Allowed lateness in Dataflow defines the additional time after the watermark that late data is accepted and processed into its appropriate window. Setting it to 1 hour ensures that events arriving up to an hour late are still included in the correct window, satisfying the requirement.

Answer

Configure the pipeline to use processing time instead of event time

Answer

Use a session window with a gap duration of 1 hour

Answer

Increase the Pub/Sub acknowledgment deadline to 1 hour