Courseiva

CCNA Pde Ingestion Processing Questions

19 of 94 questions · Page 2/2 · Pde Ingestion Processing topic · Answers revealed

Practice these questionsExam hubAll questions
76
MCQhard

A company is using Pub/Sub to ingest clickstream events and Dataflow to write to BigQuery. They observe that some events are malformed and cause the pipeline to fail. They need a solution that captures malformed events without blocking the pipeline and allows reprocessing later. Which Dataflow pattern should they implement?

A.Use a side input to filter malformed events before the main pipeline
B.Use the Reshuffle transform to reattempt failures
C.Write malformed events to a dead letter sink (e.g., another Pub/Sub topic or GCS bucket) and continue processing healthy events
D.Use logging alerts to notify the team and stop the pipeline on error
AnswerC

Dead letter sink is the correct pattern: isolate bad records and let the pipeline proceed.

Why this answer

Dead letter sinks (DLQ) are the standard pattern for handling bad records in Dataflow. The pipeline writes malformed records to a separate sink (e.g., Pub/Sub topic or GCS) for later analysis. Side inputs are for enriching data, not error handling.

Reshuffle doesn't apply. Output tags (side outputs) can also be used, but explicit dead letter pattern is more standard.

Practice this question →
77
MCQhard

A streaming pipeline ingests events from Pub/Sub, enriches them via a slow REST API call, and writes the result to BigQuery. The API has a limit of 10 requests per second per client. The pipeline processes 1000 messages per second. Which approach minimizes latency while respecting API limits?

A.Use a global window with a trigger that fires every second, and inside the DoFn limit concurrent API calls to 10.
B.Fan out the stream to multiple REST API instances using Pub/Sub topic splitting.
C.Use a Dataflow Flex Template to run multiple pipelines, each processing a subset of messages.
D.Assign each message a random key and use a sliding window of 10 seconds; the API call will be distributed across workers.
AnswerA

Groups messages into batches per second, then controls concurrency to stay within the 10 req/s limit.

Why this answer

Using a global window with a trigger every second groups 1000 messages into a batch, and then throttling concurrent API calls to 10 within the DoFn (e.g., using a fixed-size thread pool) respects the API limit while minimizing latency by processing messages in parallel up to the limit. Option B is wrong because fanning out to multiple API instances doesn't help if the limit is per client; the total requests per second across all instances would still exceed the client limit. Option C is wrong because Dataflow Flex Templates are used to run parameterized pipelines, not to solve throttling issues.

Option D is wrong because assigning a random key and using a sliding window distributes messages across workers, but without explicit throttling, the API limit could still be exceeded.

Practice this question →
78
Multi-Selecthard

Your company has a Dataproc cluster that runs Spark jobs. You need to choose between RDDs, DataFrames, and Datasets for a new job that performs complex aggregations on structured data. Which TWO statements are correct regarding performance and ease of use?

Select 2 answers
A.DataFrames and Datasets are both available in PySpark.
B.DataFrames store data in a columnar format, allowing better compression.
C.RDDs are easier to use than DataFrames for complex aggregations.
D.DataFrames are optimized by Spark's Catalyst optimizer, leading to faster execution.
E.Datasets provide compile-time type safety and are always faster than DataFrames.
AnswersB, D

DataFrames use Spark's internal binary format (Tungsten) with columnar storage, enabling efficient compression and serialization.

Why this answer

DataFrames are optimized with Catalyst optimizer and Tungsten execution, providing better performance than RDDs for structured data. Datasets combine type safety with optimized execution, but for most analytics workloads, DataFrames are sufficient and simpler.

Practice this question →
79
MCQeasy

You need to stream real-time user click events from your application into BigQuery for immediate analysis. The events must be available for query within seconds. Which approach is recommended?

A.Use Pub/Sub to Dataflow to BigQuery with the Storage Write API for high-throughput streaming.
B.Use Cloud Data Fusion to ingest streaming data from Pub/Sub into BigQuery.
C.Use Cloud Functions to receive events from Pub/Sub and insert them into BigQuery using the legacy streaming API.
D.Use Pub/Sub with a BigQuery subscription to directly write events into BigQuery.
AnswerA

This is the recommended architecture: Pub/Sub for ingestion, Dataflow for stream processing, and Storage Write API for low-latency streaming writes.

Why this answer

Pub/Sub to Dataflow to BigQuery using the Storage Write API provides the highest throughput and reliability with near-real-time latency. Legacy streaming inserts are limited and have higher latency. Direct Pub/Sub to BigQuery subscription is not a native feature.

Cloud Functions is not suitable for high-throughput streaming.

Practice this question →
80
MCQeasy

A data engineer needs to query a BigQuery table that contains an array of structs. They want to expand the array into separate rows for each element. Which SQL function should they use?

A.STRUCT
B.UNNEST
C.ARRAY_AGG
D.SPLIT
AnswerB

UNNEST expands an array into rows; it is typically used with CROSS JOIN.

Why this answer

UNNEST is used to flatten arrays into a set of rows. CROSS JOIN UNNEST is standard. STRUCT is for creating structs, ARRAY_AGG is for aggregation, and SPLIT is for strings.

The question asks to expand an array, which is exactly UNNEST.

Practice this question →
81
MCQhard

A company uses BigQuery to store event data. They need to load data from multiple sources with different schemas and expect frequent schema changes. Which approach provides the most flexibility for schema evolution while minimizing load failures and performance impact?

A.Load data as JSON files in Cloud Storage and use external tables
B.Use the Storage Write API in buffered mode with schema auto-detection
C.Use legacy streaming inserts with schema auto-detect enabled
D.Use Dataflow to preprocess and write to BigQuery using Storage Write API in committed mode
AnswerB

Buffered mode allows schema updates and auto-detection, reducing failures and handling schema evolution well.

Why this answer

Using the Storage Write API with buffered mode allows schema auto-detection and flexible schema updates without failing loads, and provides better performance than legacy streaming inserts.

Practice this question →
82
MCQhard

A Dataflow pipeline reads from Pub/Sub, applies a keyed stateful ParDo that uses state variables to deduplicate events based on event ID, and writes to BigQuery. During a pipeline update, some events are duplicated in BigQuery. The state is not preserved across updates. Which configuration ensures exactly-once semantics during updates?

A.Drain the pipeline and start the updated pipeline; all in-flight data will be processed.
B.Cancel the pipeline and restart it; Pub/Sub subscriptions will be rewound.
C.Use the Storage Write API's exactly-once delivery mode.
D.Take a snapshot of the pipeline before updating, then start the new pipeline from the snapshot.
AnswerD

Snapshots preserve the state of the pipeline, including deduplication state, allowing the new pipeline to resume without reprocessing duplicates.

Why this answer

Draining the pipeline stops it and completes processing in-flight, then the updated pipeline can start fresh. However, because state is lost, duplicates may still occur if the new pipeline processes events that were already committed. To preserve state, use snapshotting: take a snapshot before update and start the new pipeline from the snapshot.

BigQuery's Storage Write API with exactly-once semantics can help at the sink but does not prevent duplicate processing if state is lost. As long as the deduplication state is recovered from the snapshot, duplicates are avoided.

Practice this question →
83
MCQmedium

You are designing a streaming pipeline that ingests events from Pub/Sub, enriches them with a machine learning model, and writes the results to BigQuery. The ML model is deployed on Cloud Run and has a high latency (500ms per request). You need to minimize the impact of slow ML inference on the overall pipeline throughput. Which approach should you take?

A.Use Dataflow to write events to Pub/Sub, then use a separate Dataflow pipeline that batches calls to Cloud Run.
B.Increase the number of Dataflow workers to compensate for the latency.
C.Use Cloud Functions to call Cloud Run and write directly to BigQuery.
D.Use Dataflow's ParDo with synchronous calls to Cloud Run for each element.
AnswerA

Decoupling via Pub/Sub allows batching and async processing, improving throughput.

Why this answer

It uses Dataflow to batch events before sending them to Cloud Run, which amortizes the 500ms per-request latency over multiple events, significantly increasing throughput. By writing events to Pub/Sub and then processing them in a separate Dataflow pipeline with batched calls, you decouple the ingestion from the inference and avoid blocking on each individual request.

Exam trap

The trap here is that candidates assume parallelism (more Dataflow workers) or faster invocation methods (Cloud Functions) can overcome high per-request latency, when the real solution is to batch requests using Dataflow's batch processing capabilities to reduce the number of round trips.

How to eliminate wrong answers

Option B is wrong because increasing the number of Dataflow workers does not reduce the per-element latency of synchronous calls; it only adds parallelism, which can lead to excessive concurrent calls to Cloud Run and potential throttling or cost spikes. Option C is wrong because Cloud Functions are not designed for high-throughput streaming pipelines and would still make synchronous calls to Cloud Run for each event, suffering the same latency bottleneck. Option D is wrong because using ParDo with synchronous calls per element means each element waits 500ms before the next element is processed, severely limiting throughput and not leveraging batching.

Practice this question →
84
MCQmedium

A data engineer needs to create a Dataflow pipeline that reads from Pub/Sub, applies a Python transformation, and writes to BigQuery. The pipeline should be reusable across environments with different parameters. Which deployment method is most appropriate?

A.Classic Template
B.Flex Template
C.Direct pipeline submission with gcloud dataflow jobs run
D.Cloud Composer to trigger Dataflow jobs
AnswerB

Flex Templates support any SDK (including Python) and allow runtime parameters.

Why this answer

Flex Templates (Option B) are the most appropriate deployment method because they allow you to package a custom Docker image containing your Python transformation code and dependencies, making the pipeline reusable across environments with different runtime parameters. Unlike Classic Templates, Flex Templates support arbitrary pipeline code and can be parameterized at runtime via the Dataflow UI or API, which is essential for a multi-environment deployment strategy.

Exam trap

The trap here is that candidates often confuse Classic Templates with Flex Templates, assuming both support custom code, but Classic Templates are limited to Google-provided templates and cannot run arbitrary Python transformations, making Flex Templates the only correct choice for custom, reusable pipelines.

How to eliminate wrong answers

Option A is wrong because Classic Templates are pre-built, Google-provided templates that do not support custom Python transformations; they are limited to a fixed set of template parameters and cannot be easily parameterized for different environments. Option C is wrong because direct pipeline submission with gcloud dataflow jobs run does not provide a reusable, parameterized template mechanism; each submission requires the full pipeline code and configuration, making it unsuitable for repeated deployment across environments. Option D is wrong because Cloud Composer is an orchestration tool for scheduling and monitoring workflows, not a deployment method for creating reusable, parameterized Dataflow templates; it can trigger Dataflow jobs but does not solve the need for a template that can be reused with different parameters.

Practice this question →
85
MCQhard

A company uses Kafka on Dataproc to ingest streaming data. They want to process the data with Spark Structured Streaming and write results to BigQuery. The team is using Dataproc clusters. Which approach minimizes cost while maintaining performance?

A.Use a Dataproc cluster with all preemptible VMs
B.Use a single-node Dataproc cluster
C.Use a Dataproc cluster with standard master nodes and preemptible worker nodes
D.Use a Dataproc cluster with standard nodes and enable autoscaling
AnswerC

Workers can be preemptible; master should be standard for stability.

Why this answer

Preemptible VMs are cost-effective for worker nodes; master nodes should be standard for reliability.

Practice this question →
86
MCQhard

A Dataflow streaming pipeline is experiencing high latency and frequent OOM errors when processing variable-sized JSON messages from Pub/Sub. The team suspects that the autoscaling is not effective. Which feature should they enable to improve resource utilization?

A.Horizontal autoscaling
B.Dataflow Prime
C.FlexRS
D.Streaming Engine
AnswerB

Dataflow Prime offers vertical scaling and right-fitting, which helps with variable-sized messages and OOM errors.

Why this answer

Dataflow Prime is the correct choice because it provides intelligent resource management that automatically adjusts worker resources (CPU, memory) based on the pipeline's processing demands, which is critical for variable-sized JSON messages. It addresses both high latency and OOM errors by optimizing resource utilization beyond simple autoscaling, including predictive autoscaling and flexible resource scheduling to handle spikes in message size without manual tuning.

Exam trap

A common misconception is that Streaming Engine solves all streaming performance issues, but it specifically addresses shuffle and state persistence, not worker memory management for variable payloads.

How to eliminate wrong answers

Option A is wrong because Horizontal autoscaling is a basic feature already enabled by default in Dataflow; it only scales the number of workers horizontally and does not address memory inefficiencies or OOM errors caused by variable-sized messages. Option C is wrong because FlexRS is designed for batch pipelines with flexible scheduling to reduce costs, not for streaming pipelines requiring low latency and real-time processing. Option D is wrong because Streaming Engine offloads shuffle and state storage to backend services to reduce disk I/O and checkpoint latency, but it does not directly manage per-worker memory allocation or prevent OOM errors from variable-sized payloads.

Practice this question →
87
MCQeasy

A data engineer needs to transfer 500 TB of on-premises data to Google Cloud Storage. The data is stored on NAS devices and the network bandwidth is limited to 100 Mbps. What is the most cost-effective and timely transfer method?

A.Use Storage Transfer Service over the internet
B.Use a VPN connection and rsync
C.Use gsutil cp in parallel
D.Use Transfer Appliance
AnswerD

Transfer Appliance is designed for offline petabyte-scale transfers, avoiding bandwidth limitations.

Why this answer

At 100 Mbps, transferring 500 TB over the network would take over 500 days. Transfer Appliance is designed for petabyte-scale offline transfer, shipping a physical appliance to your data center. Other options are not feasible due to bandwidth constraints.

Practice this question →
88
MCQmedium

A data engineer is using Apache Spark on Dataproc to process a large dataset. They need to perform complex aggregation and transformation with high performance. The dataset has a known schema and they want to take advantage of Catalyst optimizer. Which Spark API should they use?

A.Spark SQL only
B.DataFrames
C.Datasets
D.RDDs
AnswerB

DataFrames have Catalyst optimizer, which improves performance for complex transformations.

Why this answer

DataFrames provide high-level API with Catalyst optimizer for performance, making them ideal for complex aggregations and transformations on structured data.

Practice this question →
89
MCQmedium

A data engineer needs to create a Dataflow pipeline template that can be reused across multiple environments (dev, staging, prod) with different parameters (e.g., input Pub/Sub topic, output BigQuery table). Which template type should they use?

A.Dataflow Prime
B.Flex Template
C.Classic Template
D.Cloud Composer workflow template
AnswerB

Flex Templates support custom Docker images and runtime parameters, making them suitable for multi-environment reuse.

Why this answer

Flex Templates (B) are the correct choice because they package a Dataflow pipeline as a Docker image, allowing environment-specific parameters (e.g., Pub/Sub topic, BigQuery table) to be passed at runtime via the --parameters flag. This enables true reusability across dev, staging, and prod without modifying the template code, unlike Classic Templates which require compile-time parameterization.

Exam trap

The Google Cloud exam often tests the distinction between Classic Templates (compile-time parameterization) and Flex Templates (runtime parameterization), trapping candidates who assume all templates support the same level of parameter flexibility.

How to eliminate wrong answers

Option A is wrong because Dataflow Prime is a managed service for optimizing resource utilization and autoscaling, not a template type for parameterized reuse. Option C is wrong because Classic Templates require parameters to be baked in at staging time, making them less flexible for multi-environment reuse without rebuilding the template. Option D is wrong because Cloud Composer is an Apache Airflow orchestration service used to schedule and monitor workflows, not a Dataflow template type for parameterized pipeline reuse.

Practice this question →
90
MCQeasy

A data engineer is building a Dataflow pipeline that reads from BigQuery, transforms data using Apache Beam, and writes results to Cloud Storage in Avro format. They need to ensure the pipeline can be easily redeployed with different parameters without modifying code. Which deployment method should they use?

A.Dataflow Flex Templates
B.Direct deployment using the gcloud command with parameters
C.Dataflow Classic Templates
D.Deploy as a Cloud Function triggered by Cloud Scheduler
AnswerA

Flex Templates use Docker images and support arbitrary pipeline options, including custom parameters.

Why this answer

Dataflow Flex Templates allow you to package a pipeline as a Docker image and pass runtime parameters, enabling parameterized deployments without code changes.

Practice this question →
91
MCQhard

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. Some incoming messages are malformed and fail to parse. How should you handle these messages to ensure the pipeline continues processing without data loss?

A.Configure Pub/Sub to retry indefinitely until the message is processed
B.Use a try-catch block in the pipeline and ignore malformed messages
C.Write malformed messages to a dead-letter sink (e.g., Pub/Sub topic or GCS) and continue processing
D.Set the pipeline to fail and alert the team via Cloud Monitoring
AnswerC

Why this answer

The recommended pattern is to use a dead-letter queue (e.g., a separate Pub/Sub topic or a GCS bucket) to store failed messages after a retry threshold is reached. This preserves messages for later analysis without blocking the main pipeline.

Practice this question →
92
MCQhard

A company is running a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. They notice that the number of workers is not scaling up to handle increased throughput, causing latency spikes. The pipeline uses a GlobalWindow with default triggering. What is the most likely cause of the under-scaling?

A.The pipeline includes a GroupByKey that creates a hot key, limiting parallelism
B.The Pub/Sub subscription has a large backlog, but Dataflow automatically scales to handle it
C.The pipeline uses the default worker machine type, which is too small
D.The pipeline is using legacy streaming inserts instead of the Storage Write API
AnswerA

Hot keys prevent splitting the work across workers, causing underutilization and scaling issues.

Why this answer

Dataflow's autoscaling is based on CPU utilization and throughput. If the pipeline uses a GroupByKey with hot keys, parallelism is limited and workers may not scale effectively.

Practice this question →
93
Multi-Selectmedium

A data engineer needs to perform a one-time migration of 10 TB of data from on-premises Hadoop HDFS to Cloud Storage. The network link is 1 Gbps. Which TWO services or tools should they consider? (Choose 2)

Select 2 answers
A.Dataproc with DistCp
B.Cloud Storage Transfer Service
C.BigQuery Data Transfer Service
D.gsutil rsync with parallel composite uploads
E.Transfer Appliance
AnswersA, E

Dataproc with DistCp is a standard method to copy data from HDFS to Cloud Storage, using Hadoop's distributed copy tool for efficient parallel transfer.

Why this answer

For a one-time migration of 10 TB from on-premises HDFS to Cloud Storage over a 1 Gbps link, the most appropriate services are Dataproc with DistCp and Transfer Appliance. Dataproc with DistCp is a proven method for copying data from HDFS to Cloud Storage, leveraging Apache Hadoop's distributed copy tool. Transfer Appliance is ideal for large data volumes when network bandwidth is limited, as it physically ships the data.

Cloud Storage Transfer Service does not support HDFS as a source directly, so it is not suitable. gsutil rsync with parallel composite uploads could be used but would be slower and less efficient for this volume.

Practice this question →
94
Multi-Selecthard

A company uses Workflows to orchestrate a multi-step data pipeline. One step calls an HTTP endpoint that may take up to 10 minutes, but the default Workflows timeout is too short. They also need to handle transient errors with retries. Which TWO configurations should they apply? (Choose 2)

Select 2 answers
A.Set a step timeout of 600 seconds for the HTTP call step
B.Configure a dead letter queue for failed steps
C.Use the default retry policy on the step
D.Set the workflow execution timeout to 600 seconds
E.Add a retry policy on the step with appropriate conditions for transient errors
AnswersA, E

This extends the timeout for that specific step to 10 minutes.

Why this answer

To extend the timeout, set a step timeout of 600 seconds (10 minutes). To handle transient errors, use a retry policy with appropriate conditions. Setting the entire workflow timeout to 10 minutes is not necessary if individual step timeouts are set.

The default retry policy does not cover all transient errors. Adding a dead letter queue is for event-driven patterns, not Workflows.

Practice this question →
← PreviousPage 2 of 2 · 94 questions total

Ready to test yourself?

Try a timed practice session using only Pde Ingestion Processing questions.

Start 20-question session