Databricks-Spark-Assoc Structured Streaming Practice Test 2 — 15 Questions

Question 1

Which TWO statements are true regarding the use of 'foreachBatch' in Structured Streaming?

Accepted Answer

It can be used to write data to multiple destinations in one batch.. foreachBatch is a powerful tool allowing developers to use batch-processing capabilities on streaming dataframes. It enables writing to sinks that lack native streaming support and allows for complex transformations that are not possible within the standard streaming API. Understanding how to integrate batch logic into streaming pipelines is critical for handling legacy systems and advanced data cleaning operations in Databricks environments.

Answer

It supports stateful streaming operations.

Answer

It automatically handles checkpointing for all logic.

Answer

It requires the query to be in Complete mode.

Question 2

Refer to the exhibit. You are configuring a streaming job. Based on the provided configuration, what is the impact of the 'checkpointLocation' setting?

Accepted Answer

It prevents the data from being processed multiple times.. The checkpointLocation is vital for maintaining the streaming query's progress. It stores metadata, offsets, and incremental state information. If the query stops, the checkpoint location allows the job to resume from the last offset. Without this setting, the query would lose its state upon failure, preventing the possibility of exactly-once delivery and making production-grade streaming pipelines impossible to maintain consistently.

Answer

It specifies where the actual output data is saved.

Answer

It defines the frequency of data writes to the sink.

Answer

It forces the query to use the Complete output mode.

Question 3

A data engineer is using `foreachBatch` in a Structured Streaming job to write to multiple sinks, including a Delta table and an external database. The engineer needs to ensure the operation is idempotent and handles reprocessing correctly. Which TWO statements about `foreachBatch` are correct in this context? (Choose two.)

Accepted Answer

`foreachBatch` can be used to write to multiple sinks, and each sink write can be made idempotent using the batch ID.. `foreachBatch` enables batch operations on each micro-batch and is useful for writing to multiple sinks. It provides a batch ID that can be used to achieve idempotent writes, which is essential for exactly-once semantics when sinks are not transactional. It does not automatically guarantee exactly-once, nor does it support continuous processing mode.

Answer

The function passed to `foreachBatch` is executed on the driver, so it can access driver-local variables and broadcast them.

Answer

`foreachBatch` supports continuous processing mode, allowing sub-millisecond latency.

Answer

`foreachBatch` guarantees exactly-once processing when writing to an external database without additional idempotent logic.

Question 4

A developer is testing a Structured Streaming query that reads from a rate source and writes to the console. The developer wants the query to process all available data and then stop automatically. Which trigger type should be used to achieve this behavior?

Accepted Answer

`trigger(once=True)`. The `once` trigger processes all available data in a single micro-batch and then stops the query. This is useful for testing or batch-like execution. The `availableNow` trigger also stops but processes in multiple batches. The question asks for processing all available data and then stopping; `once` is the direct answer for a single batch. The other triggers run continuously.

Answer

`trigger(continuous='1 second')`

Answer

`trigger(availableNow=True)`

Answer

`trigger(processingTime='10 seconds')`

Question 5

You are building a Structured Streaming pipeline in Databricks that reads from a Delta table using `spark.readStream.table("transactions")`. The pipeline performs a `groupBy("accountId").count()` aggregation and writes to another Delta table. You need the streaming query to emit updated counts for all accounts as new data arrives, even if an account has no new transactions in the current micro-batch. Which output mode should you use?

Accepted Answer

Update mode. The requirement is to emit updated counts for accounts as new data arrives, without re-emitting the entire result table. Update mode is designed for this: it outputs only the rows that changed in the current micro-batch. Append mode would only emit rows after they are finalized by a watermark, and complete mode would re-emit all rows every batch. Delta mode is not a valid output mode. Thus, update mode is the correct choice.

Answer

Append mode

Answer

Complete mode

Answer

Delta mode

Question 6

You are developing a Structured Streaming job in Databricks that reads from a Kafka topic and writes to a Delta table. The job uses `outputMode("append")` and a watermark of 5 minutes on the event time column. A record arrives with an event time that is 10 minutes behind the maximum event time seen so far. What happens to this record?

Accepted Answer

The record is dropped because it is later than the watermark.. A watermark of 5 minutes means that any event with a timestamp earlier than the maximum observed event time minus 5 minutes is considered late and is dropped. The record is 10 minutes behind, so it exceeds the watermark and is discarded. This ensures that state does not grow indefinitely and that results are emitted in a timely manner.

Answer

The record is included in the current micro-batch and updates the aggregation result.

Answer

The record is buffered in state until the watermark advances further.

Answer

The record triggers a reprocessing of previous micro-batches to incorporate the late data.

Question 7

A Databricks Structured Streaming job reads JSON files from a cloud storage directory using `spark.readStream.format("json").load("/mnt/raw")`. New files are added every minute, and the job must process only files that arrive after the query starts. Which option should be set to ensure that files already present when the query starts are ignored?

Accepted Answer

Set the option `ignoreExistingFiles` to `true`.. The file source in Structured Streaming provides the `ignoreExistingFiles` option specifically to skip files present at query start. Setting it to `true` ensures only newly arriving files are processed, which matches the stated requirement. Options that reference Kafka-specific settings or throughput limits do not address pre-existing files and are therefore incorrect for this scenario.

Answer

Set the option `maxFilesPerTrigger` to `1`.

Answer

Set the option `startingOffsets` to `latest`.

Answer

Set the option `includeExistingFiles` to `false`.

Question 8

A data engineer runs a Structured Streaming query that reads from a Delta table and writes to another Delta table. The query uses the default output mode. The engineer notices that the sink table contains duplicate rows for some records. Which output mode is being used, and why does it cause duplicates?

Accepted Answer

Update mode, because it writes only the rows that have been updated since the last trigger, and without a watermark, it may reprocess the same rows.. Update mode is the default output mode for streaming aggregations and emits only rows that have changed since the last trigger. Without a watermark to limit state, the engine may retain state indefinitely and re-emit rows that were already processed, leading to duplicates in the sink. This is a common cause of duplicate data in streaming pipelines that do not use watermarks or idempotent sinks.

Answer

Append mode with a watermark, because the watermark allows late data to be appended again, creating duplicates.

Answer

Complete mode, because it rewrites the entire result table on every trigger, which can duplicate rows if the sink is not idempotent.

Answer

Append mode, because it only writes new rows and never updates existing ones.

Question 9

A streaming job reads clickstream events and performs a stateful `groupBy("sessionId").count()` with a 15-minute watermark on `eventTime`. The developer notices that state store size keeps growing without bound even though watermarks are configured. Which change most directly limits the state growth?

Accepted Answer

Ensure the watermark is defined on the event-time column and that the query is not using a non-time-based aggregation key that prevents eviction.. Stateful aggregation state is only evicted when the watermark advances beyond the event time of a key and the aggregation is keyed on that event-time. If the watermark is defined on a different column, or if the aggregation does not include the event-time window, the engine has no signal to drop old keys, so the state store grows without limit. Correcting the watermark column and key alignment is the direct remedy.

Answer

Reduce the watermark to a shorter duration, such as 1 minute, so state entries expire sooner.

Answer

Enable `spark.sql.streaming.stateStore.providerClass` to the RocksDB state store so state is written to disk instead of memory.

Answer

Switch the output mode from "update" to "append" so only finalized rows are emitted and old state is discarded.

Question 10

You are monitoring a Structured Streaming query on Databricks. You want to see the number of input rows processed per second and the duration of each micro-batch. Which of the following should you use?

Accepted Answer

The Spark UI's Structured Streaming tab.. The Spark UI's Structured Streaming tab provides comprehensive metrics for running streaming queries, including input rate, processing rate, and batch durations. It is the standard tool for monitoring streaming jobs in Databricks. Other methods like `lastProgress` are programmatic and require additional handling to visualize.

Answer

The `spark.sql.streaming.metricsEnabled` configuration.

Answer

The `streamingQuery.status` method.

Answer

The `streamingQuery.lastProgress` method.

Question 11

You are using `foreachBatch` in a Structured Streaming query to write to multiple sinks and perform custom logic. Which TWO of the following statements are true regarding `foreachBatch`? (Choose two.)

Accepted Answer

`foreachBatch` allows you to use batch DataFrame operations that are not supported in streaming DataFrames.. `foreachBatch` enables batch operations on micro-batches and allows writing to multiple sinks. It does not guarantee exactly-once by itself; idempotent logic is required. It is only available on streaming DataFrames and does not mandate a specific output mode. Thus, the true statements are that it allows unsupported batch operations and supports multiple sinks.

Answer

`foreachBatch` requires the output mode to be `append`.

Answer

`foreachBatch` guarantees exactly-once processing even if the function is not idempotent.

Answer

`foreachBatch` can be used with both streaming and batch DataFrames.

Question 12

A data engineer is using `foreachBatch` in a Structured Streaming job to write to multiple sinks. Which two statements are true about `foreachBatch`? (Choose two.)

Accepted Answer

`foreachBatch` allows you to apply custom logic that is not natively supported in Structured Streaming, such as writing to a JDBC sink with merge operations.. `foreachBatch` allows developers to apply batch operations to each micro-batch of a streaming query, enabling reuse of existing batch code and writing to multiple sinks or custom destinations. It is particularly useful for operations not natively supported in Structured Streaming, such as JDBC upserts. However, it does not inherently provide exactly-once guarantees; that depends on the sink's idempotency and checkpointing.

Answer

`foreachBatch` guarantees exactly-once processing even if the function is not idempotent.

Answer

`foreachBatch` can be used with both streaming and batch DataFrames interchangeably.

Answer

`foreachBatch` processes data in a single batch, so it cannot handle large volumes of data efficiently.

Question 13

A developer wants to run a Structured Streaming query that reads from a Kafka topic and writes to a Delta table. The query must process data as soon as it arrives, with minimal latency, but should not run continuously without bounds. Which trigger type should be used to process all available data in a single micro-batch and then stop?

Accepted Answer

`trigger(once=True)`. The `once` trigger is designed to process all available data in one micro-batch and then terminate the query. This matches the requirement to process data as it arrives but stop after one execution. Other triggers either run continuously or process data in multiple batches. For a one-time batch-like run of a streaming query, `once` is the appropriate choice.

Answer

`trigger(availableNow=True)`

Answer

`trigger(processingTime='5 seconds')`

Answer

`trigger(continuous='1 second')`

Question 14

You maintain a Structured Streaming job on Databricks that reads clickstream events from an Auto Loader source and writes to a Delta table. The job runs continuously using the default trigger. Downstream consumers complain that new data only appears in the Delta table every few minutes, even though events arrive continuously. You want the sink to reflect new micro-batches as quickly as possible without changing the source. Which trigger setting should you apply?

Accepted Answer

trigger(processingTime='0 seconds'). A zero-second processing time trigger removes the artificial pause between micro-batches, so each batch begins immediately after the prior one finishes. This is the standard way to achieve the lowest micro-batch latency while remaining in the default execution mode. The other triggers either terminate the query after processing available data or switch to an experimental execution mode that does not fit this pipeline.

Answer

trigger(once=True)

Answer

trigger(continuous='1 second')

Answer

trigger(availableNow=True)

Question 15

A developer is building a Structured Streaming job that reads from a Kafka source and needs to handle late-arriving data while performing a windowed aggregation. Which TWO actions are required to ensure that late data is correctly incorporated into the aggregation results? (Choose two.)

Accepted Answer

Define a watermark on the event-time column using `.withWatermark("eventTime", "10 minutes")`.. To correctly incorporate late data into a windowed aggregation, the query must have a watermark defined on the event-time column, which allows the engine to track progress and accept late records within the threshold. Additionally, the output mode must be set to "update" or "append" so that corrected results are emitted downstream. Other options like `foreachBatch`, `maxOffsetsPerTrigger`, or state store tuning do not provide the necessary late data handling.

Answer

Configure `maxOffsetsPerTrigger` to a high value to allow more late records to be processed.

Answer

Use `foreachBatch` to manually merge late records into the aggregation result.

Answer

Enable `spark.sql.streaming.stateStore.minDeltasForSnapshot` to retain more state for late data.