Databricks-Spark-Assoc Structured Streaming Practice Test 4 — 15 Questions

Question 1

A developer is building a Structured Streaming pipeline that reads from a Kafka topic and writes to a Delta table. The pipeline must tolerate occasional downstream failures and reprocess data without duplicates. The developer sets a checkpoint location and uses the default output mode. Which statement correctly describes how the checkpoint location contributes to fault tolerance in this scenario?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The checkpoint location stores the last processed offset and query metadata, enabling the query to resume exactly where it left off after a restart.. The checkpoint location is critical for fault tolerance in Structured Streaming. It records the progress of the query, including offsets and state, so that after a failure the query can resume from the last committed offset. This, combined with a replayable source and an idempotent sink, enables exactly-once processing. The other options misattribute caching, deduplication, or data storage to the checkpoint.

Answer

The checkpoint location caches the entire streamed DataFrame in memory, allowing faster recovery after a driver restart.

Answer

The checkpoint location stores a copy of the Kafka topic's data, enabling replay from the checkpoint instead of Kafka.

Answer

The checkpoint location automatically deduplicates records in the Delta table by comparing primary keys.

Question 2

A developer writes a Structured Streaming query that reads from a Kafka topic with `spark.readStream.format("kafka")` and then calls `.writeStream.format("console").start()`. The query runs, but after a few minutes the driver logs show that the query is only processing newly arriving offsets and older messages in the topic are never read. What is the most likely cause?

Accepted Answer

The Kafka source defaults to `startingOffsets` = "latest", so the first micro-batch begins at the tail of each partition and earlier records are skipped.. When a Kafka-backed streaming query starts without an existing checkpoint, the source resolves its first offsets from `startingOffsets`. Because the default is "latest", the initial micro-batch aligns to the end of each partition, so any records produced before the query started are ignored. Setting `startingOffsets` to "earliest" or a JSON offset map is required to consume the backlog.

Answer

Kafka consumers always begin reading from the committed consumer-group offset, and Structured Streaming reuses that group offset as its start position.

Answer

The console sink silently discards any record whose Kafka `offset` value is lower than the current micro-batch ID.

Answer

The topic's retention policy removed all historical segments before the query started, so only new records remained available to read.

Question 3

You are building a Structured Streaming job that reads from a Kafka source and writes to a Delta table. You need to ensure that the job can recover from failures and process data exactly once. Which TWO of the following are required to achieve exactly-once semantics? (Choose two.)

Accepted Answer

Use a checkpoint location to store progress information.. Exactly-once semantics in Structured Streaming require a reliable checkpoint location to track progress and an idempotent sink like Delta Lake that can handle replays without duplicating data. Checkpointing ensures the query can recover from failures, while Delta's transactional writes guarantee that each batch is applied exactly once.

Answer

Enable `spark.sql.streaming.forceDeleteTempCheckpointLocation` to clean up temporary files.

Answer

Configure the Kafka source with `startingOffsets` set to `earliest`.

Answer

Set the `maxOffsetsPerTrigger` option to limit the number of records per batch.

Question 4

You are running a Structured Streaming query on Databricks that reads from a Kafka topic and writes to a Delta table. The query uses `option("maxOffsetsPerTrigger", 10000)` to limit the number of records per micro-batch. During a peak, the Kafka topic accumulates a large backlog. You notice that the query is processing data but the backlog is not decreasing. What is the most likely cause?

Accepted Answer

The `maxOffsetsPerTrigger` value is lower than the rate at which new data is arriving.. The backlog is not decreasing because the query is limited to processing 10000 records per micro-batch, which is likely less than the incoming rate. This cap is set by `maxOffsetsPerTrigger`. To reduce the backlog, you must increase this limit or the processing capacity. The other options do not address the rate mismatch: the option is not ignored, triggers do not overlap, and starting offsets only affect initial reads.

Answer

The Kafka source is not configured with `startingOffsets` set to `latest`.

Answer

The trigger interval is too short, causing the query to start new micro-batches before the previous one finishes.

Answer

The `maxOffsetsPerTrigger` option is ignored when writing to Delta Lake.

Question 5

A streaming DataFrame `df` has a watermark defined on `eventTime` with a delay of 5 minutes. The query uses `withWatermark("eventTime", "5 minutes")` and writes to a Delta table in `append` output mode. A record with event time 10:00 arrives when the watermark is at 10:10. What happens to this record?

Accepted Answer

The record is dropped because its event time is older than the watermark.. With a 5-minute watermark, any record whose event time is more than 5 minutes behind the current watermark is considered late. In append mode, such records are dropped because the system assumes all data for that time has already been emitted. The record at 10:00 arrives when the watermark is 10:10, so it is 10 minutes late and is discarded. This matches the expected watermark behavior.

Answer

The record is held in state until the next watermark update and then emitted.

Answer

The record causes the query to fail with a watermark violation error.

Answer

The record is processed and appended to the Delta table immediately.

Question 6

You are developing a Structured Streaming job that reads from a Delta table and writes to another Delta table. You need to ensure that the streaming query can recover from failures and continue processing without data loss or duplication. Which of the following must be configured?

Accepted Answer

A checkpoint location. To recover from failures and ensure exactly-once processing, a Structured Streaming query must have a checkpoint location. This location stores the progress information and state, allowing the query to resume from where it left off. Watermarks, trigger intervals, and output modes are unrelated to basic fault tolerance. Therefore, the checkpoint location is the essential configuration.

Answer

An output mode of `complete`

Answer

A watermark on the event time column

Answer

A trigger interval of `processingTime='1 second'`

Question 7

A developer is building a Structured Streaming pipeline that reads from a Kafka topic and writes to a Delta table. The pipeline must tolerate occasional downstream failures and reprocess data without duplicates. The developer sets a checkpoint location and uses the default output mode. Which statement correctly describes how the checkpoint location contributes to fault tolerance in this scenario?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The watermark allows the engine to drop state for windows that are older than the watermark, and late data within 15 minutes is still processed and can update the aggregation.. A watermark in Structured Streaming defines a threshold for how late data can arrive. It allows the engine to drop old state once the watermark passes the window end time, preventing unbounded state growth. Late data within the watermark delay is still processed and can update aggregations. This balances correctness and resource usage, enabling efficient handling of late-arriving events in streaming aggregations.

Answer

The watermark ensures exactly-once processing by deduplicating records that arrive within the 15-minute window.

Answer

The watermark forces the output mode to complete, so the entire aggregation result is recomputed on every trigger.

Answer

The watermark causes the aggregation state to be dropped immediately after 15 minutes, and any late data beyond that is ignored.

Question 8

A developer is building a Structured Streaming pipeline that reads from a Kafka topic and writes to a Delta table. The pipeline must tolerate occasional downstream failures and reprocess data without duplicates. The developer sets a checkpoint location and uses the default output mode. Which statement correctly describes how the checkpoint location contributes to fault tolerance in this scenario?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Ensure the sink is idempotent or transactional, such as Delta Lake.. Exactly-once in Structured Streaming relies on two pillars: a reliable checkpoint to track progress and an idempotent or transactional sink to handle replays. A checkpoint location stores offsets and state, enabling recovery without data loss. A sink like Delta Lake ensures that re-executed batches do not duplicate output. Together, they provide end-to-end exactly-once. Other options affect output mode, custom sink logic, or trigger frequency, none of which guarantee exactly-once.

Answer

Set the output mode to `complete`.

Answer

Use the `foreachBatch` sink to write to the Delta table.

Answer

Use `trigger(processingTime='0 seconds')` for continuous processing.

Question 9

A developer wants to start a Structured Streaming query that reads from a Delta table and writes to another Delta table, and needs the query to process all existing data in the source table on its first run. Which option should be set on the read stream?

Accepted Answer

No option is needed; the Delta source reads the initial snapshot by default on the first run when no checkpoint exists.. The Delta Lake streaming source is designed to read the full table snapshot as the first micro-batch when a query starts without a checkpoint, and then to continue with subsequent changes. No option is required to include existing data; the behavior is the default. Options like `startingVersion` and `ignoreChanges` alter other aspects of the read, not the initial snapshot inclusion.

Answer

`.option("ignoreChanges", "false")`

Answer

`.option("includeExistingData", "true")`

Answer

`.option("startingVersion", "0")`

Question 10

You are performing a stream-stream join between two streaming DataFrames, `orders` and `payments`, both with watermarks defined on their event time columns. The join condition is `orders.orderId == payments.orderId` and it is an inner join. What happens to state in this join?

Accepted Answer

State is maintained for both streams and is cleaned up based on the watermarks.. In a stream-stream inner join, the engine maintains state for both streams to match records across micro-batches. Watermarks define when state can be evicted: once the watermark passes the event time of a record plus the allowed lateness, its state is removed. This ensures that late-arriving matches within the watermark window are still processed.

Answer

State is maintained only for the right stream, as it is the stream being joined to.

Answer

State is not maintained; the join is stateless and relies on the current micro-batch.

Answer

State is maintained only for the left stream, as it is the driving stream.

Question 11

A developer is building a Structured Streaming pipeline that reads from a Kafka topic and writes to a Delta table. The pipeline must tolerate occasional downstream failures and reprocess data without duplicates. The developer sets a checkpoint location and uses the default output mode. Which statement correctly describes how the checkpoint location contributes to fault tolerance in this scenario?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The late events are dropped and not included in the output.. When a watermark is set on an event-time column, Structured Streaming uses it to determine when data is too late to be included in an aggregation. Events with timestamps older than the watermark are dropped. This behavior is intentional to bound state and ensure timely results. The other options describe mechanisms that are not part of the watermarking feature, such as buffering, erroring, or automatic dead-lettering.

Answer

The late events are buffered and processed once the watermark advances.

Answer

The late events cause the query to fail with a watermark violation error.

Answer

The late events are written to a separate dead-letter queue automatically.

Question 12

You are monitoring a Structured Streaming query in Databricks and want to see the current status, including the number of input rows per second and the batch duration. Which of the following is the most direct way to access this information?

Accepted Answer

Use the `StreamingQuery.lastProgress` method on the query object.. The `StreamingQuery.lastProgress` method provides a structured snapshot of the most recent micro-batch's metrics, including input rate and batch duration. It is the direct API for monitoring. Log files are not structured, there is no SQL command for showing streaming queries, and Delta logs do not contain streaming metrics. Thus, `lastProgress` is the correct choice.

Answer

Query the `spark.streaming` log file on the driver node.

Answer

Run `spark.sql("SHOW STREAMING QUERIES")`.

Answer

Check the Delta table's transaction log for streaming metrics.

Question 13

A developer wants to start a Structured Streaming query that reads from a Kafka topic and writes to the console for debugging. The developer uses `writeStream.format("console").start()`. What is the default trigger for this query?

Accepted Answer

Default trigger, which processes data in micro-batches as soon as the previous batch completes, without a fixed interval.. The default trigger in Structured Streaming processes data in micro-batches as soon as the previous batch completes, without a fixed interval. This is the behavior when no trigger is explicitly set. It provides continuous processing with low latency, making it suitable for most streaming use cases. Developers can override this by specifying a processing time, once, or continuous trigger.

Answer

Continuous trigger with a 1-second checkpoint interval, which provides low-latency processing.

Answer

Once trigger, which processes all available data in a single batch and then stops.

Answer

ProcessingTime trigger with an interval of 0 seconds, which processes data as fast as possible.

Question 14

You have a Structured Streaming job that reads from a Kafka topic and writes to a Delta table. You need to ensure that the job processes each record exactly once, even after failures. Which of the following should you configure?

Accepted Answer

Set the `checkpointLocation` option to a reliable storage location.. Exactly-once processing in Structured Streaming relies on checkpointing to record progress and idempotent sinks to avoid duplicates. The checkpoint location stores offset information, allowing the query to resume without reprocessing. Delta Lake supports idempotent writes when used with checkpoints. Other options like Change Data Feed or manual deduplication do not provide the necessary fault tolerance.

Answer

Configure the Kafka source with `startingOffsets` set to `earliest`.

Answer

Enable idempotent writes by setting the Delta table property `delta.enableChangeDataFeed` to true.

Answer

Use `foreachBatch` to manually deduplicate records based on a unique key.

Question 15

You are building a Structured Streaming pipeline that reads from a Delta table source and applies a stateful deduplication using dropDuplicates on a composite key. After several hours, the job fails with an error indicating that the state store has grown too large. You need to bound the state size while still removing duplicate events that arrive within a reasonable window. Which approach should you take?

Accepted Answer

Add withWatermark on the event-time column and use dropDuplicatesWithinWatermark on the key columns.. Bounding state requires a rule that tells the engine when a key will no longer be needed. A watermark on event time provides that rule, and dropDuplicatesWithinWatermark uses it to evict keys older than the watermark. This keeps deduplication correct within the allowed lateness while preventing state from growing without limit. Configuration tweaks and repartitioning do not change the fundamental retention policy.

Answer

Set the checkpoint location to a new path and restart the query so the old state is discarded.

Answer

Increase the spark.sql.streaming.stateStore.maintenanceInterval to force more frequent cleanup of old keys.

Answer

Repartition the stream by the composite key before calling dropDuplicates to spread state across more partitions.