Databricks-Spark-Assoc Structured Streaming Practice Test 3 — 15 Questions

Question 1

A developer is using `foreachBatch` in a Structured Streaming job to write each micro-batch to both a Delta table and an external JDBC database. The developer wants to ensure that the writes are idempotent and that the job can recover from failures without duplicating data. Which two actions should the developer take? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Enable checkpointing by setting a checkpoint location in the write stream.. To achieve idempotent writes with `foreachBatch`, you need to use the unique batch ID to deduplicate writes to external systems, and you must enable checkpointing to ensure the job can recover from failures without reprocessing data. These two actions together provide exactly-once semantics. The other options either reference non-existent configurations or propose inefficient or incorrect methods.

Answer

Use `foreachBatch` with a custom transaction log to track processed batch IDs.

Answer

Set the `spark.sql.streaming.foreachBatch.retry` configuration to true.

Answer

Configure the JDBC sink to use `mode("overwrite")` for each batch.

Question 2

A data engineer is running a Structured Streaming query that reads from a file source and writes to a Delta table. The query uses a checkpoint location. After a failure, the query restarts and reprocesses some data, but the engineer notices that the Delta table contains duplicate records. Which factor is most likely causing the duplicates?

Accepted Answer

The Delta table sink does not support exactly-once writes, and the query is not idempotent.. Duplicates after a restart in Structured Streaming are most commonly caused by non-idempotent sinks or improperly configured checkpoints. Delta Lake as a sink provides exactly-once guarantees when used with a reliable checkpoint, but if the sink is not transactional or the checkpoint is lost, duplicates can occur. The engineer should ensure that the checkpoint is on durable storage and that the sink supports idempotent writes.

Answer

The source file is being appended to while the query is running, causing the query to read the same file multiple times.

Answer

The query uses a watermark that is too large, causing late data to be processed multiple times.

Answer

The checkpoint location is on a local file system that is not durable across restarts.

Question 3

You are running a Structured Streaming query that performs a stateful operation with a watermark of 5 minutes. You notice that the state store is growing indefinitely, causing memory issues. Which of the following is the most likely cause?

Accepted Answer

The watermark is not being applied because the event time column is not properly specified.. Watermarks are essential for bounding state in stateful streaming operations. If the event time column is not correctly defined, the watermark cannot advance, and old state is never evicted. This results in unbounded state growth. Other factors like output mode or checkpoint performance do not prevent state cleanup when a watermark is properly applied.

Answer

The checkpoint location is on a slow storage system, causing state updates to lag.

Answer

The query uses `foreachBatch` to write to an external system, which bypasses state management.

Answer

The output mode is set to `complete`, which retains all state forever.

Question 4

A developer is building a Structured Streaming job that reads from a Delta table using `spark.readStream.table("source")`. The source Delta table is being updated with new data every 5 minutes. The developer wants the streaming job to process only the newly appended data in each micro-batch, without reprocessing old data. Which configuration is required?

Accepted Answer

Ensure a checkpoint location is set for the write stream.. To process only new data in each micro-batch when reading from a Delta table as a stream, the query must have a checkpoint location. The checkpoint stores the progress of the stream, including which files have been processed, so that subsequent micro-batches only include new data. Options like `ignoreDeletes`, `startingVersion`, and `maxFilesPerTrigger` do not fulfill this requirement; they control other aspects of the stream.

Answer

Set the option `startingVersion` to the latest version.

Answer

Set the option `ignoreDeletes` to true.

Answer

Set the option `maxFilesPerTrigger` to 1.

Question 5

You are monitoring a Structured Streaming query on Databricks and notice that the `numInputRows` metric is consistently zero, even though the source Kafka topic has new data. Which of the following is the most likely cause?

Accepted Answer

The Kafka source is configured with `startingOffsets` set to `latest`, and the query has already consumed all available data.. The most likely cause is that the query is not actually reading new data because it has already completed its processing. This can happen if the query uses a trigger that processes data only once, such as Trigger.Once, and has already finished. After completion, it will not process new data, leading to zero input rows. Other options do not directly explain why a running query would not read new data.

Answer

The query is using a trigger that only processes data once, such as `Trigger.Once`, and has already completed.

Answer

The query is using the `complete` output mode, which does not process new data.

Answer

The watermark is set too high, causing all data to be considered late.

Question 6

An analytics engineer is designing a Structured Streaming pipeline that reads from an Apache Kafka topic and writes JSON-formatted output to cloud object storage. The pipeline must maintain exactly-once processing semantics and support automatic recovery from cluster restarts. Which TWO actions are mandatory to achieve these requirements? (Choose 2)

Accepted Answer

Configure a valid checkpoint location using the .option("checkpointLocation", path) method.. Achieving end-to-end exactly-once guarantees in Spark Structured Streaming requires idempotent or transactional sinks combined with persistent checkpoint directories. The checkpoint mechanism saves the exact state and offsets, allowing the streaming query to resume seamlessly after an interruption without duplicating data processing.

Answer

Enable Delta Lake format as the sink to leverage ACID transactions and transactional metadata.

Answer

Set the output mode of the streaming query to complete mode.

Answer

Increase the driver memory allocation to at least 64GB to store all Kafka offset metadata.

Question 7

You are processing a streaming dataset of sensor readings. You need to calculate the average temperature every 10 minutes, allowing data to arrive up to 2 minutes late. Which windowing approach correctly handles this requirement in Structured Streaming?

Accepted Answer

Use window(timestamp, '10 minutes') with withWatermark('timestamp', '2 minutes').. Event-time windowing with watermarks is essential for handling late-arriving data in stream processing. By defining a window duration of 10 minutes and a watermark delay of 2 minutes, Spark maintains state for late events while discarding data older than the watermark threshold. This ensures the output remains accurate even when network latency or ingestion bottlenecks occur, preventing unbounded state growth in the processing engine.

Answer

Use window(timestamp, '10 minutes') without a watermark.

Answer

Use window(timestamp, '10 minutes') with withWatermark('timestamp', '10 minutes').

Answer

Use a trigger interval of 2 minutes with no watermark.

Question 8

Which of the following describes the function of the checkpoint location in a Structured Streaming query?

Accepted Answer

It keeps track of the offsets for fault tolerance.. Checkpoints are the backbone of fault tolerance in Structured Streaming. They store the query's progress, including offsets and metadata, in reliable storage (like DBFS or S3). This allows the query to recover from failures and resume exactly where it left off. Without checkpoints, any crash would cause the loss of processed offset information, forcing the system to reprocess all data from the beginning, which is inefficient.

Answer

It stores the physical data ingested by the stream.

Answer

It manages the Spark UI logs for performance monitoring.

Answer

It serves as a cache for shuffle operations.

Question 9

You are processing JSON data from a stream. You need to extract nested fields from the JSON structure. Which function is the most efficient and standard way to handle this in Structured Streaming?

Accepted Answer

Use the from_json function with a defined schema.. When dealing with JSON in streaming, using schema enforcement and the `from_json` function is the recommended practice. It allows you to transform the raw JSON string column into a structured format with clear types. This approach is highly efficient because it leverages Spark's Catalyst optimizer to process the nested fields, which is far superior to manual string parsing or complex regex operations.

Answer

Use a UDF to parse the JSON string.

Answer

Convert the JSON to a Map and extract keys.

Answer

Use the split function to break the JSON string.

Question 10

What is the primary difference between a micro-batch streaming query and a continuous processing query in Spark?

Accepted Answer

Continuous processing offers lower latency.. Micro-batch processing handles data in small discrete batches, providing high throughput and fault tolerance with second-level latency. Continuous processing, by contrast, runs a task continuously on each executor, enabling sub-millisecond latency. Choosing between them involves a trade-off between strict latency requirements and the operational complexity or feature limitations inherent in the continuous processing model, which does not support all operations yet.

Answer

Continuous processing supports all SQL operations.

Answer

Micro-batch processing provides lower latency.

Answer

Continuous processing is the default mode.

Question 11

Which state management strategy should you implement if you notice your streaming aggregation query is failing due to excessive memory consumption on the executor nodes?

Accepted Answer

Implement or tighten the watermark duration.. Aggregations in streaming inherently require state management. If memory is exhausted, it is often due to the state store growing too large because of late data or a lack of proper watermark cleanup. Implementing a stricter watermark and periodically cleaning the state using stateful operators like 'dropDuplicates' or correctly configured windowing is necessary. This ensures the cluster stays within its memory limits during long-term operation.

Answer

Increase the checkpoint interval.

Answer

Disable checkpointing to free memory.

Answer

Use a larger instance type for all nodes.

Question 12

Which component in Structured Streaming is responsible for providing fault tolerance and ensuring data is processed exactly once?

Accepted Answer

Checkpointing. Checkpointing is the mechanism that stores the query's metadata and progress in durable storage (like DBFS or S3). By recording the offset of the processed data, Spark can recover from failures and restart from the exact point where it left off. This is fundamental to ensuring that streaming applications are reliable and maintain exactly-once processing guarantees across restarts or cluster crashes.

Answer

The State Store

Answer

The Write Ahead Log

Answer

The Driver

Question 13

A streaming job using 'mapGroupsWithState' is failing due to excessive memory usage. Which strategy is most effective for mitigating this?

Accepted Answer

Implementing state timeouts.. The 'mapGroupsWithState' function maintains state for keys until they are explicitly timed out or removed. If keys are not removed, the state grows indefinitely, leading to memory issues. Using 'GroupStateTimeout' (either processing or event-time) allows the engine to automatically expire state after a specific duration, effectively bounding memory usage. This is a critical pattern for managing state in long-running streaming applications on Databricks.

Answer

Increasing the number of partitions.

Answer

Reducing the trigger interval.

Answer

Switching to Append mode.

Question 14

Which THREE of the following sources support streaming read operations in Spark Structured Streaming?

Accepted Answer

Delta Lake tables. Structured Streaming requires data sources to provide a way to track offsets. Delta Lake, Kafka, and File-based sources (like Parquet or JSON) are natively supported because they provide the necessary metadata for Spark to determine what data has been read and what is new. Understanding these sources is essential for designing robust streaming architectures that integrate seamlessly with the Databricks ecosystem.

Answer

Standard SQL Databases (JDBC)

Answer

In-memory lists

Question 15

An engineer is developing a Structured Streaming job that reads from an Apache Kafka source and writes the output continuously toDelta Lake using outputMode("append"). The stream occasionally experiences late-arriving data. Which downstream behavior can the engineer expect regarding the Delta Lake table?

Accepted Answer

Late-arriving data that falls behind the specified watermark threshold is dropped and never written to the Delta table.. Streaming queries using append mode require that new rows are entirely independent of previously processed outputs, meaning they are simply appended as new files. Late data arriving after the watermark threshold is dropped entirely by Spark and never written to the Delta table, preventing unbounded state growth and maintaining strict correctness guarantees.

Answer

Late-arriving data arriving after the watermark is automatically updated in-place within the existing Delta files using ACID merge operations.

Answer

Late-arriving data is buffered in an unlimited memory state store until the streaming query is manually stopped and restarted by an operator.

Answer

Late-arriving data forces the Delta Lake table to automatically switch its output mode to complete mode for that specific micro-batch.