Databricks-Spark-Assoc Structured Streaming — 30 Questions

Question 1

You are developing a Structured Streaming job in Databricks that reads JSON data from an Auto Loader source, transforms the schema, and writes the output to a Delta Lake table. During execution, downstream consumers complain that the data contains duplicates due to upstream retries. Which operation should you apply to the DataFrame to ensure exactly-once processing semantics before writing to the Delta table?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure a watermark on the event-time column and apply dropDuplicatesWithinWatermark() using the unique identifier columns.. Applying a watermark combined with dropDuplicatesWithinWatermark ensures duplicate records arriving within the specified time window are removed while maintaining bounded state. This pattern is critical for handling upstream retries in stream-to-stream or stream-to-table pipelines without causing unbounded memory consumption in stateful streaming operations.

Answer

Apply the distinct() transformation on the entire streaming DataFrame prior to writing the output to the Delta table.

Answer

Use the dropDuplicates() method with a set of unique identifier columns without specifying a watermark.

Answer

Enable Delta Lake change data feed and configure the sink to automatically deduplicate incoming records at the storage layer.

Question 2

A data engineer configures a Structured Streaming job to read from an Apache Kafka source and writes the incoming data directly to Delta Lake using the append output mode. During testing, the job throws a AnalysisException stating that streaming queries with aggregation and output mode update require a watermark. Which action resolves this issue correctly?

Accepted Answer

Add a .withWatermark() clause specifying an event-time column and threshold before performing the aggregation.. Stateful operations like aggregations require watermarking to track event time and clean up old state from memory. Without a defined watermark, Spark cannot determine when it is safe to drop stale aggregation groups, leading to planning or runtime failures. Adding a watermark ensures bounded memory usage and correct late-data handling during incremental streaming processing on Delta Lake.

Answer

Change the output mode from append to complete mode while retaining the current aggregation logic.

Answer

Increase the spark.sql.shuffle.partitions configuration property to a larger value to handle the streaming state distribution.

Answer

Switch the streaming sink format from delta to memory to execute local iterative calculations without external storage constraints.

Question 3

A streaming pipeline reads from Delta Lake using `spark.readStream.table("events")` and performs stateful aggregation using `groupBy("userId").count()`. The engineering team notices that the shuffle partitions default to 200, which causes excessive task overhead and latency. Which configuration setting should be applied to optimize the shuffle partition count specifically for this Structured Streaming query?

Accepted Answer

spark.sql.shuffle.partitions. Setting the shuffle partitions property directly impacts performance by controlling parallelism during aggregations and joins in Spark. Because streaming queries can run continuously, configuring this property via Spark Session configuration ensures optimal resource utilization without modifying downstream physical execution plans.

Answer

spark.databricks.delta.optimizeWrite.enabled

Answer

spark.streaming.concurrentJobs

Answer

spark.databricks.streaming.sql.maxFilesPerTrigger

Question 4

A data engineer is building a Structured Streaming pipeline that reads from an Apache Kafka source and writes the incoming data directly to Delta Lake. The pipeline must execute multiple streaming aggregates across event time, but downstream operational reporting requires complete updates to be written out for every micro-batch. Which output mode should be configured to meet this requirement?

Accepted Answer

Complete mode. Complete mode rewrites the entire result table to the storage sink during every trigger interval, making it necessary for aggregation queries where every historical aggregate needs to be visible. While Append mode only outputs newly appended rows, Complete mode ensures that updated calculations for previously observed windows are fully materialized in the target Delta table.

Answer

Update mode

Answer

Append mode

Answer

Overwrite mode

Question 5

A data engineer writes a Structured Streaming query that reads files from an AWS S3 bucket directory and applies transformations before writing to a Delta table. The source files are continuously dropped into the folder by an upstream ingestion process. However, the engineer notices that newly added files are completely ignored by the stream. What is the most likely cause of this behavior?

Accepted Answer

The schema of the input files was not explicitly defined or schema inference was not properly enabled for the file source.. When using file-based streaming sources like spark.readStream.format("cloudFiles"), Spark requires schema inference and tracking metadata unless explicitly bypassed, or it may fail if the file format option is incorrect. Alternatively, standard file streams require specifying a schema because automatic schema inference is disabled by default for file sources to prevent accidental job failures.

Answer

The query was started using spark.read() instead of spark.readStream() for the file path.

Answer

The trigger interval was set to process data once using .trigger(Once=True) instead of a continuous time duration.

Answer

The output mode was set to Append instead of Complete when reading file streams.

Question 6

You are developing a Structured Streaming job in Databricks that reads JSON data from an Auto Loader source, transforms the schema, and writes the output to a Delta Lake table. During execution, downstream consumers complain that the data contains duplicates due to upstream retries. Which operation should you apply to the DataFrame to ensure exactly-once processing semantics before writing to the Delta table?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Pass the option 'mergeSchema' with a value of 'true' to the Delta write stream configuration using .option("mergeSchema", "true").. Enabling the mergeSchema option on the Delta writer dynamically updates the target table structure when new columns are introduced in the incoming stream. This prevents pipeline crashes due to schema drift, ensuring continuous ingestion from dynamic upstream sources like Kafka.

Answer

Restart the streaming query after running an ALTER TABLE command manually to add the new nested field to the Delta table schema.

Answer

Configure the Kafka source reader option 'spark.sql.streaming.schemaInference' to true to automatically update downstream consumers.

Answer

Cast the incoming JSON payload into a predefined StringType column and store it raw, bypassing the schema validation checks.

Question 7

A data engineer is building a Structured Streaming pipeline that reads from an Apache Kafka source and writes the output in append mode to Delta Lake. The pipeline fails during execution due to an upstream data corruption issue, and the team needs to restart it from a specific historical offset without losing stateful aggregations. Which approach should the engineer use to resume processing correctly?

Accepted Answer

Maintain the existing checkpoint location and restart the streaming query to automatically resume from the last committed offset.. Resuming a Structured Streaming pipeline from a specific point while maintaining stateful consistency requires leveraging the checkpoint directory. Spark stores all state data, source offsets, and progress logs in the designated checkpoint location. By deleting corrupted micro-batch logs if necessary or preserving the valid checkpoint history, the query automatically recovers and resumes from the exact offset tracked in the metadata.

Answer

Delete the entire checkpoint directory and restart the streaming query with a new starting offset configuration.

Answer

Modify the Kafka source options to use startingOffsets='earliest' while pointing to the existing checkpoint location.

Answer

Switch the output mode from append to complete mode to force the pipeline to reprocess all historical data from scratch.

Question 8

When reading from a file source in Structured Streaming, which configuration setting must be explicitly enabled if you want the streaming query to process existing files in the directory before streaming new ones?

Accepted Answer

spark.readStream.option('includeExistingFiles', 'true'). By default, Structured Streaming processes only new files added to a directory after the stream starts. To ingest existing files, the 'ignoreChanges' or 'ignoreFileDeletions' options are insufficient; you must use the specific file source option that changes the behavior to include the current state of the directory. This is critical for backfilling data or starting a stream on an existing dataset without manually moving files.

Answer

spark.readStream.option('processAll', 'true')

Answer

spark.readStream.option('initialBatch', 'all')

Answer

spark.readStream.option('maxFilesPerTrigger', -1)

Question 9

Which TWO of the following are valid output modes in Structured Streaming that support aggregation operations?

Accepted Answer

Complete. Output modes define how the results of a streaming transformation are written to the sink. Aggregations, such as count or sum, require modes that can handle updating existing records or appending new finalized windows. Understanding these modes is critical for downstream systems, as some modes are incompatible with specific operations or output sinks, potentially leading to runtime errors during query execution.

Answer

Append

Answer

Overwrite

Answer

Merge

Question 10

Refer to the exhibit. You are running a streaming query on Databricks. You need to update the trigger to ensure the query processes data as soon as it becomes available, rather than waiting for a 1-minute interval. What should you modify in the configuration?

Accepted Answer

Remove the trigger parameter entirely.. The trigger configuration dictates the timing of micro-batch execution. The exhibit specifies 'ProcessingTime('1 minute')', which enforces a fixed interval. To achieve 'as soon as possible' processing, the 'AvailableNow' or 'Once' trigger (for batch-like behavior) or simply leaving the trigger unspecified (for continuous micro-batching) is required. This is a common performance optimization when low latency is a requirement for real-time dashboards or alert systems.

Answer

Change trigger to 'AvailableNow(true)'.

Answer

Set trigger to 'None'.

Answer

Set trigger to 'ProcessingTime('0 seconds')'.

Question 11

You are performing a stream-stream join between two dataframes, `orders` and `clicks`. Both streams have watermarks defined. What happens if the join condition does not include a time range constraint?

Accepted Answer

The query will fail with an AnalysisException.. Stream-stream joins in Spark require time constraints or watermarks to prevent the state store from growing indefinitely. Without a time constraint, the engine would need to buffer all past data from both sides of the join to match incoming records, which is computationally expensive and memory-intensive. Spark strictly enforces these constraints to ensure that state can be purged, preventing system-wide crashes in production environments.

Answer

The query will execute but results will be delayed by 24 hours.

Answer

The join will automatically default to an inner join with no state limit.

Answer

Only the inner join is supported in stream-stream joins.

Question 12

Which THREE of the following are true about the 'foreachBatch' sink in Structured Streaming?

Accepted Answer

It allows writing to multiple locations.. The foreachBatch sink provides significant flexibility by allowing arbitrary operations on every micro-batch as a DataFrame. This is crucial for integrating with systems that don't have native streaming connectors or for performing complex operations like multi-sink writes or custom error handling. Mastering foreachBatch is essential for advanced data engineering tasks where standard output modes prove insufficient for unique business logic requirements.

Answer

It is only available in Python.

Answer

It ignores the checkpointing mechanism.

Question 13

What happens if a streaming source produces data with a schema that differs from the one initially defined in the schema inference phase?

Accepted Answer

The query fails at runtime.. Structured Streaming requires schema consistency for most sources. If the incoming data structure deviates from the defined schema, the stream will encounter errors or produce nulls for the mismatched fields. This is a deliberate design choice to ensure data quality. Handling schema evolution requires specific configurations or using schema evolution features in Delta Lake to prevent pipeline failure during production ingestion.

Answer

Spark automatically updates the sink schema.

Answer

Spark ignores the extra fields and processes the rest.

Answer

The stream automatically restarts with the new schema.

Question 14

Which TWO of the following are valid ways to monitor a running Structured Streaming query on Databricks?

Accepted Answer

Checking the Spark UI's Streaming tab.. Monitoring is critical to ensuring streaming performance. Developers should use built-in tools like the Streaming Query UI or query listeners to keep an eye on metrics like input rates, batch duration, and state size. These tools provide real-time insights, allowing operators to diagnose bottlenecks and optimize resources for long-running streaming applications in a production environment.

Answer

Querying the system log files via SSH.

Answer

Manually counting rows in the sink table.

Answer

Restarting the cluster every hour.

Question 15

You are processing a streaming dataset with a watermark of 10 minutes. A late record arrives with an event time of 12:00, while the current watermark is 12:15. What happens to this record?

Accepted Answer

The record is dropped because it is behind the watermark.. Structured Streaming uses watermarks to manage state and handle late-arriving data. Records with an event time earlier than the current watermark minus the delay threshold are dropped to prevent unbounded state growth. Since 12:00 is less than 12:05 (the derived threshold), the record is considered too late and is discarded. Understanding this is crucial for managing data accuracy and state cleanup in production pipelines.

Answer

The record is processed and stored in the state store.

Answer

The record is sent to the dead-letter queue.

Answer

The record triggers an immediate watermark update.

Question 16

Which TWO statements are true regarding the use of 'foreachBatch' in Structured Streaming?

Accepted Answer

It can be used to write data to multiple destinations in one batch.. foreachBatch is a powerful tool allowing developers to use batch-processing capabilities on streaming dataframes. It enables writing to sinks that lack native streaming support and allows for complex transformations that are not possible within the standard streaming API. Understanding how to integrate batch logic into streaming pipelines is critical for handling legacy systems and advanced data cleaning operations in Databricks environments.

Answer

It supports stateful streaming operations.

Answer

It automatically handles checkpointing for all logic.

Answer

It requires the query to be in Complete mode.

Question 17

Refer to the exhibit. You are configuring a streaming job. Based on the provided configuration, what is the impact of the 'checkpointLocation' setting?

Accepted Answer

It prevents the data from being processed multiple times.. The checkpointLocation is vital for maintaining the streaming query's progress. It stores metadata, offsets, and incremental state information. If the query stops, the checkpoint location allows the job to resume from the last offset. Without this setting, the query would lose its state upon failure, preventing the possibility of exactly-once delivery and making production-grade streaming pipelines impossible to maintain consistently.

Answer

It specifies where the actual output data is saved.

Answer

It defines the frequency of data writes to the sink.

Answer

It forces the query to use the Complete output mode.

Question 18

A data engineer is using `foreachBatch` in a Structured Streaming job to write to multiple sinks, including a Delta table and an external database. The engineer needs to ensure the operation is idempotent and handles reprocessing correctly. Which TWO statements about `foreachBatch` are correct in this context? (Choose two.)

Accepted Answer

`foreachBatch` can be used to write to multiple sinks, and each sink write can be made idempotent using the batch ID.. `foreachBatch` enables batch operations on each micro-batch and is useful for writing to multiple sinks. It provides a batch ID that can be used to achieve idempotent writes, which is essential for exactly-once semantics when sinks are not transactional. It does not automatically guarantee exactly-once, nor does it support continuous processing mode.

Answer

The function passed to `foreachBatch` is executed on the driver, so it can access driver-local variables and broadcast them.

Answer

`foreachBatch` supports continuous processing mode, allowing sub-millisecond latency.

Answer

`foreachBatch` guarantees exactly-once processing when writing to an external database without additional idempotent logic.

Question 19

A developer is testing a Structured Streaming query that reads from a rate source and writes to the console. The developer wants the query to process all available data and then stop automatically. Which trigger type should be used to achieve this behavior?

Accepted Answer

`trigger(once=True)`. The `once` trigger processes all available data in a single micro-batch and then stops the query. This is useful for testing or batch-like execution. The `availableNow` trigger also stops but processes in multiple batches. The question asks for processing all available data and then stopping; `once` is the direct answer for a single batch. The other triggers run continuously.

Answer

`trigger(continuous='1 second')`

Answer

`trigger(availableNow=True)`

Answer

`trigger(processingTime='10 seconds')`

Question 20

You are building a Structured Streaming pipeline in Databricks that reads from a Delta table using `spark.readStream.table("transactions")`. The pipeline performs a `groupBy("accountId").count()` aggregation and writes to another Delta table. You need the streaming query to emit updated counts for all accounts as new data arrives, even if an account has no new transactions in the current micro-batch. Which output mode should you use?

Accepted Answer

Update mode. The requirement is to emit updated counts for accounts as new data arrives, without re-emitting the entire result table. Update mode is designed for this: it outputs only the rows that changed in the current micro-batch. Append mode would only emit rows after they are finalized by a watermark, and complete mode would re-emit all rows every batch. Delta mode is not a valid output mode. Thus, update mode is the correct choice.

Answer

Append mode

Answer

Complete mode

Answer

Delta mode