Develop data processing →hardMultiple Choice

DP-203 Develop data processing Practice Question

You have an Azure Databricks notebook that processes a large Delta table. The notebook uses a structured streaming query to read from the Delta table and write to another Delta table. The source table receives frequent updates and deletes. You need the streaming query to process both new data and changes (updates and deletes) from the source table. What should you do?

⚠ Common exam trap

The trap here is assuming that a standard Delta streaming read can capture updates and deletes, when it only processes new files unless Change Data Feed is enabled.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Enable Change Data Feed on the source Delta table and configure the streaming query to read the change data feed.

Delta Lake Change Data Feed (CDF) is the feature designed to capture row-level changes from a Delta table. When enabled, it records inserts, updates, and deletes, and a streaming query can read these changes as a stream. This allows downstream processing to react to all modifications, not just appends. Enabling CDF and reading the change data feed is the correct approach to meet the requirement.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Convert the source table to an external table and use `spark.readStream.format("delta").option("mergeSchema", "true")` to read all changes.
Why it's wrong here
`mergeSchema` is used to handle schema evolution when reading Delta tables, not to capture changes. Converting to an external table does not enable change tracking. This option would not provide updates or deletes to the streaming query. The streaming query would still only process new files, missing modifications and deletions entirely.
✗
Use `readStream` with the `startingVersion` option set to the latest version and `maxFilesPerTrigger` set to 1.
Why it's wrong here
Setting `startingVersion` to the latest version and limiting files per trigger controls which data files are read, but it does not provide change data. The streaming query would still only see new files appended to the table, not updates or deletes that rewrite existing files. This approach does not capture row-level changes and would miss modifications.
✗
Configure the streaming query to use `readStream` with the `ignoreDeletes` option set to true.
Why it's wrong here
The `ignoreDeletes` option allows a streaming query to continue when deletes occur in the source table by ignoring them, but it does not process updates or deletes. It is used to avoid failures when deletes happen, not to capture changes. The requirement is to process changes, so this option would not meet the need and would skip delete operations entirely.
✓
Enable Change Data Feed on the source Delta table and configure the streaming query to read the change data feed.
Why this is correct
Delta Lake Change Data Feed (CDF) records row-level changes (inserts, updates, deletes) in the source table. By enabling CDF and reading the change data feed in a streaming query, you can process all changes. This is the designed mechanism for capturing changes from a Delta table in a streaming fashion, and it supports both updates and deletes.

Go deeper

DP-203 Develop data processing Practice Question

Why each option matters

Option-by-option breakdown

Related to this question