DP-900 Describe an analytics workload on Azure Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Add a separate job to deduplicate the Delta table.

  Why it's wrong here

  Running a separate batch job to deduplicate the Delta table adds significant latency because it must scan and rewrite large volumes of data after the fact. It also requires you to define and maintain a separate deduplication state, which is complex and error-prone in a streaming pipeline. Most importantly, it cannot prevent duplicate data from being committed in the first place, so any downstream consumers that read the table before the cleanup job runs will still see duplicates. This approach is a post-processing workaround, not a reliable streaming correctness mechanism.

• ✓

  Enable checkpointing in the streaming query to store progress.

  Why this is correct

  Checkpointing in the streaming query is the correct mechanism because Structured Streaming uses a checkpoint location to persist the committed offsets and the current state of the query. When a failure occurs, the query restarts from the last committed offset, ensuring that no data is reprocessed and no data is lost—this is what enables exactly-once processing semantics. Without checkpointing, the query has no record of what has already been consumed, so it may reprocess the same input data after a restart, causing duplicates in the Delta table. In Azure Databricks, checkpointing is configured via the `checkpointLocation` option and is essential for reliable streaming.

• ✗

  Use Delta Lake's idempotent write support.

  Why it's wrong here

  Delta Lake's idempotent write support, such as using a `unique_key` or idempotent transaction IDs, does help avoid duplicate records when a write operation is retried, but it is not a substitute for checkpointing. Idempotent writes only guard against duplicate writes of the same micro-batch; they do not track source offsets or provide recovery knowledge about which data has already been read. If the streaming source is re-read due to lack of checkpointing, the entire micro-batch—including new and old records—is reprocessed, and idempotent writes may not recognize all records as seen if no deduplication key is defined. Therefore, idempotent writes are a useful complement to checkpointing, but they do not address the root cause of duplicate reads.

• ✗

  Increase the batch interval in the streaming query.

  Why it's wrong here

  Increasing the batch interval in the streaming query reduces the frequency of micro-batches, but it does not eliminate duplicates because the root cause is the absence of checkpointing, not the batch size. A longer interval means less frequent writes, but if a failure occurs between reading from the source and committing the results, the same data can still be reprocessed in the next batch. The streaming engine relies on offsets and checkpoint metadata to know exactly what has been processed; merely changing the timing of micro-batches does not provide that metadata. In fact, longer batch intervals can increase latency and memory pressure while leaving exactly-once guarantees unaddressed.

Go deeper

Related to this question