DP-203 Design and implement data storage Practice Question
A data engineering team is designing a batch processing pipeline that reads from Azure Data Lake Storage Gen2, transforms data using Azure Databricks, and writes to Azure Synapse Analytics. The pipeline must process data incrementally and handle late-arriving data up to 2 hours. Which approach should they use to track processed files?
⚠ Common exam trap
Candidates often choose Azure Data Factory with watermark columns (Option C) because it is a common incremental load pattern, but they overlook that watermark columns apply to row-based sources with change tracking, not to file-based sources where the challenge is tracking which files have been processed.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Store processed file names in a Delta table and compare with source folder listing
Storing processed file names in a Delta table allows the pipeline to track which files have already been ingested, supporting incremental processing and handling late-arriving data up to 2 hours. By comparing the current source folder listing against the Delta table, the pipeline can identify only new or late-arriving files, avoiding reprocessing and ensuring exactly-once semantics. This approach integrates seamlessly with Azure Databricks and Delta Lake's ACID transactions, providing reliable state management for batch pipelines.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use Blob Storage event triggers to invoke Azure Functions
Why it's wrong here
Blob event triggers fire per file creation and give no durable record of which files a batch run already consumed, so late-arriving data within the two-hour window is missed or reprocessed. Event-driven Functions suit real-time reaction, not incremental batch tracking.
- ✗
Use Azure Synapse Pipelines with a schedule and full load each time
Why it's wrong here
A scheduled full load rereads and rewrites the entire dataset every run, providing no incremental mechanism and no way to reconcile files arriving up to two hours late. Scheduling suits periodic complete refreshes of small datasets, not incremental pipelines with late-data handling.
- ✗
Use Azure Data Factory with watermark columns in the source
Why it's wrong here
Watermark columns exist in relational sources with a monotonic timestamp or identity column; Data Lake Storage Gen2 files carry no such column, so the watermark cannot advance. This pattern is correct for incremental loads from SQL tables, not file-based landing zones.
- ✓
Store processed file names in a Delta table and compare with source folder listing
Why this is correct
Recording processed filenames in a Delta table gives transactional, queryable state that survives failures, and comparing it against the source listing identifies new files. This supports incremental processing while allowing late-arriving files within the 2-hour window to be picked up.
Visual reference
Go deeper
Related to this question
Learn chapter
Implement Data Lake Storage
Key term
Azure Synapse Analytics
Azure Synapse Analytics is a cloud-based data integration, warehousing, and analytics service that brings together big data and data warehouse capabilities under one platform.
Key term
Azure Databricks
Azure Databricks is a fast, easy, and collaborative Apache Spark-based analytics platform optimized for Azure that lets data teams prepare data, run machine learning models, and build data pipelines using a single workspace.
About these practice questions
This DP-203 question is part of Courseiva's 509-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This DP-203 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the DP-203 exam.