A data analyst is using Azure Databricks to transform streaming data from Event Hubs. They need to ensure that if a failure occurs, the streaming job can resume processing from the last committed offset. Which checkpointing mechanism should they configure?
Structured Streaming checkpoints must be written to a durable, highly available store that preserves offsets and operator state so a query can resume exactly where it stopped after a failure. Azure Data Lake Storage Gen2 provides a hierarchical namespace, POSIX permissions, and strong consistency, making it an ideal checkpoint location for production Azure Databricks jobs. Writing checkpoints to ADLS Gen2 also enables exactly-once processing semantics, because the checkpoint includes committed offsets and transaction log metadata, allowing the streaming engine to recover without data loss or duplication.
Why this answer
Structured Streaming in Azure Databricks uses checkpointing to store the current state and offsets of the streaming query in a reliable, external storage system. By configuring checkpointing to Azure Data Lake Storage Gen2, the job can recover from failures and resume processing exactly from the last committed offset, ensuring exactly-once semantics. This is the recommended approach for production streaming workloads on Azure.
Exam trap
The trap here is that candidates often confuse DBFS with persistent storage, but DBFS is cluster-scoped and ephemeral, so checkpointing to DBFS will lose state when the cluster stops, whereas ADLS Gen2 provides durable, external checkpoint storage.
How to eliminate wrong answers
Option B is wrong because write-ahead logs on the Event Hubs namespace are used for Event Hubs internal durability and replication, not for checkpointing the consumer's offset state in Databricks. Option C is wrong because the Hive metastore is designed for storing table metadata and schema information, not for storing streaming checkpoint data or offsets. Option D is wrong because DBFS is a local ephemeral storage tied to the cluster; if the cluster is terminated or fails, checkpoint data stored in DBFS is lost, making it unsuitable for reliable failure recovery.