DP-900 Describe an analytics workload on Azure Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✓

  Use Structured Streaming with checkpointing to Azure Data Lake Storage Gen2.

  Why this is correct

  Structured Streaming checkpoints must be written to a durable, highly available store that preserves offsets and operator state so a query can resume exactly where it stopped after a failure. Azure Data Lake Storage Gen2 provides a hierarchical namespace, POSIX permissions, and strong consistency, making it an ideal checkpoint location for production Azure Databricks jobs. Writing checkpoints to ADLS Gen2 also enables exactly-once processing semantics, because the checkpoint includes committed offsets and transaction log metadata, allowing the streaming engine to recover without data loss or duplication.

• ✗

  Enable write-ahead logs on the Event Hubs namespace.

  Why it's wrong here

  Enabling write-ahead logs on the Event Hubs namespace is a misdirection: Event Hubs already uses write-ahead logging internally to durably persist ingested events, but that is completely separate from Spark Structured Streaming's checkpointing mechanism. Checkpointing in Azure Databricks is configured by specifying a checkpoint path, and it is the streaming query itself—not the event source—that writes offsets and state to that location. Toggling a WAL flag on Event Hubs does not create a checkpoint store, nor does it give you control over Spark's recovery metadata, so it cannot provide fault tolerance for the Databricks transformation.

• ✗

  Use checkpointing to Hive metastore.

  Why it's wrong here

  Checkpointing requires writing binary state files, such as offset ranges, group IDs, and the state store data, to a file system or blob storage that supports append and overwrite operations. The Hive metastore, by contrast, is a relational catalog used to store table schemas, partitions, and locations; it is not a general-purpose storage service and does not expose an API for writing checkpoint blobs. Even if you attempted to store checkpoint data as Hive table metadata, it would not be recognized by Structured Streaming's recovery logic, which expects a Hadoop-compatible filesystem path, making this option invalid by design.

• ✗

  Use checkpointing to DBFS (Databricks File System).

  Why it's wrong here

  The Databricks File System (DBFS) is a managed file system mounted on top of the cloud object store, but it can be ephemeral because it is tied to the Databricks workspace lifecycle and is often scoped to a cluster's temporary storage. In production, checkpoints stored on DBFS are at risk if the workspace is deleted or if cluster nodes are recycled, and DBFS may not provide the same performance or consistency guarantees as a dedicated external store. Furthermore, because DBFS can be mapped to underlying cloud storage that may be temporary or local, it is explicitly not recommended for checkpointing; you should point the checkpoint location to a persistent, external storage service like ADLS Gen2.

Go deeper

Related to this question