Databricks-DE-Pro Practice Test 17 — 15 Questions

Question 1

A data engineer has set up a Databricks SQL alert on a query that returns the count of failed jobs in the last hour. The alert is configured to trigger when the count exceeds 5. The engineer wants to receive notifications via email and also wants to view the alert history to understand past triggers. Which statement accurately describes the alert notification and history capabilities?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Alerts can send notifications to email, Slack, and other destinations, and the alert history is available in the Databricks SQL UI.. Databricks SQL alerts are versatile: they can notify via email, Slack, PagerDuty, or webhooks, and they maintain a history of triggers that can be reviewed in the UI. This allows teams to audit past alert conditions and responses. The other options either limit notification channels or incorrectly describe history retention, making them inaccurate.

Answer

Alerts can only send notifications to email addresses and do not retain any history of triggers.

Answer

Alerts can only be configured to trigger once and then must be manually reset, and history is only available via the API.

Answer

Alerts can send notifications only to the alert owner's email, and history is stored for 7 days.

Question 2

A data engineering team maintains a Unity Catalog metastore in a Databricks workspace. They need to provide an external partner with read-only access to a specific Delta table, but the partner's analytics platform is not Databricks and does not support the Delta Lake protocol. The partner can consume Parquet files over a REST API. Which Unity Catalog feature should the team use to share the table?

Accepted Answer

Delta Sharing. Delta Sharing is the correct choice because it is the only Unity Catalog feature designed to share Delta tables externally using an open REST protocol. It supports recipients on non-Databricks platforms by serving data as Parquet, which aligns with the partner's capabilities. The other options serve different purposes: Volumes store files, Lakehouse Federation queries external systems, and the SQL Connector is a client library.

Answer

Unity Catalog Volumes

Answer

Databricks SQL Connector for Python

Answer

Lakehouse Federation

Question 3

A data engineer needs to ensure that a Databricks job can be retried automatically if it fails due to a transient cluster error. The job is configured with a maximum of 3 retries. After the third failure, the engineer wants to receive an email notification. Which feature should be used to accomplish this?

Accepted Answer

Configure the job's `email_notifications` with `on_failure` and set the email address.. Databricks jobs support built-in email notifications that can be triggered on failure. By configuring `email_notifications` with `on_failure`, the specified recipients receive an email after the job fails, including after all retries are exhausted. This is the most direct and native method. Other options require external services or are not designed for this purpose.

Answer

Set up a Databricks SQL alert on the job's run history table.

Answer

Configure the job to use a job cluster with autoscaling and set a retry policy.

Answer

Use a webhook to call an external service that sends an email.

Question 4

What is the primary difference between sharing data via Delta Sharing compared to sharing data via Databricks-to-Databricks sharing?

Accepted Answer

Delta Sharing allows recipients outside of the Databricks ecosystem to access data.. Delta Sharing is an open protocol that allows sharing data with any client, including those outside the Databricks ecosystem, by using credential files. Databricks-to-Databricks sharing is a proprietary feature that allows seamless data access across different Unity Catalog metastores within the Databricks ecosystem, leveraging built-in authentication and governance features without requiring external credential management or token distribution.

Answer

Delta Sharing is only for real-time streaming data.

Answer

Databricks-to-Databricks sharing requires the recipient to download a credential file.

Answer

Databricks-to-Databricks sharing is limited to within the same cloud region.

Question 5

You are building a pipeline and notice that the 'Gold' layer tables are experiencing significant write latency due to frequent small file commits. What is the most effective way to resolve this while maintaining ACID integrity?

Accepted Answer

Run the 'OPTIMIZE' command periodically on the Gold layer tables.. The 'OPTIMIZE' command with 'ZORDER' is the standard way to fix the small file problem in Delta Lake. It compacts small files into larger, better-performing ones while physically organizing data to speed up reads. This is critical for Gold tables, which are typically read by BI tools. Maintaining performance here is essential for providing end-users with a responsive, high-performance experience when accessing critical business metrics and dashboards.

Answer

Increase the 'spark.sql.shuffle.partitions' to 5000 to distribute writes more widely.

Answer

Switch the table from Delta format to standard Parquet files to improve write performance.

Answer

Use the 'vacuum' command every 5 minutes to clear out the small files.

Question 6

A Data Engineer needs to ensure that a notebook job is not consuming excessive costs. Which monitoring tool provides the best view of DBU consumption per job?

Accepted Answer

Billable Usage System Table. The Databricks Billable Usage system table tracks DBU consumption at a granular level, including by job and cluster. Analyzing this data is essential for cost governance, allowing engineers to identify inefficient workflows and optimize resource allocation. By monitoring these trends, companies can ensure their data platform remains cost-effective, preventing unexpected budget overruns and providing the data necessary to justify investments in infrastructure or job refactoring.

Answer

Cluster event logs

Answer

Delta Live Tables event logs

Answer

Notebook execution history

Question 7

A data engineer is responsible for a production Databricks SQL warehouse that serves multiple teams. The engineer needs to set up monitoring to detect when query performance degrades due to resource contention. Which two metrics should the engineer monitor to identify this issue? (Choose two.)

Accepted Answer

The number of queries waiting in the queue for the warehouse.. Resource contention in a SQL warehouse manifests as queries waiting in the queue and increased query execution times. Monitoring queue length and average running time provides direct insight into whether the warehouse is oversubscribed. These metrics are available in the Databricks SQL warehouse monitoring dashboard and can be used to trigger scaling or alerting.

Answer

The total storage used by the Unity Catalog metastore.

Answer

The number of active clusters in the workspace.

Answer

The number of failed login attempts to the workspace.

Question 8

A Data Engineer needs to share a Delta table with a partner organization using Delta Sharing. The partner does not use Databricks. Which component must the Data Engineer generate to facilitate this secure connection?

Accepted Answer

A sharing credential file. Delta Sharing uses a credential file to authenticate external recipients. The provider generates a sharing token encapsulated in a JSON file, which the recipient imports into their Delta Sharing client. This ensures that the data provider maintains full control over access permissions while the recipient can query the data using standard tools without needing a Databricks account, maintaining a secure and decoupled architecture.

Answer

A shared Unity Catalog metastore link

Answer

A personal access token with REST API permissions

Answer

A cross-account IAM role in the recipient's cloud

Question 9

You are ingesting data from multiple source systems with varying file formats (JSON, CSV, Parquet) into a centralized Bronze landing zone. Which architecture pattern is the most scalable for maintaining this ingestion layer?

Accepted Answer

A separate ingestion pipeline or notebook for each source system.. A modular ingestion architecture that decouples source-specific logic from the common landing layer is the industry standard. By using parameterized notebooks or DLT pipelines for each source, you isolate potential failures, enable independent scaling, and simplify the management of schema mappings. This pattern ensures that changes in one source system do not impact the ingestion pipelines of others, creating a highly resilient and maintainable data acquisition ecosystem.

Answer

A single monolithic notebook with a massive if/else block for each source.

Answer

Ingest everything into a single raw table before applying transformations.

Answer

Use a third-party tool exclusively and bypass Databricks for ingestion.

Question 10

When using Delta Sharing to share data with a recipient, what is the best way to handle updates to the shared data?

Accepted Answer

Updates are automatically visible to the recipient.. Delta Sharing automatically reflects the state of the table at the time of the query. Because the recipient accesses the Delta table directly (or via a managed share), any new data written to the table becomes immediately available to the recipient. There is no need for the provider to manually refresh the share, which ensures data consistency and reduces the maintenance burden for the Data Engineer.

Answer

The provider must recreate the share object.

Answer

The recipient must request a new credential file.

Answer

The provider must run an 'REFRESH SHARE' command.

Question 11

A company requires data to be physically deleted from the Bronze layer for GDPR compliance. What is the correct procedure to ensure complete removal?

Accepted Answer

Run the DELETE statement and then run the VACUUM command.. Under GDPR, 'deletion' requires the actual removal of data from all storage locations. In Delta Lake, running a DELETE statement followed by a VACUUM command is necessary. DELETE removes the records from the active state, but the underlying data files remain until VACUUM removes them. This two-step process ensures the data is logically removed, and then physically purged from the storage provider, satisfying the right to be forgotten.

Answer

Simply run the DELETE statement; Delta handles physical deletion automatically.

Answer

Use the OPTIMIZE command to force the deletion of the records.

Answer

Overwriting the table with a filtered subset using the OVERWRITE option.

Question 12

A data engineer has deployed a Databricks SQL dashboard that queries a gold-layer table. The dashboard is used by executives every morning. The engineer wants to be notified if the dashboard's underlying query fails or returns zero rows, which would indicate a data pipeline issue. Which Databricks feature should they use to set up this notification?

Accepted Answer

Databricks SQL alerts. Databricks SQL alerts are designed to monitor query results and trigger notifications based on conditions such as row count, value thresholds, or query failure. By creating an alert on the dashboard's query, the engineer can receive an email if the query fails or returns zero rows, ensuring timely awareness of data pipeline issues.

Answer

Job alerts on the Databricks job that refreshes the gold table

Answer

Cluster metrics and logging in the Clusters UI

Answer

Delta Live Tables expectations

Question 13

Which statement correctly describes the relationship between Unity Catalog and Databricks SQL Warehouses when using Lakehouse Federation?

Accepted Answer

The SQL Warehouse pushes down query predicates to the external database.. Lakehouse Federation allows users to query external databases directly from Databricks SQL Warehouses. The SQL Warehouse acts as the compute engine that translates Spark SQL queries into the native dialect of the external source. Unity Catalog acts as the central governance layer that stores the connection information, credentials, and schema mapping, allowing users to query external data as if it were local tables.

Answer

The SQL Warehouse performs the data ingestion into the Unity Catalog metastore.

Answer

Unity Catalog must store the external data in a managed Delta table.

Answer

All federated queries are executed solely within the Databricks control plane.

Question 14

A data engineer is building a streaming ingestion pipeline from Apache Kafka to a Delta table. The pipeline must perform deduplication on a unique event_id field and handle late-arriving data. The engineer wants to use Structured Streaming with a watermark of 10 minutes. Which of the following approaches correctly implements deduplication and watermarking?

Accepted Answer

Use dropDuplicates with event_id and set the watermark on the event timestamp column.. The correct approach is to use dropDuplicates on the unique event_id and set a watermark on the event timestamp column. This combination allows the stream to deduplicate events within the watermark window, effectively handling late-arriving data while bounding state. It is the recommended pattern for streaming deduplication in Databricks Structured Streaming.

Answer

Use dropDuplicates with event_id and the event timestamp column without setting a watermark.

Answer

Use a windowed aggregation on event_id and timestamp, then filter out duplicates using row_number.

Answer

Use foreachBatch to manually deduplicate by querying the Delta table for existing event_id values.

Question 15

A Data Engineer is using Delta Live Tables to process a streaming source that contains duplicate records based on an 'event_id'. The engineer needs to ensure that only the latest record for each 'event_id' is retained in the target table, and the pipeline should handle late-arriving data. Which DLT feature should be used?

Accepted Answer

Apply changes with deduplication using the APPLY CHANGES API with a sequence column. The APPLY CHANGES API in DLT is designed for change data capture and deduplication. By specifying a key column and a sequence column, it ensures that only the latest record for each key is retained, even with late-arriving data. This declarative approach simplifies pipeline code and leverages DLT's built-in handling of streaming data and state management.

Answer

Use Delta Lake's MERGE INTO with a window function to rank records

Answer

Create a materialized view with a GROUP BY event_id and MAX(timestamp)

Answer

Use a streaming live table with a foreachBatch function to manually deduplicate