Databricks-DE-Pro Practice Test 19 — 15 Questions

Question 1

When designing a Data Quality framework in Databricks, what is the recommended approach for handling 'quarantined' records?

Accepted Answer

Move invalid records to a 'quarantine' table with an additional column indicating the error.. Storing invalid records in a separate, dedicated table allows for auditing and correction without blocking the main pipeline. This ensures high throughput for valid data while providing a clear path for reprocessing bad data once the underlying issue is fixed. It is a fundamental pattern for resilient data engineering, ensuring that data is never silently dropped and that stakeholders maintain confidence in the data's reliability.

Answer

Delete the bad records and notify the team via an automated Slack notification.

Answer

Update the records in place by setting the invalid fields to NULL.

Answer

Stop the entire pipeline execution to ensure no bad data reaches the final target.

Question 2

When migrating an existing Hive metastore to Unity Catalog, what is the most important security consideration regarding object naming?

Accepted Answer

The object names must fit the catalog.schema.table structure.. In Unity Catalog, the three-level namespace (catalog.schema.table) is mandatory. Existing Hive metastore objects often lack this structure and may contain characters or naming patterns that are incompatible with Unity Catalog's standards. Ensuring a clean naming convention during migration is not just a structural requirement; it is a security necessity to ensure that existing access policies can be correctly migrated and enforced in the new, more rigorous governance environment.

Answer

Table names must be all uppercase.

Answer

Legacy object owners must be deleted.

Answer

All tables must be converted to Parquet.

Question 3

Which Databricks feature provides the most granular view of data quality metrics over time for a Delta Live Tables pipeline?

Accepted Answer

DLT Event Log system table. The DLT 'event log' (stored as a system table) is the most powerful tool for analyzing data quality. It records every expectation check, allowing for historical trend analysis of data quality violations. This is critical for data governance, as it provides auditability and helps engineers identify when and where data quality has degraded over time, enabling proactive fixes to source upstream data.

Answer

Pipeline Run logs

Answer

Cluster log files

Answer

SQL warehouse query history

Question 4

A data engineer needs to inspect the logs of a long-running Databricks job that has already completed. Where should they navigate in the Databricks UI to find the driver logs?

Accepted Answer

The 'Runs' tab within the specific Job's detail page.. The Job Runs view provides a comprehensive audit trail of all executions, including specific task logs and driver output. Accessing the 'Runs' tab for a specific job allows the engineer to select a historical run and view the standard output and standard error streams. This is the primary interface for post-mortem debugging and understanding the execution context of failed tasks that are no longer actively running on a cluster.

Answer

The 'Clusters' tab under the 'Event Log' section.

Answer

The 'Workspace' folder where the notebook is stored.

Answer

The 'SQL Warehouses' monitoring dashboard.

Question 5

Refer to the exhibit. A user encounters this error when running a query. What is the correct action to resolve this issue while maintaining the security model?

Accepted Answer

Grant 'USE CATALOG' on the 'finance' catalog to the user.. In Unity Catalog, the security model is hierarchical. To access a table, the user needs 'USE CATALOG' on the catalog, 'USE SCHEMA' on the schema, and 'SELECT' on the table. The error indicates the hierarchy is broken at the top level. Granting the necessary privileges allows the user to traverse the object tree, ensuring that security is enforced consistently from the catalog down to the individual data object.

Answer

Grant 'SELECT' on the 'q1_data' table to the user.

Answer

Change the user's role to 'Account Admin' to bypass restrictions.

Answer

Move the 'q1_data' table to a catalog where the user has access.

Question 6

A data engineer is deploying a Databricks job using Databricks Asset Bundles. They want to ensure that the job uses a specific cluster configuration that is defined once and reused across multiple tasks. Which bundle feature should they use?

Accepted Answer

Define a `job_cluster` in the job's `job_clusters` section and reference it by key in each task.. The `job_clusters` section in a Databricks job definition allows specifying cluster configurations that can be referenced by multiple tasks via `job_cluster_key`. This is the correct way to define a cluster once and reuse it across tasks within the same job, ensuring consistency and simplifying maintenance. Other options either duplicate configuration or rely on external resources.

Answer

Use a `new_cluster` definition for each task, ensuring they all have identical configurations.

Answer

Define a cluster in the `resources/clusters` directory and reference it in the job.

Answer

Use an `existing_cluster_id` for all tasks, pointing to a pre-created cluster.

Question 7

Which TWO of the following are primary benefits of using Unity Catalog for data governance in Databricks?

Accepted Answer

It provides a centralized interface for managing permissions across multiple workspaces.. Unity Catalog provides a unified governance layer that simplifies security across different workspaces. Its primary value lies in centralized access control and a unified auditing capability. By providing a single source of truth for permissions and data lineage, it significantly reduces the complexity of managing large-scale data environments while ensuring compliance with internal security policies and regulatory frameworks.

Answer

It automatically encrypts all data files at rest using AES-256.

Answer

It allows users to bypass Spark SQL for direct file-system operations.

Answer

It replaces the need for IAM roles when accessing external data.

Question 8

A data engineer is configuring monitoring for a production Databricks cluster. Which TWO metrics are best suited to identify potential performance bottlenecks related to worker nodes?

Accepted Answer

Cluster CPU utilization percentage.. Monitoring worker nodes is vital for identifying bottlenecks before they cause job failures. Metrics like CPU utilization and memory usage are key indicators of node pressure. By identifying these patterns, engineers can optimize cluster configurations, adjust instance types, or tune spark code, ensuring that production workloads remain stable, performant, and cost-effective throughout their execution lifecycle.

Answer

Number of active jobs currently running in the entire workspace.

Answer

Total number of users logged into the Databricks workspace.

Answer

The version of the Databricks Runtime used by the cluster.

Question 9

A data engineer is ingesting data from an Apache Kafka topic into a Delta Lake table using Structured Streaming. The Kafka topic receives messages with a timestamp field in the value payload, but the messages can arrive out of order by up to 10 minutes. The engineer wants to perform time-windowed aggregations on the ingested data while minimizing state store overhead. Which approach should be used to handle the out-of-order data correctly?

Accepted Answer

Use withWatermark on the event-time column extracted from the Kafka value and set the watermark to 10 minutes.. To handle out-of-order data in Structured Streaming, you must use event-time processing with watermarks. Extracting the event-time column from the Kafka value and applying a watermark of 10 minutes allows the engine to wait for late data up to that threshold and then clean up state. This minimizes state store overhead and ensures correct time-windowed aggregations. Other options do not address event-time semantics or state management.

Answer

Configure the Kafka source with 'startingOffsets' set to 'latest' and enable auto-commit of offsets.

Answer

Repartition the stream by the Kafka partition ID and sort within each partition using a custom foreachBatch function.

Answer

Set the Kafka consumer property 'isolation.level' to 'read_committed' to ensure only committed messages are processed.

Question 10

Refer to the exhibit. An administrator reviews the cluster configuration JSON for an all-purpose interactive development cluster used by data engineers. Based on Databricks cost and performance optimization best practices, which specific parameter in this configuration represents the highest risk for unnecessary financial expenditure?

Accepted Answer

The auto_termination_minutes parameter is set to 120, allowing interactive compute to remain idle and bill at all-purpose rates for two full hours.. The exhibit shows an all-purpose cluster configured with a 120-minute auto-termination timeout and heavy ML-runtime packages on storage-optimized instance types. Setting auto-termination to 120 minutes (2 hours) of inactivity causes extreme financial waste when developers walk away. Interactive all-purpose clusters should typically timeout after 30 to 60 minutes of inactivity to optimize cost recovery.

Answer

The spark_version string specifies an ML runtime version containing unnecessary packages for general data engineering workloads.

Answer

The node_type_id specifies i3.xlarge storage-optimized instances which are entirely unsupported for running standard Delta Lake queries.

Answer

The autoscale max_workers limit is capped at 20, which will instantly crash any query exceeding ten gigabytes of data volume.

Question 11

A retail company uses a Databricks Lakehouse with a star schema in the Gold layer. Their fact_sales table has billions of rows and is partitioned by sale_date. Analysts frequently run queries that filter on product_id and join to dim_product. Currently, queries scanning the entire fact table are slow. To improve performance for these queries, which approach is most appropriate?

Accepted Answer

Apply Z-ORDER BY product_id on the fact_sales table.. Z-ORDER BY product_id clusters data within each partition, allowing Delta Lake to skip files that do not contain the filtered product_id values. This reduces I/O and speeds up queries that filter or join on product_id. Partitioning by product_id is impractical due to high cardinality, and the other options do not provide the needed optimization.

Answer

Partition the fact_sales table by product_id.

Answer

Convert the fact_sales table to a Parquet table and use partition pruning.

Answer

Add a bloom filter index on product_id.

Question 12

A data engineer is using Databricks Repos to manage code for a production job. They need to ensure that the job always uses the latest version of the code from a specific branch. Which Git operation should they perform before running the job?

Accepted Answer

Pull the latest changes from the remote repository into the Repo.. To ensure a Databricks job uses the latest code from a Git branch, the engineer must pull the latest changes into the Repo. This updates the working directory with the most recent commits. Committing, branching, or merging are not appropriate for simply retrieving updates. Pulling is the standard Git operation for this purpose.

Answer

Merge the remote branch into the local branch using the Repos UI.

Answer

Create a new branch from the current branch and switch to it.

Answer

Commit and push local changes to the remote repository.

Question 13

You are writing a Spark application that uses a Broadcast Hash Join. You want to force the join to use broadcast optimization for a specific table. How do you implement this in PySpark?

Accepted Answer

df1.join(broadcast(df2), 'key'). Using the 'broadcast()' function in PySpark explicitly hints to the Spark optimizer that a specific DataFrame should be broadcasted to all executors. This is highly effective when joining a large fact table with a small dimension table, as it avoids the expensive shuffle operation. Understanding this is crucial for performance tuning, as relying solely on the automatic broadcast threshold may lead to suboptimal join choices in complex queries.

Answer

df1.join(df2, 'key', 'broadcast')

Answer

spark.conf.set('spark.sql.broadcastJoin', 'true')

Answer

df1.join(hint('broadcast'), df2, 'key')

Question 14

Which of the following is the best practice for managing alerts for a mission-critical production pipeline?

Accepted Answer

Use a central notification channel connected to an incident management tool.. Centralizing alerts within Databricks and routing them to a dedicated incident management tool ensures that issues are tracked, assigned, and resolved systematically. Avoiding fragmented notification methods is key to operational maturity. This approach prevents alert fatigue, ensures that the correct personnel are notified during off-hours, and keeps a clear historical record of system issues for post-mortem analysis and continuous improvement of the data architecture.

Answer

Configure alerts to send emails to individual developers.

Answer

Create a dashboard and check it manually every hour.

Answer

Disable alerts for development environments.

Question 15

Which TWO of the following are benefits of using Unity Catalog for managing data governance in a multi-workspace environment?

Accepted Answer

Centralized access control across all workspaces.. Unity Catalog provides a unified, centralized governance layer that spans all workspaces in a Databricks account. This eliminates the need for siloed management, ensuring that users have consistent access rights, audit trails, and data discovery across the entire organization. By streamlining these processes, Unity Catalog simplifies compliance and improves collaboration while maintaining strict control over who can access which data objects across the enterprise.

Answer

Workspaces can have independent, non-overlapping governance.

Answer

Increased performance for all SQL queries.

Answer

Automatic data encryption at the application level.