Databricks-DE-Pro Practice Test 4 — 25 Questions

Question 1

A data scientist reports that their notebook takes 20 minutes to initialize, even when the cluster is running. What is the most likely reason for this high initialization time?

Accepted Answer

The notebook is installing many libraries using %pip install.. High initialization time in a running cluster is frequently caused by excessive libraries being installed at the notebook level via '%pip install'. Every time the notebook is attached or the context is refreshed, these libraries must be re-resolved and installed, which adds significant overhead. Pre-installing these libraries at the cluster level via 'cluster-scoped libraries' allows them to be available immediately upon startup, eliminating this redundant installation phase and significantly speeding up the initialization process.

Answer

The cluster is using spot instances.

Answer

The cluster has too many workers allocated.

Answer

The user is using the wrong Databricks Runtime version.

Question 2

You are optimizing a PySpark job that reads from a Delta table. You notice skewed data distribution on the 'customer_id' column, causing Task-level stragglers. Which transformation should you apply to the DataFrame to mitigate this skew during a join operation?

Accepted Answer

Salt the skewed column and perform the join.. Salting involves adding a random prefix to the join key of the skewed table and exploding the join key of the dimension table. This technique breaks down massive partitions caused by skewed keys into smaller, evenly distributed chunks. Understanding data skew is critical in Databricks for maintaining cluster stability and preventing OOM errors, ensuring that shuffle partitions are utilized efficiently across all available executors rather than overloading one core.

Answer

Increase the spark.sql.shuffle.partitions configuration dynamically.

Answer

Broadcast the skewed table to all executors.

Answer

Cache the skewed table in memory before the join.

Question 3

When using the Unity Catalog, how should an engineer properly reference a table named 'sales' located in the 'finance' schema within the 'prod_catalog' catalog using Spark SQL?

Accepted Answer

SELECT * FROM prod_catalog.finance.sales. Unity Catalog uses a three-level namespace hierarchy: catalog.schema.table. This structure provides consistent governance and discovery across the entire Databricks account. Proper referencing requires the fully qualified name to ensure Spark identifies the exact object within the catalog's permission model. This implementation is crucial for security and multi-tenant environments where namespaces must be strictly isolated to prevent unauthorized access and data leakage across production and development environments.

Answer

SELECT * FROM finance.sales

Answer

SELECT * FROM sales

Answer

SELECT * FROM catalog_prod.finance.sales

Question 4

A Data Engineer wants to monitor data quality trends over time for a critical table. Which tool provides the most native and easy-to-use visualization of these metrics?

Accepted Answer

Create a Databricks SQL dashboard using the 'expectations' table provided by DLT.. Delta Lake's integration with Databricks SQL and DLT provides native dashboards that track data quality metrics automatically. Using Expectation history logs, engineers can visualize failure rates and quality trends without building custom monitoring solutions. This visibility is essential for proactive maintenance, allowing teams to catch degrading data quality before it impacts critical downstream business reporting, ensuring higher service levels for end-users.

Answer

Export all data quality logs to an external S3 bucket and use a third-party BI tool.

Answer

Write a custom Spark job to parse the Delta transaction log and send alerts via email.

Answer

Manually query the 'information_schema' every hour to track table row counts.

Question 5

A data engineer wants to monitor the health of Delta Live Tables (DLT) pipelines and be alerted if a pipeline fails. Which approach is the most efficient and native way to achieve this?

Accepted Answer

Define a Notification Destination in the DLT pipeline settings and associate it with the 'On failure' event.. Using Notification Destinations in the DLT pipeline settings is the most native method. It allows engineers to configure email or Slack alerts directly within the UI or JSON configuration. This is crucial for production reliability, ensuring that stakeholders receive immediate notifications regarding job failures or data quality issues without needing external orchestration or custom API scripts, maintaining observability across the entire pipeline lifecycle.

Answer

Configure an external cron job to poll the Databricks Jobs API every minute for status updates.

Answer

Enable the 'Log to Workspace' feature and manually query the system logs using SQL every hour.

Answer

Use the Databricks SQL Alerts tool to monitor the underlying Delta tables for new record counts.

Question 6

Which THREE techniques are recommended for improving the performance of Spark SQL joins on large Databricks tables?

Accepted Answer

Broadcast the smaller table in a join to avoid shuffling large datasets.. Optimizing joins is critical to performance as they are often the most resource-intensive operations in Spark. Techniques like broadcasting small tables, using bucketing to avoid shuffles, and ensuring data is properly partitioned help the Spark engine execute joins efficiently. By minimizing the amount of data moved across the network (shuffling) and maximizing local processing, jobs complete faster and consume fewer compute resources, leading to a more performant and cost-effective overall data architecture.

Answer

Increase the number of partitions to the maximum possible value to ensure maximum parallelism.

Answer

Use Cross Join for every join operation to ensure no data rows are missed.

Question 7

A Databricks SQL warehouse is experiencing high costs due to idle resources. Which TWO configurations should be implemented to effectively manage and reduce warehouse costs?

Accepted Answer

Set the auto-stop duration to a very low value, such as 1 minute, for serverless SQL warehouses.. Auto-stop and serverless scaling are the primary levers for cost control in Databricks SQL. Auto-stop terminates warehouses when no queries are running, preventing billing for idle time. Serverless SQL warehouses provide faster startup times and more granular scaling, allowing users to configure aggressive auto-stop durations without compromising user experience. Together, these configurations ensure that compute resources are only consumed when active query processing is required, directly minimizing unnecessary cloud spending.

Answer

Enable multi-cluster load balancing to ensure all queries are executed on the largest instance type.

Answer

Configure SQL warehouse scaling to use the maximum cluster size at all times to avoid resizing overhead.

Answer

Disable the query cache to ensure all results are freshly computed for accurate billing metrics.

Question 8

An enterprise data team runs a large nightly batch job using a standard all-purpose cluster. The job frequently fails due to cloud provider spot instance pre-emptions and takes over four hours to complete. How should the engineer refactor this architecture for maximum cost efficiency and reliability?

Accepted Answer

Convert the workload to use a Databricks Job cluster configured with spot instances and automatic fallback to on-demand.. Migrating the workload from an all-purpose interactive cluster to a Databricks Job cluster running on spot instances with an automatic fallback mechanism ensures cost-effective batch execution. Job clusters consume lower DBU rates than all-purpose clusters, and spot instances drastically reduce infrastructure costs while fallback guarantees completion despite cloud provider interruptions.

Answer

Provision a larger all-purpose cluster with double the worker nodes to brute-force execution speed.

Answer

Upgrade the cloud provider virtual machine family to the latest generation without changing cluster types.

Answer

Increase the Apache Spark executor memory fraction and decrease shuffle partition counts.

Question 9

A data engineer needs to ensure that PII data in a Delta table is accessible only to users in the 'HR_Manager' group. Which approach provides the most granular and scalable security implementation?

Accepted Answer

Apply a row filter function to the table using Unity Catalog.. Using Unity Catalog's Row-Level Security (RLS) via SQL functions is the standard best practice. It dynamically filters rows at query time based on the execution context, such as the current user's group membership. This approach avoids duplicating data into siloed tables, minimizes maintenance overhead, and ensures consistent enforcement across different BI tools and compute resources connecting to the same catalog.

Answer

Create separate views for each group and grant access to those views.

Answer

Implement column-level masking on the PII column.

Answer

Use data partitioning to store HR data in separate folders.

Question 10

Which of the following is the most secure method for a Data Engineer to provide access to a specific Delta table for a temporary project?

Accepted Answer

Adding the user to a temporary Unity Catalog group.. Granting temporary access should be done using time-bound permissions or dedicated temporary groups. In Unity Catalog, the best approach is to manage access through a group and remove the user from that group when the project concludes. This ensures a clean, auditable, and repeatable process for managing temporary access requests, which is essential for maintaining a secure environment and avoiding the 'permission creep' that happens when users retain access indefinitely.

Answer

Granting ownership of the table to the user.

Answer

Sharing the credentials of a Service Principal.

Answer

Creating a copy of the table for the user.

Question 11

A Data Engineer needs to ensure that PII data in a Delta table is accessible only to members of the 'hr_admin' group, while allowing all other users to view the non-PII columns. Which Unity Catalog feature is the most efficient way to implement this requirement?

Accepted Answer

Apply a column mask using a SQL function in Unity Catalog.. Unity Catalog's row-level security and column-level masking allow for fine-grained access control directly at the table level. By defining a masking function or a column filter, the Data Engineer ensures that the security policy is enforced consistently across all SQL warehouses and Databricks Runtime versions. This approach centralizes governance, reduces administrative overhead compared to view-based security, and ensures that data privacy compliance is maintained without duplicating data or creating multiple table versions.

Answer

Create separate physical tables for HR and general users.

Answer

Use standard SQL views for every user to filter columns.

Answer

Assign the 'SELECT' permission on individual columns in the UI.

Question 12

A production Databricks workflow involves a task that runs a notebook. The notebook takes 15 minutes to finish, but the workflow is set to timeout after 10 minutes. What happens?

Accepted Answer

The task is terminated and marked as 'Failed' by the scheduler.. When a task exceeds its configured 'timeout' value, the Databricks scheduler forcibly terminates the task execution. This is a deliberate safety measure to prevent runaway processes from consuming cluster resources indefinitely. In production, this highlights the necessity of monitoring execution times and setting appropriate timeouts that account for normal data volume fluctuations while catching truly stuck jobs that could impact cost and resource availability.

Answer

The workflow continues to run, but logs a warning about the duration.

Answer

The workflow pauses and waits for the engineer to manually approve continuation.

Answer

The task continues to run, but is moved to a background queue.

Question 13

A data engineer is troubleshooting a Databricks Workflow where a downstream task relies on an upstream task's output. Which TWO actions ensure the data dependency is correctly handled during a failure scenario?

Accepted Answer

Configure the downstream task to use the 'depends_on' attribute to reference the upstream task ID.. Proper dependency management ensures that downstream tasks do not attempt to process incomplete or missing data from upstream failures. By using task dependencies and repair-and-rerun capabilities, engineers can isolate failures and maintain data integrity. These features are critical in production pipelines to prevent downstream corruption and ensure that data lineage remains consistent throughout the workflow execution cycle regardless of individual component failures.

Answer

Set the task timeout to zero to prevent the workflow from ever stopping during a failure.

Answer

Hardcode the file path of the upstream output into the downstream task configuration.

Answer

Disable all retries to ensure that the error log is captured immediately upon failure.

Question 14

An organization is migrating to Unity Catalog and needs to secure sensitive data. Which TWO of the following statements regarding Unity Catalog security best practices are correct?

Accepted Answer

Use groups instead of individual users for access grants.. Effective security in Unity Catalog requires a deep understanding of object ownership and the principle of least privilege. By ensuring that objects are owned by a group rather than an individual, organizations prevent access gaps when staff turnover occurs. Additionally, using service principals for automated pipelines ensures that data access is tied to the workload rather than a user, maintaining consistent security postures across environments.

Answer

Assign table ownership to individual users for better tracking.

Answer

Ensure that the metastore admin has access to all data.

Answer

Public access should be granted to the root catalog.

Question 15

An organization needs to share a dataset with a client who does not use Databricks. What is the most efficient and secure way to share this data using Unity Catalog?

Accepted Answer

Use Delta Sharing with an open sharing provider.. Delta Sharing is an open-standard protocol that allows sharing data with any recipient, regardless of their platform or environment. By using Unity Catalog to share data, the organization maintains centralized governance and audit trails while enabling the recipient to consume the data using familiar tools. This approach is highly efficient because it avoids the need for data duplication and ensures the recipient always accesses the most current version.

Answer

Export the data to CSV and upload it to a public FTP server.

Answer

Grant the client read-only access to a dedicated Databricks workspace.

Answer

Create a public API endpoint that queries the table for the client.

Question 16

You are monitoring a long-running Databricks job. You notice that the memory usage on the driver node is steadily increasing until it crashes. Which debugging action is most appropriate?

Accepted Answer

Use the Spark UI to identify tasks using 'collect()' or 'toPandas()' on large datasets.. Driver memory crashes are typically caused by collecting large datasets from workers to the driver. By identifying the 'collect' or 'toPandas' operations, you can optimize the code to process data in a distributed manner. This is critical for scaling data pipelines; understanding why the driver is overloaded ensures that jobs remain performant as data volumes grow and prevents common failures in production batch processing environments.

Answer

Increase the number of worker nodes in the cluster.

Answer

Update the cluster's Spark configuration to disable the driver's log monitoring.

Answer

Lower the 'spark.driver.maxResultSize' setting in the cluster configuration.

Question 17

A team is building a streaming pipeline that processes millions of events per second. They are using Structured Streaming with a Delta Lake sink. What is the most effective way to optimize the performance and cost of this write-heavy workload?

Accepted Answer

Enable 'delta.autoOptimize.optimizeWrite' and 'delta.autoCompact'.. For high-volume streaming, the 'optimizeWrite' and 'autoCompact' features are critical. 'optimizeWrite' ensures that data is written in optimal file sizes (typically 128MB) before committing, which prevents the creation of small files in the cloud storage. This reduces the burden on the file system and improves downstream read performance. Combined with 'autoCompact', the table remains performant for analytical queries without manual intervention, saving compute cycles and maintenance time.

Answer

Increase the trigger interval to 1 hour to batch writes.

Answer

Disable checkpointing to reduce the write latency.

Answer

Force a global sort before writing to the Delta table.

Question 18

You are developing a Delta Live Tables (DLT) pipeline and need to ensure high data quality. Which TWO of the following statements correctly describe how Expectations work within DLT?

Accepted Answer

You can use 'expect_or_fail' to stop the pipeline if a constraint is violated.. Expectations are a fundamental feature in Delta Live Tables that allow engineers to define data quality constraints directly within the DLT pipeline code. By specifying these rules, developers can monitor data health, quarantine corrupt rows, or fail pipelines when critical thresholds are breached. This declarative approach integrates governance into the data engineering workflow, ensuring that downstream consumers receive only validated, high-quality data while maintaining robust audit trails for compliance.

Answer

Expectations only support dropping rows that fail validation.

Answer

Expectations must be configured in the pipeline settings JSON file.

Answer

Expectations automatically fix invalid data values.

Question 19

Which THREE strategies should a data engineer use to optimize the debugging of failed production Databricks Jobs?

Accepted Answer

Implement structured logging within the application code to track variable states.. Efficient debugging relies on observability, modular design, and access control. By leveraging logs, modularizing code, and ensuring proper access, engineers can quickly isolate root causes. These strategies are essential for minimizing Mean Time to Recovery (MTTR) in production. Proactive monitoring and well-structured codebases allow for faster identification of failures, ensuring that business-critical pipelines remain operational and that the team can respond to incidents with precision and speed.

Answer

Grant all developers full cluster permissions to access logs directly on the nodes.

Answer

Run all production jobs as the root user to avoid permission-related errors.

Question 20

Where can a data engineer find the standard output and error logs for a specific task within a Databricks Workflow?

Accepted Answer

By clicking the 'Logs' tab within the specific task run details in the Jobs UI.. Databricks provides a comprehensive UI that aggregates logs for every task execution. By navigating to the job run and selecting the specific task, the engineer can view stdout and stderr logs directly. This is the primary method for diagnosing runtime exceptions, syntax errors, or logic failures in automated data pipelines, facilitating rapid troubleshooting without needing external tools or direct node access.

Answer

In the Databricks Filesystem (DBFS) root directory under /logs.

Answer

By querying the 'sys.logs' table in the Unity Catalog.

Answer

In the cluster configuration's 'Advanced Options' tab.