Databricks-DE-Pro Practice Test 10 — 10 Questions

Question 1

You are monitoring a long-running Databricks job. You notice that the memory usage on the driver node is steadily increasing until it crashes. Which debugging action is most appropriate?

Accepted Answer

Use the Spark UI to identify tasks using 'collect()' or 'toPandas()' on large datasets.. Driver memory crashes are typically caused by collecting large datasets from workers to the driver. By identifying the 'collect' or 'toPandas' operations, you can optimize the code to process data in a distributed manner. This is critical for scaling data pipelines; understanding why the driver is overloaded ensures that jobs remain performant as data volumes grow and prevents common failures in production batch processing environments.

Answer

Increase the number of worker nodes in the cluster.

Answer

Update the cluster's Spark configuration to disable the driver's log monitoring.

Answer

Lower the 'spark.driver.maxResultSize' setting in the cluster configuration.

Question 2

A team is building a streaming pipeline that processes millions of events per second. They are using Structured Streaming with a Delta Lake sink. What is the most effective way to optimize the performance and cost of this write-heavy workload?

Accepted Answer

Enable 'delta.autoOptimize.optimizeWrite' and 'delta.autoCompact'.. For high-volume streaming, the 'optimizeWrite' and 'autoCompact' features are critical. 'optimizeWrite' ensures that data is written in optimal file sizes (typically 128MB) before committing, which prevents the creation of small files in the cloud storage. This reduces the burden on the file system and improves downstream read performance. Combined with 'autoCompact', the table remains performant for analytical queries without manual intervention, saving compute cycles and maintenance time.

Answer

Increase the trigger interval to 1 hour to batch writes.

Answer

Disable checkpointing to reduce the write latency.

Answer

Force a global sort before writing to the Delta table.

Question 3

You are developing a Delta Live Tables (DLT) pipeline and need to ensure high data quality. Which TWO of the following statements correctly describe how Expectations work within DLT?

Accepted Answer

You can use 'expect_or_fail' to stop the pipeline if a constraint is violated.. Expectations are a fundamental feature in Delta Live Tables that allow engineers to define data quality constraints directly within the DLT pipeline code. By specifying these rules, developers can monitor data health, quarantine corrupt rows, or fail pipelines when critical thresholds are breached. This declarative approach integrates governance into the data engineering workflow, ensuring that downstream consumers receive only validated, high-quality data while maintaining robust audit trails for compliance.

Answer

Expectations only support dropping rows that fail validation.

Answer

Expectations must be configured in the pipeline settings JSON file.

Answer

Expectations automatically fix invalid data values.

Question 4

Which THREE strategies should a data engineer use to optimize the debugging of failed production Databricks Jobs?

Accepted Answer

Implement structured logging within the application code to track variable states.. Efficient debugging relies on observability, modular design, and access control. By leveraging logs, modularizing code, and ensuring proper access, engineers can quickly isolate root causes. These strategies are essential for minimizing Mean Time to Recovery (MTTR) in production. Proactive monitoring and well-structured codebases allow for faster identification of failures, ensuring that business-critical pipelines remain operational and that the team can respond to incidents with precision and speed.

Answer

Grant all developers full cluster permissions to access logs directly on the nodes.

Answer

Run all production jobs as the root user to avoid permission-related errors.

Question 5

Where can a data engineer find the standard output and error logs for a specific task within a Databricks Workflow?

Accepted Answer

By clicking the 'Logs' tab within the specific task run details in the Jobs UI.. Databricks provides a comprehensive UI that aggregates logs for every task execution. By navigating to the job run and selecting the specific task, the engineer can view stdout and stderr logs directly. This is the primary method for diagnosing runtime exceptions, syntax errors, or logic failures in automated data pipelines, facilitating rapid troubleshooting without needing external tools or direct node access.

Answer

In the Databricks Filesystem (DBFS) root directory under /logs.

Answer

By querying the 'sys.logs' table in the Unity Catalog.

Answer

In the cluster configuration's 'Advanced Options' tab.

Question 6

When partitioning a large Delta table, what is the best practice regarding the number of unique values in the partition column?

Accepted Answer

Low cardinality is preferred to avoid creating too many files.. Choosing a column with too many unique values (high cardinality) for partitioning leads to the 'small files problem.' Each unique value creates a new directory, resulting in an explosion of small files that degrades performance during reads and writes. Best practices suggest partitioning by columns with low to medium cardinality (e.g., date, region) to ensure balanced file sizes and efficient data skipping during query execution.

Answer

High cardinality is preferred for faster data skipping.

Answer

Partitioning should always be done on a primary key column.

Answer

The number of unique values does not impact performance.

Question 7

You are processing sensitive PII data in a Delta table. You need to ensure that specific columns containing PII are not readable by general data analysts while maintaining the ability to perform aggregate analysis on those rows. Which feature should you implement?

Accepted Answer

Apply Dynamic Views with functions like 'mask_hash' or 'case'.. Databricks Unity Catalog provides robust data governance, including column-level security through Dynamic Views. By defining a view that masks or filters data based on the user's role, you ensure PII protection while allowing data exploration. Managing access control via Unity Catalog is a standard professional requirement for maintaining compliance (GDPR/CCPA) and ensuring that sensitive information remains secure within the enterprise data architecture.

Answer

Use the 'ALTER TABLE DROP COLUMN' command.

Answer

Implement row-level security using Spark configurations.

Answer

Encrypt the entire storage bucket using cloud-native tools.

Question 8

A data engineer is using PySpark to cleanse a DataFrame containing customer addresses. The 'zip_code' column has some values with leading zeros that were stripped during CSV ingestion. The engineer needs to restore all zip codes to a fixed 5-character length by padding with leading zeros. Which function should be used?

Accepted Answer

lpad(zip_code, 5, '0'). The lpad function is designed to pad strings on the left to a specified length. It correctly restores leading zeros for zip codes that lost them during ingestion, ensuring all values are exactly 5 characters. Other functions either pad on the wrong side, assume numeric types, or do not achieve the required fixed length.

Answer

regexp_replace(zip_code, '^', '0')

Answer

format_string('%05d', zip_code)

Answer

rpad(zip_code, 5, '0')

Question 9

Which TWO statements regarding the use of 'APPLY CHANGES INTO' in Delta Live Tables (DLT) are correct?

Accepted Answer

It supports schema evolution automatically when adding new columns to the source stream.. APPLY CHANGES INTO is the declarative mechanism for SCD Type 1 or Type 2 processing in DLT. It manages the complexity of merging streaming data into a target table, handling late-arriving data and updates efficiently. Understanding its constraints, such as the requirement for a defined primary key and the inability to perform manual data deletions, is critical for maintaining consistent state in bronze-to-silver transformations.

Answer

It can be used to perform deletes on the target table by issuing manual DELETE commands.

Answer

It allows for multiple primary keys, but only one can be used for sequencing the updates.

Answer

It automatically creates a history table for SCD Type 1 processing by default.

Question 10

You are managing a large-scale data lakehouse. You notice that your Spark jobs are frequently failing due to disk space issues on worker nodes. Which monitoring feature should you implement to proactively capture this trend?

Accepted Answer

Monitor Spark shuffle spill metrics using the Ganglia UI or custom Spark listeners.. Implementing custom Spark listeners or utilizing the Spark UI metrics for 'Disk Spilling' is the most effective proactive measure. Disk space exhaustion is often a symptom of memory pressure, leading to excessive shuffle operations. By monitoring spill-to-disk metrics, you can identify jobs that require more memory or better partitioning, preventing job failure and optimizing compute efficiency before the disk threshold is reached.

Answer

Enable standard Databricks job success/failure notifications.

Answer

Set up a Databricks SQL alert on the system.query_history table.

Answer

Schedule a daily scan of the underlying S3 or ADLS storage buckets.