Databricks-DE-Assoc Troubleshooting, Monitoring, and Optimization Practice Test 3 — 15 Questions

Question 1

Refer to the exhibit. A data engineer receives this error when collecting data from a large transformation back to the driver node. Which approach should be used to fix this issue?

Accepted Answer

Rewrite the job to write the results to a Delta table.. The error occurs because the result set being pulled to the driver exceeds the configured limit. Collecting massive data to the driver is an anti-pattern in distributed computing as it bypasses the cluster's parallel processing capabilities. Instead of forcing data into the driver's memory, the engineer should write the output to cloud storage or a Delta table, allowing downstream processes to handle the data in a distributed, scalable manner without memory pressure.

Answer

Increase the spark.driver.maxResultSize setting.

Answer

Reduce the number of tasks in the Spark job.

Answer

Enable Spark dynamic allocation.

Question 2

A data engineer is investigating a slow-running SQL query. Which TWO metrics from the Spark UI should the engineer examine to identify data skew?

Accepted Answer

Task duration. Data skew occurs when one partition is significantly larger than others, causing one task to take much longer than the rest. By examining task duration and task input size, an engineer can spot outliers where a single executor handles a disproportionate amount of data. Identifying these metrics allows the engineer to apply remediation techniques like salting or repartitioning to balance the workload across the cluster effectively.

Answer

Executor CPU usage

Answer

Shuffle read size

Answer

Driver memory usage

Question 3

A data engineer needs to monitor the costs associated with specific projects running on a shared Databricks workspace. Which feature should the engineer use to attribute these costs accurately?

Accepted Answer

Cluster Tags. Tags allow for granular cost tracking by applying metadata to Databricks resources like clusters. By using custom tags, organizations can categorize spending by project, department, or cost center. This is vital for financial accountability in multi-tenant environments. When billing records are exported, these tags are included, enabling precise allocation of expenditures back to the specific project owners or internal clients, ensuring transparency and efficient budget management across the organization.

Answer

Notebook Versioning

Answer

Delta Lake Audit Logs

Answer

Databricks SQL Alerts

Question 4

A job is failing with a 'Disk Space' error on the worker nodes. The code performs several large joins. Which configuration should the engineer adjust to mitigate the disk space usage?

Accepted Answer

Increase spark.sql.shuffle.partitions.. Large joins often require spilling to disk when the shuffle data exceeds the available executor memory. Adjusting the spark.sql.shuffle.partitions configuration can reduce the size of individual partitions, potentially fitting them within memory. Alternatively, increasing the instance type memory or optimizing the join condition reduces the reliance on local disk storage. Managing shuffle partitions is a standard practice to balance memory pressure against the risk of disk overflow during shuffles.

Answer

Decrease spark.driver.memory.

Answer

Set spark.databricks.io.cache.enabled to false.

Answer

Increase the number of task retries.

Question 5

An engineer needs to identify the root cause of a job failure. Which THREE of the following are valid locations or methods to investigate the logs?

Accepted Answer

Cluster event logs. Troubleshooting in Databricks requires access to various log levels. The cluster event log captures infrastructure-level changes, driver logs provide application execution details, and executor logs offer task-specific debugging info. Using these three sources provides a holistic view, covering everything from cluster startup issues to specific code-level exceptions. This multi-layered approach is essential for isolating whether a problem is infrastructure-related, configuration-based, or rooted in the logic of the transformation code itself.

Answer

Notebook workspace browser history

Answer

Databricks account profile settings

Question 6

A junior data engineer notices that a scheduled Databricks job running a heavy ETL notebook is failing intermittently due to cluster driver out-of-memory errors. Which TWO configuration changes or architectural adjustments should be implemented to resolve this issue? (Select exactly TWO)

Accepted Answer

Select a larger driver node instance type with more memory capacity.. Driver out-of-memory errors typically occur when the driver node collects too much data into local memory using actions like collect() or handles excessive broadcast joins. Upgrading to a driver node with more RAM provides immediate headroom, while refactoring code to avoid pulling massive datasets to the driver prevents memory exhaustion fundamentally.

Answer

Increase the maximum number of worker nodes in the autoscaling cluster configuration.

Answer

Enable Delta caching on the cluster workers to offload data from the driver.

Answer

Decrease the Spark SQL shuffle partitions default setting to a lower number.

Question 7

A data engineer notices that a scheduled Delta Lake maintenance pipeline is running significantly slower than expected. Upon checking the table history, they see that hundreds of tiny, fragmented data files have accumulated due to frequent streaming micro-batches. Which specific optimization command should the data engineer run first to resolve this file-size bottleneck?

Accepted Answer

Invoke the OPTIMIZE table_name command to compact small files into larger, optimized data files and improve subsequent query execution efficiency.. Running OPTIMIZE reorganizes the layout of Delta Lake data by compacting small files into larger, uniform files of roughly 1 GB in size. This significantly reduces metadata overhead and improves scan performance for subsequent read queries. While VACUUM removes old physical files, it does not compact active files. OPTIMIZE directly addresses the core symptom of small file proliferation caused by frequent streaming micro-batches.

Answer

Run VACUUM table_name RETAIN 0 HOURS to instantly purge all historical file references and force file compaction across the entire dataset.

Answer

Execute REFRESH TABLE table_name to clear the local metastore cache and force Spark to rebuild the underlying file index from scratch.

Answer

Execute ALTER TABLE table_name SET TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true) to rewrite past historical micro-batches retroactively.

Question 8

A data engineer runs a Structured Streaming job that writes to a Delta table. The job processes data from a Kafka topic and uses a 10-minute watermark. After a few hours, the engineer notices that the streaming query's input rate is steady, but the processing rate has dropped significantly, and the batch duration has increased from 5 seconds to over 2 minutes. The job is running on a cluster with autoscaling enabled. Which action should the engineer take FIRST to diagnose the performance degradation?

Accepted Answer

Examine the Streaming Query progress metrics in the Spark UI, focusing on the 'addBatch' and 'walCommit' durations.. The Streaming Query progress metrics in the Spark UI are the primary tool for diagnosing Structured Streaming performance. They break down batch time into components like addBatch and walCommit, showing whether the delay is in data processing or Delta Lake commit operations. This targeted diagnosis avoids premature tuning and guides the engineer to the actual bottleneck.

Answer

Increase the number of shuffle partitions by setting spark.sql.shuffle.partitions to a higher value.

Answer

Restart the streaming job with a larger driver node to handle the increased metadata load.

Answer

Repartition the Kafka source topic to increase parallelism in the streaming query.

Question 9

A data engineer runs a nightly Databricks job that reads a large Delta table and writes aggregated results to another Delta table. The cluster logs show many small files in the source table, and the job runtime has increased steadily over weeks. The engineer wants to reduce the number of files without rewriting the entire table. Which command should be used?

Accepted Answer

OPTIMIZE sales. The small-file problem is a common cause of slow reads in Delta Lake. OPTIMIZE performs bin-packing to combine small files into larger ones, reducing the number of files that must be opened during a read. It can be run on the entire table or a subset using a WHERE clause, and it does not require rewriting the entire table, making it the appropriate maintenance command for this scenario.

Answer

ANALYZE TABLE sales COMPUTE STATISTICS

Answer

ALTER TABLE sales SET TBLPROPERTIES ('delta.autoOptimize.optimizeWrite' = 'true')

Answer

VACUUM sales RETAIN 0 HOURS

Question 10

A data engineer notices that a Databricks job processing a Delta table with 10,000 partitions runs slowly. The job filters on a column that is not the partition column, and the query plan shows that all partitions are being scanned. The engineer wants to improve performance without repartitioning the table. Which feature should be used?

Accepted Answer

Z-ORDER BY on the filter column during OPTIMIZE. When a filter is on a non-partition column, Delta Lake can still skip files if it has min/max statistics for that column and the data is clustered. Z-ORDER BY reorganizes data so that related values are stored together, improving the effectiveness of data skipping. Running OPTIMIZE with Z-ORDER BY on the filter column allows the query to read only the relevant files, avoiding a full scan of all partitions.

Answer

Enabling Change Data Feed on the table

Answer

Setting delta.dataSkippingNumIndexedCols to 0

Answer

Increasing the number of shuffle partitions

Question 11

A data engineer is investigating a Databricks job that failed overnight. The job's status in the Jobs UI shows 'Failed', and the engineer needs to view the error message and stack trace to determine the cause. Where should the engineer look to find the detailed error information for the failed run?

Accepted Answer

In the driver logs, accessible from the run's detail page in the Jobs UI.. Driver logs are the definitive source for application-level errors in Databricks jobs. They capture stdout and stderr from the driver, including full stack traces. The Jobs UI provides direct access to these logs from the run's detail page, making it the first place to check when a job fails due to an exception.

Answer

In the cluster's event log, which records all cluster scaling and termination events.

Answer

In the Databricks audit logs, which record user actions and API calls.

Answer

In the Spark UI's SQL tab, which shows the query plan and execution metrics.

Question 12

A data engineer runs a nightly Databricks job that reads from a large Delta table and writes results to another Delta table. The job has been taking progressively longer each night. The engineer examines the Spark UI and sees that the 'SQL' tab shows a single stage with many small tasks, each processing very few records, and the 'Storage' tab shows that the source table has thousands of small files. Which action should the engineer take to improve performance?

Accepted Answer

Run OPTIMIZE on the source Delta table to compact small files.. The Spark UI reveals many small tasks and thousands of small files in the source Delta table, which is a small-file problem. OPTIMIZE compacts these small files into larger ones, reducing task overhead and improving read performance. Other options either do not address the root cause or could exacerbate the issue by creating more partitions.

Answer

Enable Adaptive Query Execution (AQE) by setting spark.sql.adaptive.enabled to true.

Answer

Increase the number of shuffle partitions by setting spark.sql.shuffle.partitions to a higher value.

Answer

Repartition the source DataFrame to a smaller number of partitions before writing.

Question 13

A data engineer is troubleshooting a Databricks job that intermittently fails with a `SparkException: Job aborted due to stage failure: Task not serializable`. The job reads from a Parquet file, performs a transformation using a custom function defined in a Python class, and writes to a Delta table. The engineer suspects that the custom function is causing the issue. Which action should the engineer take to resolve the serialization error?

Accepted Answer

Refactor the custom function to avoid referencing any non-serializable objects from the enclosing class or module.. The `Task not serializable` error arises when Spark tries to serialize a closure for execution on executors but encounters objects that cannot be serialized. This often happens when a function references a non-serializable object from its enclosing scope, such as a database connection or a large data structure. Refactoring the function to avoid such references ensures that only serializable data is captured, resolving the error.

Answer

Mark the custom function with the `@staticmethod` decorator to avoid serializing the enclosing class.

Answer

Increase the driver memory to ensure that the serialized function fits in memory.

Answer

Set the Spark configuration `spark.serializer` to `org.apache.spark.serializer.KryoSerializer` to enable Kryo serialization.

Question 14

A data engineer is investigating why a Databricks job that reads from a Delta table is slow. The job performs a simple SELECT with a filter on a partition column. The engineer suspects that the table has many small files. Which Spark UI tab should be examined to confirm the number of files read?

Accepted Answer

Stages tab. The Stages tab provides per-stage metrics including the number of tasks, input size, and records read. For a scan stage, the number of tasks is often proportional to the number of files or partitions. A high task count with small input size per task suggests many small files. This makes the Stages tab the most direct place to confirm the small-file issue.

Answer

SQL tab

Answer

Environment tab

Answer

Storage tab

Question 15

A data engineer is monitoring a Databricks job and notices that the job's duration has gradually increased over the past week. The job reads a large Delta table, performs aggregations, and writes results to another Delta table. The engineer wants to identify the stage that is taking the most time. Which Spark UI tab should the engineer use to quickly identify the slowest stage?

Accepted Answer

Stages tab. The Spark UI Stages tab provides a detailed list of all stages, including their duration, number of tasks, and shuffle read/write metrics. By examining this tab, the engineer can sort stages by duration and identify the one consuming the most time. This is the most efficient way to locate the bottleneck stage and then investigate further.

Answer

Environment tab

Answer

Storage tab

Answer

Jobs tab