Databricks-DE-Assoc Troubleshooting, Monitoring, and Optimization — All Questions With Answers

Question 1mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

An engineer has discovered that a specific join operation is extremely slow due to severe data skew on the join key. Which strategy should they use to mitigate this?

Question 2hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Refer to the exhibit. An engineer observes that running VACUUM on a Delta table with a retention period of 0 hours results in an error. Why is this configuration likely failing?

Exhibit

{
  "cluster_name": "prod-cluster",
  "spark_conf": {
    "spark.databricks.delta.retentionDurationCheck.enabled": "false"
  }
}

Question 3hardmulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a Databricks job that occasionally fails with a 'java.lang.OutOfMemoryError: GC overhead limit exceeded' on the driver node. The job performs a large collect() operation on a DataFrame and then processes the results locally. Which TWO actions should the engineer take to resolve this issue? (Choose two.)

Question 4mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer notices that a Databricks job writing to a Delta table is taking much longer than expected. The job uses a MERGE INTO statement that updates a large target table from a small source DataFrame. The engineer runs DESCRIBE HISTORY on the target table and sees that each MERGE operation rewrites a large number of files. Which optimization technique should the engineer apply to improve the performance of the MERGE operation?

Question 5mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer runs a Structured Streaming job that reads from a Delta table and writes to another Delta table. The job is configured with a 5-minute trigger interval. After several hours, the engineer notices that the write latency has increased significantly and the streaming query is processing each micro-batch slower than before. The source Delta table is frequently updated with small append operations, and the target table is not optimized. Which action should the engineer take to improve the streaming job's performance?

Question 6mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a Databricks job that fails with an OutOfMemoryError on the driver. The job collects a large DataFrame to the driver using collect() before writing results. The engineer wants to resolve the failure with minimal code changes. Which action should be taken?

Question 7easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a Databricks job that fails with an error indicating that the driver node has run out of memory. The job performs a large collect() operation on a DataFrame. Which action should the engineer take to resolve this issue?

Question 8hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a Databricks job that occasionally fails with a 'FileNotFoundException' when reading from a Delta table. The table is updated by a separate streaming job that performs frequent merges and deletes. The engineer suspects that the issue is related to data retention and file cleanup. Which configuration should the engineer adjust to ensure that the reading job can still access the required historical versions of the table?

Question 9mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is optimizing a Databricks job that performs a large join between two Delta tables. The engineer notices that the job is spilling data to disk and taking a long time. Which TWO actions should the engineer take to improve performance? (Choose two.)

Question 10easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer has a Databricks job that writes to a Delta table. The job sometimes fails due to concurrent write conflicts. The engineer wants to ensure that the job can automatically retry on such conflicts. Which feature should the engineer use?

Question 11hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a Databricks job that reads from a Delta table and writes to another Delta table. The job suddenly starts failing with a `FileNotFoundException` during read operations. The engineer verifies that the source table exists and is accessible. Which action should the engineer take to resolve this error?

Question 12mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A Databricks job is failing due to 'Out of Memory' (OOM) errors during a join operation on two large datasets. Which TWO actions could help mitigate this issue?

Question 13easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which Databricks feature should a data engineer use to view the execution plan, including information about the physical operators and data lineage, to troubleshoot a slow-running SQL query?

Question 14hardmulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A streaming job using Structured Streaming is lagging significantly behind the 'current time'. Which THREE of the following could be the root cause of this processing latency?

Question 15mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

An engineer has a large table that is frequently joined with small dimension tables. To optimize this, which optimization technique should be applied to the join operation?

Question 16mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which Databricks command should you use to recover storage space by removing files that are no longer referenced by a Delta table and are older than the retention period?

Question 17easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which tool in Databricks provides real-time monitoring of cluster resource usage, including CPU, memory, and network throughput, for an active job?

Question 18mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

When configuring a Databricks Job, which TWO factors most directly influence the choice between a 'Job Cluster' and an 'All-Purpose Cluster'?

Question 19mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which Spark configuration property can be used to enable Adaptive Query Execution (AQE) in Databricks?

Question 20mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which THREE strategies are recommended to improve the performance of reading from a Delta table in Databricks?

Question 21mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer wants to monitor the health and performance of Databricks Jobs over time. Which feature should they use to visualize trends, such as job success rates and average execution times, across multiple runs?

Question 22mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is optimizing a Delta table that suffers from slow read performance due to small file sizes. Which command should the engineer execute to consolidate these small files into larger, more efficient files without altering the underlying table data?

Question 23hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Refer to the exhibit. A job fails with the provided error message. What is the most likely cause of this failure in a Databricks Delta Lake environment?

Exhibit

Error: [DELTA_FILE_NOT_FOUND] The file /mnt/data/_delta_log/00000000000000000005.json was not found.

Question 24mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a slow-running Spark job on Databricks. Which TWO metrics in the Spark UI are most useful for identifying data skew?

Question 25mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

An engineer notices that a specific notebook job is consistently taking longer to start. They observe high 'initialization' times in the job logs. Which action should the engineer take to improve startup time?

Question 26mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is analyzing a Spark job that is failing with 'Out of Memory' (OOM) errors. Which configuration parameter should be tuned to increase the amount of memory allocated to the execution of joins and aggregations?

Question 27mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which action is recommended to resolve a scenario where a Databricks Job is failing due to excessive metadata operations on a Delta table with millions of files?

Question 28hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Refer to the exhibit. An engineer applies these configurations to a cluster. What is the primary benefit of enabling the Databricks IO Cache for a workload that involves repeatedly reading the same Delta tables?

Exhibit

spark.conf.set("spark.databricks.io.cache.enabled", "true")
spark.conf.set("spark.databricks.io.cache.maxDiskUsage", "50g")

Question 29mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer needs to troubleshoot a job that is failing during the 'shuffle' phase. Which Spark UI tab should the engineer examine to analyze the shuffle partitions and identify potential imbalances?

Question 30mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer notices that a production Delta Lake table is experiencing slow read performance during concurrent write operations. The table contains millions of small files. Which action should the engineer take to resolve this performance degradation?

Question 31hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Refer to the exhibit. A data engineer receives this error when collecting data from a large transformation back to the driver node. Which approach should be used to fix this issue?

Exhibit

Error: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 1000 tasks (2048 MB) is bigger than spark.driver.maxResultSize (1024 MB)

Question 32mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is investigating a slow-running SQL query. Which TWO metrics from the Spark UI should the engineer examine to identify data skew?

Question 33easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer needs to monitor the costs associated with specific projects running on a shared Databricks workspace. Which feature should the engineer use to attribute these costs accurately?

Question 34mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A job is failing with a 'Disk Space' error on the worker nodes. The code performs several large joins. Which configuration should the engineer adjust to mitigate the disk space usage?

Question 35mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

An engineer needs to identify the root cause of a job failure. Which THREE of the following are valid locations or methods to investigate the logs?

Question 36hardmulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A junior data engineer notices that a scheduled Databricks job running a heavy ETL notebook is failing intermittently due to cluster driver out-of-memory errors. Which TWO configuration changes or architectural adjustments should be implemented to resolve this issue? (Select exactly TWO)

Question 37mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer notices that a scheduled Delta Lake maintenance pipeline is running significantly slower than expected. Upon checking the table history, they see that hundreds of tiny, fragmented data files have accumulated due to frequent streaming micro-batches. Which specific optimization command should the data engineer run first to resolve this file-size bottleneck?

Question 38mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer runs a Structured Streaming job that writes to a Delta table. The job processes data from a Kafka topic and uses a 10-minute watermark. After a few hours, the engineer notices that the streaming query's input rate is steady, but the processing rate has dropped significantly, and the batch duration has increased from 5 seconds to over 2 minutes. The job is running on a cluster with autoscaling enabled. Which action should the engineer take FIRST to diagnose the performance degradation?

Question 39mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer runs a nightly Databricks job that reads a large Delta table and writes aggregated results to another Delta table. The cluster logs show many small files in the source table, and the job runtime has increased steadily over weeks. The engineer wants to reduce the number of files without rewriting the entire table. Which command should be used?

Question 40hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer notices that a Databricks job processing a Delta table with 10,000 partitions runs slowly. The job filters on a column that is not the partition column, and the query plan shows that all partitions are being scanned. The engineer wants to improve performance without repartitioning the table. Which feature should be used?

Question 41easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is investigating a Databricks job that failed overnight. The job's status in the Jobs UI shows 'Failed', and the engineer needs to view the error message and stack trace to determine the cause. Where should the engineer look to find the detailed error information for the failed run?

Question 42mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer runs a nightly Databricks job that reads from a large Delta table and writes results to another Delta table. The job has been taking progressively longer each night. The engineer examines the Spark UI and sees that the 'SQL' tab shows a single stage with many small tasks, each processing very few records, and the 'Storage' tab shows that the source table has thousands of small files. Which action should the engineer take to improve performance?

Question 43hardmultiple choice

Study the full Python automation breakdown →

A data engineer is troubleshooting a Databricks job that intermittently fails with a `SparkException: Job aborted due to stage failure: Task not serializable`. The job reads from a Parquet file, performs a transformation using a custom function defined in a Python class, and writes to a Delta table. The engineer suspects that the custom function is causing the issue. Which action should the engineer take to resolve the serialization error?

Question 44easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is investigating why a Databricks job that reads from a Delta table is slow. The job performs a simple SELECT with a filter on a partition column. The engineer suspects that the table has many small files. Which Spark UI tab should be examined to confirm the number of files read?

Question 45easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is monitoring a Databricks job and notices that the job's duration has gradually increased over the past week. The job reads a large Delta table, performs aggregations, and writes results to another Delta table. The engineer wants to identify the stage that is taking the most time. Which Spark UI tab should the engineer use to quickly identify the slowest stage?

Question 46hardmulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is optimizing a Databricks job that performs a join between a large fact table and a small dimension table. The job is slow, and the engineer suspects that the join strategy is not optimal. Which TWO actions should the engineer take to improve performance? (Choose two.)

Question 47mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is optimizing a Databricks job that processes a large dataset. The job performs a join between a large Delta table and a small dimension table, then writes the result to a Delta table. The engineer notices that the join is causing a large shuffle and wants to reduce shuffle overhead. Which two actions should the engineer take to improve performance? (Choose two.)

Question 48hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is optimizing a Databricks job that reads from a large Delta table and performs a join with a smaller table. The job is experiencing performance issues due to shuffling. The engineer wants to reduce the amount of data shuffled during the join. Which technique should the engineer use?

Question 49mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is using Databricks Jobs to run a nightly ETL pipeline. The job occasionally fails due to a transient network error when writing to an external database. The engineer wants to automatically retry the job a few times before marking it as failed. What is the most efficient way to configure this in Databricks?

Question 50mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is monitoring a Databricks job and notices that the job's tasks are spending a significant amount of time in garbage collection (GC). The job processes large amounts of data with many small objects. Which action should the engineer take to reduce GC overhead?

Question 51hardmulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is investigating why a Databricks job that writes to a Delta table is experiencing performance degradation over time. The job performs frequent small appends. Which TWO actions should the engineer take to improve write performance? (Choose two.)

Question 52hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is monitoring a Databricks job that runs a Structured Streaming query. The engineer notices that the query's input rate is high, but the processing rate is low, and the batch duration is increasing over time. The query uses a Delta table as a source and writes to another Delta table. Which action should the engineer take to improve the streaming query's performance?

Question 53mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer runs a nightly Databricks job that ingests data into a Delta table using multiple concurrent write streams. The engineer notices that some write transactions are failing with a ConcurrentAppendException. The job writes to the same partition of the table from different tasks. Which action should the engineer take to resolve this issue?

Question 1mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

An engineer has discovered that a specific join operation is extremely slow due to severe data skew on the join key. Which strategy should they use to mitigate this?

Question 2hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Refer to the exhibit. An engineer observes that running VACUUM on a Delta table with a retention period of 0 hours results in an error. Why is this configuration likely failing?

Exhibit

{
  "cluster_name": "prod-cluster",
  "spark_conf": {
    "spark.databricks.delta.retentionDurationCheck.enabled": "false"
  }
}

Question 3hardmulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a Databricks job that occasionally fails with a 'java.lang.OutOfMemoryError: GC overhead limit exceeded' on the driver node. The job performs a large collect() operation on a DataFrame and then processes the results locally. Which TWO actions should the engineer take to resolve this issue? (Choose two.)

Question 4mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer notices that a Databricks job writing to a Delta table is taking much longer than expected. The job uses a MERGE INTO statement that updates a large target table from a small source DataFrame. The engineer runs DESCRIBE HISTORY on the target table and sees that each MERGE operation rewrites a large number of files. Which optimization technique should the engineer apply to improve the performance of the MERGE operation?

Question 5mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer runs a Structured Streaming job that reads from a Delta table and writes to another Delta table. The job is configured with a 5-minute trigger interval. After several hours, the engineer notices that the write latency has increased significantly and the streaming query is processing each micro-batch slower than before. The source Delta table is frequently updated with small append operations, and the target table is not optimized. Which action should the engineer take to improve the streaming job's performance?

Question 6mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a Databricks job that fails with an OutOfMemoryError on the driver. The job collects a large DataFrame to the driver using collect() before writing results. The engineer wants to resolve the failure with minimal code changes. Which action should be taken?

Question 7easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a Databricks job that fails with an error indicating that the driver node has run out of memory. The job performs a large collect() operation on a DataFrame. Which action should the engineer take to resolve this issue?

Question 8hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a Databricks job that occasionally fails with a 'FileNotFoundException' when reading from a Delta table. The table is updated by a separate streaming job that performs frequent merges and deletes. The engineer suspects that the issue is related to data retention and file cleanup. Which configuration should the engineer adjust to ensure that the reading job can still access the required historical versions of the table?

Question 9mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is optimizing a Databricks job that performs a large join between two Delta tables. The engineer notices that the job is spilling data to disk and taking a long time. Which TWO actions should the engineer take to improve performance? (Choose two.)

Question 10easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer has a Databricks job that writes to a Delta table. The job sometimes fails due to concurrent write conflicts. The engineer wants to ensure that the job can automatically retry on such conflicts. Which feature should the engineer use?

Question 11hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a Databricks job that reads from a Delta table and writes to another Delta table. The job suddenly starts failing with a `FileNotFoundException` during read operations. The engineer verifies that the source table exists and is accessible. Which action should the engineer take to resolve this error?

Question 12mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A Databricks job is failing due to 'Out of Memory' (OOM) errors during a join operation on two large datasets. Which TWO actions could help mitigate this issue?

Question 13easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which Databricks feature should a data engineer use to view the execution plan, including information about the physical operators and data lineage, to troubleshoot a slow-running SQL query?

Question 14hardmulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A streaming job using Structured Streaming is lagging significantly behind the 'current time'. Which THREE of the following could be the root cause of this processing latency?

Question 15mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

An engineer has a large table that is frequently joined with small dimension tables. To optimize this, which optimization technique should be applied to the join operation?

Question 16mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which Databricks command should you use to recover storage space by removing files that are no longer referenced by a Delta table and are older than the retention period?

Question 17easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which tool in Databricks provides real-time monitoring of cluster resource usage, including CPU, memory, and network throughput, for an active job?

Question 18mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

When configuring a Databricks Job, which TWO factors most directly influence the choice between a 'Job Cluster' and an 'All-Purpose Cluster'?

Question 19mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which Spark configuration property can be used to enable Adaptive Query Execution (AQE) in Databricks?

Question 20mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which THREE strategies are recommended to improve the performance of reading from a Delta table in Databricks?

Question 21mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer wants to monitor the health and performance of Databricks Jobs over time. Which feature should they use to visualize trends, such as job success rates and average execution times, across multiple runs?

Question 22mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is optimizing a Delta table that suffers from slow read performance due to small file sizes. Which command should the engineer execute to consolidate these small files into larger, more efficient files without altering the underlying table data?

Question 23hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Refer to the exhibit. A job fails with the provided error message. What is the most likely cause of this failure in a Databricks Delta Lake environment?

Exhibit

Error: [DELTA_FILE_NOT_FOUND] The file /mnt/data/_delta_log/00000000000000000005.json was not found.

Question 24mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is troubleshooting a slow-running Spark job on Databricks. Which TWO metrics in the Spark UI are most useful for identifying data skew?

Question 25mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

An engineer notices that a specific notebook job is consistently taking longer to start. They observe high 'initialization' times in the job logs. Which action should the engineer take to improve startup time?

Question 26mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is analyzing a Spark job that is failing with 'Out of Memory' (OOM) errors. Which configuration parameter should be tuned to increase the amount of memory allocated to the execution of joins and aggregations?

Question 27mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Which action is recommended to resolve a scenario where a Databricks Job is failing due to excessive metadata operations on a Delta table with millions of files?

Question 28hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Refer to the exhibit. An engineer applies these configurations to a cluster. What is the primary benefit of enabling the Databricks IO Cache for a workload that involves repeatedly reading the same Delta tables?

Exhibit

spark.conf.set("spark.databricks.io.cache.enabled", "true")
spark.conf.set("spark.databricks.io.cache.maxDiskUsage", "50g")

Question 29mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer needs to troubleshoot a job that is failing during the 'shuffle' phase. Which Spark UI tab should the engineer examine to analyze the shuffle partitions and identify potential imbalances?

Question 30mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer notices that a production Delta Lake table is experiencing slow read performance during concurrent write operations. The table contains millions of small files. Which action should the engineer take to resolve this performance degradation?

Question 31hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

Refer to the exhibit. A data engineer receives this error when collecting data from a large transformation back to the driver node. Which approach should be used to fix this issue?

Exhibit

Error: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 1000 tasks (2048 MB) is bigger than spark.driver.maxResultSize (1024 MB)

Question 32mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is investigating a slow-running SQL query. Which TWO metrics from the Spark UI should the engineer examine to identify data skew?

Question 33easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer needs to monitor the costs associated with specific projects running on a shared Databricks workspace. Which feature should the engineer use to attribute these costs accurately?

Question 34mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A job is failing with a 'Disk Space' error on the worker nodes. The code performs several large joins. Which configuration should the engineer adjust to mitigate the disk space usage?

Question 35mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

An engineer needs to identify the root cause of a job failure. Which THREE of the following are valid locations or methods to investigate the logs?

Question 36hardmulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A junior data engineer notices that a scheduled Databricks job running a heavy ETL notebook is failing intermittently due to cluster driver out-of-memory errors. Which TWO configuration changes or architectural adjustments should be implemented to resolve this issue? (Select exactly TWO)

Question 37mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer notices that a scheduled Delta Lake maintenance pipeline is running significantly slower than expected. Upon checking the table history, they see that hundreds of tiny, fragmented data files have accumulated due to frequent streaming micro-batches. Which specific optimization command should the data engineer run first to resolve this file-size bottleneck?

Question 38mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer runs a Structured Streaming job that writes to a Delta table. The job processes data from a Kafka topic and uses a 10-minute watermark. After a few hours, the engineer notices that the streaming query's input rate is steady, but the processing rate has dropped significantly, and the batch duration has increased from 5 seconds to over 2 minutes. The job is running on a cluster with autoscaling enabled. Which action should the engineer take FIRST to diagnose the performance degradation?

Question 39mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer runs a nightly Databricks job that reads a large Delta table and writes aggregated results to another Delta table. The cluster logs show many small files in the source table, and the job runtime has increased steadily over weeks. The engineer wants to reduce the number of files without rewriting the entire table. Which command should be used?

Question 40hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer notices that a Databricks job processing a Delta table with 10,000 partitions runs slowly. The job filters on a column that is not the partition column, and the query plan shows that all partitions are being scanned. The engineer wants to improve performance without repartitioning the table. Which feature should be used?

Question 41easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is investigating a Databricks job that failed overnight. The job's status in the Jobs UI shows 'Failed', and the engineer needs to view the error message and stack trace to determine the cause. Where should the engineer look to find the detailed error information for the failed run?

Question 42mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer runs a nightly Databricks job that reads from a large Delta table and writes results to another Delta table. The job has been taking progressively longer each night. The engineer examines the Spark UI and sees that the 'SQL' tab shows a single stage with many small tasks, each processing very few records, and the 'Storage' tab shows that the source table has thousands of small files. Which action should the engineer take to improve performance?

Question 43hardmultiple choice

Study the full Python automation breakdown →

A data engineer is troubleshooting a Databricks job that intermittently fails with a `SparkException: Job aborted due to stage failure: Task not serializable`. The job reads from a Parquet file, performs a transformation using a custom function defined in a Python class, and writes to a Delta table. The engineer suspects that the custom function is causing the issue. Which action should the engineer take to resolve the serialization error?

Question 44easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is investigating why a Databricks job that reads from a Delta table is slow. The job performs a simple SELECT with a filter on a partition column. The engineer suspects that the table has many small files. Which Spark UI tab should be examined to confirm the number of files read?

Question 45easymultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is monitoring a Databricks job and notices that the job's duration has gradually increased over the past week. The job reads a large Delta table, performs aggregations, and writes results to another Delta table. The engineer wants to identify the stage that is taking the most time. Which Spark UI tab should the engineer use to quickly identify the slowest stage?

Question 46hardmulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is optimizing a Databricks job that performs a join between a large fact table and a small dimension table. The job is slow, and the engineer suspects that the join strategy is not optimal. Which TWO actions should the engineer take to improve performance? (Choose two.)

Question 47mediummulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is optimizing a Databricks job that processes a large dataset. The job performs a join between a large Delta table and a small dimension table, then writes the result to a Delta table. The engineer notices that the join is causing a large shuffle and wants to reduce shuffle overhead. Which two actions should the engineer take to improve performance? (Choose two.)

Question 48hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is optimizing a Databricks job that reads from a large Delta table and performs a join with a smaller table. The job is experiencing performance issues due to shuffling. The engineer wants to reduce the amount of data shuffled during the join. Which technique should the engineer use?

Question 49mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is using Databricks Jobs to run a nightly ETL pipeline. The job occasionally fails due to a transient network error when writing to an external database. The engineer wants to automatically retry the job a few times before marking it as failed. What is the most efficient way to configure this in Databricks?

Question 50mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is monitoring a Databricks job and notices that the job's tasks are spending a significant amount of time in garbage collection (GC). The job processes large amounts of data with many small objects. Which action should the engineer take to reduce GC overhead?

Question 51hardmulti select

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is investigating why a Databricks job that writes to a Delta table is experiencing performance degradation over time. The job performs frequent small appends. Which TWO actions should the engineer take to improve write performance? (Choose two.)

Question 52hardmultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer is monitoring a Databricks job that runs a Structured Streaming query. The engineer notices that the query's input rate is high, but the processing rate is low, and the batch duration is increasing over time. The query uses a Delta table as a source and writes to another Delta table. Which action should the engineer take to improve the streaming query's performance?

Question 53mediummultiple choice

Read the full Troubleshooting, Monitoring, and Optimization explanation →

A data engineer runs a nightly Databricks job that ingests data into a Delta table using multiple concurrent write streams. The engineer notices that some write transactions are failing with a ConcurrentAppendException. The job writes to the same partition of the table from different tasks. Which action should the engineer take to resolve this issue?