Databricks-DE-Assoc Troubleshooting, Monitoring, and Optimization Practice Test 2 — 15 Questions

Question 1

Which Databricks command should you use to recover storage space by removing files that are no longer referenced by a Delta table and are older than the retention period?

Accepted Answer

VACUUM table_name RETAIN 30 DAYS;. VACUUM is the standard tool for cleaning up orphan files in Delta Lake. Because Delta Lake uses versioning, old files are kept for 'Time Travel' functionality until specifically vacuumed. Managing this storage is essential for cost control and compliance with data retention policies. Engineers must understand that once VACUUM is run, the history of the table is permanently pruned, making point-in-time recovery to those older versions impossible.

Answer

DELETE FROM table_name WHERE date < current_date() - 30;

Answer

TRUNCATE TABLE table_name;

Answer

DROP TABLE table_name;

Question 2

Which tool in Databricks provides real-time monitoring of cluster resource usage, including CPU, memory, and network throughput, for an active job?

Accepted Answer

The Ganglia UI.. The Metrics tab in the Databricks cluster UI offers a visual dashboard showing resource utilization. Monitoring these metrics helps engineers identify if a job is CPU-bound, memory-bound, or network-bound. This visibility is crucial for rightsizing clusters. By observing these trends, engineers can optimize costs by reducing cluster size for underutilized jobs or improve performance by scaling up resources for jobs that encounter significant hardware contention.

Answer

The Spark UI.

Answer

The Query Profile.

Answer

The Databricks Job Run History.

Question 3

When configuring a Databricks Job, which TWO factors most directly influence the choice between a 'Job Cluster' and an 'All-Purpose Cluster'?

Accepted Answer

The need for interactive debugging and notebook testing.. Choosing the right cluster type is vital for balancing cost and performance. Job Clusters are ephemeral, optimized for production tasks, and significantly cheaper, whereas All-Purpose Clusters are persistent and intended for interactive development. Understanding the cost implications and lifecycle management of these clusters is fundamental for any Databricks engineer responsible for managing operational budgets and ensuring efficient resource utilization across various environments.

Answer

The requirement for high availability during query execution.

Answer

The number of users concurrently using the cluster.

Answer

The type of data source (e.g., S3 vs. ADLS).

Question 4

Which Spark configuration property can be used to enable Adaptive Query Execution (AQE) in Databricks?

Accepted Answer

spark.sql.adaptive.enabled. Adaptive Query Execution (AQE) is a critical optimization framework in Spark that re-optimizes query plans at runtime based on statistics collected during shuffle stages. It enables features like coalescing shuffle partitions and dynamically switching join strategies. Enabling AQE is a best practice for modern Databricks workloads as it provides significant performance gains without manual intervention, automatically adapting to varying data distributions in production pipelines.

Answer

spark.sql.autoBroadcastJoinThreshold

Answer

spark.databricks.adaptive.execution

Answer

spark.sql.shuffle.partitions

Question 5

Which THREE strategies are recommended to improve the performance of reading from a Delta table in Databricks?

Accepted Answer

Use Z-Ordering on columns frequently used in WHERE clauses.. Optimizing reads is crucial for analytics. Techniques like Z-Ordering, Data Skipping, and Partitioning allow Spark to ignore irrelevant data files, reducing I/O. Proper file sizing ensures that Spark can efficiently read data in parallel. These strategies together minimize the amount of data scanned and transferred across the network, leading to significantly faster query results in large-scale data lake environments.

Answer

Partition the table by every column used in the query.

Answer

Always set the broadcast join threshold to -1.

Question 6

A data engineer wants to monitor the health and performance of Databricks Jobs over time. Which feature should they use to visualize trends, such as job success rates and average execution times, across multiple runs?

Accepted Answer

The Jobs Run History dashboard.. The Databricks Jobs UI provides a comprehensive view of job history, which is essential for identifying long-term performance trends and reliability issues. By analyzing these metrics, engineers can detect regressions, troubleshoot intermittent failures, and optimize costs by adjusting schedules or resources. This historic view is a cornerstone of operational maintenance in Databricks, ensuring that pipelines remain stable and performant as data volumes evolve.

Answer

The Spark SQL UI.

Answer

The Delta Lake Time Travel feature.

Answer

Cluster Ganglia Metrics.

Question 7

A data engineer is optimizing a Delta table that suffers from slow read performance due to small file sizes. Which command should the engineer execute to consolidate these small files into larger, more efficient files without altering the underlying table data?

Accepted Answer

OPTIMIZE table_name. The OPTIMIZE command is specifically designed for compacting small files into larger ones, which improves read performance by reducing metadata overhead and increasing scan efficiency. This operation is essential in Databricks environments where frequent streaming or batch writes create many small files, leading to 'small file syndrome.' By restructuring the data files while maintaining the table's logical state, OPTIMIZE ensures that downstream queries scan fewer, more optimized data blocks.

Answer

ALTER TABLE table_name REORGANIZE

Answer

VACUUM table_name

Answer

ANALYZE TABLE table_name COMPUTE STATISTICS

Question 8

Refer to the exhibit. A job fails with the provided error message. What is the most likely cause of this failure in a Databricks Delta Lake environment?

Accepted Answer

The VACUUM retention threshold was set too low.. This error occurs when the Delta log entries point to files that no longer exist, typically caused by a VACUUM operation running with a retention period shorter than the time required for long-running streaming jobs to finish. When a file is physically deleted by VACUUM, but a reader still references it via the transaction log, the read operation fails because the file object is inaccessible in the underlying storage.

Answer

The Delta table has reached the maximum file limit.

Answer

The cluster running the job lacks proper IAM permissions.

Answer

The transaction log is corrupted and requires manual deletion.

Question 9

A data engineer is troubleshooting a slow-running Spark job on Databricks. Which TWO metrics in the Spark UI are most useful for identifying data skew?

Accepted Answer

Task Duration (max vs median). Identifying data skew requires looking for significant disparities in task execution times and data distribution across partitions. When one or two tasks take drastically longer than the others while processing similar data volumes, or when partition sizes are highly uneven, skew is likely present. Monitoring these metrics allows engineers to implement techniques like salting or broadcast joins to redistribute the data and balance the workload across executors.

Answer

Executor CPU Usage

Answer

Total Cluster Memory Usage

Answer

Driver Memory Consumption

Question 10

An engineer notices that a specific notebook job is consistently taking longer to start. They observe high 'initialization' times in the job logs. Which action should the engineer take to improve startup time?

Accepted Answer

Use a Databricks Pool for the job cluster.. Initialization time in Databricks jobs often stems from the time required to pull container images, install cluster-scoped libraries, or initialize the Spark context on a new cluster. By using an existing cluster (pool) or pre-warming the cluster, the overhead of provisioning hardware and installing dependencies is removed. This optimization is crucial for meeting strict SLAs in production pipelines where every minute of latency impacts downstream processes.

Answer

Enable Auto-scaling on the cluster.

Answer

Increase the number of worker nodes.

Answer

Update the notebook code to use RDDs.

Question 11

A data engineer is analyzing a Spark job that is failing with 'Out of Memory' (OOM) errors. Which configuration parameter should be tuned to increase the amount of memory allocated to the execution of joins and aggregations?

Accepted Answer

spark.memory.fraction. Spark's memory management divides heap memory into storage and execution. Joins and aggregations occur in the execution memory. When these operations process large datasets that exceed the allocated execution memory, Spark may fail with OOM. Tuning the memory fraction configuration allows the engineer to shift the balance between storage (for caching) and execution, providing more headroom for complex shuffle-heavy operations like joins and group-by aggregations.

Answer

spark.executor.cores

Answer

spark.sql.shuffle.partitions

Answer

spark.driver.memory

Question 12

Which action is recommended to resolve a scenario where a Databricks Job is failing due to excessive metadata operations on a Delta table with millions of files?

Accepted Answer

Run OPTIMIZE to consolidate files.. When a Delta table contains millions of small files, the transaction log and metadata operations become a bottleneck. The 'list' operations required to build the state of the table consume significant time and driver memory. Implementing partition pruning or using Delta Lake's table property 'delta.enableChangeDataFeed' are not the primary solutions here. Instead, running OPTIMIZE to consolidate files is the standard way to reduce metadata overhead and improve table performance.

Answer

Increase the driver node instance size.

Answer

Disable the Delta transaction log.

Answer

Add more worker nodes to the cluster.

Question 13

Refer to the exhibit. An engineer applies these configurations to a cluster. What is the primary benefit of enabling the Databricks IO Cache for a workload that involves repeatedly reading the same Delta tables?

Accepted Answer

It speeds up repeated reads by caching data on local SSDs.. The Databricks IO Cache (also known as the disk cache) accelerates data reads by caching remote data on the local SSDs of the worker nodes. For workloads that frequently query the same tables, this eliminates the latency and network overhead of repeatedly fetching data from cloud object storage. This is particularly effective for read-heavy analytical workloads, enabling significantly faster query execution times by leveraging high-speed local disk I/O.

Answer

It enables ACID transactions for non-Delta tables.

Answer

It automatically scales the cluster based on disk usage.

Answer

It forces the cluster to store all data in memory.

Question 14

A data engineer needs to troubleshoot a job that is failing during the 'shuffle' phase. Which Spark UI tab should the engineer examine to analyze the shuffle partitions and identify potential imbalances?

Accepted Answer

The 'Stages' tab.. The 'Stages' tab in the Spark UI is the primary place to analyze shuffle performance. Within this tab, an engineer can view the 'Shuffle Read' and 'Shuffle Write' metrics for each task. By examining the partition-level distribution, the engineer can detect if the data is being shuffled unevenly, which is a common cause of stage-level bottlenecks and job failures during complex transformations like joins or window functions.

Answer

The 'Executors' tab.

Answer

The 'SQL' tab.

Answer

The 'Environment' tab.

Question 15

A data engineer notices that a production Delta Lake table is experiencing slow read performance during concurrent write operations. The table contains millions of small files. Which action should the engineer take to resolve this performance degradation?

Accepted Answer

Execute the OPTIMIZE command on the table.. Compacting small files into larger files using the OPTIMIZE command reduces metadata overhead and improves I/O efficiency. This is a critical maintenance task in Databricks because small files cause excessive object storage requests and metadata listing latency. By consolidating these files, the engine can scan data much more effectively during read operations, even while concurrent writes are occurring, as Delta Lake ensures ACID transactions remain consistent and performant.

Answer

Run the VACUUM command on the table.

Answer

Increase the number of worker nodes in the cluster.

Answer

Enable Z-Ordering on all columns in the table.