Troubleshooting Scenario Questions

Set the task timeout to zero to prevent the workflow from ever stopping during a failure.

Why it fails: Setting a timeout to zero is not a valid configuration for ensuring data integrity during dependencies. It essentially disables the monitoring of task execution time, which can lead to hung jobs consuming resources indefinitely without triggering the necessary alerts or failure handling logic required for production stability.

Hardcode the file path of the upstream output into the downstream task configuration.

Why it fails: Hardcoding paths is an anti-pattern that couples tasks tightly and creates maintenance challenges. It ignores the workflow dependency graph, making it impossible for the scheduler to manage retries or state correctly. If the upstream task fails, the downstream task might still try to read non-existent data.

Disable all retries to ensure that the error log is captured immediately upon failure.

Why it fails: Disabling retries does not assist in managing dependencies; it only guarantees the job stops at the first sign of trouble. While it might assist in capturing logs, it fails to utilize Databricks' built-in resilience features that handle transient errors automatically, thus reducing overall workflow reliability and efficiency.

System table audit logs.

Why it fails: Audit logs track user authentication and authorization events within the workspace. They do not contain technical details regarding memory allocation, execution plans, or garbage collection events, making them useless for diagnosing performance-related application crashes like OOM errors.

Workspace usage billing reports.

Why it fails: Billing reports provide insight into DBU usage and costs. They provide no technical observability into the runtime behavior, memory consumption, or stack traces of an application execution, and thus provide no value in troubleshooting an OOM exception.

The number of active sessions in the SQL warehouse.

Why it fails: SQL warehouse session counts relate to user concurrency and query queuing. While related to general system load, they provide no specific information on the memory management or heap usage of a specific job that is failing with OOM errors.

Set up a Databricks SQL alert on the target Delta table's row count to detect missing data.

Why it fails: A row count alert can indicate that data is not being written, but it does not explain why the pipeline failed. The intermittent nature of the failure requires access to error details, which a row count metric cannot provide. This method is reactive and does not help in diagnosing the underlying cause of the StreamingQueryException.

Use the Spark UI to inspect the DAG and stage details for each failed job run.

Why it fails: The Spark UI is valuable for performance tuning and understanding execution plans, but it may not retain logs for intermittent failures that occur after the cluster terminates. Delta Live Tables abstracts much of the Spark execution, and the event log is specifically designed to surface pipeline-level errors. The Spark UI alone would not provide the integrated error context needed.

Monitor the cluster's CPU and memory utilization metrics in the Databricks workspace.

Why it fails: While resource metrics can indicate performance bottlenecks, they do not provide the detailed error information needed to diagnose a StreamingQueryException. The exception could be caused by data issues, network problems, or code bugs that are not reflected in CPU or memory usage. This approach lacks the specificity required for root cause analysis.

Job notifications configured on the DLT pipeline's underlying job.

Why it fails: Job notifications can alert on pipeline failure, but they do not provide granular alerts on individual data quality expectations. The requirement is to be alerted when the pipeline fails to meet data quality expectations, which requires access to expectation metrics. Job notifications alone are insufficient.

Databricks SQL dashboard with a query on the target Delta table.

Why it fails: A SQL dashboard can visualize data in the target table, but it does not capture DLT expectation results or pipeline health metrics. It would only show the final data, not the quality checks. This approach lacks the necessary context to alert on data quality expectations.

Cluster metrics in the Databricks workspace.

Why it fails: Cluster metrics provide information about resource utilization, such as CPU and memory, but they do not include data quality metrics or expectation outcomes. They are useful for performance monitoring but cannot alert on data quality failures specific to DLT expectations.

Increase the driver node's memory to handle the failing tasks.

Why it fails: The driver node manages the Spark application but does not execute tasks. Memory issues on workers are not resolved by increasing driver memory. While driver memory is important for coordination, the error suggests worker memory problems. This action would not address the root cause.

Increase the number of partitions when reading the data to reduce per-task memory usage.

Why it fails: Increasing partitions can reduce per-task memory by processing smaller chunks, but it is not a direct diagnostic action and may not resolve the issue if the problem is skewed data or inefficient operations. It is a potential optimization but not one of the first two steps to diagnose memory issues. The question asks for actions to diagnose and resolve, but this is more of a tuning step after diagnosis.

Disable autoscaling to ensure the cluster has a fixed number of workers.

Why it fails: Disabling autoscaling would not directly address memory issues; it might even reduce available resources if the cluster scales down. Autoscaling helps handle variable workloads. The problem is likely per-executor memory, not the number of workers. Disabling autoscaling is not a diagnostic step and could worsen performance.

Disk I/O read/write throughput

Why it fails: Disk I/O throughput indicates how much data is being read from or written to disk. While excessive disk I/O can be a symptom of memory spilling, it does not directly measure memory pressure. High disk I/O could also result from other factors like large shuffles. Therefore, it is not the primary metric for detecting imminent OutOfMemoryError.

CPU utilization percentage

Why it fails: CPU utilization measures how busy the processors are. High CPU usage might indicate compute-intensive tasks but does not reveal memory constraints. A job can have low CPU utilization while still suffering from memory pressure due to large data structures. Therefore, CPU utilization is not the right metric for proactive memory monitoring.

Network throughput between executors

Why it fails: Network throughput shows data transfer during shuffles or broadcasts. While high network usage may correlate with large data movement, it does not indicate memory exhaustion. Memory pressure can occur even with low network activity. Thus, network throughput is not a reliable indicator of impending OutOfMemoryError.

Increase the executor memory by adding `spark.executor.memory=16g` to the cluster Spark configuration.

Why it fails: Increasing executor memory may temporarily alleviate the symptom but does not fix the underlying inefficiency of `groupByKey`, which shuffles all values. It can lead to larger GC pauses and higher costs. The problem is algorithmic, not just resource-related. While it might delay the failure, it is not a robust solution and does not minimize changes to the logic.

Set `spark.sql.shuffle.partitions` to a higher value to increase the number of partitions after shuffle.

Why it fails: Increasing shuffle partitions can improve parallelism but does not reduce the total shuffle volume or memory per key. With `groupByKey`, all values for a key still go to one partition, so a hot key can still cause OOM. This setting may help with overall performance but does not address the specific GC overhead caused by large per-key value lists.

Switch the job to use a larger cluster with more worker nodes to distribute the load.

Why it fails: Adding more nodes increases parallelism but does not reduce the data shuffled by `groupByKey`. Each executor still receives a large amount of data for a single key, potentially causing OOM on individual executors. The issue is data skew and lack of local aggregation, not insufficient cluster size. This approach is costly and may not resolve the error.

Increase the SQL warehouse size to provide more compute resources for the query.

Why it fails: Increasing warehouse size may help with resource contention but does not address underlying inefficiencies such as data skew or inefficient joins. It may also increase cost. The engineer should first diagnose the root cause using the query profile before scaling up, as scaling may not resolve the issue if the query is inherently inefficient.

Set the Spark configuration 'spark.sql.adaptive.enabled' to false to disable adaptive query execution.

Why it fails: Disabling adaptive query execution would likely worsen performance, as AQE dynamically optimizes query plans based on runtime statistics. It is not a recommended troubleshooting step. The default is enabled, and turning it off could lead to less efficient execution plans and longer runtimes.

Change the query to use a larger cluster by switching from a SQL warehouse to an all-purpose cluster.

Why it fails: Switching to an all-purpose cluster changes the execution environment and may not be cost-effective or aligned with the SQL warehouse usage. It also does not guarantee a fix if the query itself is inefficient. The first step should be to analyze the query profile to understand the bottleneck, rather than changing the compute platform.