Troubleshooting practice questions

A data engineer is investigating a job failure that occurred only in the production environment. Which TWO features in Databricks help in comparing the production environment to the development environment?

Trap 1: Use the 'Cluster Logs' to compare the OS kernel versions of the…

Comparing OS kernel versions is rarely the solution for job failures, as Databricks manages the underlying runtime environment. If there is a disparity, it would be due to selecting different Databricks Runtime versions, not the specific OS kernel, which is abstracted away from the end user.

Trap 2: Enable the 'Debug Mode' on the Spark Driver to see raw system calls.

Debug mode for raw system calls is not a standard feature in Databricks and would not provide useful context for comparing environment configurations. This level of debugging is excessively low-level and does not address the high-level configuration differences that typically cause environment-specific pipeline failures.

Trap 3: Query the 'workspace_users' table to see who ran the job last.

Knowing who ran a job last does not help identify configuration differences. While auditing is useful for security, the technical investigation of why a job fails requires comparing settings, libraries, and code, not the identity of the user who triggered the last execution.

A
Use the 'View as JSON' feature in the Jobs UI to compare job configurations.
Exporting and comparing job JSON configurations is an effective way to identify discrepancies in parameters, cluster sizes, or timeout settings. This allows engineers to systematically check for configuration drift between the development workspace and the production workspace, which is a common source of environment-specific failures.
B
Use the 'Cluster Logs' to compare the OS kernel versions of the nodes.
Why it fails: Comparing OS kernel versions is rarely the solution for job failures, as Databricks manages the underlying runtime environment. If there is a disparity, it would be due to selecting different Databricks Runtime versions, not the specific OS kernel, which is abstracted away from the end user.
C
Review the git branch history and configuration files in the CI/CD pipeline.
Checking the CI/CD pipeline ensures that the exact code and configuration being deployed to production matches what was tested in development. This identifies issues where different branches were accidentally merged or where environment-specific variables were incorrectly injected into the production job, causing unexpected runtime behavior.
D
Enable the 'Debug Mode' on the Spark Driver to see raw system calls.
Why it fails: Debug mode for raw system calls is not a standard feature in Databricks and would not provide useful context for comparing environment configurations. This level of debugging is excessively low-level and does not address the high-level configuration differences that typically cause environment-specific pipeline failures.
E
Query the 'workspace_users' table to see who ran the job last.
Why it fails: Knowing who ran a job last does not help identify configuration differences. While auditing is useful for security, the technical investigation of why a job fails requires comparing settings, libraries, and code, not the identity of the user who triggered the last execution.

Question 2hardmultiple choice

An engineer notices that a SQL warehouse is frequently hitting 'Max Concurrency' limits. Which log should they consult to identify which specific queries are consuming most of the warehouse resources?

Trap 1: Cluster event logs

Cluster event logs record lifecycle events such as node starts, autoscaling and configuration changes, not per-query resource consumption. They suit diagnosing cluster-level failures. Identifying concurrency-heavy queries requires the SQL warehouse query history, which shows duration and resource usage per statement.

Trap 2: Job run history logs

Job run history logs are scoped to job tasks and do not capture individual queries submitted through the SQL warehouse interface by users or BI tools. Therefore, they are insufficient for debugging concurrency issues on SQL warehouses.

Trap 3: Delta table access logs

These logs are primarily used for security auditing to see who has accessed specific data files. They lack the performance and execution plan details necessary to debug compute resource consumption or concurrency bottlenecks in a SQL warehouse.

A
Cluster event logs
Why it fails: Cluster event logs record lifecycle events such as node starts, autoscaling and configuration changes, not per-query resource consumption. They suit diagnosing cluster-level failures. Identifying concurrency-heavy queries requires the SQL warehouse query history, which shows duration and resource usage per statement.
B
Query history system table
The query history system table provides comprehensive details about every query executed, including the duration, user, warehouse, and CPU/memory footprint. It is the most appropriate source for identifying heavy queries causing concurrency limits to be reached.
C
Job run history logs
Why it fails: Job run history logs are scoped to job tasks and do not capture individual queries submitted through the SQL warehouse interface by users or BI tools. Therefore, they are insufficient for debugging concurrency issues on SQL warehouses.
D
Delta table access logs
Why it fails: These logs are primarily used for security auditing to see who has accessed specific data files. They lack the performance and execution plan details necessary to debug compute resource consumption or concurrency bottlenecks in a SQL warehouse.

Question 3hardmulti select

A data engineer is troubleshooting a Databricks Workflow where a downstream task relies on an upstream task's output. Which TWO actions ensure the data dependency is correctly handled during a failure scenario?

Trap 1: Set the task timeout to zero to prevent the workflow from ever…

Setting a timeout to zero is not a valid configuration for ensuring data integrity during dependencies. It essentially disables the monitoring of task execution time, which can lead to hung jobs consuming resources indefinitely without triggering the necessary alerts or failure handling logic required for production stability.

Trap 2: Hardcode the file path of the upstream output into the downstream…

Hardcoding paths is an anti-pattern that couples tasks tightly and creates maintenance challenges. It ignores the workflow dependency graph, making it impossible for the scheduler to manage retries or state correctly. If the upstream task fails, the downstream task might still try to read non-existent data.

Trap 3: Disable all retries to ensure that the error log is captured…

Disabling retries does not assist in managing dependencies; it only guarantees the job stops at the first sign of trouble. While it might assist in capturing logs, it fails to utilize Databricks' built-in resilience features that handle transient errors automatically, thus reducing overall workflow reliability and efficiency.

A
Configure the downstream task to use the 'depends_on' attribute to reference the upstream task ID.
The 'depends_on' attribute explicitly defines the task DAG structure within Databricks Workflows. By creating this dependency, the scheduler guarantees that the downstream task will only execute if the upstream task succeeds, preventing erroneous runs when input data is missing or corrupted due to preceding failures.
B
Set the task timeout to zero to prevent the workflow from ever stopping during a failure.
Why it fails: Setting a timeout to zero is not a valid configuration for ensuring data integrity during dependencies. It essentially disables the monitoring of task execution time, which can lead to hung jobs consuming resources indefinitely without triggering the necessary alerts or failure handling logic required for production stability.
C
Enable the 'repair and rerun' feature to target only the failed tasks in the pipeline.
The repair and rerun feature allows engineers to execute only the failed tasks and their downstream dependents. This preserves the existing successful outputs of the workflow and ensures that the system only attempts to process data that was previously blocked by the specific task failure event.
D
Hardcode the file path of the upstream output into the downstream task configuration.
Why it fails: Hardcoding paths is an anti-pattern that couples tasks tightly and creates maintenance challenges. It ignores the workflow dependency graph, making it impossible for the scheduler to manage retries or state correctly. If the upstream task fails, the downstream task might still try to read non-existent data.
E
Disable all retries to ensure that the error log is captured immediately upon failure.
Why it fails: Disabling retries does not assist in managing dependencies; it only guarantees the job stops at the first sign of trouble. While it might assist in capturing logs, it fails to utilize Databricks' built-in resilience features that handle transient errors automatically, thus reducing overall workflow reliability and efficiency.

Question 4easymultiple choice

Where can a data engineer find the standard output and error logs for a specific task within a Databricks Workflow?

Trap 1: In the Databricks Filesystem (DBFS) root directory under /logs.

DBFS is a virtual file system and does not contain the structured execution logs for individual tasks. Storing application logs here would create significant management overhead and would not be easily accessible via the Job UI, making it a poor choice for diagnosing workflow failures in production.

Trap 2: By querying the 'sys.logs' table in the Unity Catalog.

There is no 'sys.logs' table in Unity Catalog. While audit logs exist for security monitoring, they do not contain the application-level stdout or stderr logs generated by task execution. Application logs remain within the compute resource and are accessed via the job run interface in the platform UI.

Trap 3: In the cluster configuration's 'Advanced Options' tab.

The 'Advanced Options' tab in cluster configuration is used to set Spark parameters, library paths, and environment variables. It does not provide access to the execution logs of past job runs. The settings here affect the environment in which code runs, rather than reporting on its past performance.

A
In the Databricks Filesystem (DBFS) root directory under /logs.
Why it fails: DBFS is a virtual file system and does not contain the structured execution logs for individual tasks. Storing application logs here would create significant management overhead and would not be easily accessible via the Job UI, making it a poor choice for diagnosing workflow failures in production.
B
By clicking the 'Logs' tab within the specific task run details in the Jobs UI.
The Jobs UI provides a 'Logs' tab that captures stdout, stderr, and log4j outputs for every task execution. This is the official, supported way to view execution logs within the Databricks workspace, allowing engineers to quickly debug failures without leaving the browser or using complex CLI commands.
C
By querying the 'sys.logs' table in the Unity Catalog.
Why it fails: There is no 'sys.logs' table in Unity Catalog. While audit logs exist for security monitoring, they do not contain the application-level stdout or stderr logs generated by task execution. Application logs remain within the compute resource and are accessed via the job run interface in the platform UI.
D
In the cluster configuration's 'Advanced Options' tab.
Why it fails: The 'Advanced Options' tab in cluster configuration is used to set Spark parameters, library paths, and environment variables. It does not provide access to the execution logs of past job runs. The settings here affect the environment in which code runs, rather than reporting on its past performance.

Question 5easymultiple choice

Which command should be used to display the history of transactions performed on a Delta table, including operations like overwrites and updates?

Trap 1: SHOW LOGS table_name

There is no command called 'SHOW LOGS' in Databricks SQL or Spark SQL. While the transaction log files exist in the _delta_log directory, users should never interact with them directly via SQL commands; instead, they should use the appropriate metadata commands provided by the engine.

Trap 2: SELECT * FROM metadata.table_name

There is no 'metadata' schema that stores user-accessible transaction history in this format. While the information is stored in the underlying delta log, querying it directly as a standard table is not the supported way to view operation history in Databricks.

Trap 3: EXPLAIN EXTENDED table_name

The EXPLAIN command is used to view the execution plan of a SQL query, not the historical modification log of a table. It provides insights into how the optimizer intends to execute a query, which is useful for performance tuning, not for auditing data changes.

A
SHOW LOGS table_name
Why it fails: There is no command called 'SHOW LOGS' in Databricks SQL or Spark SQL. While the transaction log files exist in the _delta_log directory, users should never interact with them directly via SQL commands; instead, they should use the appropriate metadata commands provided by the engine.
B
DESCRIBE HISTORY table_name
This command returns the complete history of operations performed on the Delta table, including timestamps, operation types, user information, and version numbers. It is the primary tool for auditing Delta Lake changes and facilitating data recovery through time travel to previous table versions.
C
SELECT * FROM metadata.table_name
Why it fails: There is no 'metadata' schema that stores user-accessible transaction history in this format. While the information is stored in the underlying delta log, querying it directly as a standard table is not the supported way to view operation history in Databricks.
D
EXPLAIN EXTENDED table_name
Why it fails: The EXPLAIN command is used to view the execution plan of a SQL query, not the historical modification log of a table. It provides insights into how the optimizer intends to execute a query, which is useful for performance tuning, not for auditing data changes.

Question 6hardmultiple choice

When troubleshooting a job that frequently crashes due to 'Out of Memory' (OOM) errors, which TWO metrics or logs should be analyzed?

Trap 1: System table audit logs.

Audit logs track user authentication and authorization events within the workspace. They do not contain technical details regarding memory allocation, execution plans, or garbage collection events, making them useless for diagnosing performance-related application crashes like OOM errors.

Trap 2: Workspace usage billing reports.

Billing reports provide insight into DBU usage and costs. They provide no technical observability into the runtime behavior, memory consumption, or stack traces of an application execution, and thus provide no value in troubleshooting an OOM exception.

Trap 3: The number of active sessions in the SQL warehouse.

SQL warehouse session counts relate to user concurrency and query queuing. While related to general system load, they provide no specific information on the memory management or heap usage of a specific job that is failing with OOM errors.

A
Driver and Executor logs.
Driver and executor logs contain the stack traces and error messages that specifically indicate memory exhaustion. Reviewing these logs allows the engineer to determine whether the error occurred during a shuffle operation, a broadcast join, or while loading a large data partition into memory.
B
System table audit logs.
Why it fails: Audit logs track user authentication and authorization events within the workspace. They do not contain technical details regarding memory allocation, execution plans, or garbage collection events, making them useless for diagnosing performance-related application crashes like OOM errors.
C
Spark UI Executor memory metrics.
The Spark UI provides real-time visualization of memory usage across all executors. Monitoring how memory is consumed during specific stages helps pinpoint if a memory-intensive transformation or a skew issue is the root cause of the crashes, enabling targeted code optimizations or configuration adjustments.
D
Workspace usage billing reports.
Why it fails: Billing reports provide insight into DBU usage and costs. They provide no technical observability into the runtime behavior, memory consumption, or stack traces of an application execution, and thus provide no value in troubleshooting an OOM exception.
E
The number of active sessions in the SQL warehouse.
Why it fails: SQL warehouse session counts relate to user concurrency and query queuing. While related to general system load, they provide no specific information on the memory management or heap usage of a specific job that is failing with OOM errors.

Question 7hardmultiple choice

A data engineer is troubleshooting a Delta Live Tables pipeline that intermittently fails with 'StreamingQueryException: Job aborted due to stage failure'. The pipeline processes streaming data from a Kafka source. Which monitoring approach will best help identify the root cause of these intermittent failures?

Trap 1: Set up a Databricks SQL alert on the target Delta table's row count…

A row count alert can indicate that data is not being written, but it does not explain why the pipeline failed. The intermittent nature of the failure requires access to error details, which a row count metric cannot provide. This method is reactive and does not help in diagnosing the underlying cause of the StreamingQueryException.

Trap 2: Use the Spark UI to inspect the DAG and stage details for each…

The Spark UI is valuable for performance tuning and understanding execution plans, but it may not retain logs for intermittent failures that occur after the cluster terminates. Delta Live Tables abstracts much of the Spark execution, and the event log is specifically designed to surface pipeline-level errors. The Spark UI alone would not provide the integrated error context needed.

Trap 3: Monitor the cluster's CPU and memory utilization metrics in the…

While resource metrics can indicate performance bottlenecks, they do not provide the detailed error information needed to diagnose a StreamingQueryException. The exception could be caused by data issues, network problems, or code bugs that are not reflected in CPU or memory usage. This approach lacks the specificity required for root cause analysis.

A
Set up a Databricks SQL alert on the target Delta table's row count to detect missing data.
Why it fails: A row count alert can indicate that data is not being written, but it does not explain why the pipeline failed. The intermittent nature of the failure requires access to error details, which a row count metric cannot provide. This method is reactive and does not help in diagnosing the underlying cause of the StreamingQueryException.
B
Use the Spark UI to inspect the DAG and stage details for each failed job run.
Why it fails: The Spark UI is valuable for performance tuning and understanding execution plans, but it may not retain logs for intermittent failures that occur after the cluster terminates. Delta Live Tables abstracts much of the Spark execution, and the event log is specifically designed to surface pipeline-level errors. The Spark UI alone would not provide the integrated error context needed.
C
Enable and analyze the Delta Live Tables event log for detailed error messages and stack traces.
The Delta Live Tables event log captures detailed information about pipeline runs, including error messages, stack traces, and data quality metrics. For intermittent streaming failures, this log provides the granular context needed to pinpoint the exact cause, such as deserialization errors or Kafka connectivity issues, making it the most effective monitoring tool for this scenario.
D
Monitor the cluster's CPU and memory utilization metrics in the Databricks workspace.
Why it fails: While resource metrics can indicate performance bottlenecks, they do not provide the detailed error information needed to diagnose a StreamingQueryException. The exception could be caused by data issues, network problems, or code bugs that are not reflected in CPU or memory usage. This approach lacks the specificity required for root cause analysis.

Question 8hardmulti select

A data engineer is troubleshooting a Databricks job that fails with a 'TaskFailed' error. The job uses a cluster with autoscaling enabled. The engineer suspects that the failure is due to memory issues on the workers. Which TWO actions should the engineer take to diagnose and resolve the issue? (Choose two.)

Trap 1: Increase the driver node's memory to handle the failing tasks.

The driver node manages the Spark application but does not execute tasks. Memory issues on workers are not resolved by increasing driver memory. While driver memory is important for coordination, the error suggests worker memory problems. This action would not address the root cause.

Trap 2: Increase the number of partitions when reading the data to reduce…

Increasing partitions can reduce per-task memory by processing smaller chunks, but it is not a direct diagnostic action and may not resolve the issue if the problem is skewed data or inefficient operations. It is a potential optimization but not one of the first two steps to diagnose memory issues. The question asks for actions to diagnose and resolve, but this is more of a tuning step after diagnosis.

Trap 3: Disable autoscaling to ensure the cluster has a fixed number of…

Disabling autoscaling would not directly address memory issues; it might even reduce available resources if the cluster scales down. Autoscaling helps handle variable workloads. The problem is likely per-executor memory, not the number of workers. Disabling autoscaling is not a diagnostic step and could worsen performance.

A
Increase the driver node's memory to handle the failing tasks.
Why it fails: The driver node manages the Spark application but does not execute tasks. Memory issues on workers are not resolved by increasing driver memory. While driver memory is important for coordination, the error suggests worker memory problems. This action would not address the root cause.
B
Increase the number of partitions when reading the data to reduce per-task memory usage.
Why it fails: Increasing partitions can reduce per-task memory by processing smaller chunks, but it is not a direct diagnostic action and may not resolve the issue if the problem is skewed data or inefficient operations. It is a potential optimization but not one of the first two steps to diagnose memory issues. The question asks for actions to diagnose and resolve, but this is more of a tuning step after diagnosis.
C
Disable autoscaling to ensure the cluster has a fixed number of workers.
Why it fails: Disabling autoscaling would not directly address memory issues; it might even reduce available resources if the cluster scales down. Autoscaling helps handle variable workloads. The problem is likely per-executor memory, not the number of workers. Disabling autoscaling is not a diagnostic step and could worsen performance.
D
Check the cluster's event log for 'ExecutorLostFailure' events indicating out-of-memory errors.
The cluster event log records executor losses and out-of-memory errors. 'ExecutorLostFailure' often indicates that an executor was killed due to memory limits. Reviewing the event log helps confirm if memory is the culprit and provides details such as which executor failed and why. This is a key diagnostic step.
E
Review the Spark UI for the job's run to identify stages with high garbage collection time or spill.
The Spark UI provides detailed metrics on memory usage, including garbage collection time and disk spill. High GC time or spill indicates memory pressure. By examining the stages, the engineer can pinpoint which operations are memory-intensive and adjust accordingly, such as increasing executor memory or optimizing the query.

Question 9hardmultiple choice

A data engineer is troubleshooting a production Databricks job that intermittently fails with 'SparkOutOfMemoryError'. The job processes large datasets with skewed partitions. The engineer wants to monitor the job to proactively detect memory pressure before failures occur. Which metric should the engineer monitor on the driver and executor nodes?

Trap 1: Disk I/O read/write throughput

Disk I/O throughput indicates how much data is being read from or written to disk. While excessive disk I/O can be a symptom of memory spilling, it does not directly measure memory pressure. High disk I/O could also result from other factors like large shuffles. Therefore, it is not the primary metric for detecting imminent OutOfMemoryError.

Trap 2: CPU utilization percentage

CPU utilization measures how busy the processors are. High CPU usage might indicate compute-intensive tasks but does not reveal memory constraints. A job can have low CPU utilization while still suffering from memory pressure due to large data structures. Therefore, CPU utilization is not the right metric for proactive memory monitoring.

Trap 3: Network throughput between executors

Network throughput shows data transfer during shuffles or broadcasts. While high network usage may correlate with large data movement, it does not indicate memory exhaustion. Memory pressure can occur even with low network activity. Thus, network throughput is not a reliable indicator of impending OutOfMemoryError.

A
JVM heap memory usage
JVM heap memory usage on driver and executor nodes directly reflects memory pressure. When heap usage approaches the maximum, Spark may spill to disk or throw OutOfMemoryError. Monitoring heap usage allows the engineer to detect when memory is nearly exhausted and take action, such as increasing memory or repartitioning data, before job failure occurs.
B
Disk I/O read/write throughput
Why it fails: Disk I/O throughput indicates how much data is being read from or written to disk. While excessive disk I/O can be a symptom of memory spilling, it does not directly measure memory pressure. High disk I/O could also result from other factors like large shuffles. Therefore, it is not the primary metric for detecting imminent OutOfMemoryError.
C
CPU utilization percentage
Why it fails: CPU utilization measures how busy the processors are. High CPU usage might indicate compute-intensive tasks but does not reveal memory constraints. A job can have low CPU utilization while still suffering from memory pressure due to large data structures. Therefore, CPU utilization is not the right metric for proactive memory monitoring.
D
Network throughput between executors
Why it fails: Network throughput shows data transfer during shuffles or broadcasts. While high network usage may correlate with large data movement, it does not indicate memory exhaustion. Memory pressure can occur even with low network activity. Thus, network throughput is not a reliable indicator of impending OutOfMemoryError.

Question 10hardmultiple choice

A data engineer is troubleshooting a Databricks job that fails with a `SparkException: Job aborted due to stage failure` and the error log shows `java.lang.OutOfMemoryError: GC overhead limit exceeded` on an executor. The job processes a large dataset using a `groupByKey` operation. Which action should the engineer take to resolve the issue while minimizing changes to the existing code?

Trap 1: Increase the executor memory by adding `spark.executor.memory=16g`…

Increasing executor memory may temporarily alleviate the symptom but does not fix the underlying inefficiency of `groupByKey`, which shuffles all values. It can lead to larger GC pauses and higher costs. The problem is algorithmic, not just resource-related. While it might delay the failure, it is not a robust solution and does not minimize changes to the logic.

Trap 2: Set `spark.sql.shuffle.partitions` to a higher value to increase…

Increasing shuffle partitions can improve parallelism but does not reduce the total shuffle volume or memory per key. With `groupByKey`, all values for a key still go to one partition, so a hot key can still cause OOM. This setting may help with overall performance but does not address the specific GC overhead caused by large per-key value lists.

Trap 3: Switch the job to use a larger cluster with more worker nodes to…

Adding more nodes increases parallelism but does not reduce the data shuffled by `groupByKey`. Each executor still receives a large amount of data for a single key, potentially causing OOM on individual executors. The issue is data skew and lack of local aggregation, not insufficient cluster size. This approach is costly and may not resolve the error.

A
Increase the executor memory by adding `spark.executor.memory=16g` to the cluster Spark configuration.
Why it fails: Increasing executor memory may temporarily alleviate the symptom but does not fix the underlying inefficiency of `groupByKey`, which shuffles all values. It can lead to larger GC pauses and higher costs. The problem is algorithmic, not just resource-related. While it might delay the failure, it is not a robust solution and does not minimize changes to the logic.
B
Set `spark.sql.shuffle.partitions` to a higher value to increase the number of partitions after shuffle.
Why it fails: Increasing shuffle partitions can improve parallelism but does not reduce the total shuffle volume or memory per key. With `groupByKey`, all values for a key still go to one partition, so a hot key can still cause OOM. This setting may help with overall performance but does not address the specific GC overhead caused by large per-key value lists.
C
Replace `groupByKey` with `reduceByKey` to reduce memory pressure by combining values locally before shuffling.
`groupByKey` shuffles all values for a key without local aggregation, which can cause excessive memory usage and GC overhead. `reduceByKey` performs map-side combining, reducing the amount of data shuffled and lowering memory pressure. This change directly addresses the root cause while requiring minimal code modification, as both are transformations on key-value pairs. It is the most effective fix for the described symptom.
D
Switch the job to use a larger cluster with more worker nodes to distribute the load.
Why it fails: Adding more nodes increases parallelism but does not reduce the data shuffled by `groupByKey`. Each executor still receives a large amount of data for a single key, potentially causing OOM on individual executors. The issue is data skew and lack of local aggregation, not insufficient cluster size. This approach is costly and may not resolve the error.

Question 11hardmultiple choice

A data engineer is troubleshooting a Databricks SQL query that occasionally fails with 'Query exceeded the maximum allowed execution time' on a shared SQL warehouse. The query is a complex aggregation over a large Delta table. The engineer needs to identify the root cause and ensure the query can complete successfully. Which action should the engineer take first?

Trap 1: Increase the SQL warehouse size to provide more compute resources…

Increasing warehouse size may help with resource contention but does not address underlying inefficiencies such as data skew or inefficient joins. It may also increase cost. The engineer should first diagnose the root cause using the query profile before scaling up, as scaling may not resolve the issue if the query is inherently inefficient.

Trap 2: Set the Spark configuration 'spark.sql.adaptive.enabled' to false…

Disabling adaptive query execution would likely worsen performance, as AQE dynamically optimizes query plans based on runtime statistics. It is not a recommended troubleshooting step. The default is enabled, and turning it off could lead to less efficient execution plans and longer runtimes.

Trap 3: Change the query to use a larger cluster by switching from a SQL…

Switching to an all-purpose cluster changes the execution environment and may not be cost-effective or aligned with the SQL warehouse usage. It also does not guarantee a fix if the query itself is inefficient. The first step should be to analyze the query profile to understand the bottleneck, rather than changing the compute platform.

A
Increase the SQL warehouse size to provide more compute resources for the query.
Why it fails: Increasing warehouse size may help with resource contention but does not address underlying inefficiencies such as data skew or inefficient joins. It may also increase cost. The engineer should first diagnose the root cause using the query profile before scaling up, as scaling may not resolve the issue if the query is inherently inefficient.
B
Set the Spark configuration 'spark.sql.adaptive.enabled' to false to disable adaptive query execution.
Why it fails: Disabling adaptive query execution would likely worsen performance, as AQE dynamically optimizes query plans based on runtime statistics. It is not a recommended troubleshooting step. The default is enabled, and turning it off could lead to less efficient execution plans and longer runtimes.
C
Examine the query profile in the Databricks SQL query history to identify stages with high data skew or spill.
The query profile provides detailed execution metrics, including time spent per stage, data skew, and spill to disk. These insights help pinpoint why the query exceeds the time limit, such as an inefficient join or skewed data distribution. Addressing these issues can allow the query to complete within the limit.
D
Change the query to use a larger cluster by switching from a SQL warehouse to an all-purpose cluster.
Why it fails: Switching to an all-purpose cluster changes the execution environment and may not be cost-effective or aligned with the SQL warehouse usage. It also does not guarantee a fix if the query itself is inefficient. The first step should be to analyze the query profile to understand the bottleneck, rather than changing the compute platform.

Question 12mediummulti select

A data engineer is preparing to deploy a production Databricks Workflow that must be maintainable and auditable. The engineer wants to ensure that changes to the workflow are tracked and that failures can be diagnosed quickly. Which TWO practices should the engineer implement? (Choose two.)

Trap 1: Manually document each deployment in a shared spreadsheet

Manual documentation is error-prone and does not integrate with deployment automation or provide a reliable audit trail. It cannot enforce consistency between code and deployment. Automated version control and log retention are superior practices for maintainability and auditability, making manual spreadsheets inadequate for production workflows.

Trap 2: Use an all-purpose cluster instead of a job cluster for the workflow

All-purpose clusters are intended for interactive analysis and are more expensive and less isolated than job clusters. They do not inherently improve maintainability or auditability of workflow definitions. Production workflows should use job clusters for cost efficiency and isolation, while version control and logging address the stated requirements.

Trap 3: Configure email notifications for job success and failure

Email notifications alert recipients to job status but do not provide version control or detailed diagnostic information. They are useful for awareness but insufficient for auditing changes or deep debugging. The engineer should focus on versioned deployment and centralized logging to meet the stated requirements.

A
Manually document each deployment in a shared spreadsheet
Why it fails: Manual documentation is error-prone and does not integrate with deployment automation or provide a reliable audit trail. It cannot enforce consistency between code and deployment. Automated version control and log retention are superior practices for maintainability and auditability, making manual spreadsheets inadequate for production workflows.
B
Use an all-purpose cluster instead of a job cluster for the workflow
Why it fails: All-purpose clusters are intended for interactive analysis and are more expensive and less isolated than job clusters. They do not inherently improve maintainability or auditability of workflow definitions. Production workflows should use job clusters for cost efficiency and isolation, while version control and logging address the stated requirements.
C
Configure email notifications for job success and failure
Why it fails: Email notifications alert recipients to job status but do not provide version control or detailed diagnostic information. They are useful for awareness but insufficient for auditing changes or deep debugging. The engineer should focus on versioned deployment and centralized logging to meet the stated requirements.
D
Store the workflow definition in a Git repository and deploy using Databricks Asset Bundles
Version-controlling workflow definitions in Git and deploying via Databricks Asset Bundles provides change tracking, code review, and reproducible deployments. This practice ensures that every modification is auditable and that the deployed workflow matches the reviewed source, which is essential for maintainability and compliance in production environments.
E
Enable job run logs to be delivered to a cloud storage location for long-term retention
Delivering job logs to cloud storage ensures that diagnostic data is retained beyond the default retention period and is accessible for post-mortem analysis. This supports quick diagnosis of failures and provides an audit trail of job executions, complementing version-controlled deployment for maintainability.