Databricks-DE-Pro Monitoring and Alerting — 20 Questions

Question 1

A Data Engineer is configuring alerts for a critical production job. Which TWO actions are required to successfully set up an email notification for a job failure?

Accepted Answer

Configure the 'Notifications' section in the Job settings to include the email address.. In Databricks, job notifications are configured directly in the 'Notifications' section of the Job settings. You do not need to add an 'on_failure' trigger to the Jobs API payload for a specific task to enable email notifications; the Job-level notification settings handle this. Option C is incorrect because task-level triggers are not the standard way to configure email alerts for job failures.

Answer

Create an Alert object in the SQL workspace linked to the job ID.

Answer

Add the 'on_failure' trigger to the Jobs API payload for the specific task.

Answer

Set up a System Table audit log trigger to monitor job state changes.

Answer

Enable the 'Notify on task failure' checkbox in the cluster configuration.

Question 2

Which TWO of the following are valid destinations for Databricks SQL alerts?

Accepted Answer

Webhooks. Databricks SQL alerts are highly flexible and support multiple notification channels to integrate with existing DevOps workflows. Choosing the right destination—like a webhook for custom integrations or email for general communication—is vital for ensuring the right teams are alerted. This flexibility allows engineers to build highly responsive incident response pipelines, significantly reducing downtime and ensuring that data quality or performance issues are addressed immediately.

Answer

Direct database table updates

Answer

Direct user push notifications on the Databricks mobile app.

Answer

Automatic cluster restart triggers.

Question 3

Refer to the exhibit. If a job is failing because the cluster is reaching the 'max_workers' limit too quickly, which monitoring metric should the engineer verify to confirm if the cluster is actually utilizing these nodes efficiently?

Accepted Answer

Task-level execution time distribution. Verifying 'task_skew' and 'executor_utilization' is key. If the cluster is scaling to 8 nodes but still failing, the issue may not be a lack of resources, but rather inefficient data distribution (skew) or task overhead. Analyzing these metrics ensures that engineers don't simply increase the cluster size, which would increase costs without solving the underlying performance issues caused by unbalanced workloads.

Answer

Total number of users logged into the workspace.

Answer

The total number of files in the Delta table.

Answer

The memory overhead of the driver node.

Question 4

A data engineer is using the Databricks REST API to retrieve the event log for a specific job run. They need to fetch only the events that occurred after a certain timestamp. Which query parameter should they use?

Accepted Answer

order_by. The Databricks REST API for job run events does not support a direct timestamp filter. To fetch events after a certain timestamp, the engineer must use the order_by parameter to sort events by timestamp, then paginate through the results and filter client-side. The order_by parameter is essential for ensuring events are returned in a predictable order, enabling efficient client-side filtering. Other parameters like offset and limit are for pagination but do not filter by time.

Answer

timestamp

Answer

start_time

Answer

offset

Question 5

A data engineer runs a nightly production job on a job cluster using Databricks Runtime 14.3 LTS. The job has been intermittently failing for three days with 'Cluster terminated due to spot instance loss' in the event log, but the driver logs show no application errors. The engineer needs to reduce these failures without changing the job's runtime behavior or increasing cost significantly. Which action should the engineer take?

Accepted Answer

Configure the cluster with a mix of on-demand and spot instances using the 'First on-demand' or 'Spot with fallback' allocation strategy for the worker nodes.. The job failures are caused by spot instance reclamation, which terminates the cluster. The most effective mitigation without changing runtime behavior or greatly increasing cost is to use an allocation strategy that mixes on-demand and spot instances, such as 'First on-demand' or 'Spot with fallback'. This keeps the driver on reliable capacity and allows workers to be replaced automatically when spot instances are lost.

Answer

Enable the 'spot instance fallback' option by setting the cluster's Spark configuration 'spark.databricks.cluster.profile' to 'spot'.

Answer

Increase the cluster's autoscaling maximum workers so that replacement spot instances can be added more quickly.

Answer

Switch the entire cluster to on-demand instances and enable autoscaling to compensate for the higher cost.

Question 6

A data engineering team runs a nightly production job on Databricks. The job has been failing intermittently with a 'Cluster terminated due to idle timeout' error. The team wants to be proactively notified when the cluster is about to terminate due to inactivity so they can investigate. Which approach should they use?

Accepted Answer

Use the cluster event log to detect idle timeout events and set up a notification via a job or external monitoring.. The cluster event log is the built-in mechanism that records cluster lifecycle events, including notifications before idle termination. By monitoring this log, teams can set up alerts to act before the cluster shuts down. Other options either target the wrong monitoring surface or do not provide the required proactive notification for idle timeout.

Answer

Create a Databricks job that runs every minute and checks the cluster status via the REST API.

Answer

Configure a Databricks SQL alert on the cluster's idle timeout metric.

Answer

Enable autoscaling on the cluster and set a minimum number of workers to prevent idle timeout.

Question 7

A data engineer has deployed a Delta Live Tables pipeline that ingests streaming data from a cloud storage location. The pipeline is configured with expectations to drop invalid records. The engineer needs to monitor the number of records that were dropped due to failed expectations over time. Which approach should the engineer use to track this metric?

Accepted Answer

Query the Delta Live Tables event log for 'flow_progress' events and extract the 'dropped_records' field.. Delta Live Tables automatically logs detailed metrics to the event log, including the number of records dropped by expectations. Querying the event log for flow progress events and extracting the dropped records field provides a reliable, built-in way to monitor this metric over time without custom code or additional tables.

Answer

Configure the pipeline to write dropped records to a separate Delta table and set up a Databricks job to count them hourly.

Answer

Enable Databricks SQL alerts on the target Delta table to count rows where the expectation column is null.

Answer

Use the Databricks REST API to retrieve pipeline metrics and filter for the 'num_dropped_records' field.

Question 8

A data engineer needs to monitor the performance of a Databricks SQL warehouse and receive alerts when query latency exceeds a threshold. The warehouse is used by multiple teams for interactive queries. Which monitoring solution should the engineer implement?

Accepted Answer

Configure a Databricks SQL alert on the query history table to trigger when average latency exceeds the threshold.. Databricks SQL alerts can be created directly on queries against the query history system table, allowing for precise monitoring of query latency. This native feature simplifies alert configuration and integrates with notification destinations. Other methods either lack the necessary metrics, require custom code, or monitor the wrong layer of the stack.

Answer

Enable audit logs and analyze them for slow queries using a scheduled notebook.

Answer

Set up a cluster-level metric alert in the cloud provider's monitoring service for the SQL warehouse's underlying VMs.

Answer

Use the Databricks REST API to poll the warehouse's metrics endpoint and send notifications via a custom script.

Question 9

A data engineer is responsible for a Delta Live Tables pipeline that processes sensitive customer data. They need to monitor the pipeline for data quality issues and ensure that records failing expectations are tracked. They also want to receive alerts when the pipeline fails. Which TWO actions should the engineer take to achieve these goals? (Choose two.)

Accepted Answer

Define expectations in the pipeline using the @dlt.expect decorator and set the pipeline to drop or quarantine invalid records.. To monitor data quality in a DLT pipeline, the engineer should define expectations with @dlt.expect and query the event log for data_quality events. These two actions provide both the mechanism to capture quality metrics and the ability to analyze and alert on them. Together, they enable comprehensive monitoring of expectation failures and pipeline health.

Answer

Create a Databricks job that runs the pipeline and configure job alerts to notify on failure.

Answer

Enable cluster logging to a cloud storage location and parse logs for 'Expectation failed' messages.

Answer

Configure a Databricks SQL alert on the target table to count rows and alert if the count drops below a threshold.

Question 10

A data engineer is responsible for a mission-critical Delta Live Tables pipeline that must process data continuously. The engineer needs to configure a robust alerting strategy to ensure immediate notification if the pipeline fails or experiences significant delays. Which approach best meets this requirement?

Accepted Answer

Configure a Databricks SQL alert on the pipeline's event log to trigger when errors occur, and set the alert to notify a Slack channel.. A Databricks SQL alert on the pipeline's event log provides a flexible and proactive way to monitor for errors and delays. Routing notifications to Slack ensures the team is immediately aware. Built-in email notifications are limited, manual monitoring is unreliable, and cloud provider tools lack pipeline-level insight, making the SQL alert approach the most robust.

Answer

Manually monitor the pipeline's status in the Databricks UI and check the event log periodically.

Answer

Rely on the pipeline's built-in email notifications for failures, as they are sent automatically to the pipeline owner.

Answer

Use cloud provider monitoring tools to set alarms on the underlying compute resources, and rely on those for pipeline failure detection.

Question 11

A Data Engineer needs to monitor the health of a Delta Live Tables (DLT) pipeline. Which metric should they monitor to track the number of data quality violations over time?

Accepted Answer

expectations_violated_records. The 'expectations_violated_records' metric is specifically designed to track data quality issues in DLT pipelines. By integrating these metrics with Databricks SQL alerts or external tools like Grafana, engineers can gain visibility into data lineage and quality degradation. Monitoring these metrics is essential for maintaining pipeline reliability and ensuring that downstream data consumers receive accurate and validated data, preventing the propagation of corrupted data throughout the analytical ecosystem.

Answer

pipeline_latency_seconds

Answer

cluster_cpu_utilization

Answer

task_retry_count

Question 12

Refer to the exhibit. An engineer created this alert for a query. Under what condition will the alert status change to 'Triggered'?

Accepted Answer

When the 'duration' column value in the query output is strictly greater than 3600.. The alert is configured to monitor the 'duration' column from the specified query result. If the returned value exceeds 3600, the alert enters the 'Triggered' state. Understanding this threshold-based mechanism is vital for maintaining pipeline performance. If queries consistently exceed this time, the alert notifies the engineer to investigate potential resource contention, suboptimal query plans, or data volume spikes, ensuring the stability and timely execution of data processing tasks.

Answer

When the query returns a row count greater than 3600.

Answer

When the query execution takes longer than 3600 seconds to run.

Answer

When the query fails and returns an error code equal to 3600.

Question 13

Which Databricks feature should be used to gain observability into access patterns and security events across the entire workspace?

Accepted Answer

System tables (Audit logs). System tables (specifically the audit log system table) provide comprehensive logs of all activities within a Databricks account. These tables allow engineers and security teams to monitor access patterns, identify unauthorized actions, and perform compliance reporting. Using system tables is critical for maintaining an audit trail, detecting potential threats, and ensuring that workspace operations adhere to organizational security policies and data governance standards.

Answer

Delta Live Tables (DLT) logs

Answer

Job run history

Answer

Cluster event logs

Question 14

A Data Engineer wants to monitor cluster health proactively. Which metric is most effective for identifying that a cluster needs to be scaled up to handle increasing workload demands?

Accepted Answer

Cluster CPU utilization. Monitoring 'cluster_memory_utilization' or 'node_cpu_load' is essential for identifying saturation. When these metrics consistently approach high thresholds, it signals that the current compute power is insufficient for the data volume. Proactive scaling based on these metrics prevents job failures and performance degradation, ensuring that data pipelines meet their processing deadlines and maintain efficiency without wasting costs on over-provisioned infrastructure.

Answer

Query result set size

Answer

Delta table file count

Answer

User login frequency

Question 15

Which action allows a Data Engineer to receive a Slack notification when a Delta Live Tables pipeline finishes successfully?

Accepted Answer

Use the 'Notifications' setting in the DLT pipeline configuration.. Databricks supports native notification channels that can be configured for pipelines. By adding a webhook for a specific Slack channel to the pipeline settings, engineers can receive real-time updates on pipeline lifecycle events, including completion. This integration is vital for workflow orchestration, allowing teams to trigger subsequent processes or notify stakeholders immediately without manual intervention, thereby streamlining the overall data platform efficiency.

Answer

Configure a SQL Alert to watch the system metadata tables.

Answer

Create a Python script that polls the Jobs API every minute.

Answer

Add a post-pipeline task that sends an email to the Slack email gateway.

Question 16

When troubleshooting a job that frequently crashes due to 'Out of Memory' (OOM) errors, which TWO metrics or logs should be analyzed?

Accepted Answer

Driver and Executor logs.. Analyzing Spark UI metrics and cluster logs is critical for resolving OOM errors. Examining the 'JVM heap usage' and 'executor memory usage' helps identify if the data partition size exceeds available memory. Addressing these issues is essential for stabilizing production jobs, as OOM errors are a leading cause of pipeline failure, resulting in data gaps and missed SLAs that impact downstream business decisions.

Answer

System table audit logs.

Answer

Workspace usage billing reports.

Answer

The number of active sessions in the SQL warehouse.

Question 17

A Data Engineer needs to ensure that a notebook job is not consuming excessive costs. Which monitoring tool provides the best view of DBU consumption per job?

Accepted Answer

Billable Usage System Table. The Databricks Billable Usage system table tracks DBU consumption at a granular level, including by job and cluster. Analyzing this data is essential for cost governance, allowing engineers to identify inefficient workflows and optimize resource allocation. By monitoring these trends, companies can ensure their data platform remains cost-effective, preventing unexpected budget overruns and providing the data necessary to justify investments in infrastructure or job refactoring.

Answer

Cluster event logs

Answer

Delta Live Tables event logs

Answer

Notebook execution history

Question 18

Refer to the exhibit. The alert is intended to trigger if the data in 'my_table' is older than one hour. Which query modification correctly implements this check?

Accepted Answer

SELECT CASE WHEN max(updated_at) < now() - interval 1 hour THEN 1 ELSE 0 END. To detect stale data, the alert must compare the latest record timestamp against the current time. If the latest 'updated_at' value is less than the threshold (one hour ago), it signifies that no data has arrived in the last hour. Monitoring data freshness is critical for time-sensitive analytics, ensuring that business stakeholders are working with current data and allowing teams to troubleshoot ingestion pipeline failures proactively.

Answer

SELECT COUNT(*) FROM my_table WHERE updated_at < now() - interval 1 hour

Answer

SELECT updated_at FROM my_table ORDER BY updated_at DESC LIMIT 1

Answer

SELECT datediff(now(), max(updated_at)) FROM my_table

Question 19

Which of the following is the best practice for managing alerts for a mission-critical production pipeline?

Accepted Answer

Use a central notification channel connected to an incident management tool.. Centralizing alerts within Databricks and routing them to a dedicated incident management tool ensures that issues are tracked, assigned, and resolved systematically. Avoiding fragmented notification methods is key to operational maturity. This approach prevents alert fatigue, ensures that the correct personnel are notified during off-hours, and keeps a clear historical record of system issues for post-mortem analysis and continuous improvement of the data architecture.

Answer

Configure alerts to send emails to individual developers.

Answer

Create a dashboard and check it manually every hour.

Answer

Disable alerts for development environments.

Question 20

An engineer notices that a SQL warehouse is frequently hitting 'Max Concurrency' limits. Which log should they consult to identify which specific queries are consuming most of the warehouse resources?

Accepted Answer

Query history system table. The 'query_history' system table is the primary resource for analyzing query performance and resource consumption. By querying this table, engineers can attribute resource usage to specific users, notebooks, or warehouses. This is essential for troubleshooting concurrency bottlenecks, enabling teams to optimize expensive queries or adjust warehouse sizing to meet demand without impacting overall system performance or user productivity.

Answer

Cluster event logs

Answer

Job run history logs

Answer

Delta table access logs