Databricks-DE-Pro Monitoring and Alerting — Timed 10 Questions

Question 1

A Data Engineer is configuring alerts for a critical production job. Which TWO actions are required to successfully set up an email notification for a job failure?

Accepted Answer

Configure the 'Notifications' section in the Job settings to include the email address.. In Databricks, job notifications are configured directly in the 'Notifications' section of the Job settings. You do not need to add an 'on_failure' trigger to the Jobs API payload for a specific task to enable email notifications; the Job-level notification settings handle this. Option C is incorrect because task-level triggers are not the standard way to configure email alerts for job failures.

Answer

Create an Alert object in the SQL workspace linked to the job ID.

Answer

Add the 'on_failure' trigger to the Jobs API payload for the specific task.

Answer

Set up a System Table audit log trigger to monitor job state changes.

Answer

Enable the 'Notify on task failure' checkbox in the cluster configuration.

Question 2

Which TWO of the following are valid destinations for Databricks SQL alerts?

Accepted Answer

Webhooks. Databricks SQL alerts are highly flexible and support multiple notification channels to integrate with existing DevOps workflows. Choosing the right destination—like a webhook for custom integrations or email for general communication—is vital for ensuring the right teams are alerted. This flexibility allows engineers to build highly responsive incident response pipelines, significantly reducing downtime and ensuring that data quality or performance issues are addressed immediately.

Answer

Direct database table updates

Answer

Direct user push notifications on the Databricks mobile app.

Answer

Automatic cluster restart triggers.

Question 3

Refer to the exhibit. If a job is failing because the cluster is reaching the 'max_workers' limit too quickly, which monitoring metric should the engineer verify to confirm if the cluster is actually utilizing these nodes efficiently?

Accepted Answer

Task-level execution time distribution. Verifying 'task_skew' and 'executor_utilization' is key. If the cluster is scaling to 8 nodes but still failing, the issue may not be a lack of resources, but rather inefficient data distribution (skew) or task overhead. Analyzing these metrics ensures that engineers don't simply increase the cluster size, which would increase costs without solving the underlying performance issues caused by unbalanced workloads.

Answer

Total number of users logged into the workspace.

Answer

The total number of files in the Delta table.

Answer

The memory overhead of the driver node.

Question 4

A data engineer is using the Databricks REST API to retrieve the event log for a specific job run. They need to fetch only the events that occurred after a certain timestamp. Which query parameter should they use?

Accepted Answer

order_by. The Databricks REST API for job run events does not support a direct timestamp filter. To fetch events after a certain timestamp, the engineer must use the order_by parameter to sort events by timestamp, then paginate through the results and filter client-side. The order_by parameter is essential for ensuring events are returned in a predictable order, enabling efficient client-side filtering. Other parameters like offset and limit are for pagination but do not filter by time.

Answer

timestamp

Answer

start_time

Answer

offset

Question 5

A data engineer runs a nightly production job on a job cluster using Databricks Runtime 14.3 LTS. The job has been intermittently failing for three days with 'Cluster terminated due to spot instance loss' in the event log, but the driver logs show no application errors. The engineer needs to reduce these failures without changing the job's runtime behavior or increasing cost significantly. Which action should the engineer take?

Accepted Answer

Configure the cluster with a mix of on-demand and spot instances using the 'First on-demand' or 'Spot with fallback' allocation strategy for the worker nodes.. The job failures are caused by spot instance reclamation, which terminates the cluster. The most effective mitigation without changing runtime behavior or greatly increasing cost is to use an allocation strategy that mixes on-demand and spot instances, such as 'First on-demand' or 'Spot with fallback'. This keeps the driver on reliable capacity and allows workers to be replaced automatically when spot instances are lost.

Answer

Enable the 'spot instance fallback' option by setting the cluster's Spark configuration 'spark.databricks.cluster.profile' to 'spot'.

Answer

Increase the cluster's autoscaling maximum workers so that replacement spot instances can be added more quickly.

Answer

Switch the entire cluster to on-demand instances and enable autoscaling to compensate for the higher cost.

Question 6

A data engineering team runs a nightly production job on Databricks. The job has been failing intermittently with a 'Cluster terminated due to idle timeout' error. The team wants to be proactively notified when the cluster is about to terminate due to inactivity so they can investigate. Which approach should they use?

Accepted Answer

Use the cluster event log to detect idle timeout events and set up a notification via a job or external monitoring.. The cluster event log is the built-in mechanism that records cluster lifecycle events, including notifications before idle termination. By monitoring this log, teams can set up alerts to act before the cluster shuts down. Other options either target the wrong monitoring surface or do not provide the required proactive notification for idle timeout.

Answer

Create a Databricks job that runs every minute and checks the cluster status via the REST API.

Answer

Configure a Databricks SQL alert on the cluster's idle timeout metric.

Answer

Enable autoscaling on the cluster and set a minimum number of workers to prevent idle timeout.

Question 7

A data engineer has deployed a Delta Live Tables pipeline that ingests streaming data from a cloud storage location. The pipeline is configured with expectations to drop invalid records. The engineer needs to monitor the number of records that were dropped due to failed expectations over time. Which approach should the engineer use to track this metric?

Accepted Answer

Query the Delta Live Tables event log for 'flow_progress' events and extract the 'dropped_records' field.. Delta Live Tables automatically logs detailed metrics to the event log, including the number of records dropped by expectations. Querying the event log for flow progress events and extracting the dropped records field provides a reliable, built-in way to monitor this metric over time without custom code or additional tables.

Answer

Configure the pipeline to write dropped records to a separate Delta table and set up a Databricks job to count them hourly.

Answer

Enable Databricks SQL alerts on the target Delta table to count rows where the expectation column is null.

Answer

Use the Databricks REST API to retrieve pipeline metrics and filter for the 'num_dropped_records' field.

Question 8

A data engineer needs to monitor the performance of a Databricks SQL warehouse and receive alerts when query latency exceeds a threshold. The warehouse is used by multiple teams for interactive queries. Which monitoring solution should the engineer implement?

Accepted Answer

Configure a Databricks SQL alert on the query history table to trigger when average latency exceeds the threshold.. Databricks SQL alerts can be created directly on queries against the query history system table, allowing for precise monitoring of query latency. This native feature simplifies alert configuration and integrates with notification destinations. Other methods either lack the necessary metrics, require custom code, or monitor the wrong layer of the stack.

Answer

Enable audit logs and analyze them for slow queries using a scheduled notebook.

Answer

Set up a cluster-level metric alert in the cloud provider's monitoring service for the SQL warehouse's underlying VMs.

Answer

Use the Databricks REST API to poll the warehouse's metrics endpoint and send notifications via a custom script.

Question 9

A data engineer is responsible for a Delta Live Tables pipeline that processes sensitive customer data. They need to monitor the pipeline for data quality issues and ensure that records failing expectations are tracked. They also want to receive alerts when the pipeline fails. Which TWO actions should the engineer take to achieve these goals? (Choose two.)

Accepted Answer

Define expectations in the pipeline using the @dlt.expect decorator and set the pipeline to drop or quarantine invalid records.. To monitor data quality in a DLT pipeline, the engineer should define expectations with @dlt.expect and query the event log for data_quality events. These two actions provide both the mechanism to capture quality metrics and the ability to analyze and alert on them. Together, they enable comprehensive monitoring of expectation failures and pipeline health.

Answer

Create a Databricks job that runs the pipeline and configure job alerts to notify on failure.

Answer

Enable cluster logging to a cloud storage location and parse logs for 'Expectation failed' messages.

Answer

Configure a Databricks SQL alert on the target table to count rows and alert if the count drops below a threshold.

Question 10

A data engineer is responsible for a mission-critical Delta Live Tables pipeline that must process data continuously. The engineer needs to configure a robust alerting strategy to ensure immediate notification if the pipeline fails or experiences significant delays. Which approach best meets this requirement?

Accepted Answer

Configure a Databricks SQL alert on the pipeline's event log to trigger when errors occur, and set the alert to notify a Slack channel.. A Databricks SQL alert on the pipeline's event log provides a flexible and proactive way to monitor for errors and delays. Routing notifications to Slack ensures the team is immediately aware. Built-in email notifications are limited, manual monitoring is unreliable, and cloud provider tools lack pipeline-level insight, making the SQL alert approach the most robust.

Answer

Manually monitor the pipeline's status in the Databricks UI and check the event log periodically.

Answer

Rely on the pipeline's built-in email notifications for failures, as they are sent automatically to the pipeline owner.

Answer

Use cloud provider monitoring tools to set alarms on the underlying compute resources, and rely on those for pipeline failure detection.