Monitoring and Alerting practice questions

Create an Alert object in the SQL workspace linked to the job ID.

Why it fails: Alert objects in the Databricks SQL workspace are specifically designed for query result monitoring, not for managing individual job run states. While they offer robust alerting capabilities, they are not the appropriate mechanism for monitoring the lifecycle of a discrete job or task.

Add the 'on_failure' trigger to the Jobs API payload for the specific task.

Why it fails: The Jobs API allows for granular control over notification settings. By defining an 'on_failure' email configuration in the API payload, engineers can programmatically ensure that failures trigger immediate alerts, which is essential for CI/CD pipelines and automated infrastructure-as-code deployments of production data workflows.

Set up a System Table audit log trigger to monitor job state changes.

Why it fails: System tables provide audit logs for security and governance, not for real-time notification of job statuses. Relying on audit logs for operational alerting introduces significant latency and complexity, as audit events are not intended for immediate operational monitoring or incident response workflows.

Enable the 'Notify on task failure' checkbox in the cluster configuration.

Why it fails: Cluster configuration defines the hardware and runtime environment, not the operational alerts for a specific job or task. Notifications belong to the job orchestration layer, and associating them with clusters would result in redundant or misdirected alerts when multiple jobs use the same cluster.

Direct database table updates

Why it fails: Alerts in Databricks are designed to inform users of issues, not to modify operational data. Creating an alert that attempts to write to a table would introduce circular dependencies and potential performance degradation in the system.

Direct user push notifications on the Databricks mobile app.

Why it fails: Databricks SQL alerts deliver to email, Slack, PagerDuty, webhooks and similar configured destinations; the mobile app receives no alert push channel. It tempts because Databricks offers a mobile app for monitoring dashboards and jobs, so push notifications appear a natural extension, but alerts are not routed there.

Automatic cluster restart triggers.

Why it fails: Alerts are for notifications and observability, not for controlling infrastructure state. Restarting a cluster based on an alert would be an automated remediation action, which requires a separate workflow or orchestration mechanism, not an alert definition.

Total number of users logged into the workspace.

Why it fails: Workspace user counts say nothing about cluster node utilisation or task scheduling. To confirm whether max_workers nodes are being used efficiently, the engineer needs cluster metrics such as worker utilisation, task queue depth, or executor run time. User metrics belong to workspace administration, not job performance tuning.

The total number of files in the Delta table.

Why it fails: While file counts can impact performance, they do not help identify task-level efficiency or skew on the cluster. They are a storage metadata metric and provide no insight into how the compute nodes are sharing the processing load.

The memory overhead of the driver node.

Why it fails: Driver memory overhead only impacts the orchestration of the job. If the worker nodes are reaching their maximum limit and the job is still failing, the issue lies in the executor task execution, not the driver's memory management.

timestamp

Why it fails: timestamp is not a supported query parameter for filtering events by time. The API does not offer a direct timestamp filter. Engineers must retrieve events and filter client-side, or use the order_by parameter to sort by timestamp and then paginate.

start_time

Why it fails: start_time is not a valid query parameter for the job run events API. The API supports parameters like offset, limit, and order_by to paginate and sort results. Using start_time would result in an error or be ignored, failing to filter events as intended.

offset

Why it fails: offset is used for pagination, specifying the number of events to skip. It does not filter by timestamp. While it can help retrieve events in batches, it does not allow fetching events after a certain time. Therefore, it is not suitable for this requirement.

Enable the 'spot instance fallback' option by setting the cluster's Spark configuration 'spark.databricks.cluster.profile' to 'spot'.

Why it fails: The 'spark.databricks.cluster.profile' setting is used to indicate that a cluster is a spot instance cluster for cost attribution, but it does not enable fallback behavior. Fallback is controlled by the cluster's instance allocation strategy, not by this Spark configuration. Setting it to 'spot' alone will not prevent termination on spot loss.

Increase the cluster's autoscaling maximum workers so that replacement spot instances can be added more quickly.

Why it fails: Increasing max workers does not prevent spot instance termination; it only allows more workers to be added when demand increases. The failure is caused by spot reclamation, not by insufficient capacity. Adding more workers could increase cost without addressing the root cause, and the cluster may still terminate if the spot pool is exhausted.

Switch the entire cluster to on-demand instances and enable autoscaling to compensate for the higher cost.

Why it fails: Switching entirely to on-demand instances would eliminate spot-related failures but significantly increases cost, which the engineer wants to avoid. Autoscaling does not reduce the cost difference between on-demand and spot. A mixed strategy provides reliability while retaining cost benefits, making this option unnecessarily expensive and not aligned with the requirement.

Create a Databricks job that runs every minute and checks the cluster status via the REST API.

Why it fails: While polling the REST API can retrieve cluster state, it is not the recommended approach for detecting idle timeout warnings. The cluster event log is the authoritative source for lifecycle events and provides more granular and timely information. Polling also adds unnecessary overhead and may miss the brief warning window.

Configure a Databricks SQL alert on the cluster's idle timeout metric.

Why it fails: Databricks SQL alerts are designed to run queries against data objects such as tables or views in the Databricks SQL warehouse, not to monitor cluster lifecycle events like idle timeout. They cannot trigger on infrastructure state changes, so they cannot provide proactive notification for an impending cluster termination.

Enable autoscaling on the cluster and set a minimum number of workers to prevent idle timeout.

Why it fails: Autoscaling adjusts worker count based on workload but does not prevent idle timeout. Idle timeout is a separate configuration that terminates the cluster after a period of inactivity regardless of autoscaling. This option does not provide notification and does not address the root cause of the intermittent failures.

Configure the pipeline to write dropped records to a separate Delta table and set up a Databricks job to count them hourly.

Why it fails: While writing dropped records to a separate table is possible with custom logic, it is not a built-in feature of Delta Live Tables expectations. This approach requires additional code and maintenance, and it does not provide the built-in metrics that the event log offers. It is also less efficient and more error-prone than using the event log.

Enable Databricks SQL alerts on the target Delta table to count rows where the expectation column is null.

Why it fails: Expectations that drop records do not add a column to the target table; they simply filter out records before writing. Therefore, there is no expectation column to count. SQL alerts on the target table would not capture dropped records because those records are not written. The event log is the correct source for this metric.

Use the Databricks REST API to retrieve pipeline metrics and filter for the 'num_dropped_records' field.

Why it fails: The Databricks REST API for Delta Live Tables provides pipeline status and some metrics, but it does not expose a field named 'num_dropped_records'. The granular dropped record counts are available in the event log, not directly via the REST API. The API is better suited for pipeline-level status than for detailed record-level metrics.

Enable audit logs and analyze them for slow queries using a scheduled notebook.

Why it fails: Audit logs capture access and operation events but do not include detailed query performance metrics like latency. They are designed for security and compliance monitoring, not performance. Analyzing audit logs would not provide the necessary latency data, and a scheduled notebook adds unnecessary complexity compared to a native alert.

Set up a cluster-level metric alert in the cloud provider's monitoring service for the SQL warehouse's underlying VMs.

Why it fails: Databricks SQL warehouses are serverless and abstract the underlying infrastructure. Cloud provider monitoring may not expose query-level latency metrics and cannot easily correlate VM metrics with SQL query performance. This approach is indirect and would not provide the precise alerting needed for query latency thresholds.

Use the Databricks REST API to poll the warehouse's metrics endpoint and send notifications via a custom script.

Why it fails: While the REST API provides access to warehouse metrics, polling requires custom development and maintenance. Databricks SQL alerts offer a built-in, declarative way to monitor query latency without writing code. The API approach is more complex and error-prone, and it does not provide the same level of integration with the Databricks platform.

Create a Databricks job that runs the pipeline and configure job alerts to notify on failure.

Why it fails: DLT pipelines are not run as standard jobs; they are managed by the DLT runtime. Job alerts on a separate job would not monitor the pipeline's internal failures. DLT pipelines have their own notification settings for failure, so this approach is redundant and incorrect for monitoring data quality.

Enable cluster logging to a cloud storage location and parse logs for 'Expectation failed' messages.

Why it fails: Cluster logs may contain error messages, but they are not structured for expectation metrics. Parsing logs is fragile and lacks the aggregated view provided by the event log. This method does not integrate with alerting and is not the recommended way to monitor DLT expectations.

Configure a Databricks SQL alert on the target table to count rows and alert if the count drops below a threshold.

Why it fails: While a SQL alert on the target table can detect volume anomalies, it does not track data quality expectations specifically. It would not provide insight into which expectations failed or how many records were dropped. This approach is indirect and does not fulfill the requirement to monitor data quality issues at the expectation level.

Manually monitor the pipeline's status in the Databricks UI and check the event log periodically.

Why it fails: Manual monitoring is not reliable for mission-critical pipelines because it depends on human availability and attention. It cannot provide immediate notification of failures or delays, especially outside working hours. This approach is prone to missed incidents and does not meet the requirement for robust, automated alerting.

Rely on the pipeline's built-in email notifications for failures, as they are sent automatically to the pipeline owner.

Why it fails: While Delta Live Tables can send email notifications on failure, these are limited to the pipeline owner and may not provide immediate team-wide visibility. They also do not cover delays or performance degradation. This approach is not robust for a mission-critical pipeline requiring immediate, broad notification.

Use cloud provider monitoring tools to set alarms on the underlying compute resources, and rely on those for pipeline failure detection.

Why it fails: Cloud provider monitoring tools can track infrastructure metrics like CPU and memory but do not have visibility into Delta Live Tables pipeline-specific events such as data quality failures or processing delays. They may detect cluster issues but cannot alert on pipeline logic failures, so they are insufficient for this requirement.

pipeline_latency_seconds

Why it fails: This metric tracks the time taken for data to move through the pipeline. While important for performance monitoring and identifying bottlenecks, it does not provide insights into data quality or the number of records that failed validation checks according to defined pipeline expectations.

cluster_cpu_utilization

Why it fails: This metric measures hardware resource consumption by the underlying compute cluster. While it is vital for cost optimization and identifying scaling needs, it provides no visibility into the actual data content, quality, or whether specific records violated schema or business logic constraints.

task_retry_count

Why it fails: This metric tracks how many times a pipeline task has attempted to restart after a failure. While high retry counts indicate infrastructure or logic issues, this metric does not indicate data quality failures, which often occur during successful task runs without triggering pipeline retries.

When the query returns a row count greater than 3600.

Why it fails: The configuration explicitly targets the 'duration' column as the evaluation metric for the alert condition. The row count of the result set is irrelevant to this specific trigger configuration, which focuses exclusively on the time-based performance of the query execution.

When the query execution takes longer than 3600 seconds to run.

Why it fails: The alert is based on a specific column in the query results, not the internal query runtime metadata. While the query might return a value representing duration, the alert evaluates the data returned by the query, not the Databricks SQL execution engine's internal timer.

When the query fails and returns an error code equal to 3600.

Why it fails: Alerts in Databricks SQL are based on successful query result sets that meet a condition, not on trapping application or SQL errors. If a query fails, the alert cannot process the output, and the state will remain in the last known valid configuration.

Delta Live Tables (DLT) logs

Why it fails: DLT logs are specific to pipeline events, data quality, and execution status. They are insufficient for broader workspace security or access observability, as they do not capture non-pipeline related activities such as user login events, cluster creations, or workspace configuration changes.

Job run history

Why it fails: Job run history is useful for monitoring the success or failure of specific workflows. It does not provide visibility into broader security events, user access patterns, or administrative actions performed outside of the scope of scheduled job executions.

Cluster event logs

Why it fails: Cluster event logs track activities related to the lifecycle of specific compute clusters. While they are useful for debugging performance or configuration issues with clusters, they do not provide a comprehensive audit trail of user access or administrative actions at the workspace level.

Query result set size

Why it fails: The size of a query result set describes the volume of output, not the hardware performance during execution. While large outputs can cause network issues, they are not the primary indicator of cluster compute saturation or the need for horizontal scaling.

Delta table file count

Why it fails: The number of files in a Delta table affects storage performance, not the compute cluster's processing health. While managing small files is a best practice, increasing the file count does not trigger a need for cluster upscaling in the context of real-time monitoring.

User login frequency

Why it fails: User login frequency tracks authentication activity and is unrelated to compute cluster performance. This metric is relevant for security and capacity planning for the workspace control plane, but it provides no data on the resource consumption or performance of data processing clusters.

Configure a SQL Alert to watch the system metadata tables.

Why it fails: SQL alerts are designed to monitor query result thresholds, not pipeline state changes. Monitoring system metadata for pipeline completions via SQL alerts is an inefficient and unreliable way to manage workflow orchestration or receive real-time notifications about job statuses.

Create a Python script that polls the Jobs API every minute.

Why it fails: Polling the Jobs API is a suboptimal, resource-intensive practice that introduces unnecessary latency and maintenance overhead. Native integration features provided by Databricks are designed to handle event-driven notifications more effectively and reliably than custom-built polling scripts.

Add a post-pipeline task that sends an email to the Slack email gateway.

Why it fails: While technically possible, using a gateway is fragile and bypasses the built-in, native notification mechanisms. It adds unnecessary complexity to the pipeline design and increases the likelihood of failure if the email gateway configuration or routing rules change.

System table audit logs.

Why it fails: Audit logs track user authentication and authorization events within the workspace. They do not contain technical details regarding memory allocation, execution plans, or garbage collection events, making them useless for diagnosing performance-related application crashes like OOM errors.

Workspace usage billing reports.

Why it fails: Billing reports provide insight into DBU usage and costs. They provide no technical observability into the runtime behavior, memory consumption, or stack traces of an application execution, and thus provide no value in troubleshooting an OOM exception.

The number of active sessions in the SQL warehouse.

Why it fails: SQL warehouse session counts relate to user concurrency and query queuing. While related to general system load, they provide no specific information on the memory management or heap usage of a specific job that is failing with OOM errors.

Cluster event logs

Why it fails: Cluster event logs track cluster state changes, such as start and stop events. While they can be used to estimate uptime, they do not provide the detailed DBU usage data required for accurate cost attribution and granular job-level budget analysis.

Delta Live Tables event logs

Why it fails: These logs are specific to DLT pipelines and focus on data quality and status. They do not contain the comprehensive DBU billing information required to track costs across all notebook jobs or other types of compute tasks in the workspace.

Notebook execution history

Why it fails: Notebook execution history shows runtime duration and status, but it does not include DBU consumption data or associated costs. Relying on duration alone is an inaccurate proxy for cost, as different instance types consume DBUs at significantly different rates.

SELECT COUNT(*) FROM my_table WHERE updated_at < now() - interval 1 hour

Why it fails: This query returns the count of old records, not whether the most recent data is stale. If the table has many historical records, this query will return a high value even if new data is being ingested perfectly, leading to false-positive alerts.

SELECT updated_at FROM my_table ORDER BY updated_at DESC LIMIT 1

Why it fails: This query returns the timestamp itself, not a boolean flag or a metric that can be easily compared to a constant in a standard threshold-based alert configuration, making it difficult to trigger a boolean 'Alert' state directly.

SELECT datediff(now(), max(updated_at)) FROM my_table

Why it fails: While this returns the difference in days, it is not ideal for hourly thresholds. Even with proper conversion, the result is a number representing a duration, and if the data is fresh, the result might be zero, which is harder to alert on than a boolean flag.

Configure alerts to send emails to individual developers.

Why it fails: Sending alerts to individual developers is a poor practice that leads to silos and potential missed incidents if the individual is unavailable. It lacks an organized, team-wide approach to incident management, which is necessary for high-availability production environments.

Create a dashboard and check it manually every hour.

Why it fails: Manual checking is not scalable and is prone to human error. It fails to provide immediate notification in the event of an incident, leading to increased mean time to detect (MTTD) and negative impacts on data availability for end users.

Disable alerts for development environments.

Why it fails: Disabling alerts in development prevents engineers from identifying issues early in the development lifecycle. While production alerts are prioritized, a lack of monitoring in lower environments makes debugging harder and increases the risk of deploying broken code to production.

Cluster event logs

Why it fails: Cluster event logs record lifecycle events such as node starts, autoscaling and configuration changes, not per-query resource consumption. They suit diagnosing cluster-level failures. Identifying concurrency-heavy queries requires the SQL warehouse query history, which shows duration and resource usage per statement.

Job run history logs

Why it fails: Job run history logs are scoped to job tasks and do not capture individual queries submitted through the SQL warehouse interface by users or BI tools. Therefore, they are insufficient for debugging concurrency issues on SQL warehouses.

Delta table access logs

Why it fails: These logs are primarily used for security auditing to see who has accessed specific data files. They lack the performance and execution plan details necessary to debug compute resource consumption or concurrency bottlenecks in a SQL warehouse.