Databricks-DE-Pro Monitoring and Alerting Practice Test 2 — 15 Questions

Question 1

When troubleshooting a job that frequently crashes due to 'Out of Memory' (OOM) errors, which TWO metrics or logs should be analyzed?

Accepted Answer

Driver and Executor logs.. Analyzing Spark UI metrics and cluster logs is critical for resolving OOM errors. Examining the 'JVM heap usage' and 'executor memory usage' helps identify if the data partition size exceeds available memory. Addressing these issues is essential for stabilizing production jobs, as OOM errors are a leading cause of pipeline failure, resulting in data gaps and missed SLAs that impact downstream business decisions.

Answer

System table audit logs.

Answer

Workspace usage billing reports.

Answer

The number of active sessions in the SQL warehouse.

Question 2

A Data Engineer needs to ensure that a notebook job is not consuming excessive costs. Which monitoring tool provides the best view of DBU consumption per job?

Accepted Answer

Billable Usage System Table. The Databricks Billable Usage system table tracks DBU consumption at a granular level, including by job and cluster. Analyzing this data is essential for cost governance, allowing engineers to identify inefficient workflows and optimize resource allocation. By monitoring these trends, companies can ensure their data platform remains cost-effective, preventing unexpected budget overruns and providing the data necessary to justify investments in infrastructure or job refactoring.

Answer

Cluster event logs

Answer

Delta Live Tables event logs

Answer

Notebook execution history

Question 3

Refer to the exhibit. The alert is intended to trigger if the data in 'my_table' is older than one hour. Which query modification correctly implements this check?

Accepted Answer

SELECT CASE WHEN max(updated_at) < now() - interval 1 hour THEN 1 ELSE 0 END. To detect stale data, the alert must compare the latest record timestamp against the current time. If the latest 'updated_at' value is less than the threshold (one hour ago), it signifies that no data has arrived in the last hour. Monitoring data freshness is critical for time-sensitive analytics, ensuring that business stakeholders are working with current data and allowing teams to troubleshoot ingestion pipeline failures proactively.

Answer

SELECT COUNT(*) FROM my_table WHERE updated_at < now() - interval 1 hour

Answer

SELECT updated_at FROM my_table ORDER BY updated_at DESC LIMIT 1

Answer

SELECT datediff(now(), max(updated_at)) FROM my_table

Question 4

Which of the following is the best practice for managing alerts for a mission-critical production pipeline?

Accepted Answer

Use a central notification channel connected to an incident management tool.. Centralizing alerts within Databricks and routing them to a dedicated incident management tool ensures that issues are tracked, assigned, and resolved systematically. Avoiding fragmented notification methods is key to operational maturity. This approach prevents alert fatigue, ensures that the correct personnel are notified during off-hours, and keeps a clear historical record of system issues for post-mortem analysis and continuous improvement of the data architecture.

Answer

Configure alerts to send emails to individual developers.

Answer

Create a dashboard and check it manually every hour.

Answer

Disable alerts for development environments.

Question 5

An engineer notices that a SQL warehouse is frequently hitting 'Max Concurrency' limits. Which log should they consult to identify which specific queries are consuming most of the warehouse resources?

Accepted Answer

Query history system table. The 'query_history' system table is the primary resource for analyzing query performance and resource consumption. By querying this table, engineers can attribute resource usage to specific users, notebooks, or warehouses. This is essential for troubleshooting concurrency bottlenecks, enabling teams to optimize expensive queries or adjust warehouse sizing to meet demand without impacting overall system performance or user productivity.

Answer

Cluster event logs

Answer

Job run history logs

Answer

Delta table access logs

Question 6

Which capability is provided by Databricks' integration with cloud-native monitoring tools (e.g., CloudWatch, Azure Monitor)?

Accepted Answer

Centralized metrics collection and dashboarding.. Integrating Databricks with cloud-native monitoring provides a unified view of platform health. These tools aggregate logs and metrics from across the entire cloud environment, allowing for centralized dashboarding and advanced alerting. This integration is vital for enterprises maintaining a 'single pane of glass' strategy, as it ensures that Databricks infrastructure metrics are treated with the same governance and visibility standards as other cloud resources.

Answer

Automated data quality remediation.

Answer

Direct execution of SQL queries on the warehouse.

Answer

Fine-grained control over Spark session configurations.

Question 7

Which Databricks feature provides the most granular view of data quality metrics over time for a Delta Live Tables pipeline?

Accepted Answer

DLT Event Log system table. The DLT 'event log' (stored as a system table) is the most powerful tool for analyzing data quality. It records every expectation check, allowing for historical trend analysis of data quality violations. This is critical for data governance, as it provides auditability and helps engineers identify when and where data quality has degraded over time, enabling proactive fixes to source upstream data.

Answer

Pipeline Run logs

Answer

Cluster log files

Answer

SQL warehouse query history

Question 8

A data engineer wants to monitor the health of Delta Live Tables (DLT) pipelines and be alerted if a pipeline fails. Which approach is the most efficient and native way to achieve this?

Accepted Answer

Define a Notification Destination in the DLT pipeline settings and associate it with the 'On failure' event.. Using Notification Destinations in the DLT pipeline settings is the most native method. It allows engineers to configure email or Slack alerts directly within the UI or JSON configuration. This is crucial for production reliability, ensuring that stakeholders receive immediate notifications regarding job failures or data quality issues without needing external orchestration or custom API scripts, maintaining observability across the entire pipeline lifecycle.

Answer

Configure an external cron job to poll the Databricks Jobs API every minute for status updates.

Answer

Enable the 'Log to Workspace' feature and manually query the system logs using SQL every hour.

Answer

Use the Databricks SQL Alerts tool to monitor the underlying Delta tables for new record counts.

Question 9

Refer to the exhibit. A Databricks administrator wants to restrict access to a specific SQL Alert. Based on the JSON policy, which statement accurately describes the current permission model for this alert?

Accepted Answer

The user can see the alert, but cannot trigger a manual refresh or modify the alert settings.. The exhibit displays an Access Control List (ACL) entry granting 'CAN_VIEW' permission to a specific user for a single alert resource. In Databricks, SQL Alerts are secured via the workspace-level permission model. Understanding these JSON-based policy definitions is essential for data engineers tasked with auditing governance and ensuring that sensitive monitoring data is only accessible to authorized personnel during security reviews.

Answer

The user is allowed to modify the alert's query criteria and notification frequency.

Answer

The user has sufficient permissions to delete the alert from the Databricks SQL workspace.

Answer

This JSON policy provides the user with global access to all alerts within the SQL warehouse.

Question 10

A data engineer is configuring monitoring for a production Databricks cluster. Which TWO metrics are best suited to identify potential performance bottlenecks related to worker nodes?

Accepted Answer

Cluster CPU utilization percentage.. Monitoring worker nodes is vital for identifying bottlenecks before they cause job failures. Metrics like CPU utilization and memory usage are key indicators of node pressure. By identifying these patterns, engineers can optimize cluster configurations, adjust instance types, or tune spark code, ensuring that production workloads remain stable, performant, and cost-effective throughout their execution lifecycle.

Answer

Number of active jobs currently running in the entire workspace.

Answer

Total number of users logged into the Databricks workspace.

Answer

The version of the Databricks Runtime used by the cluster.

Question 11

You are managing a large-scale data lakehouse. You notice that your Spark jobs are frequently failing due to disk space issues on worker nodes. Which monitoring feature should you implement to proactively capture this trend?

Accepted Answer

Monitor Spark shuffle spill metrics using the Ganglia UI or custom Spark listeners.. Implementing custom Spark listeners or utilizing the Spark UI metrics for 'Disk Spilling' is the most effective proactive measure. Disk space exhaustion is often a symptom of memory pressure, leading to excessive shuffle operations. By monitoring spill-to-disk metrics, you can identify jobs that require more memory or better partitioning, preventing job failure and optimizing compute efficiency before the disk threshold is reached.

Answer

Enable standard Databricks job success/failure notifications.

Answer

Set up a Databricks SQL alert on the system.query_history table.

Answer

Schedule a daily scan of the underlying S3 or ADLS storage buckets.

Question 12

A data engineer wants to monitor the data quality of a Delta table over time. Which tool is most appropriate for this task?

Accepted Answer

Implement DLT Expectations within your pipeline code to validate data constraints.. Delta Live Tables (DLT) Expectations are the most appropriate and integrated way to enforce and monitor data quality. By defining constraints directly within the DLT pipeline code, data engineers can automatically validate data as it arrives, generate failure logs, and even quarantine invalid records. This observability is critical for maintaining high-quality datasets in a lakehouse architecture without requiring complex, separate validation workflows for every table.

Answer

Use Databricks SQL Alerts to count total rows in the table every hour.

Answer

Write a Spark job that manually scans the table for NULL values and sends an email.

Answer

Configure a cluster-level alert to trigger when the table metadata is updated.

Question 13

A data engineer is reviewing the event log of a Databricks job that has just failed. They need to determine the exact cause of the failure. Which event type in the event log indicates that a task failed due to an exception?

Accepted Answer

TASK_FAILED. The TASK_FAILED event is logged when an individual task within a job run encounters an exception. It captures the error message, stack trace, and other relevant metadata, making it the primary source for diagnosing task-level failures. Other event types like JOB_FAILED indicate a higher-level failure but lack the specific task exception details. Therefore, TASK_FAILED is the correct choice for determining the exact cause of a task failure.

Answer

RUN_FAILED

Answer

JOB_FAILED

Answer

EXECUTION_FAILED

Question 14

A data engineer wants to monitor the performance of a Databricks cluster by tracking the average CPU utilization over time. Which Databricks feature should they use to visualize this metric?

Accepted Answer

Cluster metrics dashboard in the Databricks UI. The cluster metrics dashboard in the Databricks UI is designed to display cluster performance metrics such as CPU utilization, memory usage, and network activity. It provides both real-time and historical views, allowing engineers to monitor trends and identify performance issues. Other tools like Spark UI focus on job execution, while Databricks SQL dashboards are for data visualization, not infrastructure monitoring. Therefore, the cluster metrics dashboard is the correct choice.

Answer

Spark UI

Answer

Databricks SQL dashboards

Answer

Ganglia

Question 15

A data engineer manages a Databricks SQL warehouse that serves a dashboard used by the finance team. The dashboard queries have become slow during peak hours, and the engineer suspects that some queries are scanning excessive data. Which system table should the engineer query to analyze query performance and identify expensive queries?

Accepted Answer

system.query.history. The system.query.history table is designed for query observability in Databricks SQL. It captures execution details, including query duration, rows produced, and bytes read, enabling engineers to pinpoint inefficient queries. Other system tables focus on compute metrics, audit events, or billing, and lack the query-level performance data needed to troubleshoot slow dashboards.

Answer

system.billing.usage

Answer

system.access.audit

Answer

system.compute.node_timeline