Databricks-DE-Pro Debugging and Deploying Practice Test 2 — 15 Questions

Question 1

A data engineer is automating the deployment of Databricks assets using CI/CD. The pipeline fails because the 'databricks-cli' command cannot find the workspace. What is the most likely cause?

Accepted Answer

The runner does not have the DATABRICKS_HOST and DATABRICKS_TOKEN environment variables set.. Deployment failures in CI/CD often stem from authentication or environment variable misconfiguration. The Databricks CLI requires a properly configured profile or environment variables containing the host URL and token. When these are missing, the CLI fails to authenticate or locate the workspace, preventing the deployment of code. Ensuring that the runner has access to these secure variables is foundational for a reliable DevOps lifecycle for Databricks infrastructure.

Answer

The Databricks CLI version on the build agent is incompatible with the server.

Answer

The workspace API is currently disabled for security reasons.

Answer

The CI/CD runner is missing the required library dependencies like PySpark.

Question 2

You are monitoring a long-running Databricks job. You notice that the memory usage on the driver node is steadily increasing until it crashes. Which debugging action is most appropriate?

Accepted Answer

Use the Spark UI to identify tasks using 'collect()' or 'toPandas()' on large datasets.. Driver memory crashes are typically caused by collecting large datasets from workers to the driver. By identifying the 'collect' or 'toPandas' operations, you can optimize the code to process data in a distributed manner. This is critical for scaling data pipelines; understanding why the driver is overloaded ensures that jobs remain performant as data volumes grow and prevents common failures in production batch processing environments.

Answer

Increase the number of worker nodes in the cluster.

Answer

Update the cluster's Spark configuration to disable the driver's log monitoring.

Answer

Lower the 'spark.driver.maxResultSize' setting in the cluster configuration.

Question 3

Which THREE strategies should a data engineer use to optimize the debugging of failed production Databricks Jobs?

Accepted Answer

Implement structured logging within the application code to track variable states.. Efficient debugging relies on observability, modular design, and access control. By leveraging logs, modularizing code, and ensuring proper access, engineers can quickly isolate root causes. These strategies are essential for minimizing Mean Time to Recovery (MTTR) in production. Proactive monitoring and well-structured codebases allow for faster identification of failures, ensuring that business-critical pipelines remain operational and that the team can respond to incidents with precision and speed.

Answer

Grant all developers full cluster permissions to access logs directly on the nodes.

Answer

Run all production jobs as the root user to avoid permission-related errors.

Question 4

Where can a data engineer find the standard output and error logs for a specific task within a Databricks Workflow?

Accepted Answer

By clicking the 'Logs' tab within the specific task run details in the Jobs UI.. Databricks provides a comprehensive UI that aggregates logs for every task execution. By navigating to the job run and selecting the specific task, the engineer can view stdout and stderr logs directly. This is the primary method for diagnosing runtime exceptions, syntax errors, or logic failures in automated data pipelines, facilitating rapid troubleshooting without needing external tools or direct node access.

Answer

In the Databricks Filesystem (DBFS) root directory under /logs.

Answer

By querying the 'sys.logs' table in the Unity Catalog.

Answer

In the cluster configuration's 'Advanced Options' tab.

Question 5

A data engineer is debugging a slow-running query. They notice that the data is skewed, causing one task to take significantly longer than others. Which approach effectively addresses this skew?

Accepted Answer

Add a salt column to the skewed key to distribute the data across more partitions.. Data skew occurs when one partition contains disproportionately more data than others, causing a single executor to bottleneck the entire stage. By using techniques like salt, broadcast joins, or repartitioning, the engineer can distribute the load more evenly across the cluster. This is essential for optimizing performance and preventing timeouts, ensuring that production jobs adhere to defined SLAs and utilize cluster resources efficiently.

Answer

Increase the cluster's disk size to allow for more local shuffle storage.

Answer

Change the file format from Parquet to CSV to reduce overhead.

Answer

Enable 'Auto-scaling' to automatically add more nodes during the skewed stage.

Question 6

Refer to the exhibit. A Databricks job fails with a 403 Forbidden error when trying to write to the S3 bucket. Why does this happen?

Accepted Answer

The policy is missing the 's3:PutObject' action required for writing data.. The provided IAM policy only grants 's3:GetObject' permissions, which is read-only. For a job to write data, it needs additional permissions like 's3:PutObject'. In production, strict adherence to the principle of least privilege is required; however, the policy must also enable the necessary write operations for the task to complete successfully. The 403 error is a direct consequence of this missing write-specific capability in the policy definition.

Answer

The Databricks cluster needs to be restarted to apply the new IAM policy.

Answer

The S3 bucket policy is blocking the request despite the IAM policy.

Answer

The 'Resource' ARN is missing the suffix '/*'.

Question 7

A production Databricks workflow involves a task that runs a notebook. The notebook takes 15 minutes to finish, but the workflow is set to timeout after 10 minutes. What happens?

Accepted Answer

The task is terminated and marked as 'Failed' by the scheduler.. When a task exceeds its configured 'timeout' value, the Databricks scheduler forcibly terminates the task execution. This is a deliberate safety measure to prevent runaway processes from consuming cluster resources indefinitely. In production, this highlights the necessity of monitoring execution times and setting appropriate timeouts that account for normal data volume fluctuations while catching truly stuck jobs that could impact cost and resource availability.

Answer

The workflow continues to run, but logs a warning about the duration.

Answer

The workflow pauses and waits for the engineer to manually approve continuation.

Answer

The task continues to run, but is moved to a background queue.

Question 8

A data engineer is investigating a job failure that occurred only in the production environment. Which TWO features in Databricks help in comparing the production environment to the development environment?

Accepted Answer

Use the 'View as JSON' feature in the Jobs UI to compare job configurations.. Comparing environment configurations is essential for troubleshooting parity issues. Databricks provides tools like the Jobs UI for export and version control integration for code, which help engineers identify subtle differences between environments. Ensuring parity is critical for reducing 'it works on my machine' scenarios. By utilizing these tools, engineers can verify cluster settings, library versions, and code versions to isolate why a process fails in production but succeeds in development.

Answer

Use the 'Cluster Logs' to compare the OS kernel versions of the nodes.

Answer

Enable the 'Debug Mode' on the Spark Driver to see raw system calls.

Answer

Query the 'workspace_users' table to see who ran the job last.

Question 9

Refer to the exhibit. Why is Task D marked as 'Skipped'?

Accepted Answer

Task D has a dependency on Task B, which failed.. In Databricks Workflows, a task is 'Skipped' if its upstream dependencies have not been met, typically because a required parent task failed. Because Task B failed, the downstream Task D—which likely depends on Task B—cannot execute. This mechanism prevents the workflow from processing invalid or incomplete data, maintaining the integrity of the data pipeline and preventing downstream failures from compounding existing upstream errors.

Answer

Task D reached its execution timeout limit before it could start.

Answer

The cluster running Task D was terminated by an administrator.

Answer

Task D was manually disabled in the job configuration.

Question 10

What is the primary benefit of using a Job Cluster instead of an All-Purpose Cluster for production workloads?

Accepted Answer

Job clusters are more cost-effective and provide better workload isolation.. Job clusters are specifically designed for automated production workloads. They are cheaper because they are ephemeral, terminating as soon as the job finishes, and they provide better isolation, ensuring that production jobs do not interfere with other development tasks. This is a critical best practice for cost management and system reliability, ensuring that production pipelines run in a clean, predictable environment that scales according to actual need.

Answer

Job clusters support more advanced libraries than All-Purpose clusters.

Answer

Job clusters can be manually resized while the job is currently running.

Answer

Job clusters are required to access data stored in the Unity Catalog.

Question 11

Refer to the exhibit. Which action is the most appropriate to resolve this memory-related failure during the job execution?

Accepted Answer

Upgrade the cluster to an instance type with higher memory capacity.. The error indicates an Out of Memory (OOM) condition where the container exceeded its allocated memory limit. Increasing the instance type to a memory-optimized VM size provides more RAM per executor, allowing Spark to process larger partitions without triggering the YARN memory killer. This is a common bottleneck in memory-intensive operations like wide transformations, where adjusting the cluster configuration is more effective than attempting to optimize complex query logic.

Answer

Decrease the number of partitions using spark.sql.shuffle.partitions.

Answer

Enable autoscaling to add more nodes to the cluster.

Answer

Change the file format from Parquet to CSV for better performance.

Question 12

A data engineer is preparing to deploy a production Databricks workflow. Which TWO best practices should be implemented to ensure maintainability and robust error handling?

Accepted Answer

Use Databricks Git folders for version control of production code.. Maintaining production environments requires strict adherence to modular code design and comprehensive observability. Using version control for notebooks and job configurations ensures that every change is tracked, audited, and reversible. Simultaneously, implementing robust notification alerts for job failures allows engineering teams to respond proactively to issues. These two practices collectively minimize the risk of deployment errors and reduce the Mean Time to Resolution (MTTR) when unexpected failures occur in production pipelines.

Answer

Hardcode credentials directly into the notebook for ease of access.

Answer

Avoid using libraries and rely only on built-in Spark functions.

Answer

Deploy code directly from the workspace to production.

Question 13

Refer to the exhibit. Which configuration change is required to enable multiple instances of this job to run simultaneously?

Accepted Answer

Set max_concurrent_runs to a value greater than 1 in the job settings.. The error message explicitly states that the job does not allow concurrent runs. By default, many jobs are configured for single-instance execution to prevent data contention or race conditions. To allow overlapping executions, the 'max_concurrent_runs' parameter must be adjusted in the job settings. This is a critical configuration for scenarios where independent data partitions need to be processed in parallel to meet aggressive latency requirements in a production system.

Answer

Increase the timeout threshold for the job.

Answer

Add more workers to the cluster assigned to the job.

Answer

Enable 'Retries' in the job configuration.

Question 14

A data engineer needs to inspect the logs of a long-running Databricks job that has already completed. Where should they navigate in the Databricks UI to find the driver logs?

Accepted Answer

The 'Runs' tab within the specific Job's detail page.. The Job Runs view provides a comprehensive audit trail of all executions, including specific task logs and driver output. Accessing the 'Runs' tab for a specific job allows the engineer to select a historical run and view the standard output and standard error streams. This is the primary interface for post-mortem debugging and understanding the execution context of failed tasks that are no longer actively running on a cluster.

Answer

The 'Clusters' tab under the 'Event Log' section.

Answer

The 'Workspace' folder where the notebook is stored.

Answer

The 'SQL Warehouses' monitoring dashboard.

Question 15

A data engineer is deploying a Databricks job using Databricks Asset Bundles. They want to ensure that the job uses a specific cluster configuration that is defined once and reused across multiple tasks. Which bundle feature should they use?

Accepted Answer

Define a `job_cluster` in the job's `job_clusters` section and reference it by key in each task.. The `job_clusters` section in a Databricks job definition allows specifying cluster configurations that can be referenced by multiple tasks via `job_cluster_key`. This is the correct way to define a cluster once and reuse it across tasks within the same job, ensuring consistency and simplifying maintenance. Other options either duplicate configuration or rely on external resources.

Answer

Use a `new_cluster` definition for each task, ensuring they all have identical configurations.

Answer

Define a cluster in the `resources/clusters` directory and reference it in the job.

Answer

Use an `existing_cluster_id` for all tasks, pointing to a pre-created cluster.