Databricks-DE-Pro Debugging and Deploying — Timed 10 Questions

Question 1

A data engineer is debugging a Databricks Job that frequently fails due to transient network issues while reading from an external S3 bucket. Which configuration should the engineer implement to improve job resilience?

10 minute time limit — choose an answer to begin.

Accepted Answer

Set the 'max_retries' attribute in the task configuration to a value greater than zero.. Databricks Jobs use the 'max_retries' field within the 'retry_policy' object of the Job API, but it is not a top-level attribute as implied by the option. Furthermore, the question asks for a configuration to improve resilience against transient network issues during reads; while retries are correct, the specific configuration for Databricks Jobs is 'max_retries' inside a 'retry_policy' block, and the explanation incorrectly describes it as a simple task attribute.

Answer

Increase the driver node instance type to provide more memory for buffering S3 connections.

Answer

Implement a try-catch block within the Python script to manually restart the cluster.

Answer

Use a standard cluster instead of a job cluster to ensure persistent connectivity to S3.

Question 2

A data engineer is using Databricks Asset Bundles (DABs) to deploy a project. The deployment fails because the local configuration differs from the remote environment. What is the best way to synchronize the environment?

Accepted Answer

Update the bundle target configuration and run 'databricks bundle deploy'.. Databricks Asset Bundles (DABs) are declarative. When local configuration differs from the remote environment, the 'databricks bundle deploy' command calculates the difference and applies the necessary changes to the workspace to match the local configuration. Option B is the correct approach as it ensures the target configuration is accurate before triggering the reconciliation process.

Answer

Manually delete all resources in the workspace and redeploy the bundle.

Answer

Force the deployment by setting the 'ignore-errors' flag to true.

Answer

Use the 'databricks bundle import' command to pull the remote state locally.

Question 3

A developer is writing code that reads a large amount of data from a table and applies a complex transformation. The job succeeds locally but fails in the production job cluster. What is the most likely reason?

Accepted Answer

The local notebook has implicit access to cached data not available in production.. Interactive notebook environments (often referred to as 'local' development in this context) maintain state across cell executions. A developer might have cached intermediate results or created temporary views in previous steps that are not explicitly defined in the production script. When the job runs in a production cluster, it starts with a fresh SparkSession. If the complex transformation relied on those implicit caches to stay within memory limits or to bypass expensive re-computations, the production job will fail. This is a common 'gotcha' when moving from interactive development to automated jobs.

Answer

The production cluster is using a different version of the Python interpreter.

Answer

The production cluster is configured to use a single node, which is faster.

Answer

The Spark UI is disabled in the production job cluster for performance.

Question 4

A data engineer is debugging a Databricks Job that fails intermittently due to cluster startup delays. Which strategy best minimizes this impact on production reliability?

Accepted Answer

Attach the job to an existing instance pool.. Utilizing instance pools pre-allocates a set of idle, ready-to-use instances, effectively reducing cluster start-up times to near-zero. This approach is critical for mission-critical jobs where strict SLAs must be met. By decoupling cluster provisioning from job execution, engineers ensure that transient cloud provider capacity issues do not trigger cascading failures in automated ETL pipelines, thereby improving overall system stability and reducing time-to-completion metrics for downstream dependencies.

Answer

Increase the maximum number of workers in the cluster configuration.

Answer

Enable cluster termination after 10 minutes of inactivity.

Answer

Switch to a single-node cluster to eliminate node communication overhead.

Question 5

A data engineer has a Databricks Asset Bundle (DAB) project with a job defined in `resources/jobs.yml`. After running `databricks bundle validate`, they see the error: `unknown field: job_clusters`. They are using Databricks CLI version 0.205.0. What is the most likely cause of this error?

Accepted Answer

The Databricks CLI version is too old and does not support the `job_clusters` field in the bundle schema.. The error occurs because the installed Databricks CLI version does not include `job_clusters` in its bundle schema. Asset Bundles are version-dependent, and fields are added over time. Upgrading the CLI ensures compatibility with newer bundle features. The other options misdiagnose the cause as a missing field, indentation, or missing metadata, which would produce different errors.

Answer

The `job_clusters` field is not supported in the bundle schema for the CLI version; they should use `new_cluster` instead.

Answer

The bundle configuration is missing a `bundle` section that defines the CLI version compatibility.

Answer

The `job_clusters` field is defined at the wrong indentation level, causing the YAML parser to misinterpret it.

Question 6

A data engineer has a Databricks job that reads from a Delta table and writes to another Delta table. The job sometimes fails with a 'FileNotFoundException'. They suspect that the source table is being overwritten by another process while the job is running. Which Delta Lake feature should they use to ensure the job reads a consistent snapshot of the source table?

Accepted Answer

Set the Spark configuration 'spark.databricks.delta.snapshotIsolation' to 'true' to enable snapshot isolation.. Delta Lake provides snapshot isolation for reads by default, meaning a query sees a consistent version of the table. However, if another process overwrites the table and removes old files, a long-running read may fail with FileNotFoundException. To prevent this, the job should either use time travel to pin to a specific version or ensure that vacuum does not delete files needed for the read. Setting 'spark.databricks.delta.snapshotIsolation' to 'true' reinforces this behavior, but it is already the default. The key is to manage file retention and possibly use time travel.

Answer

Use the 'mergeSchema' option when reading the table to handle schema changes.

Answer

Configure the job to read the table with the 'ignoreFileNotFound' option set to 'true'.

Answer

Enable Delta Lake time travel by setting the Spark configuration 'spark.databricks.delta.timeTravel.enabled' to 'true'.

Question 7

A data engineer is deploying a new production Databricks job using Databricks Asset Bundles (DABs). The bundle includes a job with a schedule and a task that runs a Python wheel. After running `databricks bundle deploy -t prod`, the job appears in the workspace but the schedule is not active. What is the most likely cause?

Accepted Answer

The schedule is defined in the bundle but the job's `schedule` property is set to `PAUSED` in the target override.. The schedule not being active after deployment often results from a target-specific override in the bundle configuration that sets the schedule status to PAUSED. Databricks Asset Bundles allow per-target customizations, and a common practice is to pause schedules in production until explicitly enabled. The other options either describe issues that would prevent deployment or are unrelated to schedule activation.

Answer

The job was deployed with `--fail-on-active-runs` flag, which pauses schedules by default.

Answer

The bundle target `prod` does not have a `workspace.host` configured in `databricks.yml`.

Answer

The job cluster does not have autoscaling enabled, so the schedule cannot be activated.

Question 8

A data engineer is troubleshooting a Databricks Workflow job that fails intermittently with 'Cluster terminated: Cloud provider launch failure'. The job uses a job cluster with a fixed number of workers. Which change is most likely to improve reliability?

Accepted Answer

Configure the job cluster to use spot instances with fallback to on-demand. Cloud provider launch failures are caused by the inability to provision requested instances, often due to spot instance unavailability. Configuring spot instances with on-demand fallback lets Databricks use alternative capacity when spot is scarce, directly improving the reliability of job cluster launches for intermittent failures.

Answer

Add a cluster init script to pre-install dependencies

Answer

Increase the driver node size

Answer

Enable autoscaling on the job cluster

Question 9

A data engineer is using Databricks Repos to manage a project. They need to ensure that the production job always uses the latest committed version of the code from the main branch. The job is configured to run from a Repo path. Which Git reference should be specified in the job configuration to achieve this?

Accepted Answer

The remote branch reference `refs/heads/main`. To ensure a job always uses the latest committed version from the main branch, the job should reference the branch (e.g., `main` or `refs/heads/main`). Databricks Repos will check out the latest commit on that branch when the job runs. Using a specific commit SHA pins to a fixed version, and tags are static. The full ref `refs/heads/main` is explicit and correct.

Answer

The tag `latest`

Answer

A specific commit SHA

Answer

The branch name `main`

Question 10

A data engineer maintains a Databricks Workflow that runs a notebook task referencing an external JAR library stored in DBFS at dbfs:/libs/etl-utils-1.2.jar. The notebook runs successfully in development on an all-purpose cluster because the engineer installed the library manually. In production, the Workflow task fails immediately with ClassNotFoundException. The cluster policy for the job cluster does not permit installing libraries through the UI. What should the engineer do to resolve the failure?

Accepted Answer

Add the JAR as a task library on the notebook task in the Workflow definition, specifying the dbfs:/libs/etl-utils-1.2.jar path.. Task-level libraries in a Workflow definition ensure that required JARs are installed on the job cluster before execution, independent of cluster policy restrictions. This is the supported, automated method for making libraries available to notebook tasks. Manually installing or using init scripts introduces fragility and does not guarantee classpath inclusion.

Answer

Modify the cluster policy to allow library installation, then install the JAR manually on the job cluster before each run.

Answer

Configure the job cluster to use an init script that downloads the JAR from DBFS and places it in the Spark classpath.

Answer

Use %pip install in the notebook to install the JAR from DBFS at the start of each task execution.