Debugging and Deploying practice questions

Increase the driver node instance type to provide more memory for buffering S3 connections.

Why it fails: Increasing driver memory primarily addresses out-of-memory errors or scheduling bottlenecks rather than transient network connectivity issues. While more memory helps with massive metadata operations, it does not provide an automated mechanism for re-attempting failed network requests during data ingestion tasks from external storage.

Implement a try-catch block within the Python script to manually restart the cluster.

Why it fails: Manually restarting a cluster from within a notebook or script is an anti-pattern that creates complex state management issues. It interrupts the job lifecycle and does not guarantee that the subsequent retry will resolve the connectivity issue, whereas built-in job retries are managed by the platform.

Use a standard cluster instead of a job cluster to ensure persistent connectivity to S3.

Why it fails: Job clusters are specifically optimized for ephemeral workloads and provide the same connectivity capabilities as standard clusters. Switching to a standard cluster does not mitigate transient network failures or provide better error handling for S3 interactions; it merely increases costs and ignores the best practices for production.

Manually delete all resources in the workspace and redeploy the bundle.

Why it fails: Manually deleting resources is destructive and contradicts the purpose of using infrastructure-as-code. DABs are designed to manage the lifecycle of resources incrementally. Manual intervention increases the risk of downtime and accidental deletion of production data or dependencies that are not managed by the bundle itself.

Force the deployment by setting the 'ignore-errors' flag to true.

Why it fails: Forcing a deployment by ignoring errors is dangerous, as it can hide critical configuration mismatches that could lead to runtime failures. The goal of the deployment process is to achieve a consistent state, and ignoring errors essentially blinds the engineer to the actual differences between the local and remote environments.

Use the 'databricks bundle import' command to pull the remote state locally.

Why it fails: The 'import' command is not used to synchronize configurations in this manner. DABs follow an 'as-code' paradigm where the local repository is the source of truth. Modifying the local files to match the remote workspace would overwrite your version-controlled configuration, leading to technical debt and configuration management issues.

The production cluster is using a different version of the Python interpreter.

Why it fails: While version mismatches can cause syntax errors, they usually manifest as immediate failures rather than intermittent or resource-based ones. It is standard practice in Databricks to use consistent runtimes. If the environment is correctly set up, the interpreter version should not be the bottleneck for data-intensive processing.

The production cluster is configured to use a single node, which is faster.

Why it fails: Single-node clusters are not faster for large-scale distributed tasks; they are intended for small workloads. Using a single node for large datasets is an anti-pattern that would lead to immediate failure, not a scenario where the job 'succeeds locally' but fails in production because of speed.

The Spark UI is disabled in the production job cluster for performance.

Why it fails: Disabling the Spark UI has no impact on the successful execution of the job code. The job would still attempt to run the transformations, and the resource limits would be hit regardless of whether the UI is enabled or not. This is not a cause for the logic failure.

Increase the maximum number of workers in the cluster configuration.

Why it fails: Increasing the maximum worker count addresses scaling capacity during runtime but does not mitigate the initial latency caused by provisioning new virtual machines from the cloud provider. This setting only controls how the cluster reacts to workload spikes after the cluster has successfully initialized and transitioned to a running state.

Enable cluster termination after 10 minutes of inactivity.

Why it fails: Auto-termination is a cost-optimization feature that shuts down clusters when idle. While it saves money, it forces subsequent jobs to incur cold-start penalties when the cluster must be recreated. This setting actively works against the goal of reducing startup latency for frequently scheduled production jobs.

Switch to a single-node cluster to eliminate node communication overhead.

Why it fails: Single-node clusters are intended for lightweight development and testing. They lack the parallelism required for production ETL pipelines and do not resolve the fundamental issue of infrastructure startup latency. Furthermore, they are not designed to handle the data volumes typical of enterprise-grade Databricks production environments.

The `job_clusters` field is not supported in the bundle schema for the CLI version; they should use `new_cluster` instead.

Why it fails: In Databricks Asset Bundles, `job_clusters` is a valid top-level field for defining reusable clusters in a job. The error indicates a schema mismatch, not that the field is unsupported. Using `new_cluster` would be incorrect because that is for task-level cluster definitions, and it would not resolve the unknown field error.

The bundle configuration is missing a `bundle` section that defines the CLI version compatibility.

Why it fails: The `bundle` section in `databricks.yml` specifies the bundle name and other metadata, but it does not control schema version compatibility. The error is about an unknown field in the job definition, not about missing bundle metadata. Adding a `bundle` section would not fix the validation error.

The `job_clusters` field is defined at the wrong indentation level, causing the YAML parser to misinterpret it.

Why it fails: While YAML indentation errors can cause parsing issues, the error message `unknown field: job_clusters` indicates that the field is recognized by the parser but not allowed by the schema. If it were an indentation issue, the error would likely be a YAML syntax error, not an unknown field error.

Use the 'mergeSchema' option when reading the table to handle schema changes.

Why it fails: The 'mergeSchema' option is used to evolve the schema when reading Parquet files or Delta tables, but it does not address consistency issues. It allows reading data with different schemas, but the FileNotFoundException is caused by files being deleted or replaced during the read, not by schema mismatches. This option would not prevent the error.

Configure the job to read the table with the 'ignoreFileNotFound' option set to 'true'.

Why it fails: There is no 'ignoreFileNotFound' option in Delta Lake. This is not a valid configuration. Spark may have options like 'ignoreMissingFiles' for Parquet, but Delta Lake does not support this. Using such an option would not be appropriate and could lead to silent data loss. The correct approach is to ensure snapshot isolation and manage file retention.

Enable Delta Lake time travel by setting the Spark configuration 'spark.databricks.delta.timeTravel.enabled' to 'true'.

Why it fails: Time travel allows querying previous versions of a table, but it does not automatically provide a consistent snapshot for concurrent reads. The job would still need to specify a version or timestamp, and the FileNotFoundException could still occur if files are removed by vacuum. This setting alone does not prevent concurrent overwrites from affecting the read.

The job was deployed with `--fail-on-active-runs` flag, which pauses schedules by default.

Why it fails: The `--fail-on-active-runs` flag causes the deployment to fail if there are active runs, but it does not pause or deactivate schedules. This flag is used to prevent conflicts during deployment, not to alter schedule state. The scenario says the job appears with an inactive schedule, so this flag is irrelevant and would not cause that behavior.

The bundle target `prod` does not have a `workspace.host` configured in `databricks.yml`.

Why it fails: The `workspace.host` is required for authentication and deployment, but if it were missing, the deploy command would fail entirely, not deploy the job without an active schedule. The job is present, so the host is configured. This option is a plausible misdirection because host configuration is part of DABs setup, but it does not explain a deployed job with an inactive schedule.

The job cluster does not have autoscaling enabled, so the schedule cannot be activated.

Why it fails: Autoscaling is unrelated to whether a job schedule is active. A job can have a fixed-size cluster and still run on a schedule. The cluster configuration does not control the schedule's enabled state. This distractor confuses cluster scaling with job scheduling, which are independent features in Databricks.

Add a cluster init script to pre-install dependencies

Why it fails: Init scripts run after the cluster is provisioned and do not affect whether the cloud provider can launch instances. A launch failure occurs before init scripts execute. While init scripts are useful for dependency management, they cannot mitigate provisioning failures. The engineer should adjust instance configuration to improve launch reliability.

Increase the driver node size

Why it fails: A larger driver node addresses driver memory or CPU bottlenecks, but a cloud provider launch failure occurs when the cloud cannot provision the requested instances. Increasing driver size may even worsen the issue by requesting a less available instance type. The engineer should focus on instance availability and retry behavior, not driver capacity.

Enable autoscaling on the job cluster

Why it fails: Autoscaling adjusts worker count based on workload but does not resolve the inability to launch instances. If the cloud provider cannot provision the requested instance type, autoscaling will not help and may introduce additional launch attempts. The root cause is instance availability, so the engineer should address instance type and fallback options instead.

The tag `latest`

Why it fails: Git tags are static references and do not automatically update to the latest commit. A tag named `latest` would need to be manually moved or recreated to point to a new commit. Databricks Repos does not treat `latest` as a dynamic reference. Therefore, this would not ensure the job uses the latest committed version unless the tag is updated externally.

A specific commit SHA

Why it fails: Using a specific commit SHA pins the job to an exact version of the code, ensuring reproducibility. However, it does not automatically use the latest committed version; it uses a fixed version. To always use the latest, you would need to update the SHA manually or via automation. This option is the opposite of the requirement.

The branch name `main`

Why it fails: Specifying the branch name `main` in a job configuration means the job will use the latest commit on that branch at the time the job is triggered. However, if the job is configured with a branch, it will pull the latest code each run, which might not be a specific committed version. The requirement is to use the latest committed version, but using a branch can lead to non-reproducible runs if new commits are pushed. It does not guarantee a fixed version.

Modify the cluster policy to allow library installation, then install the JAR manually on the job cluster before each run.

Why it fails: Even if the policy is relaxed, manual installation on an ephemeral job cluster is not persistent; the cluster terminates after the run and the library is lost. This approach is not automated and would require external orchestration, making the Workflow unreliable and defeating the purpose of job clusters.

Configure the job cluster to use an init script that downloads the JAR from DBFS and places it in the Spark classpath.

Why it fails: Init scripts can copy files, but they run before the Spark driver starts and do not automatically add JARs to the Spark classpath. The engineer would need to set spark.jars or similar configuration, which is more complex and error-prone than attaching the library directly to the task.

Use %pip install in the notebook to install the JAR from DBFS at the start of each task execution.

Why it fails: %pip install expects a Python package index or wheel, not a raw JAR file. While %pip can install local wheels, it cannot install a JAR directly. The JAR must be attached as a library, so this approach fails with a pip error rather than resolving the missing class.

Use the Azure Key Vault SDK directly in the Python wheel to fetch the secret at runtime.

Why it fails: Using the Azure Key Vault SDK directly requires managing authentication credentials for Key Vault, which could lead to hardcoding or exposing those credentials. Databricks secret scopes abstract this and handle authentication securely. While possible, it introduces additional complexity and potential security risks. The native secret scope integration is preferred for simplicity and security.

Embed the secret in the Python wheel as a constant and obfuscate it with base64 encoding.

Why it fails: Embedding a secret in the wheel, even obfuscated, is insecure because the wheel can be decompiled or inspected. Base64 encoding is not encryption and provides no security. This approach violates the requirement to avoid hardcoding and would expose the secret to anyone with access to the wheel. It is never recommended for production secrets.

Set the secret as an environment variable in the cluster's Spark configuration using `spark.executorEnv.DB_PASSWORD`.

Why it fails: Setting a secret directly in the Spark configuration as an environment variable would expose it in plain text in the cluster configuration and potentially in logs. This violates the requirement to avoid hardcoding and exposure. While environment variables can be used, they should be populated via secret references, not directly. This option is insecure and not recommended.

Increase the number of shuffle partitions by setting spark.sql.shuffle.partitions to a higher value, such as 4000.

Why it fails: Increasing shuffle partitions creates more, smaller partitions, but a skewed key still maps to a single partition, so the hot partition remains large. This can reduce memory per partition for non-skewed data but does not solve the skew-induced OOM. The task processing the skewed key will still receive a disproportionate amount of data.

Repartition the DataFrame on a different column before the groupBy to distribute data evenly across partitions.

Why it fails: Repartitioning on a different column changes the distribution but does not guarantee even distribution for the groupBy key. If the new column is also skewed, the problem persists. Moreover, repartitioning adds a full shuffle and may not align with the groupBy, leading to incorrect results or additional shuffles.

Cache the DataFrame before the groupBy to avoid recomputation and reduce memory pressure.

Why it fails: Caching stores data in memory or disk but does not address the root cause of skew. The groupBy will still create a large partition for the skewed key, and caching may consume additional memory, potentially worsening the OOM. Caching is beneficial for iterative algorithms, not for skew mitigation.

Add a bundle section to the databricks.yml file with a name and include the path to the configuration.

Why it fails: A bundle section with a name is required in databricks.yml, but the error indicates the file cannot be found, not that it is malformed. Adding a bundle section does not help if the CLI cannot locate the file. The issue is with file discovery, not content.

Run the command with the --config-file flag pointing to the absolute path of databricks.yml.

Why it fails: While the --config-file flag can specify a custom configuration file, the default behavior should find databricks.yml. If the file is in the current directory, the CLI should locate it without the flag. Using the flag might work, but it is a workaround, not the root cause. The error suggests authentication or directory issues.

Rename the configuration file to bundle.yml, as the CLI expects that filename by default.

Why it fails: Databricks Asset Bundles use databricks.yml as the default configuration file, not bundle.yml. Renaming the file would cause the CLI to not recognize it as a bundle configuration, leading to the same or a different error. The correct filename is databricks.yml.

Set the task timeout to zero to prevent the workflow from ever stopping during a failure.

Why it fails: Setting a timeout to zero is not a valid configuration for ensuring data integrity during dependencies. It essentially disables the monitoring of task execution time, which can lead to hung jobs consuming resources indefinitely without triggering the necessary alerts or failure handling logic required for production stability.

Hardcode the file path of the upstream output into the downstream task configuration.

Why it fails: Hardcoding paths is an anti-pattern that couples tasks tightly and creates maintenance challenges. It ignores the workflow dependency graph, making it impossible for the scheduler to manage retries or state correctly. If the upstream task fails, the downstream task might still try to read non-existent data.

Disable all retries to ensure that the error log is captured immediately upon failure.

Why it fails: Disabling retries does not assist in managing dependencies; it only guarantees the job stops at the first sign of trouble. While it might assist in capturing logs, it fails to utilize Databricks' built-in resilience features that handle transient errors automatically, thus reducing overall workflow reliability and efficiency.

The cluster is running an outdated version of the Spark runtime.

Why it fails: An outdated runtime would typically trigger incompatibility errors or library missing errors rather than a table-not-found exception. The error clearly specifies a metadata lookup failure within the catalog service, which is independent of the specific Spark version running on the cluster nodes during job execution.

The job is running under a service principal that lacks permissions to the Hive Metastore.

Why it fails: Insufficient permissions usually result in an 'AccessDenied' or 'AuthorizationException' rather than a 'TableNotFound' error. The system is successfully querying the catalog but failing to locate the specific table object, implying a configuration or environment mismatch rather than a security or identity-based restriction issue.

The cluster has insufficient memory to load the table metadata.

Why it fails: Metadata operations are lightweight and managed by the metastore service, not the cluster's RAM. Insufficient memory would manifest as a 'java.lang.OutOfMemoryError' during the processing stage, not as a failure to resolve a table name. The error is strictly related to catalog resolution and object location.

The Databricks CLI version on the build agent is incompatible with the server.

Why it fails: While CLI versioning can cause issues, the 'cannot find workspace' error specifically points to a connection or authentication failure. Incompatibility errors typically result in command-specific error messages or warnings about deprecated API calls, rather than a failure to locate the workspace itself during the authentication phase.

The workspace API is currently disabled for security reasons.

Why it fails: If the API were disabled, the error would specifically return a '403 Forbidden' or '401 Unauthorized' response from the server. The CLI indicates it cannot find the workspace, which usually implies that the client-side configuration lacks the necessary endpoint information to even attempt the initial connection.

The CI/CD runner is missing the required library dependencies like PySpark.

Why it fails: The Databricks CLI is a standalone binary that does not depend on PySpark for its execution or workspace connection. Missing libraries would only become an issue later if the pipeline attempted to execute Spark code, not during the initial authentication and workspace discovery phase of the deployment.

Increase the number of worker nodes in the cluster.

Why it fails: Increasing worker count does not alleviate memory pressure on the driver node. In fact, if the driver is collecting data from a larger number of workers, the memory bottleneck might worsen. Driver-side issues must be addressed by changing how data is gathered or processed, not by adding workers.

Update the cluster's Spark configuration to disable the driver's log monitoring.

Why it fails: Disabling log monitoring does not resolve memory leaks or inefficient code patterns. While it might marginally reduce the memory footprint, it is a dangerous practice that obscures root causes and prevents the engineer from monitoring the health of the application. It does not address the underlying architectural problem.

Lower the 'spark.driver.maxResultSize' setting in the cluster configuration.

Why it fails: Lowering the maxResultSize will only cause the job to fail faster with a 'Result size exceeded' error. It does not solve the underlying issue of why the driver is being asked to handle more data than it can manage. This setting is a safeguard, not a solution for performance.

Grant all developers full cluster permissions to access logs directly on the nodes.

Why it fails: Granting excessive permissions violates the principle of least privilege and introduces security risks. Production environments should use centralized logging services rather than giving individual developers direct access to cluster nodes. Debugging should be performed via standard interface tools, not by logging into underlying infrastructure components directly.

Run all production jobs as the root user to avoid permission-related errors.

Why it fails: Running jobs as the root user is a critical security vulnerability and is not a debugging strategy. Errors related to permissions should be resolved by configuring appropriate IAM roles or service principals, not by bypassing the security model, which exposes the entire workspace to unauthorized file modifications.

In the Databricks Filesystem (DBFS) root directory under /logs.

Why it fails: DBFS is a virtual file system and does not contain the structured execution logs for individual tasks. Storing application logs here would create significant management overhead and would not be easily accessible via the Job UI, making it a poor choice for diagnosing workflow failures in production.

By querying the 'sys.logs' table in the Unity Catalog.

Why it fails: There is no 'sys.logs' table in Unity Catalog. While audit logs exist for security monitoring, they do not contain the application-level stdout or stderr logs generated by task execution. Application logs remain within the compute resource and are accessed via the job run interface in the platform UI.

In the cluster configuration's 'Advanced Options' tab.

Why it fails: The 'Advanced Options' tab in cluster configuration is used to set Spark parameters, library paths, and environment variables. It does not provide access to the execution logs of past job runs. The settings here affect the environment in which code runs, rather than reporting on its past performance.

Increase the cluster's disk size to allow for more local shuffle storage.

Why it fails: Increasing disk size does not solve the underlying problem of uneven data distribution. The bottleneck remains at the CPU and memory level of the overloaded executor. The job will still suffer from the same performance degradation because the data processing load is not being shared among other executors.

Change the file format from Parquet to CSV to reduce overhead.

Why it fails: CSV is a text-based, uncompressed format that is significantly less efficient than Parquet. Switching to CSV would increase I/O overhead and memory consumption, potentially making the skew issue worse. Parquet's columnar structure is optimized for performance, and it is not the cause of data skew in join operations.

Enable 'Auto-scaling' to automatically add more nodes during the skewed stage.

Why it fails: Auto-scaling adds nodes to the cluster, but it cannot re-partition data that is already being processed by a single task. Even with more nodes, the skewed partition will still be assigned to one executor, leaving other nodes idle while waiting for that single, overloaded task to finish processing.