Scenario practice questions

A data engineer is investigating a job failure that occurred only in the production environment. Which TWO features in Databricks help in comparing the production environment to the development environment?

Trap 1: Use the 'Cluster Logs' to compare the OS kernel versions of the…

Comparing OS kernel versions is rarely the solution for job failures, as Databricks manages the underlying runtime environment. If there is a disparity, it would be due to selecting different Databricks Runtime versions, not the specific OS kernel, which is abstracted away from the end user.

Trap 2: Enable the 'Debug Mode' on the Spark Driver to see raw system calls.

Debug mode for raw system calls is not a standard feature in Databricks and would not provide useful context for comparing environment configurations. This level of debugging is excessively low-level and does not address the high-level configuration differences that typically cause environment-specific pipeline failures.

Trap 3: Query the 'workspace_users' table to see who ran the job last.

Knowing who ran a job last does not help identify configuration differences. While auditing is useful for security, the technical investigation of why a job fails requires comparing settings, libraries, and code, not the identity of the user who triggered the last execution.

A
Use the 'View as JSON' feature in the Jobs UI to compare job configurations.
Exporting and comparing job JSON configurations is an effective way to identify discrepancies in parameters, cluster sizes, or timeout settings. This allows engineers to systematically check for configuration drift between the development workspace and the production workspace, which is a common source of environment-specific failures.
B
Use the 'Cluster Logs' to compare the OS kernel versions of the nodes.
Why it fails: Comparing OS kernel versions is rarely the solution for job failures, as Databricks manages the underlying runtime environment. If there is a disparity, it would be due to selecting different Databricks Runtime versions, not the specific OS kernel, which is abstracted away from the end user.
C
Review the git branch history and configuration files in the CI/CD pipeline.
Checking the CI/CD pipeline ensures that the exact code and configuration being deployed to production matches what was tested in development. This identifies issues where different branches were accidentally merged or where environment-specific variables were incorrectly injected into the production job, causing unexpected runtime behavior.
D
Enable the 'Debug Mode' on the Spark Driver to see raw system calls.
Why it fails: Debug mode for raw system calls is not a standard feature in Databricks and would not provide useful context for comparing environment configurations. This level of debugging is excessively low-level and does not address the high-level configuration differences that typically cause environment-specific pipeline failures.
E
Query the 'workspace_users' table to see who ran the job last.
Why it fails: Knowing who ran a job last does not help identify configuration differences. While auditing is useful for security, the technical investigation of why a job fails requires comparing settings, libraries, and code, not the identity of the user who triggered the last execution.

Question 2hardmulti select

A Data Engineer is implementing a medallion architecture. Which THREE steps are critical for effectively implementing a high-quality 'Silver' layer from 'Bronze' data?

Trap 1: Apply business logic and complex transformations to create…

Aggregation should generally happen at the Gold layer. The Silver layer is intended to be a refined, tabular version of the source data, kept at the grain of the input. Creating summaries at the Silver layer reduces the granularity and flexibility of the data, making it less useful for general analysis.

Trap 2: Convert all column names to uppercase to ensure case-insensitive…

While standardizing column names is a good practice, it is not a 'critical' step for data quality in the same way as deduplication or type enforcement. It is an optional naming convention rather than a fundamental requirement for ensuring data integrity or structural validity in a medallion architecture.

A
Enforce strict schema validation on incoming Bronze files.
Schema enforcement at the Silver layer is critical to ensure data consistency. By validating that columns match expected types and structures, you prevent downstream failures in analytical queries and reporting tools, which often lack the robustness to handle unexpected schema changes or malformed data types automatically.
B
Perform deduplication to ensure unique records based on business keys.
Data quality at the Silver level requires uniqueness. Deduplication ensures that metrics are calculated based on accurate counts and that analytical models are not biased by repeated data points. This is foundational for providing reliable business insights and avoiding costly mistakes in decision-making based on inflated or incorrect datasets.
C
Apply business logic and complex transformations to create aggregate summary tables.
Why it fails: Aggregation should generally happen at the Gold layer. The Silver layer is intended to be a refined, tabular version of the source data, kept at the grain of the input. Creating summaries at the Silver layer reduces the granularity and flexibility of the data, making it less useful for general analysis.
D
Convert all column names to uppercase to ensure case-insensitive consistency.
Why it fails: While standardizing column names is a good practice, it is not a 'critical' step for data quality in the same way as deduplication or type enforcement. It is an optional naming convention rather than a fundamental requirement for ensuring data integrity or structural validity in a medallion architecture.
E
Standardize data formats (e.g., timestamps, currency codes) across all source systems.
Standardization is crucial for cross-system analysis. By normalizing formats like date/time and currency in the Silver layer, you enable joining and comparing data from disparate sources, which is the primary purpose of the Silver layer: creating a unified, trustworthy dataset that serves as the foundation for downstream analytical workloads.

Question 3hardmulti select

A data engineer is troubleshooting a Databricks Workflow where a downstream task relies on an upstream task's output. Which TWO actions ensure the data dependency is correctly handled during a failure scenario?

Trap 1: Set the task timeout to zero to prevent the workflow from ever…

Setting a timeout to zero is not a valid configuration for ensuring data integrity during dependencies. It essentially disables the monitoring of task execution time, which can lead to hung jobs consuming resources indefinitely without triggering the necessary alerts or failure handling logic required for production stability.

Trap 2: Hardcode the file path of the upstream output into the downstream…

Hardcoding paths is an anti-pattern that couples tasks tightly and creates maintenance challenges. It ignores the workflow dependency graph, making it impossible for the scheduler to manage retries or state correctly. If the upstream task fails, the downstream task might still try to read non-existent data.

Trap 3: Disable all retries to ensure that the error log is captured…

Disabling retries does not assist in managing dependencies; it only guarantees the job stops at the first sign of trouble. While it might assist in capturing logs, it fails to utilize Databricks' built-in resilience features that handle transient errors automatically, thus reducing overall workflow reliability and efficiency.

A
Configure the downstream task to use the 'depends_on' attribute to reference the upstream task ID.
The 'depends_on' attribute explicitly defines the task DAG structure within Databricks Workflows. By creating this dependency, the scheduler guarantees that the downstream task will only execute if the upstream task succeeds, preventing erroneous runs when input data is missing or corrupted due to preceding failures.
B
Set the task timeout to zero to prevent the workflow from ever stopping during a failure.
Why it fails: Setting a timeout to zero is not a valid configuration for ensuring data integrity during dependencies. It essentially disables the monitoring of task execution time, which can lead to hung jobs consuming resources indefinitely without triggering the necessary alerts or failure handling logic required for production stability.
C
Enable the 'repair and rerun' feature to target only the failed tasks in the pipeline.
The repair and rerun feature allows engineers to execute only the failed tasks and their downstream dependents. This preserves the existing successful outputs of the workflow and ensures that the system only attempts to process data that was previously blocked by the specific task failure event.
D
Hardcode the file path of the upstream output into the downstream task configuration.
Why it fails: Hardcoding paths is an anti-pattern that couples tasks tightly and creates maintenance challenges. It ignores the workflow dependency graph, making it impossible for the scheduler to manage retries or state correctly. If the upstream task fails, the downstream task might still try to read non-existent data.
E
Disable all retries to ensure that the error log is captured immediately upon failure.
Why it fails: Disabling retries does not assist in managing dependencies; it only guarantees the job stops at the first sign of trouble. While it might assist in capturing logs, it fails to utilize Databricks' built-in resilience features that handle transient errors automatically, thus reducing overall workflow reliability and efficiency.

Question 4mediummultiple choice

A data engineer is configuring an Auto Loader stream to ingest JSON files from an S3 bucket into a Bronze Delta table. The source bucket contains both .json and .json.gz files, and the engineer wants to ensure that only .json files are processed. Which parameter should be set to achieve this?

Trap 1: cloudFiles.schemaLocation

The cloudFiles.schemaLocation parameter specifies a location to store the inferred schema and metadata for the stream. It is used for schema evolution and checkpointing, not for filtering files. Setting this parameter would not affect which files are ingested; it only manages schema information and does not restrict file selection based on extension.

Trap 2: cloudFiles.includeExistingFiles

This parameter controls whether Auto Loader processes files that already exist in the source location when the stream starts. It does not filter by file extension. Setting it to true would ingest all existing files regardless of format, and setting it to false would ignore existing files. It is unrelated to selecting only .json files and would not prevent .json.gz files from being ingested.

Trap 3: cloudFiles.format

The cloudFiles.format parameter specifies the format of the source files, such as json, csv, or parquet. It tells Auto Loader how to parse the file contents, but it does not filter which files are picked up based on their extension. All files in the directory would still be considered, and .json.gz files would also be read as JSON, potentially causing issues.

A
cloudFiles.schemaLocation
Why it fails: The cloudFiles.schemaLocation parameter specifies a location to store the inferred schema and metadata for the stream. It is used for schema evolution and checkpointing, not for filtering files. Setting this parameter would not affect which files are ingested; it only manages schema information and does not restrict file selection based on extension.
B
cloudFiles.includeExistingFiles
Why it fails: This parameter controls whether Auto Loader processes files that already exist in the source location when the stream starts. It does not filter by file extension. Setting it to true would ingest all existing files regardless of format, and setting it to false would ignore existing files. It is unrelated to selecting only .json files and would not prevent .json.gz files from being ingested.
C
cloudFiles.format
Why it fails: The cloudFiles.format parameter specifies the format of the source files, such as json, csv, or parquet. It tells Auto Loader how to parse the file contents, but it does not filter which files are picked up based on their extension. All files in the directory would still be considered, and .json.gz files would also be read as JSON, potentially causing issues.
D
cloudFiles.pathGlobFilter
The cloudFiles.pathGlobFilter parameter allows you to specify a glob pattern to filter files based on their path or extension. By setting it to '*.json', Auto Loader will only process files ending with .json, excluding .json.gz files. This is the correct way to selectively ingest files by extension in Auto Loader, ensuring only the desired files are read.

Question 5hardmultiple choice

A data engineer is troubleshooting a Delta Live Tables pipeline that intermittently fails with 'StreamingQueryException: Job aborted due to stage failure'. The pipeline processes streaming data from a Kafka source. Which monitoring approach will best help identify the root cause of these intermittent failures?

Trap 1: Set up a Databricks SQL alert on the target Delta table's row count…

A row count alert can indicate that data is not being written, but it does not explain why the pipeline failed. The intermittent nature of the failure requires access to error details, which a row count metric cannot provide. This method is reactive and does not help in diagnosing the underlying cause of the StreamingQueryException.

Trap 2: Use the Spark UI to inspect the DAG and stage details for each…

The Spark UI is valuable for performance tuning and understanding execution plans, but it may not retain logs for intermittent failures that occur after the cluster terminates. Delta Live Tables abstracts much of the Spark execution, and the event log is specifically designed to surface pipeline-level errors. The Spark UI alone would not provide the integrated error context needed.

Trap 3: Monitor the cluster's CPU and memory utilization metrics in the…

While resource metrics can indicate performance bottlenecks, they do not provide the detailed error information needed to diagnose a StreamingQueryException. The exception could be caused by data issues, network problems, or code bugs that are not reflected in CPU or memory usage. This approach lacks the specificity required for root cause analysis.

A
Set up a Databricks SQL alert on the target Delta table's row count to detect missing data.
Why it fails: A row count alert can indicate that data is not being written, but it does not explain why the pipeline failed. The intermittent nature of the failure requires access to error details, which a row count metric cannot provide. This method is reactive and does not help in diagnosing the underlying cause of the StreamingQueryException.
B
Use the Spark UI to inspect the DAG and stage details for each failed job run.
Why it fails: The Spark UI is valuable for performance tuning and understanding execution plans, but it may not retain logs for intermittent failures that occur after the cluster terminates. Delta Live Tables abstracts much of the Spark execution, and the event log is specifically designed to surface pipeline-level errors. The Spark UI alone would not provide the integrated error context needed.
C
Enable and analyze the Delta Live Tables event log for detailed error messages and stack traces.
The Delta Live Tables event log captures detailed information about pipeline runs, including error messages, stack traces, and data quality metrics. For intermittent streaming failures, this log provides the granular context needed to pinpoint the exact cause, such as deserialization errors or Kafka connectivity issues, making it the most effective monitoring tool for this scenario.
D
Monitor the cluster's CPU and memory utilization metrics in the Databricks workspace.
Why it fails: While resource metrics can indicate performance bottlenecks, they do not provide the detailed error information needed to diagnose a StreamingQueryException. The exception could be caused by data issues, network problems, or code bugs that are not reflected in CPU or memory usage. This approach lacks the specificity required for root cause analysis.

Question 6easymultiple choice

A data engineer is building a Silver layer table that combines data from multiple Bronze tables. The engineer wants to ensure that the Silver table only contains the most recent version of each record based on a 'last_updated' timestamp. Which Delta Lake operation should be used to achieve this?

Trap 1: DELETE then INSERT the new records

Deleting and re-inserting is non-atomic and can lead to temporary data loss or inconsistencies for concurrent readers. It also does not guarantee that only the latest record is kept if multiple versions exist. MERGE is preferred because it handles both updates and inserts in a single transaction.

Trap 2: Use Delta Lake time travel to revert to a previous version

Time travel is for querying historical snapshots, not for maintaining a current view. Reverting to a previous version would discard newer data, which is the opposite of what is needed. It does not help in deduplicating records based on timestamps; it is a read feature, not a write operation.

Trap 3: INSERT OVERWRITE with the entire dataset

INSERT OVERWRITE replaces the entire table or partition, which is inefficient for incremental updates and loses history. It does not handle record-level deduplication based on timestamps. This approach would require reprocessing all data and is not suitable for maintaining a current view with minimal compute.

A
DELETE then INSERT the new records
Why it fails: Deleting and re-inserting is non-atomic and can lead to temporary data loss or inconsistencies for concurrent readers. It also does not guarantee that only the latest record is kept if multiple versions exist. MERGE is preferred because it handles both updates and inserts in a single transaction.
B
Use Delta Lake time travel to revert to a previous version
Why it fails: Time travel is for querying historical snapshots, not for maintaining a current view. Reverting to a previous version would discard newer data, which is the opposite of what is needed. It does not help in deduplicating records based on timestamps; it is a read feature, not a write operation.
C
INSERT OVERWRITE with the entire dataset
Why it fails: INSERT OVERWRITE replaces the entire table or partition, which is inefficient for incremental updates and loses history. It does not handle record-level deduplication based on timestamps. This approach would require reprocessing all data and is not suitable for maintaining a current view with minimal compute.
D
MERGE INTO with a condition that updates when the source timestamp is greater
MERGE INTO allows you to update existing records when the incoming data has a newer timestamp and insert new records when they don't exist. This ensures that the Silver table always reflects the latest version. It is the standard way to upsert data in Delta Lake while maintaining ACID compliance.

Question 7mediummultiple choice

A data engineering team is running a nightly batch job on a Databricks job cluster that processes a 10 TB Delta table. The job reads the entire table, performs transformations, and writes results to another Delta table. The team notices that the job takes 4 hours and consumes significant DBUs. They want to reduce runtime and cost without changing the business logic. The table is partitioned by ingestion date, but queries often filter on a high-cardinality column 'customer_id'. Which optimization technique is most appropriate to improve performance and reduce cost?

Trap 1: Enable Delta Lake caching by running CACHE TABLE on the source…

Caching the entire 10 TB table in memory is impractical because it would require a cluster large enough to hold the data, increasing cost. Also, caching is ephemeral and not persisted across job runs, so the nightly job would still need to read from storage. This does not address the underlying data layout issue and may cause memory pressure, leading to spilling and slower performance.

Trap 2: Increase the cluster size by adding more worker nodes to the job…

Adding more workers may speed up the job but also increases DBU consumption, raising cost. Without addressing data layout, the job still reads unnecessary data, so scaling out is inefficient. This approach does not solve the root cause of slow performance due to poor data skipping on the high-cardinality filter column, and it may not yield linear speedup due to I/O bottlenecks.

Trap 3: Convert the table to a Parquet table and use partition pruning on…

Converting to Parquet loses Delta Lake features like ACID transactions and time travel, which may be required. Partitioning by 'customer_id' would create too many small partitions due to high cardinality, causing performance degradation. This approach is not cost-effective and may break downstream processes. It also does not leverage Databricks optimizations like Z-ORDER or data skipping.

A
Enable Delta Lake caching by running CACHE TABLE on the source table before the job.
Why it fails: Caching the entire 10 TB table in memory is impractical because it would require a cluster large enough to hold the data, increasing cost. Also, caching is ephemeral and not persisted across job runs, so the nightly job would still need to read from storage. This does not address the underlying data layout issue and may cause memory pressure, leading to spilling and slower performance.
B
Increase the cluster size by adding more worker nodes to the job cluster.
Why it fails: Adding more workers may speed up the job but also increases DBU consumption, raising cost. Without addressing data layout, the job still reads unnecessary data, so scaling out is inefficient. This approach does not solve the root cause of slow performance due to poor data skipping on the high-cardinality filter column, and it may not yield linear speedup due to I/O bottlenecks.
C
Optimize file layout using Z-ORDER BY on the 'customer_id' column.
Z-ORDER BY clusters data on the 'customer_id' column, improving data skipping for queries that filter on that column. This reduces the amount of data read during the nightly job, leading to faster runtime and lower DBU consumption. It is a cost-effective optimization because it reorganizes existing data without changing business logic, and the benefits persist across runs until the data is rewritten.
D
Convert the table to a Parquet table and use partition pruning on 'customer_id'.
Why it fails: Converting to Parquet loses Delta Lake features like ACID transactions and time travel, which may be required. Partitioning by 'customer_id' would create too many small partitions due to high cardinality, causing performance degradation. This approach is not cost-effective and may break downstream processes. It also does not leverage Databricks optimizations like Z-ORDER or data skipping.

Question 8hardmultiple choice

Refer to the exhibit. Which configuration change is required to enable multiple instances of this job to run simultaneously?

Exhibit

CLI Output: databricks jobs run-now --job-id 5678
{ "error": "INVALID_PARAMETER_VALUE", "message": "The job 5678 is currently running and does not allow concurrent runs." }

Trap 1: Increase the timeout threshold for the job.

The timeout setting defines how long a job can run before it is automatically cancelled. It does not control concurrency. Modifying the timeout will not allow a second instance of the job to start while the first is already running; it only changes the termination criteria.

Trap 2: Add more workers to the cluster assigned to the job.

Scaling out the cluster increases the compute resources available to the job, but it does not change the concurrency policy defined for the job itself. Even with a massive cluster, the job remains restricted by the concurrency limit, which is a logic control, not a resource control.

Trap 3: Enable 'Retries' in the job configuration.

Retries are intended for handling transient failures by automatically restarting a failed task or job. They are unrelated to the policy governing concurrent execution. Enabling retries will not permit multiple simultaneous runs of the same job identifier, as that is a separate configuration parameter designed for managing execution flow.

A
Increase the timeout threshold for the job.
Why it fails: The timeout setting defines how long a job can run before it is automatically cancelled. It does not control concurrency. Modifying the timeout will not allow a second instance of the job to start while the first is already running; it only changes the termination criteria.
B
Set max_concurrent_runs to a value greater than 1 in the job settings.
The max_concurrent_runs parameter controls how many instances of a job can execute at the same time. Setting this to a value higher than 1 allows the job to be triggered even if a previous run is still in progress, enabling parallel processing of independent data workflows.
C
Add more workers to the cluster assigned to the job.
Why it fails: Scaling out the cluster increases the compute resources available to the job, but it does not change the concurrency policy defined for the job itself. Even with a massive cluster, the job remains restricted by the concurrency limit, which is a logic control, not a resource control.
D
Enable 'Retries' in the job configuration.
Why it fails: Retries are intended for handling transient failures by automatically restarting a failed task or job. They are unrelated to the policy governing concurrent execution. Enabling retries will not permit multiple simultaneous runs of the same job identifier, as that is a separate configuration parameter designed for managing execution flow.