Databricks-DE-Assoc Practice Test 12 — 15 Questions

Question 1

A data engineer is designing a Delta Lake pipeline that processes streaming sales transactions. The schema evolves frequently, and the pipeline must handle these changes without manual intervention. Which feature should the engineer enable to support automatic schema updates while preventing data corruption?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Enable 'mergeSchema' in the DataFrameWriter options.. Schema evolution is critical in production pipelines to accommodate changing source data structures. Delta Lake handles this automatically when .option('mergeSchema', 'true') is used during write operations. This ensures that new columns are added to the metadata without requiring a table rebuild. Understanding this mechanism is vital for maintaining robust, automated pipelines in Databricks environments where source systems are prone to upstream schema modifications.

Answer

Set the table property 'delta.autoOptimize.optimizeWrite' to true.

Answer

Use the ALTER TABLE command to manually add missing columns before writing.

Answer

Set the 'overwriteSchema' option to true during every write.

Question 2

A data engineer needs to run a Databricks notebook that processes data stored in an external ADLS Gen2 location. The notebook must access the data securely without embedding credentials in the notebook code. The engineer has already configured a Unity Catalog external location with a storage credential. Which method should the engineer use to read the data?

Accepted Answer

Reference the external location path directly in the notebook using the Unity Catalog external location URL.. Unity Catalog external locations are designed to provide secure access to cloud storage using storage credentials managed by Unity Catalog. Once an external location is configured, data engineers can reference the path directly in their notebooks, and Unity Catalog handles authentication and authorization. This approach avoids embedding credentials in code, aligns with security best practices, and ensures that access is governed by Unity Catalog permissions.

Answer

Use a personal access token (PAT) stored in Databricks secrets to authenticate to ADLS Gen2.

Answer

Set Spark configuration spark.hadoop.fs.azure.account.oauth2.client.id and related properties in the notebook to authenticate.

Answer

Mount the ADLS Gen2 container to DBFS using a service principal and then read the data from the mount point.

Question 3

A data engineer has configured a Databricks Job with multiple dependent tasks forming a linear pipeline. Task A extracts data, Task B transforms it, and Task C loads it into a gold table. The pipeline runs daily. The team notices that if Task B fails due to an intermittent schema validation issue, the entire job run fails, but they want Task C to execute conditionally only if Task B succeeds, while alerting the on-call engineer immediately upon any failure. How should the task dependencies and conditional execution be configured?

Accepted Answer

Ensure Task C has a dependency pointing exclusively to Task B, and configure a job-level email notification for failures.. Configuring explicit task dependencies via the UI or API ensures that downstream tasks only execute upon the successful completion of their parents. By linking Task C strictly to Task B's success, you prevent corrupting downstream layers with partial or failed upstream transformations. This workflow pattern is essential for maintaining reliable data engineering pipelines in production Lakeflow environments.

Answer

Set Task C to run on failure of Task B to capture the exception state before alerting.

Answer

Convert Task B and Task C into a single notebook task and handle exceptions internally with try-except blocks.

Answer

Disable the task dependency between Task B and Task C and run them in parallel with different cluster configurations.

Question 4

When utilizing Databricks Asset Bundles, how should secrets (e.g., API keys, database credentials) be handled to ensure security during CI/CD?

Accepted Answer

Use the bundle's secret reference syntax to map variables to a secret scope.. Hardcoding secrets in configuration files is a critical security vulnerability. Databricks Asset Bundles support referencing secrets through the 'secrets' scope, which securely retrieves credentials at runtime. By decoupling sensitive data from the version-controlled YAML files, teams maintain security, avoid leaking credentials in Git, and simplify secret rotation. This practice is essential for enterprise security and ensures that sensitive infrastructure data is managed in alignment with the principle of least privilege.

Answer

Hardcode the secrets in the bundle's YAML configuration files.

Answer

Store secrets as environment variables in the Git repository.

Answer

Ask all developers to manually enter secrets in the production UI.

Question 5

A data engineer needs to configure a Databricks Job to orchestrate a data pipeline that includes a Python task, a SQL task, and a notebook task. The pipeline requires passing a dynamic run identifier from the Python task to the subsequent SQL and notebook tasks. Which mechanism should the data engineer use to achieve this task-to-task dependency parameter passing?

Accepted Answer

Use the dbutils.jobs.taskValues.set method in the Python task and reference it downstream using the task value syntax.. Databricks workflows natively support passing values between tasks using task values. A Python task can set a task value using dbutils.jobs.taskValues.set, which downstream tasks can then reference using standard task value interpolation syntax. This eliminates external storage dependencies, ensuring robust, serverless orchestration directly managed by the Databricks control plane.

Answer

Write the dynamic identifier to a shared Unity Catalog volume and read it inside the downstream tasks.

Answer

Store the identifier as an environment variable in the cluster configuration shared by all three tasks.

Answer

Modify the global job parameters dynamically via the Databricks REST API from within the running Python script.

Question 6

A data engineer wants to move data from a 'Bronze' table to a 'Silver' table while performing data cleaning. They want to ensure that this process is only executed once per data batch. Which approach is best for this requirement?

Accepted Answer

Use Structured Streaming with Delta as source and sink. Using Spark Structured Streaming with a Delta Lake source and sink provides built-in exactly-once processing guarantees. The streaming engine manages offsets and state, ensuring that every record is processed exactly once even in the event of a failure. This is the recommended pattern for moving data across the Medallion architecture, providing a robust, fault-tolerant way to implement complex data transformations and business logic consistently.

Answer

Use a daily batch job with a 'WHERE' clause on timestamp

Answer

Use a simple 'INSERT OVERWRITE' every hour

Answer

Use a manual Python script to fetch data and write it back

Question 7

Which TWO of the following statements accurately describe the behavior of Delta Lake table constraints and enforcement?

Accepted Answer

CHECK constraints are enforced during write operations for both batch and streaming jobs.. Delta Lake supports NOT NULL constraints and CHECK constraints to enforce data quality at the write layer. These constraints are vital for maintaining schema integrity. Understanding their behavior is essential for engineers designing robust data pipelines, as they ensure that invalid data is rejected before it is committed to the transaction log, maintaining a single source of truth.

Answer

Delta Lake automatically enforces NOT NULL constraints on all columns by default.

Answer

Adding a CHECK constraint to a table will retroactively validate all existing data in that table.

Answer

Constraints can only be defined on numeric data types within a Delta table.

Question 8

Why should developers avoid using 'notebook' references in production pipelines that point to the 'Shared' folder for shared development work?

Accepted Answer

It lacks the version control and access isolation required for production.. The 'Shared' folder is meant for collaborative development, not for hosting production-ready code. Pipelines referencing this folder are susceptible to unpredictable changes, as any developer with access can modify the code. In CI/CD, production code should reside in a controlled, versioned, and immutable location. Using the 'Shared' folder for production pipelines violates the separation of concerns and increases the risk of unauthorized or accidental changes breaking critical data workflows.

Answer

Databricks jobs cannot execute notebooks from the 'Shared' directory.

Answer

The 'Shared' directory is deleted automatically every 24 hours.

Answer

Notebooks in the 'Shared' directory are limited to a smaller file size.

Question 9

What must be configured to allow Databricks to access cloud storage on behalf of a user without the user needing to provide their own cloud credentials?

Accepted Answer

Storage Credential. A storage credential acts as a secure wrapper around cloud-provider authentication (like an IAM role or service account). By creating a storage credential in Unity Catalog, the administrator gives Databricks the permission to access the storage. Users then interact with 'External Locations' that reference these credentials, effectively decoupling user identity from raw cloud infrastructure access, which is the cornerstone of Unity Catalog's secure data governance model.

Answer

Credential Passthrough

Answer

Table ACLs

Answer

Instance Profiles

Question 10

Refer to the exhibit. A Databricks administrator is using the Unity Catalog JSON policy to manage access. If the 'data_scientist_1' user attempts to execute an 'UPDATE' command on the 'orders' table, what will be the result?

Accepted Answer

The command will fail due to insufficient privileges on the table.. The policy explicitly grants only 'SELECT' and 'DESCRIBE' permissions. In Unity Catalog, if an action is not explicitly permitted, it is denied by default. Therefore, the update attempt will be rejected. This is critical for data engineers to understand because Unity Catalog enforces a strict least-privilege security model, ensuring that only authorized users can modify data, which protects the integrity of tables in a shared environment.

Answer

The command will succeed because the user is part of the 'dev_analytics' schema.

Answer

The command will succeed because 'SELECT' implies 'UPDATE' in Unity Catalog.

Answer

The command will succeed if the cluster is running in Single User mode.

Question 11

A data engineer needs to join two massive datasets. One dataset is very small (10MB), and the other is very large (1TB). To ensure the join operation is performed as efficiently as possible, which join strategy should be enforced?

Accepted Answer

Enforce a Broadcast Hash Join.. Selecting the correct join strategy is vital for preventing data shuffles, which are expensive. A Broadcast join is perfect for scenarios where one side is small enough to fit in memory on every worker node. By broadcasting the small table, the engine avoids a massive shuffle of the large table, resulting in significantly faster performance and reduced network traffic.

Answer

Enforce a Sort-Merge Join.

Answer

Enforce a Cartesian Product Join.

Answer

Enforce a Shuffle Hash Join.

Question 12

Which capability is provided by Databricks' integration with MLflow?

Accepted Answer

Experiment tracking and model lifecycle management. MLflow is an open-source platform that integrates seamlessly with Databricks to manage the machine learning lifecycle. It provides tools for experiment tracking, model registry, and deployment. This integration is vital for data scientists, as it allows them to log experiments, compare results, and manage model versions, ensuring that machine learning projects are reproducible and ready for production deployment within the collaborative workspace.

Answer

Automatic creation of SQL database schemas

Answer

Low-level kernel manipulation

Answer

Automatic hardware upgrades for clusters

Question 13

Which TWO of the following statements correctly describe the behavior of the Delta Lake 'MERGE' operation when handling schema evolution?

Accepted Answer

The 'mergeSchema' option must be enabled for any new columns to be added to the target table during a MERGE.. Schema evolution in Delta Lake allows for flexible data structures. By using MERGE with specific options, engineers can automate updates to target table structures as incoming data evolves. Understanding these behaviors is critical for maintaining robust ELT pipelines where source system changes might otherwise break downstream consumers or cause data ingestion failures that require manual schema repairs.

Answer

MERGE operations automatically drop columns in the target table if they are missing from the source DataFrame.

Answer

MERGE operations support renaming columns in the target table using the 'RENAME' keyword inside the 'WHEN MATCHED' clause.

Answer

The 'overwriteSchema' option is required for MERGE to successfully insert new columns into the target table.

Question 14

Refer to the exhibit. The merge operation is failing in your pipeline. What is the root cause of this error, and how should it be resolved?

Accepted Answer

The source DataFrame must be aggregated or deduplicated to ensure one match per target ID.. The error occurs because the join condition in the MERGE statement is not unique relative to the source data. When the source contains multiple records with the same identifier that exists in the target, the engine cannot decide which source record to use for the update. To resolve this, you must deduplicate the source data before performing the merge, ensuring each key has exactly one entry.

Answer

The target table does not have a primary key defined.

Answer

The target table is too large to perform a MERGE and should be replaced with a full overwrite.

Answer

The cluster configuration needs more memory to handle the join operation.

Question 15

What is the primary function of an 'Access Connector' for Azure Databricks when using Unity Catalog?

Accepted Answer

It allows Databricks to access storage without managing account keys directly.. An Access Connector is a dedicated Azure resource that provides a secure, identity-based bridge between Unity Catalog and Azure Data Lake Storage (ADLS). It manages the identity used by Databricks to read and write data, ensuring that Unity Catalog can enforce access policies without requiring the storage account keys to be shared directly with users or hard-coded into notebooks.

Answer

It acts as a firewall between the Databricks workspace and the public internet.

Answer

It automatically scales the compute resources of the cluster.

Answer

It provides a user interface for browsing the storage account content.