Databricks-DE-Assoc Practice Test 2 — 25 Questions

Question 1

A data engineer needs to ingest data from a legacy SQL Server database into a Delta Lake bronze table. The ingestion must be performant and support parallel reads from the source table. What is the best practice for configuring the JDBC connection in this scenario?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure 'partitionColumn', 'lowerBound', 'upperBound', and 'numPartitions'.. Parallelizing JDBC reads is essential for moving large datasets from relational databases to Databricks. By providing partitioning columns and boundaries, Spark can spawn multiple executors to read different segments of the data simultaneously. This significantly reduces the time required for the initial load and improves the overall throughput of the ingestion pipeline.

Answer

Use a single executor to ensure data consistency during the read.

Answer

Rely on the default JDBC settings for automatic parallelization.

Answer

Use the COPY INTO command to read directly from the JDBC source.

Question 2

Which security feature in Databricks allows administrators to mask sensitive data, such as email addresses or social security numbers, in query results based on user-defined functions?

Accepted Answer

Dynamic Data Masking. Dynamic Data Masking allows you to define a masking function that hides or transforms data based on the user's role. By applying these functions to specific columns in a view or table, sensitive information is obscured in the results returned to the user, providing an effective way to maintain compliance with data privacy regulations like GDPR or HIPAA.

Answer

Row-Level Security

Answer

Table ACLs

Answer

Workspace-level isolation

Question 3

A data engineering team is building a medallion architecture in Databricks. In the Silver layer, streaming data from Kafka must be cleaned, deduplicated, and written into a Delta table. Which Structured Streaming output mode should the engineer select to ensure append-only storage of fully processed, stateful deduplicated records?

Accepted Answer

Append output mode. When performing stateful operations such as deduplication with dropDuplicates() on a streaming DataFrame, Structured Streaming requires the output mode to be set to append. This mode allows newly emitted unique rows to be written continuously to the target Delta table without requiring complete table rewrites, fitting the streaming append-only nature of the Silver medallion layer.

Answer

Complete output mode

Answer

Update output mode

Answer

Overwrite output mode

Question 4

A data engineering team needs to ingest millions of small JSON files from an S3 bucket into a Delta Lake table. The solution must provide incremental loading, support schema evolution, and automatically scale to handle increasing file volumes without manual tracking of processed files. Which tool is best suited for this requirement?

Accepted Answer

Auto Loader using the cloudFiles source in Structured Streaming.. Auto Loader is the recommended tool for incremental ingestion from cloud storage. It scales to millions of files using either directory listing or file notifications. Unlike standard Spark sources, it tracks processed files in a checkpoint, ensuring exactly-once semantics. This automation reduces operational overhead when managing unpredictable data volumes and evolving schema structures in production environments.

Answer

The standard Apache Spark DataFrame reader using the .load() method.

Answer

The COPY INTO SQL command with the mergeSchema option enabled.

Answer

A Python loop that iterates through filenames and uses INSERT INTO.

Question 5

Refer to the exhibit. A data engineer is attempting to ingest JSON data into a Delta table. Based on the error log, what is the most appropriate transformation step to implement before loading this data into the production table?

Accepted Answer

Apply a 'coalesce' function or a 'filter' transformation on the DataFrame before writing to the Delta table.. The exhibit shows a failure due to null values in a required field during ingestion. In production pipelines, failing at the ingestion layer stops data flow. Implementing a transformation step to handle nulls, either by providing default values or filtering, ensures the pipeline is resilient. This is a core Data Engineering principle: handle dirty source data before it reaches the target, preventing ingestion failures.

Answer

Use the 'DROP TABLE' command to clear the target and recreate it with nullable constraints.

Answer

Modify the target table schema using 'ALTER TABLE' to change the column to nullable.

Answer

Set the 'spark.sql.execution.nullValue' configuration to 'ignore' for the session.

Question 6

A junior data engineer writes a PySpark transformation that reads a Parquet dataset, filters out inactive users, and appends the resulting DataFrame to an existing Delta Lake table. However, the data engineer notices duplicate records appearing in the target table after multiple runs. Which technique should be implemented to ensure idempotency?

Accepted Answer

Implement a Delta Lake MERGE operation using a unique identifier to conditionally insert or update records.. Using Delta Lake's MERGE INTO operation allows data engineers to perform upserts by matching incoming records against existing table rows using a unique business key. This prevents duplicates during repeated pipeline executions, guaranteeing idempotency. Traditional append operations simply add records without checking for prior existence, which is a common root cause of duplicate data issues in incremental batch pipelines.

Answer

Use DataFrameWriter's mode("overwrite") with replaceWhere option matching the updated partition keys.

Answer

Increase the shuffle partition count using spark.sql("SET spark.sql.shuffle.partitions = 200").

Answer

Execute the VACUUM command immediately after every write operation completes.

Question 7

A data engineer is working on a Bronze-to-Silver transformation. Which THREE of the following practices are recommended to optimize performance and data quality during this stage?

Accepted Answer

Apply data quality checks to filter out or quarantine malformed records before writing to Silver.. Effective Bronze-to-Silver transformations are the foundation of a robust Lakehouse. By combining data validation, efficient partitioning, and incremental processing, engineers can minimize resource consumption while ensuring the data is reliable. These practices ensure the pipeline remains performant as volume scales and that the downstream Gold tables contain clean, query-optimized data, which is critical for business intelligence and data science use cases.

Answer

Always use 'SELECT *' to ensure the entire schema is captured in the Silver table.

Answer

Perform heavy joins and aggregations directly on the Bronze table.

Question 8

A data engineering team needs to ingest streaming data from Kafka into a Delta table while maintaining exactly-once processing guarantees and low latency. Which Databricks Intelligence Platform feature should they utilize to build this streaming pipeline declaratively?

Accepted Answer

Delta Live Tables. Delta Live Tables (DLT) is a declarative framework that simplifies ETL pipeline development. By defining data sources and target tables using SQL or Python, DLT manages infrastructure deployment, task orchestration, and error handling automatically, while leveraging structured streaming under the hood for low-latency, exactly-once ingestion.

Answer

Databricks Job clusters with manual spark-submit scripts

Answer

Databricks SQL query history

Answer

Unity Catalog lineage graphs

Question 9

A data engineer needs to share a subset of a table with an external partner who does not have access to the internal Databricks workspace. What is the most appropriate method to achieve this?

Accepted Answer

Use Delta Sharing to share a table.. Delta Sharing is the secure, open-standard solution for sharing data externally. It allows the recipient to access data directly from cloud storage without needing to provision an account in the provider's Databricks workspace. This process is secure because it uses short-lived tokens and does not require complex credential management, ensuring that data stays governed and that the provider retains control over what is being shared.

Answer

Create a service principal with restricted access.

Answer

Export data to CSV and email it.

Answer

Grant the partner access to a shared S3 bucket.

Question 10

A data engineer needs to automate the ingestion and transformation of data within Databricks with minimal manual intervention. Which feature is most appropriate for orchestrating these data workflows?

Accepted Answer

Databricks Workflows. Databricks Workflows is a fully managed orchestration service that allows users to create, schedule, and monitor data pipelines. It is essential because it integrates seamlessly with the platform, allowing engineers to chain together notebooks, SQL queries, and Delta Live Tables pipelines into complex, multi-step workflows. This automation reduces manual toil and improves the reliability of data production environments in the Lakehouse.

Answer

Delta Live Tables (DLT)

Answer

Unity Catalog UI

Answer

Databricks SQL Warehouse

Question 11

When a Data Engineer uses a 'Repair and Rerun' functionality on a failed Databricks Job, what happens?

Accepted Answer

It restarts from the failed task.. Repair and Rerun is a critical feature for managing complex DAG workflows. It allows engineers to restart a job from the specific failed task, inheriting the successful results of prior tasks. This saves significant time and compute costs by avoiding redundant processing of already completed data, which is essential for maintaining efficient pipelines and preventing unnecessary data re-computation in production environments.

Answer

It deletes all previous successful task logs.

Answer

It resets all task statuses to 'Pending'.

Answer

It requires a new cluster to be provisioned.

Question 12

Which identity management approach is mandatory for the implementation of Unity Catalog within a Databricks environment?

Accepted Answer

Account-level identities. Unity Catalog requires users to be managed at the account level, rather than at the individual workspace level. This synchronization ensures that permissions and identities are consistent across all workspaces in the account, which is a fundamental requirement for the unified governance model that Unity Catalog provides to the Databricks platform.

Answer

Workspace-local users

Answer

Local SQL-only credentials

Answer

Personal access tokens for every user

Question 13

A data engineer needs to perform an upsert on a target Delta table using a source DataFrame. Which operation provides the most robust mechanism to handle duplicates and updates in a single pass?

Accepted Answer

Using the 'MERGE INTO' command.. The MERGE operation is the standard industry practice for upsert patterns in Delta Lake. It allows for atomic updates, inserts, and deletions in a single transaction. This prevents partial writes and maintains the integrity of the data stream, which is crucial in production ELT workflows where data often arrives out of order or contains overlapping records that must be reconciled.

Answer

Using 'INSERT INTO' combined with a 'DELETE' operation.

Answer

Using 'overwrite' mode in the DataFrameWriter.

Answer

Using 'append' mode with custom filter logic.

Question 14

Which Databricks compute resource is specifically optimized for running BI dashboards and SQL queries?

Accepted Answer

SQL Warehouse. Databricks SQL Warehouses are compute environments designed to deliver high-performance SQL execution, making them ideal for BI tools and dashboarding. Understanding the difference between SQL Warehouses and traditional data engineering clusters is important, as it helps engineers choose the right compute resource to match the workload, ensuring better query performance and cost efficiency for end-users who need to interact with data via SQL.

Answer

All-Purpose Cluster

Answer

Job Cluster

Answer

Delta Live Tables Pipeline

Question 15

An organization is adopting the Databricks Intelligence Platform and wants to leverage Mosaic AI for building custom machine learning models. Which feature allows data engineers to track machine learning experiments, log parameters, and manage model artifacts reliably?

Accepted Answer

MLflow. MLflow is the open-source platform integrated natively into Databricks Mosaic AI for managing the end-to-end machine learning lifecycle. It provides comprehensive tracking of experiments, parameters, metrics, and registered model versions, ensuring reproducibility and smooth transition from development to production environments within the data lakehouse architecture.

Answer

Databricks SQL dashboards

Answer

Unity Catalog volumes

Answer

Delta Live Tables

Question 16

A data engineer needs to create a Delta table in Unity Catalog that will be used by multiple teams. The engineer wants to ensure that the table is governed by Unity Catalog and that access can be controlled using SQL GRANT statements. What is the correct way to create the table?

Accepted Answer

CREATE TABLE my_catalog.my_schema.my_table (id INT, name STRING) USING DELTA;. To create a managed Delta table in Unity Catalog, use the CREATE TABLE statement with a three-part name (catalog.schema.table) and specify USING DELTA. This ensures the table is governed by Unity Catalog and can be secured with GRANT statements.

Answer

CREATE EXTERNAL TABLE my_catalog.my_schema.my_table (id INT, name STRING) USING DELTA LOCATION '/mnt/data/my_table';

Answer

CREATE TABLE my_catalog.my_schema.my_table AS SELECT * FROM parquet.`/mnt/data/my_table`;

Answer

CREATE TABLE my_table (id INT, name STRING) USING DELTA LOCATION '/mnt/data/my_table';

Question 17

Refer to the exhibit. A data engineer is deploying an automated job. Based on the provided configuration, what is the primary benefit of using the 'autoscale' attribute in this cluster definition?

Accepted Answer

It enables the cluster to adjust resources dynamically based on current job load.. The 'autoscale' configuration allows the Databricks Intelligence Platform to dynamically adjust the number of worker nodes based on the workload demands. By setting a minimum and maximum range, the cluster automatically adds nodes during intensive processing and removes them during idle periods. This is critical for optimizing costs and performance, ensuring that compute resources are consumed only when necessary without requiring manual intervention by the engineer.

Answer

It forces the cluster to always use 8 nodes to ensure maximum performance.

Answer

It ensures that the cluster uses specific instance types defined in the policy.

Answer

It mandates the use of Spot Instances to reduce total cost of ownership.

Question 18

When a data engineer needs to automate a recurring ETL job, which Databricks tool is most appropriate for orchestrating tasks and handling dependencies?

Accepted Answer

Databricks Workflows. Databricks Workflows allows engineers to create, run, and monitor multi-task jobs. It supports dependencies, retries, and notifications, making it the standard choice for orchestrating complex data pipelines. By using Workflows, engineers can ensure that data transformation tasks occur in the correct sequence, with robust error handling and monitoring, which is critical for maintaining reliable production data pipelines in the Databricks Intelligence Platform.

Answer

Databricks Notebooks

Answer

Delta Live Tables

Answer

MLflow Models

Question 19

A Databricks job is failing due to 'Out of Memory' (OOM) errors during a join operation on two large datasets. Which TWO actions could help mitigate this issue?

Accepted Answer

Increase the cluster's worker node instance type with more memory.. OOM errors during joins often occur because the driver or worker nodes lack enough memory to handle the shuffle or broadcast operations. Increasing the cluster memory allows for larger data partitions to reside in memory, while adjusting the broadcast join threshold prevents the optimizer from attempting to broadcast datasets that exceed available node capacity. These configurations are essential for stabilizing large-scale ETL pipelines that process high-volume data.

Answer

Decrease the number of partitions in the Spark cluster.

Answer

Enable dynamic allocation for the cluster.

Answer

Use the cache() method on every DataFrame in the pipeline.

Question 20

A data engineer needs to share a Delta table managed by Unity Catalog with external partners who do not have access to the Databricks workspace. Which feature should be used to securely grant read-only access to this table without replicating the data?

Accepted Answer

Create a Delta Sharing share, add the table to the share, and grant access to a recipient object configured for the partners.. Delta Sharing is an open protocol for secure data sharing that allows organizations to share data directly from cloud storage without copying files. Unity Catalog integrates natively with Delta Sharing, enabling governance teams to manage access for external recipients securely using tokens and activation links, making it the ideal solution for cross-organization data collaboration.

Answer

Configure cross-account IAM role assumption to grant direct AWS S3 bucket read permissions to the external partners.

Answer

Set up a Databricks SQL endpoint and create individual workspace user accounts for every external partner.

Answer

Export the Delta table to CSV format and upload the files to an external SFTP server on a nightly schedule.