Databricks-DE-Assoc Practice Test 4 — 10 Questions

Question 1

A junior data engineer writes a PySpark transformation that reads a Parquet dataset, filters out inactive users, and appends the resulting DataFrame to an existing Delta Lake table. However, the data engineer notices duplicate records appearing in the target table after multiple runs. Which technique should be implemented to ensure idempotency?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Implement a Delta Lake MERGE operation using a unique identifier to conditionally insert or update records.. Using Delta Lake's MERGE INTO operation allows data engineers to perform upserts by matching incoming records against existing table rows using a unique business key. This prevents duplicates during repeated pipeline executions, guaranteeing idempotency. Traditional append operations simply add records without checking for prior existence, which is a common root cause of duplicate data issues in incremental batch pipelines.

Answer

Use DataFrameWriter's mode("overwrite") with replaceWhere option matching the updated partition keys.

Answer

Increase the shuffle partition count using spark.sql("SET spark.sql.shuffle.partitions = 200").

Answer

Execute the VACUUM command immediately after every write operation completes.

Question 2

A data engineer is working on a Bronze-to-Silver transformation. Which THREE of the following practices are recommended to optimize performance and data quality during this stage?

Accepted Answer

Apply data quality checks to filter out or quarantine malformed records before writing to Silver.. Effective Bronze-to-Silver transformations are the foundation of a robust Lakehouse. By combining data validation, efficient partitioning, and incremental processing, engineers can minimize resource consumption while ensuring the data is reliable. These practices ensure the pipeline remains performant as volume scales and that the downstream Gold tables contain clean, query-optimized data, which is critical for business intelligence and data science use cases.

Answer

Always use 'SELECT *' to ensure the entire schema is captured in the Silver table.

Answer

Perform heavy joins and aggregations directly on the Bronze table.

Question 3

A data engineering team needs to ingest streaming data from Kafka into a Delta table while maintaining exactly-once processing guarantees and low latency. Which Databricks Intelligence Platform feature should they utilize to build this streaming pipeline declaratively?

Accepted Answer

Delta Live Tables. Delta Live Tables (DLT) is a declarative framework that simplifies ETL pipeline development. By defining data sources and target tables using SQL or Python, DLT manages infrastructure deployment, task orchestration, and error handling automatically, while leveraging structured streaming under the hood for low-latency, exactly-once ingestion.

Answer

Databricks Job clusters with manual spark-submit scripts

Answer

Databricks SQL query history

Answer

Unity Catalog lineage graphs

Question 4

A data engineer needs to share a subset of a table with an external partner who does not have access to the internal Databricks workspace. What is the most appropriate method to achieve this?

Accepted Answer

Use Delta Sharing to share a table.. Delta Sharing is the secure, open-standard solution for sharing data externally. It allows the recipient to access data directly from cloud storage without needing to provision an account in the provider's Databricks workspace. This process is secure because it uses short-lived tokens and does not require complex credential management, ensuring that data stays governed and that the provider retains control over what is being shared.

Answer

Create a service principal with restricted access.

Answer

Export data to CSV and email it.

Answer

Grant the partner access to a shared S3 bucket.

Question 5

A data engineer needs to automate the ingestion and transformation of data within Databricks with minimal manual intervention. Which feature is most appropriate for orchestrating these data workflows?

Accepted Answer

Databricks Workflows. Databricks Workflows is a fully managed orchestration service that allows users to create, schedule, and monitor data pipelines. It is essential because it integrates seamlessly with the platform, allowing engineers to chain together notebooks, SQL queries, and Delta Live Tables pipelines into complex, multi-step workflows. This automation reduces manual toil and improves the reliability of data production environments in the Lakehouse.

Answer

Delta Live Tables (DLT)

Answer

Unity Catalog UI

Answer

Databricks SQL Warehouse

Question 6

When a Data Engineer uses a 'Repair and Rerun' functionality on a failed Databricks Job, what happens?

Accepted Answer

It restarts from the failed task.. Repair and Rerun is a critical feature for managing complex DAG workflows. It allows engineers to restart a job from the specific failed task, inheriting the successful results of prior tasks. This saves significant time and compute costs by avoiding redundant processing of already completed data, which is essential for maintaining efficient pipelines and preventing unnecessary data re-computation in production environments.

Answer

It deletes all previous successful task logs.

Answer

It resets all task statuses to 'Pending'.

Answer

It requires a new cluster to be provisioned.

Question 7

Which identity management approach is mandatory for the implementation of Unity Catalog within a Databricks environment?

Accepted Answer

Account-level identities. Unity Catalog requires users to be managed at the account level, rather than at the individual workspace level. This synchronization ensures that permissions and identities are consistent across all workspaces in the account, which is a fundamental requirement for the unified governance model that Unity Catalog provides to the Databricks platform.

Answer

Workspace-local users

Answer

Local SQL-only credentials

Answer

Personal access tokens for every user

Question 8

A data engineer needs to perform an upsert on a target Delta table using a source DataFrame. Which operation provides the most robust mechanism to handle duplicates and updates in a single pass?

Accepted Answer

Using the 'MERGE INTO' command.. The MERGE operation is the standard industry practice for upsert patterns in Delta Lake. It allows for atomic updates, inserts, and deletions in a single transaction. This prevents partial writes and maintains the integrity of the data stream, which is crucial in production ELT workflows where data often arrives out of order or contains overlapping records that must be reconciled.

Answer

Using 'INSERT INTO' combined with a 'DELETE' operation.

Answer

Using 'overwrite' mode in the DataFrameWriter.

Answer

Using 'append' mode with custom filter logic.

Question 9

Which Databricks compute resource is specifically optimized for running BI dashboards and SQL queries?

Accepted Answer

SQL Warehouse. Databricks SQL Warehouses are compute environments designed to deliver high-performance SQL execution, making them ideal for BI tools and dashboarding. Understanding the difference between SQL Warehouses and traditional data engineering clusters is important, as it helps engineers choose the right compute resource to match the workload, ensuring better query performance and cost efficiency for end-users who need to interact with data via SQL.

Answer

All-Purpose Cluster

Answer

Job Cluster

Answer

Delta Live Tables Pipeline

Question 10

An organization is adopting the Databricks Intelligence Platform and wants to leverage Mosaic AI for building custom machine learning models. Which feature allows data engineers to track machine learning experiments, log parameters, and manage model artifacts reliably?

Accepted Answer

MLflow. MLflow is the open-source platform integrated natively into Databricks Mosaic AI for managing the end-to-end machine learning lifecycle. It provides comprehensive tracking of experiments, parameters, metrics, and registered model versions, ensuring reproducibility and smooth transition from development to production environments within the data lakehouse architecture.

Answer

Databricks SQL dashboards

Answer

Unity Catalog volumes

Answer

Delta Live Tables