Databricks-DE-Pro Practice Exam 4 — 20 Questions

Question 1

Refer to the exhibit. The engineer wants to replace only one specific partition in the 'orders' table. What is the best method in Databricks?

Accepted Answer

Enable dynamic partition overwrite mode and perform an INSERT OVERWRITE.. Using 'partitionOverwriteMode=dynamic' with a standard INSERT OVERWRITE operation is the correct way to replace a single partition without affecting the entire table. The default behavior is 'static', which overwrites the whole table. By setting the dynamic mode, the system identifies the partitions present in the incoming data and replaces only those, ensuring minimal impact and preventing accidental data loss across the entire dataset during the overwrite process.

Answer

Run a DELETE statement for the partition and then an APPEND.

Answer

Use the MERGE command to overwrite only the records in the specific partition.

Answer

Set 'spark.databricks.delta.retentionDurationCheck.enabled' to false.

Question 2

You are migrating a legacy ETL process to Delta Live Tables (DLT). You have an existing table defined with a complex transformation that involves a custom Python function using a third-party library. How should you structure this in DLT to ensure the function is available and correctly applied?

Accepted Answer

Define the function inside the same notebook but outside the 'dlt.table' function.. DLT pipelines require explicit management of dependencies. Using the 'dlt.table' decorator alongside proper library installation via cluster configuration or pip requirements ensures that Python functions are serialized and distributed to worker nodes. Understanding how DLT handles dependencies and decorators is critical for building reliable, production-grade pipelines that maintain parity with legacy logic while leveraging the automatic orchestration and quality management features inherent in the DLT framework.

Answer

Hardcode the library path using 'sys.path.append'.

Answer

You cannot use custom Python functions in DLT.

Answer

Use the 'spark.udf.register' method globally.

Question 3

Refer to the exhibit. You are using Auto Loader to ingest data with evolving schemas. After running the job for a week, you realize that new columns added to the source JSON are not being captured in the destination table. What must you add to the configuration?

Accepted Answer

Add 'cloudFiles.schemaEvolutionMode': 'addCol'.. By default, Auto Loader schema inference only detects the schema during the initial load. To capture schema evolution, you must explicitly enable 'cloudFiles.schemaEvolutionMode'. Without this parameter, Auto Loader ignores new fields to protect downstream consumers from breaking changes. Enabling 'addCol' allows the schema to expand dynamically, ensuring the target Delta table reflects the structure of the incoming data files as they arrive.

Answer

Add 'cloudFiles.schemaEvolutionMode': 'rescue'.

Answer

Add 'cloudFiles.maxFiles': '1000'.

Answer

Add 'cloudFiles.allowOverwrites': 'true'.

Question 4

When ingesting data using Auto Loader, what is the purpose of the 'cloudFiles.schemaLocation' parameter?

Accepted Answer

It stores metadata about the inferred schema and tracks evolution.. The schema location is vital for Auto Loader because it stores the inferred schema and tracks schema evolution history. By persisting this information in a managed location, Auto Loader can resume ingestion after a job restart without re-inferring the schema from scratch. This ensures consistency and prevents potential ingestion failures caused by schema drift when processing new files in a long-running stream.

Answer

It specifies the target directory where the processed Delta table data is stored.

Answer

It defines the temporary directory used for shuffling large datasets during joins.

Answer

It is used to cache the raw JSON files before they are parsed.

Question 5

You are migrating a legacy CSV-based ETL process to Databricks. The source CSV files contain inconsistent date formats. Which approach provides the most scalable way to handle these inconsistencies during the bronze-to-silver transformation?

Accepted Answer

Use the 'to_timestamp' function with an array of acceptable formats to parse the column.. Using Spark's `to_timestamp` with multiple format strings or a custom UDF is the standard, scalable way to handle format drift. By applying this logic in the Silver layer, you preserve the raw data in the Bronze layer while ensuring the refined data is standardized. This strategy follows the Medallion architecture pattern, allowing for lineage tracking and the ability to reprocess data if requirements change or better parsing logic is developed.

Answer

Convert all columns to string type to avoid parsing errors, and handle formatting in the visualization tool.

Answer

Use the 'from_unixtime' function with a hardcoded string format for all records.

Answer

Delete all rows with invalid date formats using a 'drop' expectation in a DLT pipeline.

Question 6

A data engineer needs to restrict access to personally identifiable information (PII) columns in a Unity Catalog table for a group of analysts. Which Unity Catalog feature should be used to enforce this policy while ensuring data remains queryable?

Accepted Answer

Define a column mask using SQL functions to redact data for the analyst group.. Dynamic views or Row-level security (RLS) and Column-level security (CLS) are essential for compliance. By utilizing SQL functions like current_user() within a defined mask, you ensure that PII is obscured based on user identity. This approach centralizes security within the data platform, preventing the need to duplicate datasets for different access levels, which significantly reduces the administrative burden and minimizes the risk of unauthorized data exposure in production environments.

Answer

Apply a static mask using a custom UDF during data ingestion.

Answer

Grant the analyst group SELECT access on the table and instruct them to cast PII columns to null.

Answer

Create separate physical tables for analysts containing only non-sensitive columns.

Question 7

A Data Engineer needs to verify that the column 'user_id' is unique in a critical Gold table. What is the most efficient, non-blocking way to perform this check in a production environment?

Accepted Answer

Define a primary key constraint in the Delta table definition.. Using a Delta constraint or a DLT expectation is the most efficient way to enforce uniqueness at the storage layer. Unlike a full table scan query, which is reactive and performance-intensive, native constraints are checked during the commit process, providing proactive quality assurance. This ensures that the table never enters an invalid state, which is vital for the integrity of downstream machine learning models and reporting applications.

Answer

Run a daily query: SELECT COUNT(user_id) - COUNT(DISTINCT user_id).

Answer

Use a custom UDF to check for duplicates inside a Spark streaming loop.

Answer

Perform a left-anti join with the previous day's data every time the pipeline runs.

Question 8

A Data Engineer is building a Delta Live Tables (DLT) pipeline to ingest raw JSON data. They need to ensure that records missing the required 'user_id' field are dropped while simultaneously capturing these discarded records in a separate table for auditing purposes. Which approach achieves this in DLT?

Accepted Answer

Define two separate tables in the pipeline, one filtering for valid records and one for invalid records using the NOT condition.. To handle data quality in DLT, the 'expect_violation_or_drop' constraint is not a standard clause. Instead, the 'expect_or_drop' constraint removes invalid records, but does not preserve them. The correct architectural pattern involves using a separate pipeline or query that filters for the inverse condition (where 'user_id' is null) and writes those records to an 'expect_all_fail' target table, maintaining strict lineage and auditing for schema non-compliance.

Answer

Apply the 'expect_or_drop' constraint and use a trigger to log dropped records.

Answer

Use the 'expect_or_fail' constraint to halt the pipeline and flag the error.

Answer

Configure a DLT pipeline to use 'expect_all_drop' to automatically split the data stream.

Question 9

A Data Engineer needs to monitor the health of a Delta Live Tables (DLT) pipeline. Which metric should they monitor to track the number of data quality violations over time?

Accepted Answer

expectations_violated_records. The 'expectations_violated_records' metric is specifically designed to track data quality issues in DLT pipelines. By integrating these metrics with Databricks SQL alerts or external tools like Grafana, engineers can gain visibility into data lineage and quality degradation. Monitoring these metrics is essential for maintaining pipeline reliability and ensuring that downstream data consumers receive accurate and validated data, preventing the propagation of corrupted data throughout the analytical ecosystem.

Answer

pipeline_latency_seconds

Answer

cluster_cpu_utilization

Answer

task_retry_count

Question 10

A data engineer is setting up a new Unity Catalog metastore. What is the primary purpose of the 'Metastore Admin' role?

Accepted Answer

To define the root storage location and manage top-level catalogs.. The Metastore Admin is the highest-privilege role in Unity Catalog, responsible for the initial configuration and top-level governance settings. This role is essential for establishing the security foundation of the platform, including catalog creation and cross-workspace access control. Proper management of this role is critical to prevent privilege escalation and ensure that only authorized personnel can define the organization's overall data governance strategy.

Answer

To create and manage compute clusters for all users.

Answer

To monitor and respond to daily system health alerts.

Answer

To approve all requests for new user sign-ups.

Question 11

When designing a streaming pipeline using Structured Streaming, which THREE of the following are necessary to ensure 'exactly-once' processing semantics in Databricks?

Accepted Answer

Using a source system that supports replaying data (e.g., Kafka or Delta).. Exactly-once processing requires that the source system, the processing engine, and the sink system all support fault-tolerant checkpoints and idempotency. Databricks handles this through checkpointing and write-ahead logs. Understanding these components is critical for data engineers to ensure data consistency in critical financial or operational systems, preventing the common pitfalls of duplicate entries or missed data during cluster restarts or transient failures.

Answer

Setting 'spark.sql.shuffle.partitions' to 1.

Answer

Disabling the Delta Lake write-ahead log.

Question 12

A Data Engineer is using Databricks Asset Bundles (DABs) to manage a project. Where should the engineer define the job settings, such as clusters, schedules, and task dependencies?

Accepted Answer

In a databricks.yml file within the project root.. Databricks Asset Bundles use YAML files, specifically those located in the bundles configuration folder, to manage infrastructure as code. Defining job settings in the `databricks.yml` file allows for version control and consistent deployments across different environments (dev, staging, prod). This approach ensures that the entire project state is replicable and managed through a standardized CI/CD pipeline, reducing manual configuration errors and increasing the speed of environment deployments.

Answer

In a hardcoded JSON file located in the /tmp/ directory.

Answer

Inside the Python code using the dbutils.job() library.

Answer

By manually clicking through the Databricks UI after deployment.

Question 13

Refer to the exhibit. An engineer has configured the cluster settings as shown. What is the expected impact on the Delta table's performance and write operations?

Accepted Answer

Read performance will improve over time as files are automatically compacted and optimized for downstream queries.. The settings `optimizeWrite` and `autoCompact` are crucial for maintaining healthy Delta tables. `optimizeWrite` rearranges data for optimal file size before writing, while `autoCompact` automatically merges small files after writes. These configurations reduce the need for manual maintenance, lower storage costs, and ensure read queries perform optimally by preventing file fragmentation. This proactive approach is standard practice for high-performance Delta Lake implementations that minimize manual intervention.

Answer

Write latency will increase, but read performance will remain unaffected.

Answer

Storage costs will increase significantly due to the creation of temporary duplicate data files.

Answer

The cluster will fail to start because these configurations are deprecated in recent Databricks runtimes.

Question 14

A data engineer is tasked with ensuring that sensitive information in a 'customer' table is masked for all users except the 'Data_Science' group. What is the correct Unity Catalog feature to implement?

Accepted Answer

Apply a masking policy using a SQL function to the table column.. Dynamic data masking in Unity Catalog allows for the creation of masking functions that return obfuscated values based on the current user's role. By assigning these functions to columns, administrators ensure that sensitive data is protected while remaining available for authorized users. This approach is highly effective for maintaining data usability without compromising the security of PII.

Answer

Create a view that performs a CASE statement to mask the column.

Answer

Use the 'DROP COLUMN' command to remove sensitive columns.

Answer

Encrypt the column using an external library before writing to Delta.

Question 15

Refer to the exhibit. An engineer observes that queries filtering on 'customer_id' are running slowly despite Z-Ordering. What is the most likely cause?

Accepted Answer

The queries lack filters on 'region' or 'date', preventing effective partition pruning.. The exhibit shows that the table is partitioned by 'region' and 'date', while Z-Ordering is applied to 'customer_id'. If queries filter on 'customer_id' but do not provide 'region' or 'date', the engine must scan all partitions. Z-Ordering is only effective within each partition. If the data is not well-clustered or the partitions are too large, the engine cannot skip files effectively, leading to high latency during execution.

Answer

The partition columns should include 'customer_id' to improve the pruning speed.

Answer

Z-Ordering must be performed on the partition columns instead of the join columns.

Answer

The file format should be changed to Parquet to improve individual file read performance.

Question 16

A data scientist reports that their notebook takes 20 minutes to initialize, even when the cluster is running. What is the most likely reason for this high initialization time?

Accepted Answer

The notebook is installing many libraries using %pip install.. High initialization time in a running cluster is frequently caused by excessive libraries being installed at the notebook level via '%pip install'. Every time the notebook is attached or the context is refreshed, these libraries must be re-resolved and installed, which adds significant overhead. Pre-installing these libraries at the cluster level via 'cluster-scoped libraries' allows them to be available immediately upon startup, eliminating this redundant installation phase and significantly speeding up the initialization process.

Answer

The cluster is using spot instances.

Answer

The cluster has too many workers allocated.

Answer

The user is using the wrong Databricks Runtime version.

Question 17

You are optimizing a PySpark job that reads from a Delta table. You notice skewed data distribution on the 'customer_id' column, causing Task-level stragglers. Which transformation should you apply to the DataFrame to mitigate this skew during a join operation?

Accepted Answer

Salt the skewed column and perform the join.. Salting involves adding a random prefix to the join key of the skewed table and exploding the join key of the dimension table. This technique breaks down massive partitions caused by skewed keys into smaller, evenly distributed chunks. Understanding data skew is critical in Databricks for maintaining cluster stability and preventing OOM errors, ensuring that shuffle partitions are utilized efficiently across all available executors rather than overloading one core.

Answer

Increase the spark.sql.shuffle.partitions configuration dynamically.

Answer

Broadcast the skewed table to all executors.

Answer

Cache the skewed table in memory before the join.

Question 18

When using the Unity Catalog, how should an engineer properly reference a table named 'sales' located in the 'finance' schema within the 'prod_catalog' catalog using Spark SQL?

Accepted Answer

SELECT * FROM prod_catalog.finance.sales. Unity Catalog uses a three-level namespace hierarchy: catalog.schema.table. This structure provides consistent governance and discovery across the entire Databricks account. Proper referencing requires the fully qualified name to ensure Spark identifies the exact object within the catalog's permission model. This implementation is crucial for security and multi-tenant environments where namespaces must be strictly isolated to prevent unauthorized access and data leakage across production and development environments.

Answer

SELECT * FROM finance.sales

Answer

SELECT * FROM sales

Answer

SELECT * FROM catalog_prod.finance.sales

Question 19

A Data Engineer wants to monitor data quality trends over time for a critical table. Which tool provides the most native and easy-to-use visualization of these metrics?

Accepted Answer

Create a Databricks SQL dashboard using the 'expectations' table provided by DLT.. Delta Lake's integration with Databricks SQL and DLT provides native dashboards that track data quality metrics automatically. Using Expectation history logs, engineers can visualize failure rates and quality trends without building custom monitoring solutions. This visibility is essential for proactive maintenance, allowing teams to catch degrading data quality before it impacts critical downstream business reporting, ensuring higher service levels for end-users.

Answer

Export all data quality logs to an external S3 bucket and use a third-party BI tool.

Answer

Write a custom Spark job to parse the Delta transaction log and send alerts via email.

Answer

Manually query the 'information_schema' every hour to track table row counts.

Question 20

A data engineer wants to monitor the health of Delta Live Tables (DLT) pipelines and be alerted if a pipeline fails. Which approach is the most efficient and native way to achieve this?

Accepted Answer

Define a Notification Destination in the DLT pipeline settings and associate it with the 'On failure' event.. Using Notification Destinations in the DLT pipeline settings is the most native method. It allows engineers to configure email or Slack alerts directly within the UI or JSON configuration. This is crucial for production reliability, ensuring that stakeholders receive immediate notifications regarding job failures or data quality issues without needing external orchestration or custom API scripts, maintaining observability across the entire pipeline lifecycle.

Answer

Configure an external cron job to poll the Databricks Jobs API every minute for status updates.

Answer

Enable the 'Log to Workspace' feature and manually query the system logs using SQL every hour.

Answer

Use the Databricks SQL Alerts tool to monitor the underlying Delta tables for new record counts.