Databricks-DE-Pro Data Transformation, Cleansing, Quality — 10 Questions

Question 1

A Data Engineer is building a pipeline using Delta Live Tables (DLT) to clean IoT sensor data. Which TWO of the following statements regarding the implementation of Expectations are correct?

Accepted Answer

Expectations must be applied using standard Python decorators within the DLT pipeline code.. DLT Expectations provide a declarative way to handle data quality in both SQL and Python. There are three primary policies: 'expect' (logs the violation but allows the record to pass), 'expect or drop' (logs the violation and drops the record), and 'expect or fail' (logs the violation and fails the update). This flexibility allows engineers to monitor data quality (using the 'expect' policy) without interrupting the pipeline flow or losing data.

Answer

When an expectation fails, the entire DLT pipeline will stop processing immediately.

Answer

Expectations are only compatible with Bronze-layer tables in a Medallion architecture.

Answer

Expectations require a manual SQL job to trigger the validation logic on every batch.

Question 2

Refer to the exhibit. A pipeline job is failing because of a schema mismatch between the source data and the Delta table. Which solution allows the pipeline to succeed without manually changing the source data or the existing table schema?

Accepted Answer

Transform the DataFrame by explicitly casting the 'ts' column to a TIMESTAMP type before the write operation.. The explanation incorrectly discusses 'mergeSchema' and 'evolving data', which are irrelevant to resolving a type mismatch (e.g., String vs Timestamp). The correct answer (B) is correct because explicit casting resolves the type conflict at the DataFrame level before the write operation, which is the only way to handle this without schema evolution or manual table changes.

Answer

Set the spark configuration 'spark.databricks.delta.schema.autoMerge.enabled' to true.

Answer

Run an ALTER TABLE command to change the 'ts' column type to STRING before the job starts.

Answer

Use 'overwriteSchema' to force the Delta table to adapt to the incoming DataFrame.

Question 3

You are optimizing a pipeline that processes high-volume JSON data. Which THREE techniques will improve the performance and quality of the transformation layer?

Accepted Answer

Convert the JSON source into Parquet during the Bronze layer ingestion.. To optimize a pipeline processing high-volume JSON data, you should convert the raw data into an efficient columnar format like Parquet (Option B), explicitly define schemas using DDL strings with functions like 'from_json' or Auto Loader options to ensure stability (Option D), and leverage Auto Loader's schema evolution capabilities rather than inferring schemas repeatedly. Option A is an anti-pattern for production scale.

Answer

Use 'cloudFiles.inferSchema' to automate schema discovery during ingestion.

Answer

Always set the 'numPartitions' to 1 to ensure that all data is processed in a single batch.

Answer

Perform a 'repartition' on every column before writing to the Delta table.

Question 4

A data engineering team is optimizing a massive Delta Lake table partitioned by date and clustered by customer_id using Liquid Clustering. The table experiences frequent updates and deletes based on streaming CDC feeds. Which underlying Delta Lake feature allows this pattern to perform efficiently without causing file compaction bottlenecks?

Accepted Answer

Delta Lake deletion vectors avoid rewriting entire parquet data files during row-level updates and deletes by recording changes in separate bitmap files.. Deletion vectors are a storage optimization that allows Delta Lake to mark rows as deleted or changed without rewriting the entire Parquet data file. In the traditional 'copy-on-write' model, updating a single row requires rewriting the whole file, which leads to write amplification and compaction bottlenecks during frequent CDC (Change Data Capture) operations. Deletion vectors store these changes in a separate bitmap file, significantly improving write performance for updates and deletes.

Answer

Predictive Optimization automatically compacts small files created by frequent CDC updates in the background without manual intervention.

Answer

Uniform (UniForm) automatically translates Delta Lake transaction logs into Iceberg metadata formats for external engine compatibility.

Answer

Z-ordering indices cluster data across multiple columns within partitions to maximize data skipping during read operations.

Question 5

Refer to the exhibit. A Data Engineer is reviewing a DLT configuration file. What is the impact of the 'on_violation' setting on the pipeline?

Accepted Answer

The pipeline halts execution entirely, preventing the current batch from being committed.. In Databricks Delta Live Tables, the 'on_violation' key does not exist as a top-level configuration. Data quality constraints are defined using the 'expectations' parameter, where the 'fail' constraint (e.g., 'expect_or_fail') causes the pipeline to halt. The question incorrectly references a non-existent configuration key.

Answer

The pipeline ignores the violation and continues processing, logging only the error.

Answer

Only the violating rows are removed from the result set, but the pipeline keeps running.

Answer

The pipeline flags the rows as invalid but writes them to the target table anyway.

Question 6

A Data Engineer needs to perform a complex data cleaning operation that involves joining a streaming table with a large, static lookup table. The lookup table is updated daily via a separate batch process. What is the most efficient way to ensure the join is performant and consistent within a DLT pipeline?

Accepted Answer

Use the broadcast join hint on the static lookup table to optimize performance.. Broadcasting is the most performant strategy for joining a streaming table with a static lookup table because it avoids a costly shuffle of the streaming data. While Spark's auto-broadcast threshold is 10MB, using a broadcast hint allows for larger tables (up to several GBs) to be broadcasted, provided they fit in the executor's memory. In a DLT pipeline, the static side of a stream-static join is re-evaluated for each micro-batch, ensuring that daily updates to the lookup table are picked up (consistency) without needing a stateful stream-to-stream join.

Answer

Perform a standard shuffle join to ensure accuracy with the daily updated data.

Answer

Convert the static lookup table into a streaming source and use a streaming-to-streaming join.

Answer

Cache the lookup table at the start of the pipeline and use it in a cross join.

Question 7

A data engineer is building a Delta Live Tables pipeline that ingests raw JSON events and must produce a cleansed Silver table. The pipeline must handle malformed records without failing, capture metrics about data quality, and allow downstream analysts to query only valid records. Which two actions should the engineer take? (Choose two.)

Accepted Answer

Define the Silver table with @dlt.table and use @dlt.expect to track invalid records without dropping them, then create a separate view filtering valid records.. The pipeline must not fail on malformed records, must capture quality metrics, and must allow querying valid records. Using @dlt.expect with a separate view (or @dlt.expect_or_drop with a quarantine flow) achieves this. Both approaches record metrics and prevent pipeline failure. The combination of expect_or_drop plus quarantine also preserves invalid data for analysis, which is often required in production pipelines.

Answer

Define expectations using @dlt.expect_or_drop to filter out invalid records and record metrics in the event log.

Answer

Use @dlt.expect_or_fail to enforce strict quality and halt the pipeline on any invalid record.

Answer

Define expectations using the @dlt.expect decorator with the 'drop' action to remove invalid records from the Silver table.

Question 8

A Data Engineer is working with a Delta table that contains a column `email` with inconsistent casing and leading/trailing spaces. The engineer needs to standardize the email addresses by trimming spaces and converting to lowercase. Which SQL expression correctly transforms the `email` column in a SELECT statement?

Accepted Answer

SELECT lower(trim(email)) AS email_clean FROM table. The correct expression uses `trim` to remove leading and trailing spaces and `lower` to convert to lowercase, achieving standardization. The order of these functions is not critical, but the combination ensures the email is cleansed. Other options either miss case conversion, use verbose alternatives, or rely on regex that does not handle all whitespace or case.

Answer

SELECT trim(lower(email)) AS email_clean FROM table

Answer

SELECT lcase(rtrim(ltrim(email))) AS email_clean FROM table

Answer

SELECT regexp_replace(email, '^\s+|\s+$', '') AS email_clean FROM table

Question 9

A data engineer is implementing a data quality check on a streaming Delta table that receives late-arriving events. The engineer needs to detect records where the event timestamp is more than 2 hours behind the current watermark. Which approach correctly identifies such records for quarantine?

Accepted Answer

Use the event_time column and compare it to current_timestamp() - INTERVAL 2 HOURS in a filter condition.. To identify records that are more than 2 hours behind the current time, the engineer should compare the event timestamp to the current processing time. Using current_timestamp() - INTERVAL 2 HOURS provides a dynamic threshold that correctly flags late-arriving events. This approach works in streaming and batch, and does not rely on watermark internals.

Answer

Use a watermark on event_time and filter records where event_time < current_watermark - INTERVAL 2 HOURS.

Answer

Use the event_time column and compare it to the current watermark using the watermark() function.

Answer

Use the event_time column and compare it to the maximum event_time in the table minus 2 hours.

Question 10

A Data Engineer is building a Databricks job to cleanse a large dataset. The engineer needs to identify and handle duplicate records, missing values, and outliers. Which TWO techniques are most appropriate for addressing these data quality issues in a scalable way? (Choose two.)

Accepted Answer

Use Delta Lake's MERGE INTO to deduplicate records based on a unique key.. MERGE INTO and dropDuplicates() are both scalable, Spark-native techniques for handling duplicates. MERGE INTO allows for sophisticated deduplication with conditions, while dropDuplicates() efficiently removes exact duplicates. Other options involve non-scalable single-node operations or are not directly focused on deduplication and missing values in a scalable job context.

Answer

Implement outlier detection using the IQR method on a single node.

Answer

Use pandas to fill missing values with the mean of each column.

Answer

Use Delta Live Tables expectations to drop rows with null values.