A candidate must design transformations that cleanse, validate, and join data correctly on Databricks. The most important thing is knowing how Delta Lake and Lakeflow pipelines handle invalid records by default, and applying skew mitigation like salting when joins on large tables are involved.
Start practicing
Data Transformation, Cleansing, Quality — choose a session length
Free · No account required
Domain overview
This domain covers building reliable transformations on Databricks: cleansing and validating data, handling skew and nested types, and enforcing quality in Delta Lake and Lakeflow Spark Declarative Pipelines. Questions present concrete scenarios—self-joins on skewed tables, clickstream STRUCT columns, JSON sensor ingestion, and MERGE constraint violations—and ask you to choose the correct technique or predict default behavior.
Exam objectives
Applying salting and skew hints to optimize self-joins on large skewed Delta tables
Flattening and casting nested STRUCT fields from Delta clickstream payloads in Databricks SQL
Using Lakeflow Spark Declarative Pipelines expectations to drop or quarantine invalid records
Predicting Delta Lake default behavior when MERGE rows violate CHECK or NOT NULL constraints
Assuming Delta MERGE silently drops bad rows; by default constraint violations raise errors and can abort the transaction unless handled.
Repartitioning or broadcasting a skewed self-join without salting the key, which leaves straggler tasks and does not fix skew.
Treating nested STRUCT fields as flat columns instead of using dot notation or explode, causing schema and analysis errors.
Click any question to see the full explanation and answer options, or start a focused practice session above.
A Data Engineer needs to enforce a NOT NULL constraint on a specific column in a Delta table while maintaining the ability to perform high-performance streaming writes. Which approach is the most efficient and native method to ensure this data quality requirement?
2Refer to the exhibit. A Data Engineer is attempting to merge data into a table with these constraints defined. If the incoming batch contains rows that violate these rules, what is the default behavior of the Delta Lake engine during the merge operation?
3You are migrating a legacy CSV-based ETL process to Databricks. The source CSV files contain inconsistent date formats. Which approach provides the most scalable way to handle these inconsistencies during the bronze-to-silver transformation?
4You need to perform a deduplication task on a streaming source that includes late-arriving data. Which Delta Lake feature is best suited to manage this while ensuring efficient state cleanup?
5A Data Engineer is implementing a medallion architecture. Which THREE steps are critical for effectively implementing a high-quality 'Silver' layer from 'Bronze' data?
6You are tasked with handling PII (Personally Identifiable Information) in your data pipeline. Which approach is best for protecting this data while maintaining the ability to perform analytics?
7A Data Engineer wants to monitor data quality trends over time for a critical table. Which tool provides the most native and easy-to-use visualization of these metrics?
8When designing a Data Quality framework in Databricks, what is the recommended approach for handling 'quarantined' records?
9You are performing a complex data transformation involving a self-join on a large, skewed table. Which technique is most effective for preventing data skew and improving join performance?
10A Data Engineer needs to verify that the column 'user_id' is unique in a critical Gold table. What is the most efficient, non-blocking way to perform this check in a production environment?
11When refining data in a Medallion architecture, why is it recommended to perform schema enforcement as early as possible in the Bronze layer?
12You are building a pipeline and notice that the 'Gold' layer tables are experiencing significant write latency due to frequent small file commits. What is the most effective way to resolve this while maintaining ACID integrity?
13A Data Engineer is building a Delta Live Tables (DLT) pipeline to ingest raw JSON data. They need to ensure that records missing the required 'user_id' field are dropped while simultaneously capturing these discarded records in a separate table for auditing purposes. Which approach achieves this in DLT?
14Which TWO statements regarding the use of 'APPLY CHANGES INTO' in Delta Live Tables (DLT) are correct?
15A Data Engineer is building a Databricks SQL pipeline that ingests clickstream events from a Delta table. The events table contains a nested column `payload` of type STRUCT with fields `page_id` (STRING), `duration` (INT), and `referrer` (STRING). The engineer needs to flatten the `payload` fields into top-level columns and drop any records where `page_id` is NULL. Which SQL expression accomplishes this transformation while preserving all other columns?
16A Data Engineer is using Delta Live Tables (DLT) to build a pipeline that ingests JSON files from cloud storage. The engineer defines a streaming table with expectations to enforce data quality. The expectation `@dlt.expect_or_drop("valid_timestamp", "timestamp IS NOT NULL")` is applied. During a pipeline run, 5% of records have a NULL timestamp. What is the outcome for those records, and how does it affect the pipeline?
17A Data Engineer is working on a Delta Live Tables (DLT) pipeline that ingests JSON files from cloud storage. The pipeline must drop rows where the 'email' column is null and also flag rows where 'age' is negative as invalid, but still process them. Which combination of DLT expectations should be used?
18A data engineer is using PySpark to cleanse a DataFrame containing customer addresses. The 'zip_code' column has some values with leading zeros that were stripped during CSV ingestion. The engineer needs to restore all zip codes to a fixed 5-character length by padding with leading zeros. Which function should be used?
19A Data Engineer is building a Lakeflow Spark Declarative Pipelines pipeline that ingests JSON sensor events. The pipeline must drop records where the `sensor_id` is NULL, ensure that `event_time` is not in the future, and continue processing without failing the update. Which combination of expectations should be used?
20A Data Engineer is using Delta Live Tables to process a streaming source that contains duplicate records based on an 'event_id'. The engineer needs to ensure that only the latest record for each 'event_id' is retained in the target table, and the pipeline should handle late-arriving data. Which DLT feature should be used?
21A Data Engineer is using Delta Live Tables to process a stream of financial transactions. The pipeline must ensure that each `transaction_id` appears only once in the target table, even if the source stream contains duplicates due to at-least-once ingestion. The engineer wants to use the `APPLY CHANGES` API. Which combination of settings will achieve this with minimal data loss?
22A data engineer is using PySpark to cleanse a large dataset of customer records. The DataFrame `df` contains a string column `phone` with values like '123-456-7890', '(123) 456-7890', and '1234567890'. The engineer needs to standardize these to digits only (e.g., '1234567890'). Which transformation should be used?
23A Data Engineer is tasked with cleaning a dataset in Databricks. The dataset contains a column 'phone_number' with various formats, including parentheses, dashes, and spaces. The engineer needs to standardize all phone numbers to a digits-only format (e.g., '1234567890'). Which approach is most efficient and scalable?
24A data engineer is using Delta Lake to manage a table that receives frequent updates and deletes. The engineer notices that query performance has degraded over time due to many small files. Which command should be used to optimize the table by compacting small files and improving query performance?
25A Data Engineer is using Delta Live Tables to process a stream of user events. The `user_id` column should be unique in the target table, but the source may contain duplicate events due to retries. The engineer wants to keep only the latest event for each `user_id` based on the `event_timestamp`. Which Delta Live Tables feature should be used?
A candidate must design transformations that cleanse, validate, and join data correctly on Databricks. The most important thing is knowing how Delta Lake and Lakeflow pipelines handle invalid records by default, and applying skew mitigation like salting when joins on large tables are involved.
The Courseiva Databricks-DE-Pro question bank contains 25 questions in the Data Transformation, Cleansing, Quality domain. Click any question to see the full explanation and answer breakdown.
Start with a 10-question focused session to identify your baseline accuracy in this domain. Read every explanation — even for questions you answer correctly — to understand the reasoning. Once you score consistently above 80%, move to a 20–30 question session to confirm depth before moving to the next domain.
Yes — the session launcher on this page draws questions exclusively from the Data Transformation, Cleansing, Quality domain. Choose 10, 20, 30, or 50 questions for a focused session, or click individual questions to review them one by one.
Save your results, see per-domain analytics, and get readiness scores — free, for every certification.
Sign Up FreeFree forever · Every certification included