Data Transformation, Cleansing, Quality

You are building a pipeline and notice that the 'Gold' layer tables are experiencing significant write latency due to frequent small file commits. What is the most effective way to resolve this while maintaining ACID integrity?

A Data Engineer is using Delta Live Tables to process a stream of user events. The `user_id` column should be unique in the target table, but the source may contain duplicate events due to retries. The engineer wants to keep only the latest event for each `user_id` based on the `event_timestamp`. Which Delta Live Tables feature should be used?

A Data Engineer is using Delta Live Tables to process a streaming source that contains duplicate records based on an 'event_id'. The engineer needs to ensure that only the latest record for each 'event_id' is retained in the target table, and the pipeline should handle late-arriving data. Which DLT feature should be used?

A Data Engineer is tasked with cleaning a dataset in Databricks. The dataset contains a column 'phone_number' with various formats, including parentheses, dashes, and spaces. The engineer needs to standardize all phone numbers to a digits-only format (e.g., '1234567890'). Which approach is most efficient and scalable?

A Data Engineer needs to enforce a NOT NULL constraint on a specific column in a Delta table while maintaining the ability to perform high-performance streaming writes. Which approach is the most efficient and native method to ensure this data quality requirement?

A Data Engineer is implementing a medallion architecture. Which THREE steps are critical for effectively implementing a high-quality 'Silver' layer from 'Bronze' data?

A Data Engineer is using Delta Live Tables to process a stream of financial transactions. The pipeline must ensure that each `transaction_id` appears only once in the target table, even if the source stream contains duplicates due to at-least-once ingestion. The engineer wants to use the `APPLY CHANGES` API. Which combination of settings will achieve this with minimal data loss?

Refer to the exhibit. A Data Engineer is attempting to merge data into a table with these constraints defined. If the incoming batch contains rows that violate these rules, what is the default behavior of the Delta Lake engine during the merge operation?

When designing a Data Quality framework in Databricks, what is the recommended approach for handling 'quarantined' records?

A Data Engineer needs to verify that the column 'user_id' is unique in a critical Gold table. What is the most efficient, non-blocking way to perform this check in a production environment?

A data engineer is using PySpark to cleanse a DataFrame containing customer addresses. The 'zip_code' column has some values with leading zeros that were stripped during CSV ingestion. The engineer needs to restore all zip codes to a fixed 5-character length by padding with leading zeros. Which function should be used?

A data engineer is using Delta Lake to manage a table that receives frequent updates and deletes. The engineer notices that query performance has degraded over time due to many small files. Which command should be used to optimize the table by compacting small files and improving query performance?

You need to perform a deduplication task on a streaming source that includes late-arriving data. Which Delta Lake feature is best suited to manage this while ensuring efficient state cleanup?

When refining data in a Medallion architecture, why is it recommended to perform schema enforcement as early as possible in the Bronze layer?

A Data Engineer is using Delta Live Tables (DLT) to build a pipeline that ingests JSON files from cloud storage. The engineer defines a streaming table with expectations to enforce data quality. The expectation `@dlt.expect_or_drop("valid_timestamp", "timestamp IS NOT NULL")` is applied. During a pipeline run, 5% of records have a NULL timestamp. What is the outcome for those records, and how does it affect the pipeline?

You are tasked with handling PII (Personally Identifiable Information) in your data pipeline. Which approach is best for protecting this data while maintaining the ability to perform analytics?

Which TWO statements regarding the use of 'APPLY CHANGES INTO' in Delta Live Tables (DLT) are correct?

A data engineer is using PySpark to cleanse a large dataset of customer records. The DataFrame `df` contains a string column `phone` with values like '123-456-7890', '(123) 456-7890', and '1234567890'. The engineer needs to standardize these to digits only (e.g., '1234567890'). Which transformation should be used?

A Data Engineer wants to monitor data quality trends over time for a critical table. Which tool provides the most native and easy-to-use visualization of these metrics?

You are performing a complex data transformation involving a self-join on a large, skewed table. Which technique is most effective for preventing data skew and improving join performance?

You are migrating a legacy CSV-based ETL process to Databricks. The source CSV files contain inconsistent date formats. Which approach provides the most scalable way to handle these inconsistencies during the bronze-to-silver transformation?

A Data Engineer is working on a Delta Live Tables (DLT) pipeline that ingests JSON files from cloud storage. The pipeline must drop rows where the 'email' column is null and also flag rows where 'age' is negative as invalid, but still process them. Which combination of DLT expectations should be used?

A Data Engineer is building a Databricks SQL pipeline that ingests clickstream events from a Delta table. The events table contains a nested column `payload` of type STRUCT with fields `page_id` (STRING), `duration` (INT), and `referrer` (STRING). The engineer needs to flatten the `payload` fields into top-level columns and drop any records where `page_id` is NULL. Which SQL expression accomplishes this transformation while preserving all other columns?

A Data Engineer is building a Delta Live Tables (DLT) pipeline to ingest raw JSON data. They need to ensure that records missing the required 'user_id' field are dropped while simultaneously capturing these discarded records in a separate table for auditing purposes. Which approach achieves this in DLT?

A Data Engineer is building a Lakeflow Spark Declarative Pipelines pipeline that ingests JSON sensor events. The pipeline must drop records where the `sensor_id` is NULL, ensure that `event_time` is not in the future, and continue processing without failing the update. Which combination of expectations should be used?