Data Transformation and Modeling

A data engineer is designing a Delta Lake Bronze-to-Silver pipeline in Databricks and needs to ensure that downstream consumers receive high-quality data. Which TWO data quality enforcement mechanisms are natively supported in Delta Live Tables using expectations?

A data engineer is designing a Bronze-to-Silver transformation pipeline using Delta Lake. They need to ensure that the Silver table contains only records where the 'transaction_id' is not null and the 'amount' is positive. Which technique best ensures data quality at this stage?

A data engineer is migrating legacy batch jobs to Delta Live Tables (DLT). They want to optimize performance for a complex join operation between two large tables. Which TWO strategies should they implement to improve the join efficiency?

You are tasked with handling late-arriving data in a streaming pipeline that performs windowed aggregations. Which approach ensures that the output remains accurate while balancing memory usage?

When implementing a Medallion Architecture, what is the primary purpose of the 'Silver' layer?

A team is preparing to optimize their Databricks data transformation pipeline. Which THREE of the following actions are considered best practices for optimizing Delta Lake performance?

Refer to the exhibit. The merge operation is failing in your pipeline. What is the root cause of this error, and how should it be resolved?

You are building a pipeline where a Bronze table contains JSON data with a nested 'user_info' struct. You need to promote this to a Silver table where 'user_id' is a top-level column. Which approach is the most efficient for this transformation?

Your organization requires that all data processing pipelines enforce a strict schema to prevent corrupt data from landing in the Silver layer. Which feature should be configured to ensure that only data matching the expected schema is written?

What is the primary benefit of using Data Live Tables (DLT) for managing dependencies between tables in a pipeline?

A data engineer wants to use 'Expectations' in Delta Live Tables to monitor data quality. What happens if a record violates an expectation defined with the 'fail' constraint?

Refer to the exhibit. The streaming pipeline is experiencing memory issues because the state store size increases continuously. What is the most effective way to address this while maintaining aggregation accuracy?

A data engineer is designing a Delta Live Tables (DLT) pipeline. They need to ensure that records with missing values in the 'customer_id' column are dropped during the ingestion process. Which constraint syntax should be used?

Which TWO of the following statements correctly describe the behavior of the Delta Lake 'MERGE' operation when handling schema evolution?

Refer to the exhibit. A data engineer is attempting to ingest JSON data into a Delta table. Based on the error log, what is the most appropriate transformation step to implement before loading this data into the production table?

Which of the following is the primary benefit of using 'Auto Loader' (cloudFiles) for data ingestion in Databricks compared to standard batch processing?

A data engineer is working on a Bronze-to-Silver transformation. Which THREE of the following practices are recommended to optimize performance and data quality during this stage?

A data engineer needs to join a small 'lookup' table with a large 'transactions' table in a Spark job. Which transformation strategy will provide the best performance in a cluster environment?

Refer to the exhibit. A data engineer is configuring a streaming pipeline. Which outcome will these specific configurations have on the target table's performance?

What is the primary function of a Delta Lake 'Vacuum' operation?

A data engineer needs to perform an upsert on a target Delta table using a source DataFrame. Which operation provides the most robust mechanism to handle duplicates and updates in a single pass?

Refer to the exhibit. An engineer notices that queries filtering by 'customer_id' are running slowly on the 'orders' table. Based on the exhibit, what is the most appropriate action to resolve this?

A data engineer is working with a large Delta table and notices that queries filtering by 'region_id' are performing slowly. The table is currently partitioned by 'date'. Which strategy should the engineer use to optimize query performance for 'region_id' filtering without increasing the number of partitions?

Which TWO of the following statements accurately describe the behavior of Delta Lake table constraints and enforcement?

Refer to the exhibit. A data engineer is reviewing the configuration of a Delta table that is frequently queried by 'customer_id'. Given the current metadata, which action will provide the most significant improvement to query performance?

A data engineering team is migrating a legacy data warehouse to Databricks. They want to ensure that their raw data is ingested into a 'Bronze' table in its original format. Which approach is most recommended for this ingestion layer?

A data engineer is using Structured Streaming to ingest data from a Kafka topic. They want to ensure that if the pipeline fails, it can resume exactly where it left off, without processing duplicate data. Which component enables this functionality?

Refer to the exhibit. A data engineer is running these commands before performing heavy write operations into a Delta table. What is the primary benefit of enabling these configurations?

A data engineer wants to move data from a 'Bronze' table to a 'Silver' table while performing data cleaning. They want to ensure that this process is only executed once per data batch. Which approach is best for this requirement?

An analytics team needs to frequently query a large Delta table by a high-cardinality customer_id column and a date column. To optimize query performance and reduce data scanning during filtering, how should the data engineer structure the table layout?

A junior data engineer writes a PySpark transformation that reads a Parquet dataset, filters out inactive users, and appends the resulting DataFrame to an existing Delta Lake table. However, the data engineer notices duplicate records appearing in the target table after multiple runs. Which technique should be implemented to ensure idempotency?

A data engineer needs to optimize the layout of a massive Delta Lake table that suffers from poor query performance due to a large number of small files and unsorted data records. Which TWO operations should the engineer execute to resolve these performance bottlenecks?

A data engineering team is building a medallion architecture in Databricks. In the Silver layer, streaming data from Kafka must be cleaned, deduplicated, and written into a Delta table. Which Structured Streaming output mode should the engineer select to ensure append-only storage of fully processed, stateful deduplicated records?

A data engineer is working on a Delta Lake table that has accumulated millions of small files due to frequent streaming updates. This fragmentation has significantly degraded query performance. Which operation should the engineer execute to optimize file layout without altering table data?

A data engineer is designing a Delta Lake pipeline that processes streaming sales transactions. The schema evolves frequently, and the pipeline must handle these changes without manual intervention. Which feature should the engineer enable to support automatic schema updates while preventing data corruption?

Which TWO of the following scenarios are valid use cases for utilizing Delta Lake's Change Data Feed (CDF)?

Refer to the exhibit. A data engineer is attempting to run a VACUUM command on a table, but the command fails with an error indicating that the retention period is too short. Given the configuration, what is the most appropriate action the engineer should take to safely remove files older than 7 days?

Which object type in Databricks Unity Catalog acts as the top-level container for organizing schemas and tables, providing a unified namespace for data assets?

A data engineer is working with a large, partitioned table and needs to perform a complex transformation. Which THREE of the following strategies will optimize query performance for this transformation?

A data engineer needs to join two massive datasets. One dataset is very small (10MB), and the other is very large (1TB). To ensure the join operation is performed as efficiently as possible, which join strategy should be enforced?

A data engineer is building a Databricks job that processes millions of small JSON files landed in cloud storage each hour. The job currently spends most of its runtime on file listing and task scheduling overhead. The engineer wants to improve throughput without changing the downstream table schema. Which change should be made to the ingestion step?

A data engineer has a Delta table named `sales` with columns `sale_id`, `customer_id`, `amount`, and `sale_date`. They need to create a new table that contains only the `customer_id` and the total `amount` per customer for all sales in 2023. Which SQL statement correctly creates this aggregated table?

A data engineer maintains a Delta table where each row represents a customer record, and updates arrive continuously as change data capture events. The engineer needs to apply inserts, updates, and deletes from a staging table into the target table in a single atomic operation, matching records on customer_id. Which Delta Lake operation should be used?

A data engineer maintains a Delta table named inventory.products with columns product_id, category, price, and updated_at. The engineer needs to create a new table that contains one row per category with the average price and the most recently updated product_id in that category. The query must be efficient and use only standard Databricks SQL. Which statement should the engineer run?

A data engineer is implementing a Type 2 slowly changing dimension in Delta Lake for a customers table. The table has columns customer_id, name, address, effective_date, end_date, and is_current. When a customer's address changes, the engineer wants to expire the existing current row and insert a new current row in a single atomic operation. Which Delta Lake feature should the engineer use?

A data engineer is creating a Silver table in a Delta Live Tables pipeline. The pipeline must continuously ingest new files from a cloud storage location as they arrive, and the engineer wants to avoid reprocessing files that were already ingested. Which approach should be used to read the source data?

A data engineer has a Delta table named silver_events with columns event_id (string), event_ts (timestamp), and payload (string). The table is partitioned by event_date (derived from event_ts). The engineer needs to update the payload column for all events that occurred on '2024-06-01' based on a mapping table named updates (event_id, new_payload). Which PySpark operation should be used to perform this update efficiently while preserving Delta Lake ACID guarantees?

A data engineer is building a Delta Live Tables pipeline that ingests streaming data from a Kafka topic into a bronze table, then applies a series of transformations to produce a silver table. The engineer notices that the pipeline is reprocessing all data from the beginning of the Kafka topic on each run, causing high latency. The Kafka topic has a retention period of 7 days, and the pipeline is configured to use the default settings. What is the most likely cause of this behavior?

A data engineer is working with a Delta table that contains a column named raw_data of type STRING, which holds JSON strings. The engineer needs to extract specific fields from this JSON and store them as separate columns in a new Delta table. Which approach is most efficient and maintains data quality?

A data engineer needs to create a Silver Delta table that contains only distinct, non-null `customer_id` values from a Bronze table, and the result must be refreshed idempotently each night. Which statement best satisfies the requirement?

A data engineer is building a Gold aggregate table that summarizes daily sales by product category. The Silver source is a streaming Delta table that receives late-arriving events up to 48 hours old. The engineer needs the Gold table to always reflect the most accurate aggregates, including corrections for late data, without full recomputation. Which approach is most appropriate?

A data engineer is using Delta Live Tables to build a pipeline. They need to create a table that contains the latest record for each customer based on a `last_updated` timestamp. The source is a streaming table with append-only data. Which Delta Live Tables operation should be used to achieve this?

A data engineer is building a Silver table in Delta Lake from a Bronze table that contains raw JSON events. The engineer needs to flatten a nested struct column named 'device' with fields 'type' and 'os', and also extract a field from an array of structs named 'events'. The goal is to produce a clean, denormalized Silver table. Which PySpark operation should the engineer use to achieve this transformation efficiently?

A data engineer is working with a Delta table that contains a column 'timestamp' of type timestamp. The table is partitioned by date. The engineer needs to run a query that filters on a specific date range and also on a high-cardinality column 'user_id'. The query is performing poorly. Which optimization technique should the engineer apply to improve query performance?