Databricks-DE-Pro Data Ingestion and Acquisition Practice Test 2 — 15 Questions

Question 1

A data engineer is designing an ingestion pipeline using Databricks Auto Loader to ingest files from a cloud storage location into a Delta table. The engineer needs to ensure that the pipeline is resilient to schema changes and can handle files that contain corrupt records. Which two features should the engineer configure? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure a rescuedDataColumn to capture records that do not match the schema.. To handle schema changes and corrupt records in Auto Loader, the engineer should configure schema evolution to 'addNewColumns' (or 'rescue') and enable a rescuedDataColumn. The 'rescue' mode automatically captures malformed records into a rescued column, preventing stream failure. The rescuedDataColumn option provides a place to store those records. Together, they ensure resilience to both schema evolution and corrupt data, making these two features the correct choices.

Answer

Set cloudFiles.schemaLocation to a reliable storage path to store the inferred schema.

Answer

Set cloudFiles.schemaEvolutionMode to 'addNewColumns' to automatically add new columns to the schema.

Answer

Enable cloudFiles.allowOverwrites to allow files to be overwritten in the source location.

Question 2

Refer to the exhibit. You are using Auto Loader to ingest data with evolving schemas. After running the job for a week, you realize that new columns added to the source JSON are not being captured in the destination table. What must you add to the configuration?

Accepted Answer

Add 'cloudFiles.schemaEvolutionMode': 'addCol'.. By default, Auto Loader schema inference only detects the schema during the initial load. To capture schema evolution, you must explicitly enable 'cloudFiles.schemaEvolutionMode'. Without this parameter, Auto Loader ignores new fields to protect downstream consumers from breaking changes. Enabling 'addCol' allows the schema to expand dynamically, ensuring the target Delta table reflects the structure of the incoming data files as they arrive.

Answer

Add 'cloudFiles.schemaEvolutionMode': 'rescue'.

Answer

Add 'cloudFiles.maxFiles': '1000'.

Answer

Add 'cloudFiles.allowOverwrites': 'true'.

Question 3

Which approach is most appropriate for ingesting data from a JDBC source into Delta Lake where the source table has no 'updated_at' or 'version' column for incremental loading?

Accepted Answer

Perform a full overwrite of the Delta table for every load.. When a source table lacks a watermark column, you cannot use standard incremental loading techniques. The most robust approach is a full overwrite of the destination table, ensuring that the target remains a faithful copy of the source. While this can be resource-intensive for large tables, it is the only way to ensure data integrity without primary keys or timestamps to track changes.

Answer

Use the 'partitionColumn' parameter with a random UUID.

Answer

Enable streaming ingestion using the JDBC source readStream API.

Answer

Use the 'fetchSize' parameter to optimize the load.

Question 4

Your organization is ingesting sensitive PII data. You need to ensure that personal identifiers are masked during the ingestion process before they are stored in the Bronze layer of your Medallion architecture. What is the best practice for this?

Accepted Answer

Apply masking logic within the initial streaming ingestion transformation.. Implementing masking at the ingestion layer using Delta Live Tables (DLT) expectations or standard Spark transformations ensures that sensitive data is never written in plaintext to the Bronze layer. Applying transformations during the 'Acquisition' phase is a critical security practice, ensuring that governance requirements are met before data becomes available to analysts, thereby reducing the risk of accidental exposure and maintaining data privacy compliance across the pipeline.

Answer

Mask the data using Delta Lake column masking after the data reaches the Silver layer.

Answer

Configure Unity Catalog to mask columns only for specific users.

Answer

Use a post-ingestion job to delete PII columns.

Question 5

Which TWO of the following are primary benefits of using Delta Live Tables (DLT) for data ingestion over standard Structured Streaming pipelines?

Accepted Answer

Declarative pipeline management and automated dependency handling.. DLT simplifies the operational complexity of data pipelines by providing automatic infrastructure management and built-in quality controls. It abstracts the configuration required for managing checkpoints, scaling, and handling schema drift, allowing engineers to focus on defining transformations. The 'Expectations' framework and declarative pipeline management provide superior observability and data quality enforcement compared to manual, imperative coding in standard Spark Structured Streaming.

Answer

DLT supports significantly higher throughput than Structured Streaming.

Answer

DLT is the only way to read from cloud object storage.

Answer

DLT supports non-Delta storage formats for all outputs.

Question 6

When ingesting data using Auto Loader, what is the purpose of the 'cloudFiles.schemaLocation' parameter?

Accepted Answer

It stores metadata about the inferred schema and tracks evolution.. The schema location is vital for Auto Loader because it stores the inferred schema and tracks schema evolution history. By persisting this information in a managed location, Auto Loader can resume ingestion after a job restart without re-inferring the schema from scratch. This ensures consistency and prevents potential ingestion failures caused by schema drift when processing new files in a long-running stream.

Answer

It specifies the target directory where the processed Delta table data is stored.

Answer

It defines the temporary directory used for shuffling large datasets during joins.

Answer

It is used to cache the raw JSON files before they are parsed.

Question 7

Which THREE of the following are essential components of an effective ingestion monitoring strategy in Databricks?

Accepted Answer

Tracking the 'numInputRows' metric in the streaming query progress.. Monitoring ingestion requires visibility into both infrastructure health and data quality. By tracking metrics like micro-batch latency, file processing rates, and record-level validation through expectations, you gain a holistic view of the pipeline. These components allow engineers to identify bottlenecks, respond to data quality drops, and ensure SLAs are met consistently, which is critical for maintaining reliable downstream analytics in a data lakehouse architecture.

Answer

Monitoring the total number of files in the cloud storage bucket.

Answer

Regularly restarting the cluster to clear cache.

Question 8

When ingesting data from a Kafka topic into Delta Lake, what is the best way to handle out-of-order data arriving in the stream?

Accepted Answer

Implement a watermark on the event time column.. In streaming systems, data can arrive delayed. Using Watermarking allows the system to define a threshold for how long it will wait for late data. By keeping state for the specified duration, the engine can correctly aggregate or join records that arrived out of order. This is a standard and essential technique in Spark Structured Streaming to ensure the accuracy of time-windowed operations in high-throughput environments.

Answer

Disable all aggregations to avoid processing errors.

Answer

Increase the 'spark.sql.shuffle.partitions' to 5000.

Answer

Use the 'Trigger.Once' execution mode.

Question 9

You are ingesting data from multiple source systems with varying file formats (JSON, CSV, Parquet) into a centralized Bronze landing zone. Which architecture pattern is the most scalable for maintaining this ingestion layer?

Accepted Answer

A separate ingestion pipeline or notebook for each source system.. A modular ingestion architecture that decouples source-specific logic from the common landing layer is the industry standard. By using parameterized notebooks or DLT pipelines for each source, you isolate potential failures, enable independent scaling, and simplify the management of schema mappings. This pattern ensures that changes in one source system do not impact the ingestion pipelines of others, creating a highly resilient and maintainable data acquisition ecosystem.

Answer

A single monolithic notebook with a massive if/else block for each source.

Answer

Ingest everything into a single raw table before applying transformations.

Answer

Use a third-party tool exclusively and bypass Databricks for ingestion.

Question 10

You are designing an ingestion pipeline that must handle massive bursts of data at irregular intervals. Which feature should you prioritize to ensure the ingestion process remains cost-effective?

Accepted Answer

Use auto-scaling clusters with a minimum of zero workers.. Using Photon-accelerated clusters with auto-scaling is the most effective way to handle bursty workloads. By configuring the cluster to scale out when the queue of files is large and scale down during idle periods, you maximize resource utilization. This approach ensures you have enough compute to meet latency SLAs during bursts while minimizing costs by scaling to zero or minimal size when no data is being ingested, optimizing for both performance and budget.

Answer

Use a fixed-size cluster with a large number of nodes.

Answer

Run the pipeline continuously on a single-node cluster.

Answer

Increase the 'spark.sql.shuffle.partitions' to 10000.

Question 11

What is the primary advantage of using Delta Lake as the sink for your data ingestion pipelines compared to raw Parquet files?

Accepted Answer

Delta Lake provides ACID transactions and schema enforcement.. The primary advantage of Delta Lake over raw Parquet is its support for ACID transactions and schema enforcement. These features prevent data corruption during concurrent writes and ensure that only clean, well-structured data enters the lake. This reliability is fundamental for enterprise data pipelines, as it eliminates common issues like partial writes, corrupted files, and downstream failures caused by unexpected schema changes, which are difficult to manage with raw Parquet.

Answer

Delta Lake offers faster read performance for all queries.

Answer

Delta Lake is required for all streaming sources.

Answer

Delta Lake automatically compresses data by 90% more than Parquet.

Question 12

A data engineer is building a streaming ingestion pipeline using Databricks Auto Loader to ingest JSON files from cloud storage into a Delta Bronze table. The pipeline must handle schema evolution without failing and must minimize the number of files that require reprocessing when the schema changes. The engineer wants to configure Auto Loader appropriately. Which two configuration settings should be used to achieve these requirements? (Choose two.)

Accepted Answer

Set cloudFiles.schemaLocation to a dedicated directory in cloud storage.. To handle schema evolution without failing and minimize reprocessing, Auto Loader must be configured to add new columns dynamically and to store schema metadata persistently. Setting the schema evolution mode to addNewColumns ensures new fields are incorporated, while specifying a schema location allows Auto Loader to track schema changes and avoid reprocessing already-ingested files. Together, these settings provide robust schema evolution with efficient incremental processing.

Answer

Set cloudFiles.inferColumnTypes to true.

Answer

Set cloudFiles.useNotifications to true.

Answer

Set cloudFiles.format to 'parquet'.

Question 13

A data engineer is building a streaming ingestion pipeline from Apache Kafka to a Delta table. The pipeline must perform deduplication on a unique event_id field and handle late-arriving data. The engineer wants to use Structured Streaming with a watermark of 10 minutes. Which of the following approaches correctly implements deduplication and watermarking?

Accepted Answer

Use dropDuplicates with event_id and set the watermark on the event timestamp column.. The correct approach is to use dropDuplicates on the unique event_id and set a watermark on the event timestamp column. This combination allows the stream to deduplicate events within the watermark window, effectively handling late-arriving data while bounding state. It is the recommended pattern for streaming deduplication in Databricks Structured Streaming.

Answer

Use dropDuplicates with event_id and the event timestamp column without setting a watermark.

Answer

Use a windowed aggregation on event_id and timestamp, then filter out duplicates using row_number.

Answer

Use foreachBatch to manually deduplicate by querying the Delta table for existing event_id values.

Question 14

A data engineer is ingesting data from an Apache Kafka topic into a Delta Lake table using Structured Streaming. The Kafka topic receives messages with a timestamp field in the value payload, but the ingestion must handle late-arriving data and produce correct aggregations. The engineer wants to ensure that watermarks are applied correctly. Which approach should be used?

Accepted Answer

Parse the timestamp field from the Kafka value payload, cast it to a timestamp, and apply a watermark on that column.. For correct event-time processing with Kafka and Delta Lake, the event timestamp must come from the message payload, not from Kafka metadata or processing time. Parsing that field and applying a watermark on it enables Structured Streaming to manage late-arriving data accurately. This ensures that aggregations and stateful operations reflect the true event time and that late data is handled according to the defined watermark.

Answer

Use the Kafka timestamp column provided by the Kafka source, and apply a watermark on that column.

Answer

Use the current_timestamp() function to generate a processing-time column and apply a watermark on that column.

Answer

Set the Kafka source option 'startingOffsets' to 'earliest' and rely on the default watermark.

Question 15

A data engineer is configuring an Auto Loader stream to ingest JSON files from an S3 bucket into a Bronze Delta table. The source bucket contains both .json and .json.gz files, and the engineer wants to ensure that only .json files are processed. Which parameter should be set to achieve this?

Accepted Answer

cloudFiles.pathGlobFilter. Auto Loader provides the cloudFiles.pathGlobFilter option to filter files using glob patterns. Setting it to '*.json' ensures that only files with the .json extension are processed, while other files like .json.gz are ignored. This is the intended mechanism for file selection based on path patterns in Auto Loader, making it the correct choice for this scenario.

Answer

cloudFiles.schemaLocation

Answer

cloudFiles.includeExistingFiles

Answer

cloudFiles.format