Databricks-DE-Pro Practice Test 18 — 15 Questions

Question 1

Which capability is provided by Databricks' integration with cloud-native monitoring tools (e.g., CloudWatch, Azure Monitor)?

Accepted Answer

Centralized metrics collection and dashboarding.. Integrating Databricks with cloud-native monitoring provides a unified view of platform health. These tools aggregate logs and metrics from across the entire cloud environment, allowing for centralized dashboarding and advanced alerting. This integration is vital for enterprises maintaining a 'single pane of glass' strategy, as it ensures that Databricks infrastructure metrics are treated with the same governance and visibility standards as other cloud resources.

Answer

Automated data quality remediation.

Answer

Direct execution of SQL queries on the warehouse.

Answer

Fine-grained control over Spark session configurations.

Question 2

A financial services firm maintains a Delta Lake table of account transactions that must support both current-state queries and full audit history of every change, including corrections that arrive days later. Regulators require the ability to query the table as it existed at any prior date. Which Delta Lake capability should the engineer rely on to satisfy the audit requirement?

Accepted Answer

Time travel using the transaction log to query historical versions of the table by version number or timestamp. Delta Lake's transaction log records each commit atomically, enabling time travel to any retained version or timestamp. Because late corrections create new versions while prior versions remain queryable, auditors can reconstruct the exact table state at a prior date, which is precisely what the regulatory requirement demands.

Answer

Relying on the underlying cloud object store's versioning to recover previous Parquet files

Answer

Maintaining a separate archive table populated by a nightly full copy of the transactions table

Answer

Enabling Change Data Feed to capture row-level changes and querying the feed instead of the table

Question 3

You have a large Spark DataFrame that you need to filter and save as multiple smaller Parquet files based on the values in a 'region' column. Which method should you use to optimize the file layout for subsequent queries?

Accepted Answer

Use the 'partitionBy' option in the write command.. Partitioning by a low-cardinality column like 'region' physically organizes data into folders. This allows engines to perform 'partition pruning', skipping irrelevant folders during query execution. This is a standard optimization in Databricks for any dataset with clear categorical groupings, as it directly reduces the I/O cost of queries that filter by the partition key, improving performance significantly for analysts and dashboard tools.

Answer

Use the 'repartition' method on the DataFrame before writing.

Answer

Use the 'sortWithinPartitions' method.

Answer

Use the 'coalesce' method.

Question 4

A Data Engineer needs to enforce a NOT NULL constraint on a specific column in a Delta table while maintaining the ability to perform high-performance streaming writes. Which approach is the most efficient and native method to ensure this data quality requirement?

Accepted Answer

Add a CHECK constraint to the table using the ALTER TABLE ADD CONSTRAINT command.. Using Delta Lake's CHECK constraints is the most efficient way to enforce data quality at write time. By defining a constraint, the Delta engine validates every incoming row against the business logic before committing the transaction. This avoids post-process cleanup jobs, reduces storage waste, and ensures that downstream consumers always receive valid data, which is critical for maintaining robust data pipelines in a Lakehouse architecture.

Answer

Apply a filter transformation in the DataFrame after the data is written to the table.

Answer

Use an external Delta Live Tables expectation to quarantine bad records.

Answer

Perform a manual check within the Spark Structured Streaming loop before appending to the sink.

Question 5

Refer to the exhibit. Why is Task D marked as 'Skipped'?

Accepted Answer

Task D has a dependency on Task B, which failed.. In Databricks Workflows, a task is 'Skipped' if its upstream dependencies have not been met, typically because a required parent task failed. Because Task B failed, the downstream Task D—which likely depends on Task B—cannot execute. This mechanism prevents the workflow from processing invalid or incomplete data, maintaining the integrity of the data pipeline and preventing downstream failures from compounding existing upstream errors.

Answer

Task D reached its execution timeout limit before it could start.

Answer

The cluster running Task D was terminated by an administrator.

Answer

Task D was manually disabled in the job configuration.

Question 6

A Data Engineer needs to perform an 'upsert' operation on a Delta table. Which command provides the most efficient way to merge new data with existing records in a single transactional step?

Accepted Answer

MERGE INTO target USING source ON condition WHEN MATCHED .... The MERGE INTO command is the standard SQL construct for performing upserts in Delta Lake. It handles both updates to existing records and the insertion of new records based on a join condition. This single, atomic statement simplifies complex logic that would otherwise require multiple steps, ensuring that the target table remains in a consistent state throughout the entire upsert operation in high-concurrency production environments.

Answer

INSERT INTO ... ON DUPLICATE KEY UPDATE

Answer

UPDATE target SET ...; INSERT INTO target ...

Answer

OVERWRITE TABLE target SELECT * FROM source

Question 7

A Data Engineer is using Delta Live Tables to process a stream of financial transactions. The pipeline must ensure that each `transaction_id` appears only once in the target table, even if the source stream contains duplicates due to at-least-once ingestion. The engineer wants to use the `APPLY CHANGES` API. Which combination of settings will achieve this with minimal data loss?

Accepted Answer

Use `APPLY CHANGES INTO target` with `KEYS (transaction_id)` and `SEQUENCE BY timestamp`, and set `STORED AS SCD TYPE 1`.. To deduplicate and keep only the latest record per `transaction_id`, `APPLY CHANGES` with SCD Type 1 and a sequence column is correct. SCD Type 1 performs upserts, overwriting existing rows, so each key appears once. SCD Type 2 would keep history, resulting in multiple rows. The sequence column ensures the latest record is applied deterministically.

Answer

Use `APPLY CHANGES INTO target` with `KEYS (transaction_id)` and omit the `SEQUENCE BY` clause, and set `STORED AS SCD TYPE 1`.

Answer

Use `APPLY CHANGES INTO target` with `KEYS (transaction_id)` and `SEQUENCE BY timestamp`, and set `STORED AS SCD TYPE 2`.

Answer

Use `APPLY CHANGES INTO target` with `KEYS (transaction_id)` and `SEQUENCE BY timestamp`, and set `STORED AS SCD TYPE 2` with `IGNORE NULLS`.

Question 8

Which TWO of the following are valid ways to pass configuration parameters to a Databricks Job task? (Select TWO)

Accepted Answer

Using dbutils.widgets.text to read parameters injected via the Job UI.. Databricks Jobs offer multiple ways to parameterize tasks. Using task-level parameters allows for dynamic behavior without changing code. CLI options and the Job API are standard methods for injecting these values. Understanding these mechanisms is essential for creating reusable, modular pipelines that can adapt to different environments or data sources without hardcoding variables directly into the notebook or Python scripts.

Answer

Hardcoding the variables in a global configuration class.

Answer

Using a temporary local file to store configuration values.

Answer

Reading directly from the user's home directory path.

Question 9

A data engineer is optimizing a PySpark job that processes a large DataFrame and writes the result to a Delta table. The job currently uses repartition(100) before writing, but the output consists of many small files. The engineer wants to reduce the number of output files without shuffling the entire dataset again. Which approach should be used?

Accepted Answer

Use coalesce(10) before writing to reduce the number of partitions without a full shuffle.. coalesce is the correct choice because it reduces the number of partitions without a full shuffle, making it efficient for reducing output file count. repartition would cause an unnecessary shuffle, and changing shuffle partitions or using partitionBy does not achieve the goal of reducing files without a shuffle.

Answer

Use partitionBy("column") to write the data into subdirectories, which reduces the number of files per directory.

Answer

Set spark.sql.shuffle.partitions to 10 before writing.

Answer

Use repartition(10) before writing to redistribute data evenly and reduce file count.

Question 10

A data team is using Liquid Clustering on a Delta table. How does this feature improve performance compared to traditional Z-Ordering or Partitioning?

Accepted Answer

It automatically adjusts the data layout based on changing query patterns without manual maintenance.. Liquid Clustering provides dynamic data layout optimization that automatically adapts to query patterns without requiring manual maintenance or complex partition strategies. Unlike static partitioning, which can lead to data skew or file management issues, Liquid Clustering reorganizes data based on actual usage, significantly reducing the burden on engineers. It is a more flexible, modern solution that simplifies performance tuning while ensuring that read queries remain fast across varying data access patterns.

Answer

It eliminates the need for any clustering keys by automatically indexing every column.

Answer

It forces all data to be stored on a single partition to maximize local cache hits.

Answer

It replaces Delta Lake's file management, making the OPTIMIZE command unnecessary.

Question 11

A data engineer is designing an ETL pipeline processing high-frequency streaming data into Delta tables on Databricks. The pipeline experiences frequent small file creation and high metadata overhead, degrading query performance. Which optimization technique should the engineer implement to resolve this issue?

Accepted Answer

Enable spark.databricks.delta.optimizeWrite.enabled and spark.databricks.delta.autoCompact.enabled.. Implementing automated compaction via optimized write and Auto Compact merges small files during writes, maintaining optimal file sizes around 128MB. This technique is critical for streaming workloads because frequent micro-batches naturally produce excessive tiny files that severely degrade both metadata listing times and subsequent read scan performance across production environments.

Answer

Increase the Delta table version retention period to keep historic snapshots longer.

Answer

Enable predictive optimization to automatically manage compaction and vacuum operations.

Answer

Switch the table format from Delta to Apache Parquet to leverage native cloud storage indexing.

Question 12

A data engineer wants to ensure that all data in a specific catalog is encrypted at rest. Which feature should they verify is enabled within the Unity Catalog metastore configuration?

Accepted Answer

Customer-managed keys (CMK) for managed storage.. Customer-managed keys (CMK) for managed storage are the standard governance tool for ensuring that data at rest is encrypted according to organizational security policies. By configuring CMK, the organization maintains control over the encryption lifecycle, satisfying compliance requirements. This feature is critical for highly regulated industries where the entity, rather than the cloud provider, must maintain the ultimate authority over data access and encryption standards in the cloud.

Answer

Workspace-level access control lists.

Answer

Unity Catalog lineage tracking.

Answer

Table access control (TAC).

Question 13

A data engineer has a Unity Catalog managed table `sales.raw.transactions` that contains a column `customer_email` with PII. Analysts in the `marketing_analysts` group need to query the table for aggregate reporting but must never see individual email addresses. The engineer wants to enforce this dynamically without creating a separate view or copy of the data. Which Unity Catalog feature should the engineer use?

Accepted Answer

Apply a column mask to `customer_email` using a user-defined function that returns NULL for members of `marketing_analysts`.. A column mask in Unity Catalog applies a user-defined function at query time to transform or redact column values based on the invoking user's identity or group memberships. This allows the same table to return different values for different users without duplicating data or creating separate views. It is the native mechanism for dynamic column-level security and satisfies the requirement to hide email addresses from marketing analysts while still permitting aggregate queries.

Answer

Create a row filter on `sales.raw.transactions` that excludes rows where `customer_email` is not null.

Answer

Enable attribute-based access control (ABAC) at the catalog level to automatically mask all PII columns for all non-admin users.

Answer

Revoke SELECT on the table from `marketing_analysts` and grant SELECT only on a view that omits `customer_email`.

Question 14

Your organization runs numerous batch data engineering pipelines using standard Databricks jobs. Finance reports indicate that compute costs are inflated due to cluster startup times and rigid over-provisioning. Which optimization approach provides the best balance of cost savings and execution reliability for scheduled production batch jobs?

Accepted Answer

Migrate scheduled pipeline execution to Databricks Jobs compute leveraging job clusters configured with spot instances for workers.. Transitioning scheduled production batch pipelines from interactive all-purpose clusters to Databricks Jobs compute with spot instance integration delivers massive financial savings. Jobs compute provides lower compute unit pricing compared to interactive workspaces, while spot instances discount infrastructure further, and robust retry mechanisms handle any cloud-level pre-emptions gracefully.

Answer

Keep using interactive all-purpose clusters but implement notebook-level python scripts to manually stop clusters when jobs finish.

Answer

Refactor all batch jobs to execute exclusively on Serverless SQL Warehouses regardless of workload dependencies.

Answer

Reduce the executor memory allocation below default recommendations to force Spark to spill data to disk more frequently.

Question 15

Which TWO of the following are benefits of using the Databricks Delta Lake 'Optimize' command? (Select TWO)

Accepted Answer

It compacts small files into larger, more efficient Parquet files.. The OPTIMIZE command compacts small files into larger files, which significantly improves read performance by reducing the metadata overhead of listing thousands of small files. It also allows for Z-Ordering, which collocated related information in the same files, enabling data skipping during query execution. These optimizations are crucial for maintaining high-performance data lakes as datasets grow over time in production environments.

Answer

It automatically shrinks the data volume by dropping null values.

Answer

It converts Parquet files to CSV format for better compatibility.

Answer

It automatically updates the table statistics for the CBO.