Databricks-DE-Pro Developing Code (Python/SQL) Practice Test 3 — 15 Questions

Question 1

Your team is using a shared cluster for development. A user reports that their job is slow because the cluster memory is frequently filled by large data broadcasts. What configuration adjustment should you make to prevent this issue across all jobs on the cluster?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Decrease the 'spark.sql.autoBroadcastJoinThreshold'.. Controlling the broadcast threshold prevents Spark from automatically attempting to broadcast large tables that exceed the available memory, which is the most common cause of OOMs on shared clusters. Setting this value correctly ensures that only appropriately small tables are broadcast, forcing the engine to use a join shuffle instead. This maintains stability for all users on the shared cluster, preventing one user's job from impacting others.

Answer

Increase 'spark.driver.memory'.

Answer

Increase the number of shuffle partitions.

Answer

Enable 'spark.sql.adaptive.enabled'.

Question 2

Refer to the exhibit. You are appending data to an existing Delta table. What is the most likely cause of this error, and how should you resolve it?

Accepted Answer

The incoming data's 'price' column has a higher precision than the table schema.. This error occurs because of a data type mismatch between the incoming DataFrame and the existing table schema, specifically a change in precision or scale. Databricks enforces schema safety to prevent data corruption. Resolving this requires either casting the incoming data to match the target schema or using the 'mergeSchema' option if the goal is to allow evolution, provided the change is safe and intended for the application.

Answer

The table is locked by another process.

Answer

The table schema is corrupted and needs to be repaired.

Answer

The user does not have write access to the table.

Question 3

Which THREE of the following are benefits of using Delta Lake over standard Parquet files for your data lake storage?

Accepted Answer

ACID transactions for concurrent reads and writes.. Delta Lake adds a transactional layer (the log) on top of Parquet files, enabling ACID transactions, time travel, and schema enforcement. These features solve the most common challenges with raw data lakes, such as partial writes and data corruption. As a Databricks professional, you must prioritize Delta Lake to build reliable, scalable architectures that support robust data governance and high-performance analytical queries without the risks of file-based data inconsistency.

Answer

Faster cold-start reads by caching data on the driver.

Answer

Allows the use of SQL as the only way to interact with data.

Question 4

Which command should be used to display the history of transactions performed on a Delta table, including operations like overwrites and updates?

Accepted Answer

DESCRIBE HISTORY table_name. The 'DESCRIBE HISTORY' command is the standard way to inspect the transaction log of a Delta table. This is vital for auditing, time-travel debugging, and understanding the sequence of operations that led to the current state of the table. Understanding this command is essential for troubleshooting data quality issues and verifying that pipeline operations performed as expected during backfills or incremental loads.

Answer

SHOW LOGS table_name

Answer

SELECT * FROM metadata.table_name

Answer

EXPLAIN EXTENDED table_name

Question 5

Which THREE of the following are benefits of using Delta Lake over standard Parquet files in Databricks?

Accepted Answer

ACID transactions. Delta Lake provides a robust layer of reliability and performance features on top of Parquet. ACID transactions ensure data consistency during concurrent writes. Schema enforcement prevents data corruption by rejecting non-conforming writes, while time travel allows for auditing and disaster recovery. These capabilities are fundamental to modern Lakehouse architectures, as they bridge the gap between reliable data warehousing and the flexibility of cloud object storage.

Answer

Automatic file compaction to remove Parquet footer bloat

Answer

Automatic conversion from CSV to Parquet on read

Question 6

You are writing a Spark application that uses a Broadcast Hash Join. You want to force the join to use broadcast optimization for a specific table. How do you implement this in PySpark?

Accepted Answer

df1.join(broadcast(df2), 'key'). Using the 'broadcast()' function in PySpark explicitly hints to the Spark optimizer that a specific DataFrame should be broadcasted to all executors. This is highly effective when joining a large fact table with a small dimension table, as it avoids the expensive shuffle operation. Understanding this is crucial for performance tuning, as relying solely on the automatic broadcast threshold may lead to suboptimal join choices in complex queries.

Answer

df1.join(df2, 'key', 'broadcast')

Answer

spark.conf.set('spark.sql.broadcastJoin', 'true')

Answer

df1.join(hint('broadcast'), df2, 'key')

Question 7

Which of the following describes the behavior of the 'OPTIMIZE' command in Databricks?

Accepted Answer

It rewrites data files into a more efficient layout for reads.. The 'OPTIMIZE' command performs file compaction, which consolidates small files into larger, optimally sized files. This improves scan performance significantly by reducing the metadata overhead of opening and reading many tiny files. It also supports Z-Ordering, which co-locates related data in the same files, further enhancing the efficiency of data skipping and improving query speed for common filter predicates.

Answer

It recomputes statistics to improve query planning.

Answer

It clears the transaction log to reduce storage costs.

Answer

It converts non-Delta tables to Delta format.

Question 8

When partitioning a large Delta table, what is the best practice regarding the number of unique values in the partition column?

Accepted Answer

Low cardinality is preferred to avoid creating too many files.. Choosing a column with too many unique values (high cardinality) for partitioning leads to the 'small files problem.' Each unique value creates a new directory, resulting in an explosion of small files that degrades performance during reads and writes. Best practices suggest partitioning by columns with low to medium cardinality (e.g., date, region) to ensure balanced file sizes and efficient data skipping during query execution.

Answer

High cardinality is preferred for faster data skipping.

Answer

Partitioning should always be done on a primary key column.

Answer

The number of unique values does not impact performance.

Question 9

When running a PySpark job, you receive an 'Out of Memory (OOM)' error during a shuffle operation. Which configuration is the most appropriate to address this first?

Accepted Answer

Increase spark.sql.shuffle.partitions. OOM errors during shuffles are frequently caused by partitions that are too large for the executor's memory. Increasing 'spark.sql.shuffle.partitions' is the primary corrective action, as it splits the data into a larger number of smaller partitions. This reduces the memory footprint of each individual task, allowing them to fit within the executor's heap memory and preventing the job from crashing due to memory exhaustion during data movement.

Answer

Increase spark.driver.memory

Answer

Decrease spark.executor.memory

Answer

Set spark.sql.shuffle.partitions to 1

Question 10

Which TWO of the following are valid ways to trigger a job in Databricks?

Accepted Answer

Using the 'Jobs' UI to trigger a run.. Databricks provides multiple interfaces for job orchestration. The 'Jobs' UI in the workspace allows for interactive configuration and scheduling. Alternatively, the Databricks REST API provides a programmatic way to trigger jobs, which is crucial for CI/CD pipelines and integrating Databricks with external orchestrators like Airflow or Azure Data Factory. Using these methods ensures that pipelines can be automated, monitored, and integrated into broader enterprise data workflows.

Answer

Executing the 'run_job' command inside a notebook cell.

Answer

Running the 'dbutils.jobs.run()' method.

Answer

Creating a file named 'trigger.job' in DBFS.

Question 11

A Data Engineer is developing a Delta Live Tables (DLT) pipeline using Python. They need to ensure that records failing a specific data quality check are dropped, but the pipeline continues to process the remaining valid records. Which expectation syntax should the engineer implement?

Accepted Answer

@dlt.expect_or_drop('col_check', 'id IS NOT NULL'). The 'expect_or_drop' constraint is specifically designed for DLT to handle data quality failures gracefully. By using this expectation, the pipeline marks the record as invalid and removes it from the target dataset without halting the entire execution. This is critical in production environments where partial data ingestion is preferred over pipeline failure, ensuring high availability and robust data processing workflows for downstream users.

Answer

@dlt.expect_or_fail('col_check', 'id IS NOT NULL')

Answer

@dlt.expect('col_check', 'id IS NOT NULL')

Answer

@dlt.validate('col_check', 'id IS NOT NULL')

Question 12

Which TWO of the following are valid ways to pass configuration parameters to a Databricks Job task? (Select TWO)

Accepted Answer

Using dbutils.widgets.text to read parameters injected via the Job UI.. Databricks Jobs offer multiple ways to parameterize tasks. Using task-level parameters allows for dynamic behavior without changing code. CLI options and the Job API are standard methods for injecting these values. Understanding these mechanisms is essential for creating reusable, modular pipelines that can adapt to different environments or data sources without hardcoding variables directly into the notebook or Python scripts.

Answer

Hardcoding the variables in a global configuration class.

Answer

Using a temporary local file to store configuration values.

Answer

Reading directly from the user's home directory path.

Question 13

When using the Unity Catalog, how should an engineer properly reference a table named 'sales' located in the 'finance' schema within the 'prod_catalog' catalog using Spark SQL?

Accepted Answer

SELECT * FROM prod_catalog.finance.sales. Unity Catalog uses a three-level namespace hierarchy: catalog.schema.table. This structure provides consistent governance and discovery across the entire Databricks account. Proper referencing requires the fully qualified name to ensure Spark identifies the exact object within the catalog's permission model. This implementation is crucial for security and multi-tenant environments where namespaces must be strictly isolated to prevent unauthorized access and data leakage across production and development environments.

Answer

SELECT * FROM finance.sales

Answer

SELECT * FROM sales

Answer

SELECT * FROM catalog_prod.finance.sales

Question 14

Which THREE features are provided by Delta Lake when compared to standard Parquet files? (Select THREE)

Accepted Answer

ACID transactions. Delta Lake adds a transaction log (the _delta_log) to standard Parquet files, enabling ACID transactions, time travel, and schema enforcement. These features solve common data engineering pain points such as partial writes, data corruption, and the inability to revert to previous versions of data. Mastering these capabilities is essential for building reliable data lakes that provide the same consistency and reliability as traditional data warehouses.

Answer

Automatic indexing for all columns

Answer

Built-in support for real-time web socket streaming

Question 15

A Data Engineer needs to perform an 'upsert' operation on a Delta table. Which command provides the most efficient way to merge new data with existing records in a single transactional step?

Accepted Answer

MERGE INTO target USING source ON condition WHEN MATCHED .... The MERGE INTO command is the standard SQL construct for performing upserts in Delta Lake. It handles both updates to existing records and the insertion of new records based on a join condition. This single, atomic statement simplifies complex logic that would otherwise require multiple steps, ensuring that the target table remains in a consistent state throughout the entire upsert operation in high-concurrency production environments.

Answer

INSERT INTO ... ON DUPLICATE KEY UPDATE

Answer

UPDATE target SET ...; INSERT INTO target ...

Answer

OVERWRITE TABLE target SELECT * FROM source