Databricks-DE-Pro Quick Quiz — 10 Questions

Question 1

A data engineer is investigating a job failure that occurred only in the production environment. Which TWO features in Databricks help in comparing the production environment to the development environment?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the 'View as JSON' feature in the Jobs UI to compare job configurations.. Comparing environment configurations is essential for troubleshooting parity issues. Databricks provides tools like the Jobs UI for export and version control integration for code, which help engineers identify subtle differences between environments. Ensuring parity is critical for reducing 'it works on my machine' scenarios. By utilizing these tools, engineers can verify cluster settings, library versions, and code versions to isolate why a process fails in production but succeeds in development.

Answer

Use the 'Cluster Logs' to compare the OS kernel versions of the nodes.

Answer

Enable the 'Debug Mode' on the Spark Driver to see raw system calls.

Answer

Query the 'workspace_users' table to see who ran the job last.

Question 2

A data engineer is optimizing a Delta Lake table that experiences high read latency due to many small files. Which command should be executed to physically reorganize the data layout to improve query performance?

Accepted Answer

OPTIMIZE table_name. The OPTIMIZE command is the primary tool for compacting small files into larger, efficiently sized files in Delta Lake. This improves read performance by reducing metadata overhead and enabling efficient data skipping. Compaction is a critical maintenance task for tables with frequent streaming or small-batch writes, ensuring that downstream analytical queries remain performant and cost-effective by reducing the number of I/O operations required during data retrieval.

Answer

VACUUM table_name

Answer

ANALYZE TABLE table_name COMPUTE STATISTICS

Answer

REORG TABLE table_name APPLY (PURGE)

Question 3

A data engineer deploys a Databricks Job that runs a notebook task. The notebook writes to a Delta table in Unity Catalog. The job fails with the error: 'PERMISSION_DENIED: User does not have USE CATALOG on catalog 'prod'.' The engineer confirms the job's service principal has USE CATALOG granted on the catalog. Which configuration should the engineer check next?

Accepted Answer

Whether the job is configured to run as the service principal or as a different user. Unity Catalog privileges are enforced based on the identity that executes the job. When a service principal has USE CATALOG but the job runs as a different user or principal, the error still occurs. Checking the job's run-as configuration and aligning it with the granted identity resolves the mismatch and allows the notebook to access the catalog.

Answer

The notebook's default language setting

Answer

The job cluster's Spark config for spark.databricks.acl.enabled

Answer

The cluster's autoscaling min and max worker counts

Question 4

A data engineer is configuring a Delta Live Tables (DLT) pipeline that processes streaming data from Apache Kafka. The pipeline performs a series of transformations and writes to a Delta table. The engineer notices that the pipeline is experiencing high latency and wants to optimize it for cost and performance. The pipeline is set to continuous mode. Which configuration change is most effective to reduce cost while maintaining acceptable latency?

Accepted Answer

Switch the pipeline to triggered mode and schedule it to run every 5 minutes.. Switching from continuous to triggered mode with a 5-minute schedule reduces cost by allowing the cluster to shut down between runs. Continuous mode keeps the cluster running 24/7, incurring constant DBU charges. Triggered mode with a schedule balances latency and cost, making it the most effective change for reducing cost while maintaining acceptable latency.

Answer

Increase the number of Kafka partitions to improve parallelism.

Answer

Configure the pipeline to use Photon-optimized clusters.

Answer

Enable autoscaling on the DLT cluster and set a minimum number of workers.

Question 5

A data engineer needs to read a CSV file from cloud storage into a Spark DataFrame in Databricks. The file has a header row and uses commas as delimiters. The engineer wants to infer the schema automatically. Which code snippet correctly reads the file?

Accepted Answer

spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("path/to/file.csv"). The correct method chain uses spark.read.format("csv") to specify the format, then .option("header", "true") and .option("inferSchema", "true") to set the options, and finally .load(path). This is the standard and correct way to read a CSV file with schema inference in Spark.

Answer

spark.read.option("format", "csv").option("header", "true").option("inferSchema", "true").load("path/to/file.csv")

Answer

spark.read.csv("path/to/file.csv").option("header", "true").option("inferSchema", "true")

Answer

spark.read.format("csv").load("path/to/file.csv", header=True, inferSchema=True)

Question 6

Your team is using a shared cluster for development. A user reports that their job is slow because the cluster memory is frequently filled by large data broadcasts. What configuration adjustment should you make to prevent this issue across all jobs on the cluster?

Accepted Answer

Decrease the 'spark.sql.autoBroadcastJoinThreshold'.. Controlling the broadcast threshold prevents Spark from automatically attempting to broadcast large tables that exceed the available memory, which is the most common cause of OOMs on shared clusters. Setting this value correctly ensures that only appropriately small tables are broadcast, forcing the engine to use a join shuffle instead. This maintains stability for all users on the shared cluster, preventing one user's job from impacting others.

Answer

Increase 'spark.driver.memory'.

Answer

Increase the number of shuffle partitions.

Answer

Enable 'spark.sql.adaptive.enabled'.

Question 7

Which TWO of the following are primary benefits of using Delta Live Tables (DLT) for data ingestion over standard Structured Streaming pipelines?

Accepted Answer

Declarative pipeline management and automated dependency handling.. DLT simplifies the operational complexity of data pipelines by providing automatic infrastructure management and built-in quality controls. It abstracts the configuration required for managing checkpoints, scaling, and handling schema drift, allowing engineers to focus on defining transformations. The 'Expectations' framework and declarative pipeline management provide superior observability and data quality enforcement compared to manual, imperative coding in standard Spark Structured Streaming.

Answer

DLT supports significantly higher throughput than Structured Streaming.

Answer

DLT is the only way to read from cloud object storage.

Answer

DLT supports non-Delta storage formats for all outputs.

Question 8

When running a PySpark job, you receive an 'Out of Memory (OOM)' error during a shuffle operation. Which configuration is the most appropriate to address this first?

Accepted Answer

Increase spark.sql.shuffle.partitions. OOM errors during shuffles are frequently caused by partitions that are too large for the executor's memory. Increasing 'spark.sql.shuffle.partitions' is the primary corrective action, as it splits the data into a larger number of smaller partitions. This reduces the memory footprint of each individual task, allowing them to fit within the executor's heap memory and preventing the job from crashing due to memory exhaustion during data movement.

Answer

Increase spark.driver.memory

Answer

Decrease spark.executor.memory

Answer

Set spark.sql.shuffle.partitions to 1

Question 9

An engineer notices that a SQL warehouse is frequently hitting 'Max Concurrency' limits. Which log should they consult to identify which specific queries are consuming most of the warehouse resources?

Accepted Answer

Query history system table. The 'query_history' system table is the primary resource for analyzing query performance and resource consumption. By querying this table, engineers can attribute resource usage to specific users, notebooks, or warehouses. This is essential for troubleshooting concurrency bottlenecks, enabling teams to optimize expensive queries or adjust warehouse sizing to meet demand without impacting overall system performance or user productivity.

Answer

Cluster event logs

Answer

Job run history logs

Answer

Delta table access logs

Question 10

A data engineering team is modeling a large Delta Lake fact table that stores clickstream events. Analysts frequently run queries that filter by event_date and then aggregate by user_id, and the table receives continuous appends plus occasional late-arriving corrections. The team wants to reduce bytes scanned and improve join performance. Which two design choices are most appropriate? (Choose two.)

Accepted Answer

Partition the table by event_date and apply Z-ORDER on user_id within each partition. The workload filters on event_date and aggregates by user_id, so layout must accelerate both. Partitioning by event_date with Z-ORDER on user_id gives reliable partition pruning plus co-located user data, while liquid clustering on both columns achieves similar skipping without rigid high-cardinality directories and can be adapted incrementally as access patterns change.

Answer

Convert the table to a Parquet external table and rely on the query engine to infer partitioning

Answer

Enable Change Data Feed on the table so late-arriving corrections are captured automatically

Answer

Partition the table by user_id to maximize file skipping for user-level aggregations