Databricks-DE-Pro Practice Questions

Question 1

A data engineer is investigating a job failure that occurred only in the production environment. Which TWO features in Databricks help in comparing the production environment to the development environment?

Accepted Answer

Use the 'View as JSON' feature in the Jobs UI to compare job configurations.. Comparing environment configurations is essential for troubleshooting parity issues. Databricks provides tools like the Jobs UI for export and version control integration for code, which help engineers identify subtle differences between environments. Ensuring parity is critical for reducing 'it works on my machine' scenarios. By utilizing these tools, engineers can verify cluster settings, library versions, and code versions to isolate why a process fails in production but succeeds in development.

Answer

Use the 'Cluster Logs' to compare the OS kernel versions of the nodes.

Answer

Enable the 'Debug Mode' on the Spark Driver to see raw system calls.

Answer

Query the 'workspace_users' table to see who ran the job last.

Question 2

A data engineer is optimizing a Delta Lake table that experiences high read latency due to many small files. Which command should be executed to physically reorganize the data layout to improve query performance?

Accepted Answer

OPTIMIZE table_name. The OPTIMIZE command is the primary tool for compacting small files into larger, efficiently sized files in Delta Lake. This improves read performance by reducing metadata overhead and enabling efficient data skipping. Compaction is a critical maintenance task for tables with frequent streaming or small-batch writes, ensuring that downstream analytical queries remain performant and cost-effective by reducing the number of I/O operations required during data retrieval.

Answer

VACUUM table_name

Answer

ANALYZE TABLE table_name COMPUTE STATISTICS

Answer

REORG TABLE table_name APPLY (PURGE)

Question 3

A data engineer deploys a Databricks Job that runs a notebook task. The notebook writes to a Delta table in Unity Catalog. The job fails with the error: 'PERMISSION_DENIED: User does not have USE CATALOG on catalog 'prod'.' The engineer confirms the job's service principal has USE CATALOG granted on the catalog. Which configuration should the engineer check next?

Accepted Answer

Whether the job is configured to run as the service principal or as a different user. Unity Catalog privileges are enforced based on the identity that executes the job. When a service principal has USE CATALOG but the job runs as a different user or principal, the error still occurs. Checking the job's run-as configuration and aligning it with the granted identity resolves the mismatch and allows the notebook to access the catalog.

Answer

The notebook's default language setting

Answer

The job cluster's Spark config for spark.databricks.acl.enabled

Answer

The cluster's autoscaling min and max worker counts

Question 4

A data engineer is configuring a Delta Live Tables (DLT) pipeline that processes streaming data from Apache Kafka. The pipeline performs a series of transformations and writes to a Delta table. The engineer notices that the pipeline is experiencing high latency and wants to optimize it for cost and performance. The pipeline is set to continuous mode. Which configuration change is most effective to reduce cost while maintaining acceptable latency?

Accepted Answer

Switch the pipeline to triggered mode and schedule it to run every 5 minutes.. Switching from continuous to triggered mode with a 5-minute schedule reduces cost by allowing the cluster to shut down between runs. Continuous mode keeps the cluster running 24/7, incurring constant DBU charges. Triggered mode with a schedule balances latency and cost, making it the most effective change for reducing cost while maintaining acceptable latency.

Answer

Increase the number of Kafka partitions to improve parallelism.

Answer

Configure the pipeline to use Photon-optimized clusters.

Answer

Enable autoscaling on the DLT cluster and set a minimum number of workers.

Question 5

A data engineer needs to read a CSV file from cloud storage into a Spark DataFrame in Databricks. The file has a header row and uses commas as delimiters. The engineer wants to infer the schema automatically. Which code snippet correctly reads the file?

Accepted Answer

spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("path/to/file.csv"). The correct method chain uses spark.read.format("csv") to specify the format, then .option("header", "true") and .option("inferSchema", "true") to set the options, and finally .load(path). This is the standard and correct way to read a CSV file with schema inference in Spark.

Answer

spark.read.option("format", "csv").option("header", "true").option("inferSchema", "true").load("path/to/file.csv")

Answer

spark.read.csv("path/to/file.csv").option("header", "true").option("inferSchema", "true")

Answer

spark.read.format("csv").load("path/to/file.csv", header=True, inferSchema=True)

Question 6

Your team is using a shared cluster for development. A user reports that their job is slow because the cluster memory is frequently filled by large data broadcasts. What configuration adjustment should you make to prevent this issue across all jobs on the cluster?

Accepted Answer

Decrease the 'spark.sql.autoBroadcastJoinThreshold'.. Controlling the broadcast threshold prevents Spark from automatically attempting to broadcast large tables that exceed the available memory, which is the most common cause of OOMs on shared clusters. Setting this value correctly ensures that only appropriately small tables are broadcast, forcing the engine to use a join shuffle instead. This maintains stability for all users on the shared cluster, preventing one user's job from impacting others.

Answer

Increase 'spark.driver.memory'.

Answer

Increase the number of shuffle partitions.

Answer

Enable 'spark.sql.adaptive.enabled'.

Question 7

Which TWO of the following are primary benefits of using Delta Live Tables (DLT) for data ingestion over standard Structured Streaming pipelines?

Accepted Answer

Declarative pipeline management and automated dependency handling.. DLT simplifies the operational complexity of data pipelines by providing automatic infrastructure management and built-in quality controls. It abstracts the configuration required for managing checkpoints, scaling, and handling schema drift, allowing engineers to focus on defining transformations. The 'Expectations' framework and declarative pipeline management provide superior observability and data quality enforcement compared to manual, imperative coding in standard Spark Structured Streaming.

Answer

DLT supports significantly higher throughput than Structured Streaming.

Answer

DLT is the only way to read from cloud object storage.

Answer

DLT supports non-Delta storage formats for all outputs.

Question 8

When running a PySpark job, you receive an 'Out of Memory (OOM)' error during a shuffle operation. Which configuration is the most appropriate to address this first?

Accepted Answer

Increase spark.sql.shuffle.partitions. OOM errors during shuffles are frequently caused by partitions that are too large for the executor's memory. Increasing 'spark.sql.shuffle.partitions' is the primary corrective action, as it splits the data into a larger number of smaller partitions. This reduces the memory footprint of each individual task, allowing them to fit within the executor's heap memory and preventing the job from crashing due to memory exhaustion during data movement.

Answer

Increase spark.driver.memory

Answer

Decrease spark.executor.memory

Answer

Set spark.sql.shuffle.partitions to 1

Question 9

An engineer notices that a SQL warehouse is frequently hitting 'Max Concurrency' limits. Which log should they consult to identify which specific queries are consuming most of the warehouse resources?

Accepted Answer

Query history system table. The 'query_history' system table is the primary resource for analyzing query performance and resource consumption. By querying this table, engineers can attribute resource usage to specific users, notebooks, or warehouses. This is essential for troubleshooting concurrency bottlenecks, enabling teams to optimize expensive queries or adjust warehouse sizing to meet demand without impacting overall system performance or user productivity.

Answer

Cluster event logs

Answer

Job run history logs

Answer

Delta table access logs

Question 10

A data engineering team is modeling a large Delta Lake fact table that stores clickstream events. Analysts frequently run queries that filter by event_date and then aggregate by user_id, and the table receives continuous appends plus occasional late-arriving corrections. The team wants to reduce bytes scanned and improve join performance. Which two design choices are most appropriate? (Choose two.)

Accepted Answer

Partition the table by event_date and apply Z-ORDER on user_id within each partition. The workload filters on event_date and aggregates by user_id, so layout must accelerate both. Partitioning by event_date with Z-ORDER on user_id gives reliable partition pruning plus co-located user data, while liquid clustering on both columns achieves similar skipping without rigid high-cardinality directories and can be adapted incrementally as access patterns change.

Answer

Convert the table to a Parquet external table and rely on the query engine to infer partitioning

Answer

Enable Change Data Feed on the table so late-arriving corrections are captured automatically

Answer

Partition the table by user_id to maximize file skipping for user-level aggregations

Question 11

A data engineer is implementing fine-grained access control on a Delta table in Unity Catalog that contains sensitive customer data. The requirement is to mask the `credit_card` column for all users except members of the `finance` group, and to filter out rows where the `region` column is not in the user's allowed regions. Which two Unity Catalog features should the engineer use? (Choose two.)

Accepted Answer

Row filter using a SQL UDF that checks the user's allowed regions against the `region` column.. Unity Catalog provides native row filters and column masks for fine-grained access control. A column mask with a SQL UDF can conditionally reveal or obscure the credit card column based on group membership, while a row filter with a SQL UDF can restrict rows by region. Together they implement the required masking and filtering directly on the table without requiring a separate view.

Answer

Table ACLs that grant SELECT only on specific columns.

Answer

Workspace-level IP access list to restrict access to the table.

Answer

Dynamic view that joins the table with a mapping table of user regions.

Question 12

A Data Engineer needs to encrypt data at rest within a Databricks workspace that uses a customer-managed key (CMK). What is the primary purpose of this configuration?

Accepted Answer

To control the lifecycle of the encryption keys used for storage.. Using a customer-managed key (CMK) provides an additional layer of security and control over data at rest in cloud storage, such as S3 or ADLS. By managing the key in the cloud provider's Key Management Service (KMS), the organization can revoke access to the data at any time by disabling the key. This is a common requirement for high-compliance industries that need verifiable control over their encrypted data.

Answer

To increase the read throughput of Delta tables.

Answer

To enable transient data encryption for cluster nodes.

Answer

To bypass the need for Unity Catalog permissions.

Question 13

You are performing a complex data transformation involving a self-join on a large, skewed table. Which technique is most effective for preventing data skew and improving join performance?

Accepted Answer

Apply a 'salt' to the join key to redistribute the skewed data across more partitions.. Skewed joins are a primary cause of performance failure in distributed computing. Using a 'salted' key—adding a random prefix to the join key—distributes data evenly across partitions during the join. This prevents 'hot' partitions where one executor processes the majority of the data, significantly speeding up the query and preventing OOM (Out of Memory) errors, ensuring consistent performance for large-scale analytical tasks.

Answer

Increase the number of executors in the cluster to handle the load.

Answer

Use the 'repartition' hint on the join keys to force uniform distribution.

Answer

Convert the join into a broadcast join to prevent shuffling entirely.

Question 14

You are designing an ingestion pipeline that must handle massive bursts of data at irregular intervals. Which feature should you prioritize to ensure the ingestion process remains cost-effective?

Accepted Answer

Use auto-scaling clusters with a minimum of zero workers.. Using Photon-accelerated clusters with auto-scaling is the most effective way to handle bursty workloads. By configuring the cluster to scale out when the queue of files is large and scale down during idle periods, you maximize resource utilization. This approach ensures you have enough compute to meet latency SLAs during bursts while minimizing costs by scaling to zero or minimal size when no data is being ingested, optimizing for both performance and budget.

Answer

Use a fixed-size cluster with a large number of nodes.

Answer

Run the pipeline continuously on a single-node cluster.

Answer

Increase the 'spark.sql.shuffle.partitions' to 10000.

Question 15

An organization wants to restrict data access to only allow connections from specific corporate IP ranges. Which Databricks feature should be configured to implement this network security requirement?

Accepted Answer

IP Access Lists.. Network-level security is a cornerstone of enterprise data governance. By using IP access lists, Databricks allows administrators to define a whitelist of allowed CIDR blocks. This ensures that even if a user has valid credentials, they cannot access the Databricks workspace unless they are connecting from a trusted corporate network, effectively mitigating the risk of unauthorized access from public or malicious locations.

Answer

Unity Catalog access controls.

Answer

Cluster-level Spark configurations.

Answer

Workspace-level SSO integration.

Databricks Certified Data Engineer Professional Databricks-DE-Pro practice test

Three ways to study

Study Databricks-DE-Pro by domain

Study Databricks-DE-Pro by topic

Data Ingestion and Acquisition practice questions

Data Governance practice questions

Monitoring and Alerting practice questions

Debugging and Deploying practice questions

Data Security and Compliance practice questions

Data Sharing and Federation practice questions

Developing Code (Python/SQL) practice questions

Data Transformation, Cleansing, Quality practice questions

Data Modelling practice questions

Cost and Performance Optimization practice questions

Databricks-DE-Pro fundamentals practice questions

Databricks-DE-Pro scenario practice questions

Databricks Certified Data Engineer Professional practice questions