Databricks-DE-Pro Hard Questions — 25 Difficult Questions

Question 1

A data engineer is investigating a job failure that occurred only in the production environment. Which TWO features in Databricks help in comparing the production environment to the development environment?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the 'View as JSON' feature in the Jobs UI to compare job configurations.. Comparing environment configurations is essential for troubleshooting parity issues. Databricks provides tools like the Jobs UI for export and version control integration for code, which help engineers identify subtle differences between environments. Ensuring parity is critical for reducing 'it works on my machine' scenarios. By utilizing these tools, engineers can verify cluster settings, library versions, and code versions to isolate why a process fails in production but succeeds in development.

Answer

Use the 'Cluster Logs' to compare the OS kernel versions of the nodes.

Answer

Enable the 'Debug Mode' on the Spark Driver to see raw system calls.

Answer

Query the 'workspace_users' table to see who ran the job last.

Question 2

A data engineer is configuring a Delta Live Tables (DLT) pipeline that processes streaming data from Apache Kafka. The pipeline performs a series of transformations and writes to a Delta table. The engineer notices that the pipeline is experiencing high latency and wants to optimize it for cost and performance. The pipeline is set to continuous mode. Which configuration change is most effective to reduce cost while maintaining acceptable latency?

Accepted Answer

Switch the pipeline to triggered mode and schedule it to run every 5 minutes.. Switching from continuous to triggered mode with a 5-minute schedule reduces cost by allowing the cluster to shut down between runs. Continuous mode keeps the cluster running 24/7, incurring constant DBU charges. Triggered mode with a schedule balances latency and cost, making it the most effective change for reducing cost while maintaining acceptable latency.

Answer

Increase the number of Kafka partitions to improve parallelism.

Answer

Configure the pipeline to use Photon-optimized clusters.

Answer

Enable autoscaling on the DLT cluster and set a minimum number of workers.

Question 3

Refer to the exhibit. If a job is failing because the cluster is reaching the 'max_workers' limit too quickly, which monitoring metric should the engineer verify to confirm if the cluster is actually utilizing these nodes efficiently?

Accepted Answer

Task-level execution time distribution. Verifying 'task_skew' and 'executor_utilization' is key. If the cluster is scaling to 8 nodes but still failing, the issue may not be a lack of resources, but rather inefficient data distribution (skew) or task overhead. Analyzing these metrics ensures that engineers don't simply increase the cluster size, which would increase costs without solving the underlying performance issues caused by unbalanced workloads.

Answer

Total number of users logged into the workspace.

Answer

The total number of files in the Delta table.

Answer

The memory overhead of the driver node.

Question 4

Your team is using a shared cluster for development. A user reports that their job is slow because the cluster memory is frequently filled by large data broadcasts. What configuration adjustment should you make to prevent this issue across all jobs on the cluster?

Accepted Answer

Decrease the 'spark.sql.autoBroadcastJoinThreshold'.. Controlling the broadcast threshold prevents Spark from automatically attempting to broadcast large tables that exceed the available memory, which is the most common cause of OOMs on shared clusters. Setting this value correctly ensures that only appropriately small tables are broadcast, forcing the engine to use a join shuffle instead. This maintains stability for all users on the shared cluster, preventing one user's job from impacting others.

Answer

Increase 'spark.driver.memory'.

Answer

Increase the number of shuffle partitions.

Answer

Enable 'spark.sql.adaptive.enabled'.

Question 5

Which TWO of the following are primary benefits of using Delta Live Tables (DLT) for data ingestion over standard Structured Streaming pipelines?

Accepted Answer

Declarative pipeline management and automated dependency handling.. DLT simplifies the operational complexity of data pipelines by providing automatic infrastructure management and built-in quality controls. It abstracts the configuration required for managing checkpoints, scaling, and handling schema drift, allowing engineers to focus on defining transformations. The 'Expectations' framework and declarative pipeline management provide superior observability and data quality enforcement compared to manual, imperative coding in standard Spark Structured Streaming.

Answer

DLT supports significantly higher throughput than Structured Streaming.

Answer

DLT is the only way to read from cloud object storage.

Answer

DLT supports non-Delta storage formats for all outputs.

Question 6

When running a PySpark job, you receive an 'Out of Memory (OOM)' error during a shuffle operation. Which configuration is the most appropriate to address this first?

Accepted Answer

Increase spark.sql.shuffle.partitions. OOM errors during shuffles are frequently caused by partitions that are too large for the executor's memory. Increasing 'spark.sql.shuffle.partitions' is the primary corrective action, as it splits the data into a larger number of smaller partitions. This reduces the memory footprint of each individual task, allowing them to fit within the executor's heap memory and preventing the job from crashing due to memory exhaustion during data movement.

Answer

Increase spark.driver.memory

Answer

Decrease spark.executor.memory

Answer

Set spark.sql.shuffle.partitions to 1

Question 7

An engineer notices that a SQL warehouse is frequently hitting 'Max Concurrency' limits. Which log should they consult to identify which specific queries are consuming most of the warehouse resources?

Accepted Answer

Query history system table. The 'query_history' system table is the primary resource for analyzing query performance and resource consumption. By querying this table, engineers can attribute resource usage to specific users, notebooks, or warehouses. This is essential for troubleshooting concurrency bottlenecks, enabling teams to optimize expensive queries or adjust warehouse sizing to meet demand without impacting overall system performance or user productivity.

Answer

Cluster event logs

Answer

Job run history logs

Answer

Delta table access logs

Question 8

A data engineering team is modeling a large Delta Lake fact table that stores clickstream events. Analysts frequently run queries that filter by event_date and then aggregate by user_id, and the table receives continuous appends plus occasional late-arriving corrections. The team wants to reduce bytes scanned and improve join performance. Which two design choices are most appropriate? (Choose two.)

Accepted Answer

Partition the table by event_date and apply Z-ORDER on user_id within each partition. The workload filters on event_date and aggregates by user_id, so layout must accelerate both. Partitioning by event_date with Z-ORDER on user_id gives reliable partition pruning plus co-located user data, while liquid clustering on both columns achieves similar skipping without rigid high-cardinality directories and can be adapted incrementally as access patterns change.

Answer

Convert the table to a Parquet external table and rely on the query engine to infer partitioning

Answer

Enable Change Data Feed on the table so late-arriving corrections are captured automatically

Answer

Partition the table by user_id to maximize file skipping for user-level aggregations

Question 9

A data engineer is implementing fine-grained access control on a Delta table in Unity Catalog that contains sensitive customer data. The requirement is to mask the `credit_card` column for all users except members of the `finance` group, and to filter out rows where the `region` column is not in the user's allowed regions. Which two Unity Catalog features should the engineer use? (Choose two.)

Accepted Answer

Row filter using a SQL UDF that checks the user's allowed regions against the `region` column.. Unity Catalog provides native row filters and column masks for fine-grained access control. A column mask with a SQL UDF can conditionally reveal or obscure the credit card column based on group membership, while a row filter with a SQL UDF can restrict rows by region. Together they implement the required masking and filtering directly on the table without requiring a separate view.

Answer

Table ACLs that grant SELECT only on specific columns.

Answer

Workspace-level IP access list to restrict access to the table.

Answer

Dynamic view that joins the table with a mapping table of user regions.

Question 10

A Data Engineer needs to encrypt data at rest within a Databricks workspace that uses a customer-managed key (CMK). What is the primary purpose of this configuration?

Accepted Answer

To control the lifecycle of the encryption keys used for storage.. Using a customer-managed key (CMK) provides an additional layer of security and control over data at rest in cloud storage, such as S3 or ADLS. By managing the key in the cloud provider's Key Management Service (KMS), the organization can revoke access to the data at any time by disabling the key. This is a common requirement for high-compliance industries that need verifiable control over their encrypted data.

Answer

To increase the read throughput of Delta tables.

Answer

To enable transient data encryption for cluster nodes.

Answer

To bypass the need for Unity Catalog permissions.

Question 11

You are performing a complex data transformation involving a self-join on a large, skewed table. Which technique is most effective for preventing data skew and improving join performance?

Accepted Answer

Apply a 'salt' to the join key to redistribute the skewed data across more partitions.. Skewed joins are a primary cause of performance failure in distributed computing. Using a 'salted' key—adding a random prefix to the join key—distributes data evenly across partitions during the join. This prevents 'hot' partitions where one executor processes the majority of the data, significantly speeding up the query and preventing OOM (Out of Memory) errors, ensuring consistent performance for large-scale analytical tasks.

Answer

Increase the number of executors in the cluster to handle the load.

Answer

Use the 'repartition' hint on the join keys to force uniform distribution.

Answer

Convert the join into a broadcast join to prevent shuffling entirely.

Question 12

Refer to the exhibit. A Databricks job fails with a 403 Forbidden error when trying to write to the S3 bucket. Why does this happen?

Accepted Answer

The policy is missing the 's3:PutObject' action required for writing data.. The provided IAM policy only grants 's3:GetObject' permissions, which is read-only. For a job to write data, it needs additional permissions like 's3:PutObject'. In production, strict adherence to the principle of least privilege is required; however, the policy must also enable the necessary write operations for the task to complete successfully. The 403 error is a direct consequence of this missing write-specific capability in the policy definition.

Answer

The Databricks cluster needs to be restarted to apply the new IAM policy.

Answer

The S3 bucket policy is blocking the request despite the IAM policy.

Answer

The 'Resource' ARN is missing the suffix '/*'.

Question 13

A data engineer is designing a solution to share a Delta table with an external partner organization. The partner uses a different Databricks account and must be able to read the table, but the data must not be copied outside the provider's cloud storage. The provider uses Unity Catalog and wants to minimize operational overhead while ensuring the partner sees only the shared table. Which Unity Catalog feature should the engineer use?

Accepted Answer

Delta Sharing. Delta Sharing is designed for cross-organization data sharing without copying data. The provider defines a share, adds the table, and grants the recipient access. The recipient can then read the shared data using their own Databricks workspace or other compatible clients. The data stays in the provider's storage, and the provider retains control over access. This meets the requirements with minimal overhead.

Answer

A foreign table that points to the partner's storage location.

Answer

Grant SELECT on the table to the partner's service principal and configure cross-account IAM access.

Answer

A deep clone of the table into a storage location accessible by the partner.

Question 14

A data engineer is tasked with reducing compute costs for an interactive SQL analytics workspace that runs sporadic, highly unpredictable queries. The jobs experience cold start delays and occasional out-of-memory errors due to sudden concurrency spikes. Which TWO strategies should the engineer implement to balance cost efficiency and performance?

Accepted Answer

Migrate interactive SQL workloads to Databricks Serverless Compute to dynamically scale resources and eliminate idle billing.. Implementing serverless compute eliminates idle cluster costs while absorbing concurrency spikes instantly through elastic scaling. Enabling Photon on standard clusters provides vectorized query execution that speeds up scans and aggregations, reducing runtime and lowering total execution costs. Together, these choices optimize both resource utilization and user concurrency responsiveness.

Answer

Configure single-node clusters with maximum autoscaling limits to handle unpredictable concurrency peaks without cluster management overhead.

Answer

Provision pools of pre-warmed idle driver nodes to ensure zero-second startup latency for all analytical queries.

Answer

Disable automatic cluster termination and keep all worker nodes running 24/7 to guarantee immediate resource availability.

Question 15

A data engineer is using PySpark to process a large DataFrame and needs to reduce the number of partitions before writing to a Delta table to avoid creating too many small files. The DataFrame currently has 2000 partitions, each about 10 MB. The engineer wants to reduce the number of partitions to approximately 200 while minimizing data shuffling. Which approach is most appropriate?

Accepted Answer

Use coalesce(200) to reduce the number of partitions without a full shuffle.. The correct answer is to use coalesce(200). coalesce reduces the number of partitions by merging existing partitions without a full shuffle, which is efficient for decreasing partition count. It is particularly useful when the partitions are already reasonably sized and you just want to reduce the number of output files. repartition would work but causes a full shuffle, which is unnecessary here. The other options involve shuffles or are not designed for simple partition reduction.

Answer

Use repartition(200) to evenly redistribute the data across 200 partitions.

Answer

Use repartitionByRange(200) on a key column to control data distribution.

Answer

Use bucketBy(200, 'id') and saveAsTable to create 200 buckets.

Question 16

A data engineer is implementing a data quality check on a streaming Delta table that receives late-arriving events. The engineer needs to detect records where the event timestamp is more than 2 hours behind the current watermark. Which approach correctly identifies such records for quarantine?

Accepted Answer

Use the event_time column and compare it to current_timestamp() - INTERVAL 2 HOURS in a filter condition.. To identify records that are more than 2 hours behind the current time, the engineer should compare the event timestamp to the current processing time. Using current_timestamp() - INTERVAL 2 HOURS provides a dynamic threshold that correctly flags late-arriving events. This approach works in streaming and batch, and does not rely on watermark internals.

Answer

Use a watermark on event_time and filter records where event_time < current_watermark - INTERVAL 2 HOURS.

Answer

Use the event_time column and compare it to the current watermark using the watermark() function.

Answer

Use the event_time column and compare it to the maximum event_time in the table minus 2 hours.

Question 17

A Data Engineer is implementing a data quality framework for a Delta table that receives streaming updates. The engineer needs to ensure that any record with a negative `amount` is not written to the table, but also wants to capture these invalid records in a separate quarantine table for later analysis. The pipeline must not fail on invalid records. Which approach using Delta Live Tables achieves this?

Accepted Answer

Define two expectations: one with `@dp.expect_or_drop("valid_amount", "amount >= 0")` on the main table, and another with `@dp.expect("invalid_amount", "amount < 0")` on a separate quarantine table that reads from the same source.. To drop invalid records from the main table while capturing them in a quarantine table, use `expect_or_drop` on the main table and define a separate quarantine table that filters the source for invalid records. This keeps the pipeline running and provides a repository for bad data. Using `expect_all_or_fail` would halt the pipeline, and relying on the event log does not preserve the actual records.

Answer

Use `@dp.expect_or_drop("valid_amount", "amount >= 0")` on the main table, and rely on the event log to retrieve dropped records for quarantine.

Answer

Use `@dp.expect_all_or_fail({"valid_amount": "amount >= 0"})` on the main table, and configure a separate quarantine table using `@dp.expect_all({"invalid_amount": "amount < 0"})`.

Answer

Use `@dp.expect("valid_amount", "amount >= 0")` on the main table, and create a quarantine table that uses `@dp.expect_or_drop("invalid_amount", "amount < 0")`.

Question 18

Which THREE of the following are benefits of using Delta Lake over standard Parquet files for your data lake storage?

Accepted Answer

ACID transactions for concurrent reads and writes.. Delta Lake adds a transactional layer (the log) on top of Parquet files, enabling ACID transactions, time travel, and schema enforcement. These features solve the most common challenges with raw data lakes, such as partial writes and data corruption. As a Databricks professional, you must prioritize Delta Lake to build reliable, scalable architectures that support robust data governance and high-performance analytical queries without the risks of file-based data inconsistency.

Answer

Faster cold-start reads by caching data on the driver.

Answer

Allows the use of SQL as the only way to interact with data.

Question 19

Which TWO of the following are true regarding Unity Catalog's ability to govern external locations?

Accepted Answer

External locations require a storage credential to function.. Unity Catalog can manage access to external cloud storage locations by creating 'External Locations'. This allows administrators to grant specific permissions to users to read from or write to these storage buckets without providing them with direct cloud provider credentials. This abstraction is vital for security, as it centralizes control and auditing of data access within the platform while keeping the underlying storage infrastructure shielded from the users.

Answer

Users can directly mount external locations as DBFS paths.

Answer

External locations are automatically created for every S3 bucket in the account.

Answer

External locations are only supported for Delta-formatted data.

Question 20

A data engineer is optimizing a Spark job that reads from a large Delta table and performs a join with a smaller dimension table. The job is running slowly, and the engineer suspects data skew and shuffle overhead are the main issues. Which two techniques should the engineer apply to improve performance and reduce cost? (Choose two.)

Accepted Answer

Enable adaptive query execution (AQE) and set spark.sql.adaptive.skewJoin.enabled to true.. Broadcast join eliminates shuffle for the smaller table, and enabling AQE with skew join optimization dynamically handles skew in the larger table. Together, they reduce shuffle overhead and mitigate stragglers, improving performance and lowering cost. Other options either introduce unnecessary shuffles or do not directly address skew.

Answer

Increase the number of shuffle partitions to 2000.

Answer

Cache the larger table in memory before the join.

Answer

Repartition the larger table on the join key before the join.