Databricks-DE-Pro Practice Questions 1 — 50 Questions

Question 1

A data engineer is investigating a job failure that occurred only in the production environment. Which TWO features in Databricks help in comparing the production environment to the development environment?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the 'View as JSON' feature in the Jobs UI to compare job configurations.. Comparing environment configurations is essential for troubleshooting parity issues. Databricks provides tools like the Jobs UI for export and version control integration for code, which help engineers identify subtle differences between environments. Ensuring parity is critical for reducing 'it works on my machine' scenarios. By utilizing these tools, engineers can verify cluster settings, library versions, and code versions to isolate why a process fails in production but succeeds in development.

Answer

Use the 'Cluster Logs' to compare the OS kernel versions of the nodes.

Answer

Enable the 'Debug Mode' on the Spark Driver to see raw system calls.

Answer

Query the 'workspace_users' table to see who ran the job last.

Question 2

A data engineer is optimizing a Delta Lake table that experiences high read latency due to many small files. Which command should be executed to physically reorganize the data layout to improve query performance?

Accepted Answer

OPTIMIZE table_name. The OPTIMIZE command is the primary tool for compacting small files into larger, efficiently sized files in Delta Lake. This improves read performance by reducing metadata overhead and enabling efficient data skipping. Compaction is a critical maintenance task for tables with frequent streaming or small-batch writes, ensuring that downstream analytical queries remain performant and cost-effective by reducing the number of I/O operations required during data retrieval.

Answer

VACUUM table_name

Answer

ANALYZE TABLE table_name COMPUTE STATISTICS

Answer

REORG TABLE table_name APPLY (PURGE)

Question 3

A data engineer deploys a Databricks Job that runs a notebook task. The notebook writes to a Delta table in Unity Catalog. The job fails with the error: 'PERMISSION_DENIED: User does not have USE CATALOG on catalog 'prod'.' The engineer confirms the job's service principal has USE CATALOG granted on the catalog. Which configuration should the engineer check next?

Accepted Answer

Whether the job is configured to run as the service principal or as a different user. Unity Catalog privileges are enforced based on the identity that executes the job. When a service principal has USE CATALOG but the job runs as a different user or principal, the error still occurs. Checking the job's run-as configuration and aligning it with the granted identity resolves the mismatch and allows the notebook to access the catalog.

Answer

The notebook's default language setting

Answer

The job cluster's Spark config for spark.databricks.acl.enabled

Answer

The cluster's autoscaling min and max worker counts

Question 4

A data engineer is configuring a Delta Live Tables (DLT) pipeline that processes streaming data from Apache Kafka. The pipeline performs a series of transformations and writes to a Delta table. The engineer notices that the pipeline is experiencing high latency and wants to optimize it for cost and performance. The pipeline is set to continuous mode. Which configuration change is most effective to reduce cost while maintaining acceptable latency?

Accepted Answer

Switch the pipeline to triggered mode and schedule it to run every 5 minutes.. Switching from continuous to triggered mode with a 5-minute schedule reduces cost by allowing the cluster to shut down between runs. Continuous mode keeps the cluster running 24/7, incurring constant DBU charges. Triggered mode with a schedule balances latency and cost, making it the most effective change for reducing cost while maintaining acceptable latency.

Answer

Increase the number of Kafka partitions to improve parallelism.

Answer

Configure the pipeline to use Photon-optimized clusters.

Answer

Enable autoscaling on the DLT cluster and set a minimum number of workers.

Question 5

A data engineer needs to read a CSV file from cloud storage into a Spark DataFrame in Databricks. The file has a header row and uses commas as delimiters. The engineer wants to infer the schema automatically. Which code snippet correctly reads the file?

Accepted Answer

spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("path/to/file.csv"). The correct method chain uses spark.read.format("csv") to specify the format, then .option("header", "true") and .option("inferSchema", "true") to set the options, and finally .load(path). This is the standard and correct way to read a CSV file with schema inference in Spark.

Answer

spark.read.option("format", "csv").option("header", "true").option("inferSchema", "true").load("path/to/file.csv")

Answer

spark.read.csv("path/to/file.csv").option("header", "true").option("inferSchema", "true")

Answer

spark.read.format("csv").load("path/to/file.csv", header=True, inferSchema=True)

Question 6

Your team is using a shared cluster for development. A user reports that their job is slow because the cluster memory is frequently filled by large data broadcasts. What configuration adjustment should you make to prevent this issue across all jobs on the cluster?

Accepted Answer

Decrease the 'spark.sql.autoBroadcastJoinThreshold'.. Controlling the broadcast threshold prevents Spark from automatically attempting to broadcast large tables that exceed the available memory, which is the most common cause of OOMs on shared clusters. Setting this value correctly ensures that only appropriately small tables are broadcast, forcing the engine to use a join shuffle instead. This maintains stability for all users on the shared cluster, preventing one user's job from impacting others.

Answer

Increase 'spark.driver.memory'.

Answer

Increase the number of shuffle partitions.

Answer

Enable 'spark.sql.adaptive.enabled'.

Question 7

Which TWO of the following are primary benefits of using Delta Live Tables (DLT) for data ingestion over standard Structured Streaming pipelines?

Accepted Answer

Declarative pipeline management and automated dependency handling.. DLT simplifies the operational complexity of data pipelines by providing automatic infrastructure management and built-in quality controls. It abstracts the configuration required for managing checkpoints, scaling, and handling schema drift, allowing engineers to focus on defining transformations. The 'Expectations' framework and declarative pipeline management provide superior observability and data quality enforcement compared to manual, imperative coding in standard Spark Structured Streaming.

Answer

DLT supports significantly higher throughput than Structured Streaming.

Answer

DLT is the only way to read from cloud object storage.

Answer

DLT supports non-Delta storage formats for all outputs.

Question 8

When running a PySpark job, you receive an 'Out of Memory (OOM)' error during a shuffle operation. Which configuration is the most appropriate to address this first?

Accepted Answer

Increase spark.sql.shuffle.partitions. OOM errors during shuffles are frequently caused by partitions that are too large for the executor's memory. Increasing 'spark.sql.shuffle.partitions' is the primary corrective action, as it splits the data into a larger number of smaller partitions. This reduces the memory footprint of each individual task, allowing them to fit within the executor's heap memory and preventing the job from crashing due to memory exhaustion during data movement.

Answer

Increase spark.driver.memory

Answer

Decrease spark.executor.memory

Answer

Set spark.sql.shuffle.partitions to 1

Question 9

An engineer notices that a SQL warehouse is frequently hitting 'Max Concurrency' limits. Which log should they consult to identify which specific queries are consuming most of the warehouse resources?

Accepted Answer

Query history system table. The 'query_history' system table is the primary resource for analyzing query performance and resource consumption. By querying this table, engineers can attribute resource usage to specific users, notebooks, or warehouses. This is essential for troubleshooting concurrency bottlenecks, enabling teams to optimize expensive queries or adjust warehouse sizing to meet demand without impacting overall system performance or user productivity.

Answer

Cluster event logs

Answer

Job run history logs

Answer

Delta table access logs

Question 10

A data engineering team is modeling a large Delta Lake fact table that stores clickstream events. Analysts frequently run queries that filter by event_date and then aggregate by user_id, and the table receives continuous appends plus occasional late-arriving corrections. The team wants to reduce bytes scanned and improve join performance. Which two design choices are most appropriate? (Choose two.)

Accepted Answer

Partition the table by event_date and apply Z-ORDER on user_id within each partition. The workload filters on event_date and aggregates by user_id, so layout must accelerate both. Partitioning by event_date with Z-ORDER on user_id gives reliable partition pruning plus co-located user data, while liquid clustering on both columns achieves similar skipping without rigid high-cardinality directories and can be adapted incrementally as access patterns change.

Answer

Convert the table to a Parquet external table and rely on the query engine to infer partitioning

Answer

Enable Change Data Feed on the table so late-arriving corrections are captured automatically

Answer

Partition the table by user_id to maximize file skipping for user-level aggregations

Question 11

A data engineer is implementing fine-grained access control on a Delta table in Unity Catalog that contains sensitive customer data. The requirement is to mask the `credit_card` column for all users except members of the `finance` group, and to filter out rows where the `region` column is not in the user's allowed regions. Which two Unity Catalog features should the engineer use? (Choose two.)

Accepted Answer

Row filter using a SQL UDF that checks the user's allowed regions against the `region` column.. Unity Catalog provides native row filters and column masks for fine-grained access control. A column mask with a SQL UDF can conditionally reveal or obscure the credit card column based on group membership, while a row filter with a SQL UDF can restrict rows by region. Together they implement the required masking and filtering directly on the table without requiring a separate view.

Answer

Table ACLs that grant SELECT only on specific columns.

Answer

Workspace-level IP access list to restrict access to the table.

Answer

Dynamic view that joins the table with a mapping table of user regions.

Question 12

A Data Engineer needs to encrypt data at rest within a Databricks workspace that uses a customer-managed key (CMK). What is the primary purpose of this configuration?

Accepted Answer

To control the lifecycle of the encryption keys used for storage.. Using a customer-managed key (CMK) provides an additional layer of security and control over data at rest in cloud storage, such as S3 or ADLS. By managing the key in the cloud provider's Key Management Service (KMS), the organization can revoke access to the data at any time by disabling the key. This is a common requirement for high-compliance industries that need verifiable control over their encrypted data.

Answer

To increase the read throughput of Delta tables.

Answer

To enable transient data encryption for cluster nodes.

Answer

To bypass the need for Unity Catalog permissions.

Question 13

You are performing a complex data transformation involving a self-join on a large, skewed table. Which technique is most effective for preventing data skew and improving join performance?

Accepted Answer

Apply a 'salt' to the join key to redistribute the skewed data across more partitions.. Skewed joins are a primary cause of performance failure in distributed computing. Using a 'salted' key—adding a random prefix to the join key—distributes data evenly across partitions during the join. This prevents 'hot' partitions where one executor processes the majority of the data, significantly speeding up the query and preventing OOM (Out of Memory) errors, ensuring consistent performance for large-scale analytical tasks.

Answer

Increase the number of executors in the cluster to handle the load.

Answer

Use the 'repartition' hint on the join keys to force uniform distribution.

Answer

Convert the join into a broadcast join to prevent shuffling entirely.

Question 14

You are designing an ingestion pipeline that must handle massive bursts of data at irregular intervals. Which feature should you prioritize to ensure the ingestion process remains cost-effective?

Accepted Answer

Use auto-scaling clusters with a minimum of zero workers.. Using Photon-accelerated clusters with auto-scaling is the most effective way to handle bursty workloads. By configuring the cluster to scale out when the queue of files is large and scale down during idle periods, you maximize resource utilization. This approach ensures you have enough compute to meet latency SLAs during bursts while minimizing costs by scaling to zero or minimal size when no data is being ingested, optimizing for both performance and budget.

Answer

Use a fixed-size cluster with a large number of nodes.

Answer

Run the pipeline continuously on a single-node cluster.

Answer

Increase the 'spark.sql.shuffle.partitions' to 10000.

Question 15

An organization wants to restrict data access to only allow connections from specific corporate IP ranges. Which Databricks feature should be configured to implement this network security requirement?

Accepted Answer

IP Access Lists.. Network-level security is a cornerstone of enterprise data governance. By using IP access lists, Databricks allows administrators to define a whitelist of allowed CIDR blocks. This ensures that even if a user has valid credentials, they cannot access the Databricks workspace unless they are connecting from a trusted corporate network, effectively mitigating the risk of unauthorized access from public or malicious locations.

Answer

Unity Catalog access controls.

Answer

Cluster-level Spark configurations.

Answer

Workspace-level SSO integration.

Question 16

Refer to the exhibit. A Databricks job fails with a 403 Forbidden error when trying to write to the S3 bucket. Why does this happen?

Accepted Answer

The policy is missing the 's3:PutObject' action required for writing data.. The provided IAM policy only grants 's3:GetObject' permissions, which is read-only. For a job to write data, it needs additional permissions like 's3:PutObject'. In production, strict adherence to the principle of least privilege is required; however, the policy must also enable the necessary write operations for the task to complete successfully. The 403 error is a direct consequence of this missing write-specific capability in the policy definition.

Answer

The Databricks cluster needs to be restarted to apply the new IAM policy.

Answer

The S3 bucket policy is blocking the request despite the IAM policy.

Answer

The 'Resource' ARN is missing the suffix '/*'.

Question 17

A data engineer is designing a solution to share a Delta table with an external partner organization. The partner uses a different Databricks account and must be able to read the table, but the data must not be copied outside the provider's cloud storage. The provider uses Unity Catalog and wants to minimize operational overhead while ensuring the partner sees only the shared table. Which Unity Catalog feature should the engineer use?

Accepted Answer

Delta Sharing. Delta Sharing is designed for cross-organization data sharing without copying data. The provider defines a share, adds the table, and grants the recipient access. The recipient can then read the shared data using their own Databricks workspace or other compatible clients. The data stays in the provider's storage, and the provider retains control over access. This meets the requirements with minimal overhead.

Answer

A foreign table that points to the partner's storage location.

Answer

Grant SELECT on the table to the partner's service principal and configure cross-account IAM access.

Answer

A deep clone of the table into a storage location accessible by the partner.

Question 18

A data engineer is debugging a slow-running query. They notice that the data is skewed, causing one task to take significantly longer than others. Which approach effectively addresses this skew?

Accepted Answer

Add a salt column to the skewed key to distribute the data across more partitions.. Data skew occurs when one partition contains disproportionately more data than others, causing a single executor to bottleneck the entire stage. By using techniques like salt, broadcast joins, or repartitioning, the engineer can distribute the load more evenly across the cluster. This is essential for optimizing performance and preventing timeouts, ensuring that production jobs adhere to defined SLAs and utilize cluster resources efficiently.

Answer

Increase the cluster's disk size to allow for more local shuffle storage.

Answer

Change the file format from Parquet to CSV to reduce overhead.

Answer

Enable 'Auto-scaling' to automatically add more nodes during the skewed stage.

Question 19

A Data Engineer is developing a Delta Live Tables (DLT) pipeline using Python. They need to ensure that records failing a specific data quality check are dropped, but the pipeline continues to process the remaining valid records. Which expectation syntax should the engineer implement?

Accepted Answer

@dlt.expect_or_drop('col_check', 'id IS NOT NULL'). The 'expect_or_drop' constraint is specifically designed for DLT to handle data quality failures gracefully. By using this expectation, the pipeline marks the record as invalid and removes it from the target dataset without halting the entire execution. This is critical in production environments where partial data ingestion is preferred over pipeline failure, ensuring high availability and robust data processing workflows for downstream users.

Answer

@dlt.expect_or_fail('col_check', 'id IS NOT NULL')

Answer

@dlt.expect('col_check', 'id IS NOT NULL')

Answer

@dlt.validate('col_check', 'id IS NOT NULL')

Question 20

A data engineer is tasked with reducing compute costs for an interactive SQL analytics workspace that runs sporadic, highly unpredictable queries. The jobs experience cold start delays and occasional out-of-memory errors due to sudden concurrency spikes. Which TWO strategies should the engineer implement to balance cost efficiency and performance?

Accepted Answer

Migrate interactive SQL workloads to Databricks Serverless Compute to dynamically scale resources and eliminate idle billing.. Implementing serverless compute eliminates idle cluster costs while absorbing concurrency spikes instantly through elastic scaling. Enabling Photon on standard clusters provides vectorized query execution that speeds up scans and aggregations, reducing runtime and lowering total execution costs. Together, these choices optimize both resource utilization and user concurrency responsiveness.

Answer

Configure single-node clusters with maximum autoscaling limits to handle unpredictable concurrency peaks without cluster management overhead.

Answer

Provision pools of pre-warmed idle driver nodes to ensure zero-second startup latency for all analytical queries.

Answer

Disable automatic cluster termination and keep all worker nodes running 24/7 to guarantee immediate resource availability.