Databricks-DE-Assoc Practice Test 4 — 25 Questions

Question 1

A data engineer needs to optimize the layout of a massive Delta Lake table that suffers from poor query performance due to a large number of small files and unsorted data records. Which TWO operations should the engineer execute to resolve these performance bottlenecks?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Execute the OPTIMIZE command to compact small files into larger files.. Optimizing a Delta table requires addressing both small file consolidation and multidimensional physical organization. Running OPTIMIZE compacts small files into larger, uniform file sizes, while combining it with Liquid Clustering or Z-Ordering organizes data records physically to maximize data skipping during filter queries. Together, these maintenance operations significantly reduce read input/output overhead across cloud object storage.

Answer

Execute the VACUUM command with a retention period of zero hours.

Answer

Run REPARTITION on the Delta table using the Spark DataFrame API directly.

Answer

Enable predictive optimization on the Databricks Unity Catalog managed table.

Question 2

A data engineer is working on a Delta Lake table that has accumulated millions of small files due to frequent streaming updates. This fragmentation has significantly degraded query performance. Which operation should the engineer execute to optimize file layout without altering table data?

Accepted Answer

Invoke the OPTIMIZE command to compact small parquet files into larger files using bin-packing.. Running the OPTIMIZE command on a Delta table compacts many small files into larger, optimized files using the bin-packing algorithm. This directly addresses fragmentation issues caused by frequent streaming micro-batches, improving read efficiency and scan speeds. Data engineers frequently schedule optimization jobs as part of bronze-to-silver and silver-to-gold maintenance pipelines in production environments to maintain optimal query performance and reduce cloud storage request latency.

Answer

Execute REFRESH TABLE to update the file metadata cache across all cluster worker nodes.

Answer

Run the VACUUM command with a retention threshold of zero hours to purge all historical data files.

Answer

Execute ALTER TABLE SET TBLPROPERTIES to enable automatic file compaction on every write transaction.

Question 3

An engineer needs to identify the root cause of a job failure. Which THREE of the following are valid locations or methods to investigate the logs?

Accepted Answer

Cluster event logs. Troubleshooting in Databricks requires access to various log levels. The cluster event log captures infrastructure-level changes, driver logs provide application execution details, and executor logs offer task-specific debugging info. Using these three sources provides a holistic view, covering everything from cluster startup issues to specific code-level exceptions. This multi-layered approach is essential for isolating whether a problem is infrastructure-related, configuration-based, or rooted in the logic of the transformation code itself.

Answer

Notebook workspace browser history

Answer

Databricks account profile settings

Question 4

Refer to the exhibit. A data engineer is attempting to run a VACUUM command on a table, but the command fails with an error indicating that the retention period is too short. Given the configuration, what is the most appropriate action the engineer should take to safely remove files older than 7 days?

Accepted Answer

Execute VACUUM table_name RETAIN 168 HOURS.. The configuration restricts the vacuuming of files to ensure Time Travel remains available. Databricks prevents setting a retention period shorter than the log retention to avoid data loss. To safely vacuum, the engineer must temporarily adjust the configuration or ensure the retention period respects the safety threshold. Understanding these safety constraints is critical for data lifecycle management and cost control in cloud storage environments.

Answer

Set 'spark.databricks.delta.retentionDurationCheck.enabled' to false.

Answer

Delete the underlying parquet files directly from cloud storage.

Answer

Run the OPTIMIZE command followed by REFRESH TABLE.

Question 5

A data engineer needs to share a table in Unity Catalog with external partners who do not have access to the Databricks workspace. Which feature should the data engineer configure?

Accepted Answer

Create a Delta Sharing recipient and publish tables to a sharing share object.. Delta Sharing enables secure data sharing with external organizations without requiring recipients to have access to your Databricks workspace or even be on the same cloud platform. It leverages cloud object storage credentials and open protocols to ensure governed data exchange.

Answer

Configure cross-workspace table linking using standard Spark mount points.

Answer

Create a shared Unity Catalog metastore and provision external IAM database users.

Answer

Export the Delta table files to CSV format and upload them to a public cloud bucket.

Question 6

A data engineer is designing a pipeline on Databricks that requires ACID transactions and schema enforcement for streaming data. Which storage abstraction should they use to ensure data reliability and support time travel?

Accepted Answer

Delta Lake tables. Delta Lake provides the foundation for the Databricks Lakehouse, offering ACID transactions, schema enforcement, and versioning capabilities. By utilizing Delta tables, engineers can perform time travel via transaction logs to audit data or recover from accidental deletions. This storage layer is essential for maintaining high-quality data throughout the pipeline lifecycle, preventing partial writes and ensuring consistent reads for downstream analytics users in the Lakehouse environment.

Answer

Parquet files in S3 without a transaction log

Answer

Hive-style partitioned CSV files

Answer

A standard memory-mapped temporary table

Question 7

A data engineer is designing a Delta Lake Bronze-to-Silver pipeline in Databricks and needs to ensure that downstream consumers receive high-quality data. Which TWO data quality enforcement mechanisms are natively supported in Delta Live Tables using expectations?

Accepted Answer

CONSTRAINT expectation_name EXPECT (column_name IS NOT NULL) ON VIOLATION DROP ROW. Delta Live Tables provides native expectation syntax to monitor and enforce data quality constraints directly within pipeline definitions. Data engineers can configure expectations to either drop invalid records or fail the pipeline execution when violations occur, ensuring robust governance. These declarative validation rules are critical for maintaining trusted data assets across modern enterprise Medallion architectures without writing complex custom validation code.

Answer

ASSERT expectation_name ON VIOLATION RETRY

Answer

FILTER expectation_name WHERE column_name IS NOT NULL

Answer

VALIDATE expectation_name ON ERROR IGNORE

Question 8

Your organization requires that all data processing pipelines enforce a strict schema to prevent corrupt data from landing in the Silver layer. Which feature should be configured to ensure that only data matching the expected schema is written?

Accepted Answer

Rely on the default Delta Lake schema enforcement mechanism.. Schema enforcement (often called schema validation) is the standard mechanism in Delta Lake to protect downstream tables. By setting 'mergeSchema' to false or relying on default behavior, you ensure that any incoming data that deviates from the predefined target schema will cause the pipeline to fail. This is critical for data quality in production, preventing unexpected data structures from breaking analytical models and downstream dashboards.

Answer

Enable 'spark.databricks.delta.schema.autoMerge.enabled'.

Answer

Use a JSON schema file during the read process to discard invalid records.

Answer

Set the table property 'delta.columnMapping.mode' to 'name'.

Question 9

Refer to the exhibit. The streaming pipeline is experiencing memory issues because the state store size increases continuously. What is the most effective way to address this while maintaining aggregation accuracy?

Accepted Answer

Add a watermark to the stream and use a windowed aggregation instead of a global count.. Continuous state growth is a common issue when grouping by high-cardinality keys like 'user_id' in streaming without time-bound constraints. By introducing a watermark and a window, you inform Spark which state is eligible for eviction. This allows the engine to periodically clean up expired state, ensuring the pipeline remains stable and memory usage stays within the bounds of the allocated cluster resources.

Answer

Increase the cluster memory and executor count indefinitely.

Answer

Change the output mode to 'Append' to flush the state to the sink immediately.

Answer

Use the 'dropDuplicates' method instead of 'groupBy' to simplify the state.

Question 10

A data engineer needs to join a small 'lookup' table with a large 'transactions' table in a Spark job. Which transformation strategy will provide the best performance in a cluster environment?

Accepted Answer

Use the 'broadcast()' function from 'pyspark.sql.functions' on the small table.. In distributed computing, joining a large table with a small one causes a shuffle if not handled correctly. By using a broadcast join, the engineer forces the small table to be copied to every node, eliminating the need for a network-heavy shuffle of the large table. This technique is a standard optimization for improving query latency and overall throughput in data pipelines.

Answer

Increase the 'spark.sql.shuffle.partitions' value to 1000.

Answer

Sort both tables by the join key before performing the join.

Answer

Repartition both tables to the same number of partitions.

Question 11

Your team is migrating a manual Databricks job to a CI/CD pipeline. You need to ensure the job configuration is version-controlled and deployed programmatically. Which approach aligns with Databricks best practices?

Accepted Answer

Define the job configuration in YAML files using Databricks Asset Bundles and deploy via the CLI.. Using Databricks Asset Bundles (DABs) is the recommended approach for modern CI/CD. It treats infrastructure as code, allowing you to define jobs, pipelines, and workflows in YAML files. This ensures consistency across environments like development, staging, and production. By decoupling the job definition from the workspace UI, you gain auditability, reproducibility, and the ability to integrate seamlessly with Git workflows, reducing manual configuration errors and accelerating release cycles for data engineering teams.

Answer

Manually export the job JSON via the UI and copy-paste it into the production workspace.

Answer

Use the Databricks UI to update production jobs directly to ensure changes are applied immediately.

Answer

Write a custom Python script that uses the workspace API to recreate the job from scratch every time.

Question 12

Which Spark configuration property can be used to enable Adaptive Query Execution (AQE) in Databricks?

Accepted Answer

spark.sql.adaptive.enabled. Adaptive Query Execution (AQE) is a critical optimization framework in Spark that re-optimizes query plans at runtime based on statistics collected during shuffle stages. It enables features like coalescing shuffle partitions and dynamically switching join strategies. Enabling AQE is a best practice for modern Databricks workloads as it provides significant performance gains without manual intervention, automatically adapting to varying data distributions in production pipelines.

Answer

spark.sql.autoBroadcastJoinThreshold

Answer

spark.databricks.adaptive.execution

Answer

spark.sql.shuffle.partitions

Question 13

Which of the following is a recommended strategy for managing library dependencies in a CI/CD pipeline for Databricks?

Accepted Answer

Include a requirements.txt file and install libraries as part of the job definition.. Managing libraries through a centralized repository or a requirements.txt file ensures that the exact same versions are installed consistently across all environments. This avoids the 'dependency hell' where code fails because of incompatible library versions. By integrating dependency management into the CI/CD build process, teams can audit used versions, ensure security compliance, and guarantee that the production pipeline runs with the exact dependencies that were tested during the development phase.

Answer

Install libraries manually on the cluster via the Databricks UI.

Answer

Use the latest version of all libraries to ensure best performance.

Answer

Store library binary files directly in the Git repository.

Question 14

Which THREE strategies are recommended to improve the performance of reading from a Delta table in Databricks?

Accepted Answer

Use Z-Ordering on columns frequently used in WHERE clauses.. Optimizing reads is crucial for analytics. Techniques like Z-Ordering, Data Skipping, and Partitioning allow Spark to ignore irrelevant data files, reducing I/O. Proper file sizing ensures that Spark can efficiently read data in parallel. These strategies together minimize the amount of data scanned and transferred across the network, leading to significantly faster query results in large-scale data lake environments.

Answer

Partition the table by every column used in the query.

Answer

Always set the broadcast join threshold to -1.

Question 15

Refer to the exhibit. A CI/CD pipeline running a Databricks CLI command fails with the error shown. What is the most likely cause?

Accepted Answer

The path provided in the deployment script does not exist in the target Databricks workspace.. This error indicates that the automation script is attempting to reference a file or directory path that is missing in the target workspace. In CI/CD, this frequently happens when paths are hardcoded for local development environments rather than being abstracted. Using relative paths or workspace-agnostic configurations within Databricks Asset Bundles prevents these environment-specific failures, ensuring that the deployment process remains robust regardless of where the code is being executed during the pipeline run.

Answer

The Databricks CLI is not installed on the runner.

Answer

The Personal Access Token has expired.

Answer

The user does not have permissions to write to the root directory.

Question 16

A data engineer is configuring Unity Catalog governance for a multi-department organization. Which TWO actions require the metastore admin or catalog owner to have a workspace-independent metastore assigned? (Choose TWO)

Accepted Answer

Binding a workspace to a Unity Catalog metastore to enable catalog-level data access.. A Unity Catalog metastore is the top-level container for metadata, governing three levels of namespacing (catalog, schema, table). Certain administrative tasks like assigning a metastore to workspaces or managing root storage locations require specific privileges and architectural scopes, ensuring centralized security governance across multiple workspaces.

Answer

Creating a new catalog inside the Unity Catalog metastore.

Answer

Granting the SELECT privilege on a specific schema to a group of data analysts.

Answer

Defining a custom dynamic view with column-level masking functions for auditing.

Question 17

Which TWO of the following are benefits of using Unity Catalog over the legacy Hive Metastore?

Accepted Answer

Unified governance across all workspaces.. Unity Catalog represents a modern approach to data governance, solving the fragmentation caused by having separate metastores in every workspace. By providing a centralized, account-level view of all data, it enables enterprise-wide discovery, consistent security policies, and deep integration with lineage and audit features. This shift simplifies the data engineer's job by reducing configuration overhead while significantly improving the organization's security posture and audit readiness for compliance requirements.

Answer

Support for more SQL dialects.

Answer

Increased query performance.

Answer

Lower storage costs.

Question 18

An engineer notices that a specific notebook job is consistently taking longer to start. They observe high 'initialization' times in the job logs. Which action should the engineer take to improve startup time?

Accepted Answer

Use a Databricks Pool for the job cluster.. Initialization time in Databricks jobs often stems from the time required to pull container images, install cluster-scoped libraries, or initialize the Spark context on a new cluster. By using an existing cluster (pool) or pre-warming the cluster, the overhead of provisioning hardware and installing dependencies is removed. This optimization is crucial for meeting strict SLAs in production pipelines where every minute of latency impacts downstream processes.

Answer

Enable Auto-scaling on the cluster.

Answer

Increase the number of worker nodes.

Answer

Update the notebook code to use RDDs.

Question 19

An analytics team needs to frequently query a large Delta table by a high-cardinality customer_id column and a date column. To optimize query performance and reduce data scanning during filtering, how should the data engineer structure the table layout?

Accepted Answer

Partition the table by date and apply Liquid Clustering on customer_id.. Partitioning the Delta table by the low-cardinality date column avoids the creation of excessive tiny files, while Liquid Clustering on customer_id organizes the data layout dynamically based on clustering keys. This hybrid approach leverages the strengths of both features: partition pruning for temporal ranges and liquid clustering for high-cardinality point lookups, significantly improving downstream query execution efficiency.

Answer

Partition the table by both customer_id and date simultaneously.

Answer

Apply Z-Order clustering exclusively on date without any partitioning.

Answer

Rely entirely on Spark AQE to automatically index customer_id at runtime.

Question 20

A data engineering team uses Databricks Asset Bundles (DABs) to manage a job that must deploy to both a staging and a production workspace. The team wants to avoid hardcoding workspace-specific values such as the cluster ID and the storage path in databricks.yml. Which approach should they use?

Accepted Answer

Define variables in the bundle and set target-specific values in databricks.yml under targets, then deploy with databricks bundle deploy -t <target>.. Databricks Asset Bundles use a declarative databricks.yml with variables and targets. Declaring variables once and overriding them per target lets one bundle deploy consistently to staging and production without hardcoded workspace-specific values. Selecting the target at deploy time applies the correct values, keeps a single source of truth in Git, and avoids drift from manual edits or post-deployment patching.

Answer

Run databricks bundle deploy once, then use the Databricks REST API to patch the job's cluster ID and storage path after each deployment.

Answer

Store the cluster ID and storage path in a Databricks secret scope and reference them with dbutils.secrets.get inside the job's notebook code.

Answer

Create a separate Git branch for each workspace and manually edit the cluster ID and storage path in databricks.yml before each deployment.