Databricks-DE-Assoc Practice Test 9 — 10 Questions

Question 1

A data engineer is designing a pipeline on Databricks that requires ACID transactions and schema enforcement for streaming data. Which storage abstraction should they use to ensure data reliability and support time travel?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Delta Lake tables. Delta Lake provides the foundation for the Databricks Lakehouse, offering ACID transactions, schema enforcement, and versioning capabilities. By utilizing Delta tables, engineers can perform time travel via transaction logs to audit data or recover from accidental deletions. This storage layer is essential for maintaining high-quality data throughout the pipeline lifecycle, preventing partial writes and ensuring consistent reads for downstream analytics users in the Lakehouse environment.

Answer

Parquet files in S3 without a transaction log

Answer

Hive-style partitioned CSV files

Answer

A standard memory-mapped temporary table

Question 2

A data engineer is designing a Delta Lake Bronze-to-Silver pipeline in Databricks and needs to ensure that downstream consumers receive high-quality data. Which TWO data quality enforcement mechanisms are natively supported in Delta Live Tables using expectations?

Accepted Answer

CONSTRAINT expectation_name EXPECT (column_name IS NOT NULL) ON VIOLATION DROP ROW. Delta Live Tables provides native expectation syntax to monitor and enforce data quality constraints directly within pipeline definitions. Data engineers can configure expectations to either drop invalid records or fail the pipeline execution when violations occur, ensuring robust governance. These declarative validation rules are critical for maintaining trusted data assets across modern enterprise Medallion architectures without writing complex custom validation code.

Answer

ASSERT expectation_name ON VIOLATION RETRY

Answer

FILTER expectation_name WHERE column_name IS NOT NULL

Answer

VALIDATE expectation_name ON ERROR IGNORE

Question 3

Your organization requires that all data processing pipelines enforce a strict schema to prevent corrupt data from landing in the Silver layer. Which feature should be configured to ensure that only data matching the expected schema is written?

Accepted Answer

Rely on the default Delta Lake schema enforcement mechanism.. Schema enforcement (often called schema validation) is the standard mechanism in Delta Lake to protect downstream tables. By setting 'mergeSchema' to false or relying on default behavior, you ensure that any incoming data that deviates from the predefined target schema will cause the pipeline to fail. This is critical for data quality in production, preventing unexpected data structures from breaking analytical models and downstream dashboards.

Answer

Enable 'spark.databricks.delta.schema.autoMerge.enabled'.

Answer

Use a JSON schema file during the read process to discard invalid records.

Answer

Set the table property 'delta.columnMapping.mode' to 'name'.

Question 4

Refer to the exhibit. The streaming pipeline is experiencing memory issues because the state store size increases continuously. What is the most effective way to address this while maintaining aggregation accuracy?

Accepted Answer

Add a watermark to the stream and use a windowed aggregation instead of a global count.. Continuous state growth is a common issue when grouping by high-cardinality keys like 'user_id' in streaming without time-bound constraints. By introducing a watermark and a window, you inform Spark which state is eligible for eviction. This allows the engine to periodically clean up expired state, ensuring the pipeline remains stable and memory usage stays within the bounds of the allocated cluster resources.

Answer

Increase the cluster memory and executor count indefinitely.

Answer

Change the output mode to 'Append' to flush the state to the sink immediately.

Answer

Use the 'dropDuplicates' method instead of 'groupBy' to simplify the state.

Question 5

A data engineer needs to join a small 'lookup' table with a large 'transactions' table in a Spark job. Which transformation strategy will provide the best performance in a cluster environment?

Accepted Answer

Use the 'broadcast()' function from 'pyspark.sql.functions' on the small table.. In distributed computing, joining a large table with a small one causes a shuffle if not handled correctly. By using a broadcast join, the engineer forces the small table to be copied to every node, eliminating the need for a network-heavy shuffle of the large table. This technique is a standard optimization for improving query latency and overall throughput in data pipelines.

Answer

Increase the 'spark.sql.shuffle.partitions' value to 1000.

Answer

Sort both tables by the join key before performing the join.

Answer

Repartition both tables to the same number of partitions.

Question 6

Your team is migrating a manual Databricks job to a CI/CD pipeline. You need to ensure the job configuration is version-controlled and deployed programmatically. Which approach aligns with Databricks best practices?

Accepted Answer

Define the job configuration in YAML files using Databricks Asset Bundles and deploy via the CLI.. Using Databricks Asset Bundles (DABs) is the recommended approach for modern CI/CD. It treats infrastructure as code, allowing you to define jobs, pipelines, and workflows in YAML files. This ensures consistency across environments like development, staging, and production. By decoupling the job definition from the workspace UI, you gain auditability, reproducibility, and the ability to integrate seamlessly with Git workflows, reducing manual configuration errors and accelerating release cycles for data engineering teams.

Answer

Manually export the job JSON via the UI and copy-paste it into the production workspace.

Answer

Use the Databricks UI to update production jobs directly to ensure changes are applied immediately.

Answer

Write a custom Python script that uses the workspace API to recreate the job from scratch every time.

Question 7

Which Spark configuration property can be used to enable Adaptive Query Execution (AQE) in Databricks?

Accepted Answer

spark.sql.adaptive.enabled. Adaptive Query Execution (AQE) is a critical optimization framework in Spark that re-optimizes query plans at runtime based on statistics collected during shuffle stages. It enables features like coalescing shuffle partitions and dynamically switching join strategies. Enabling AQE is a best practice for modern Databricks workloads as it provides significant performance gains without manual intervention, automatically adapting to varying data distributions in production pipelines.

Answer

spark.sql.autoBroadcastJoinThreshold

Answer

spark.databricks.adaptive.execution

Answer

spark.sql.shuffle.partitions

Question 8

Which of the following is a recommended strategy for managing library dependencies in a CI/CD pipeline for Databricks?

Accepted Answer

Include a requirements.txt file and install libraries as part of the job definition.. Managing libraries through a centralized repository or a requirements.txt file ensures that the exact same versions are installed consistently across all environments. This avoids the 'dependency hell' where code fails because of incompatible library versions. By integrating dependency management into the CI/CD build process, teams can audit used versions, ensure security compliance, and guarantee that the production pipeline runs with the exact dependencies that were tested during the development phase.

Answer

Install libraries manually on the cluster via the Databricks UI.

Answer

Use the latest version of all libraries to ensure best performance.

Answer

Store library binary files directly in the Git repository.

Question 9

Which THREE strategies are recommended to improve the performance of reading from a Delta table in Databricks?

Accepted Answer

Use Z-Ordering on columns frequently used in WHERE clauses.. Optimizing reads is crucial for analytics. Techniques like Z-Ordering, Data Skipping, and Partitioning allow Spark to ignore irrelevant data files, reducing I/O. Proper file sizing ensures that Spark can efficiently read data in parallel. These strategies together minimize the amount of data scanned and transferred across the network, leading to significantly faster query results in large-scale data lake environments.

Answer

Partition the table by every column used in the query.

Answer

Always set the broadcast join threshold to -1.

Question 10

Refer to the exhibit. A CI/CD pipeline running a Databricks CLI command fails with the error shown. What is the most likely cause?

Accepted Answer

The path provided in the deployment script does not exist in the target Databricks workspace.. This error indicates that the automation script is attempting to reference a file or directory path that is missing in the target workspace. In CI/CD, this frequently happens when paths are hardcoded for local development environments rather than being abstracted. Using relative paths or workspace-agnostic configurations within Databricks Asset Bundles prevents these environment-specific failures, ensuring that the deployment process remains robust regardless of where the code is being executed during the pipeline run.

Answer

The Databricks CLI is not installed on the runner.

Answer

The Personal Access Token has expired.

Answer

The user does not have permissions to write to the root directory.