Databricks-DE-Assoc Practice Test 20 — 15 Questions

Question 1

A data engineering team stores all production notebooks and job definitions in a Git repository. They want every merge to the `main` branch to automatically deploy the updated notebooks to the production Databricks workspace without any manual copy/paste. Which approach should they implement?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure a Databricks Repo in the production workspace that tracks the `main` branch, and run a CI/CD pipeline that calls the Repos API to update the repo to the latest commit after each merge.. Deploying from Git requires the production workspace to consume the repository as the source of truth. A Databricks Repo that tracks the `main` branch, updated by a CI/CD pipeline through the Repos API after each merge, achieves automated, traceable deployment of notebooks and job definitions without manual copying.

Answer

Schedule a Databricks Job that executes `git pull` inside a notebook on a fixed cron schedule.

Answer

Enable automatic Git synchronization on the production workspace so Databricks pulls changes from the repository on its own.

Answer

Use the Databricks CLI to export notebooks from the development workspace and import them into production on every merge.

Question 2

A data engineer is creating a Silver table in a Delta Live Tables pipeline. The pipeline must continuously ingest new files from a cloud storage location as they arrive, and the engineer wants to avoid reprocessing files that were already ingested. Which approach should be used to read the source data?

Accepted Answer

Use Auto Loader with cloudFiles as the streaming source and let it track discovered files in its state store.. Auto Loader with cloudFiles is the purpose-built streaming source for incremental ingestion from cloud storage. It discovers new files, records processed files in a persistent state store so they are not read twice, and integrates with Delta Live Tables streaming tables. This provides continuous ingestion with exactly-once processing semantics and optional schema evolution.

Answer

Use dbutils.fs.ls(path) in a loop and append new file paths to a Delta table before reading them.

Answer

Define the source as a batch table and schedule the pipeline to run every minute with a full refresh.

Answer

Use spark.read.format("parquet").load(path) inside a streaming table definition.

Question 3

Refer to the exhibit. An engineer notices that queries filtering by 'customer_id' are running slowly on the 'orders' table. Based on the exhibit, what is the most appropriate action to resolve this?

Accepted Answer

Execute 'OPTIMIZE orders ZORDER BY (customer_id, order_date)'.. The exhibit shows that the Z-Ordering configuration is present but the status is 'PENDING', suggesting that an optimization has been defined but not executed. Running the 'OPTIMIZE' command is a standard maintenance task to physically reorganize the data files based on the specified columns. This improves query performance by enabling data skipping for queries filtering on 'customer_id', which is essential for responsive dashboard performance.

Answer

Run the 'VACUUM' command on the 'orders' table immediately.

Answer

Drop the table and re-ingest the data with a different partition strategy.

Answer

Update the 'spark.sql.shuffle.partitions' setting to 2000.

Question 4

A data engineer is building a Gold aggregate table that summarizes daily sales by product category. The Silver source is a streaming Delta table that receives late-arriving events up to 48 hours old. The engineer needs the Gold table to always reflect the most accurate aggregates, including corrections for late data, without full recomputation. Which approach is most appropriate?

Accepted Answer

Use `foreachBatch` with a MERGE that upserts the recomputed aggregates for affected date and category keys from the latest micro-batch.. `foreachBatch` combined with a keyed MERGE gives per-micro-batch control, so only aggregates for keys touched by the batch are recomputed and upserted. This accommodates late-arriving events within the batch window without rewriting the entire Gold table. Complete mode, append mode, or watermark-bounded update mode either recompute everything or fail to persist corrections durably.

Answer

Use a watermark of 48 hours with `update` output mode to emit only changed aggregate rows.

Answer

Use a Structured Streaming query with `complete` output mode writing to the Gold Delta table.

Answer

Use `append` output mode and let the BI layer sum the raw daily rows at query time.

Question 5

Refer to the exhibit. If 'task1' fails due to a timeout, what happens to 'task2'?

Accepted Answer

It will be skipped by the job scheduler.. In Databricks Jobs, dependencies are strictly evaluated based on the success of preceding tasks. If a task fails—regardless of whether it was a timeout or a code error—the dependent tasks in the directed acyclic graph (DAG) will not be triggered. This behavior ensures that downstream data quality is protected from incomplete or stale upstream data processing, which is a fundamental requirement for maintaining reliable data pipelines.

Answer

It will run anyway because it is independent.

Answer

It will pause until task1 is restarted.

Answer

It will run with a 'warning' status.

Question 6

Which command is used to query the history of a Delta table to perform time travel?

Accepted Answer

DESCRIBE HISTORY table_name. The 'DESCRIBE HISTORY' command provides a detailed log of all transactions performed on a Delta table, including operations like inserts, updates, and deletes. This is essential for auditing and reverting to previous states if data corruption or accidental deletions occur. By understanding the history, engineers can verify the lineage and integrity of their data, which is a core benefit of the Delta Lake storage layer.

Answer

SELECT * FROM TABLE_HISTORY

Answer

SHOW LOGS FROM table_name

Answer

GET_DELTA_VERSIONS(table_name)

Question 7

Which of the following is the best practice for managing service principals in a Databricks workspace?

Accepted Answer

Assign the service principal a dedicated identity with scoped permissions.. Service principals are non-human identities used for automated processes. The best practice is to assign them only the minimum privileges required for the task (least privilege), store their credentials in a secure secrets manager (like Databricks Secrets or Azure Key Vault), and treat them as distinct entities from human users to ensure that automated jobs are isolated and easily auditable.

Answer

Use the workspace admin's personal access token for all automated jobs.

Answer

Store service principal credentials in plaintext variables within notebook code.

Answer

Grant the service principal 'Admin' rights to avoid configuration errors.

Question 8

Refer to the exhibit. A data engineer is configuring a streaming pipeline. Which outcome will these specific configurations have on the target table's performance?

Accepted Answer

It will prevent the creation of small files, resulting in faster downstream read performance.. These settings are crucial for maintaining the performance of Delta tables that receive constant small writes. Auto-compacting and optimized writes prevent the 'small file problem,' where thousands of tiny files degrade read performance. By configuring these in a streaming context, the engineer ensures the table remains query-efficient over time, which is essential for BI users who require fast dashboard load times from the target Delta tables.

Answer

It will increase the latency of the streaming write significantly.

Answer

It will reduce the storage cost by automatically deleting old files.

Answer

It will force the pipeline to process data in batches rather than continuously.

Question 9

A data engineer is using Databricks SQL to analyze data stored in a Delta table. The engineer wants to optimize query performance by leveraging Delta Lake features. Which TWO actions should the engineer take to improve query performance on the Delta table? (Choose two.)

Accepted Answer

Run the OPTIMIZE command to compact small files.. OPTIMIZE compacts small files into larger ones, reducing I/O and improving data skipping. Z-ORDER BY colocates data on a specified column, further enhancing data skipping for queries that filter on that column. Together, these Delta Lake features directly improve query performance by optimizing the physical layout of data. Other options either do not directly address file layout or would remove Delta Lake benefits.

Answer

Convert the table to Parquet format.

Answer

Enable auto-compaction on the Delta table.

Answer

Increase the cluster size by adding more workers.

Question 10

Which Databricks feature should a data engineer use to view the execution plan, including information about the physical operators and data lineage, to troubleshoot a slow-running SQL query?

Accepted Answer

The Spark UI SQL tab.. The Spark UI (specifically the SQL tab) provides a comprehensive graphical and textual representation of the query execution plan. By examining the 'Analyzed Plan' and 'Physical Plan', engineers can identify bottlenecks like expensive joins, full table scans, or lack of pruning. Mastering these diagnostic tools is essential for optimizing query performance and ensuring that execution logic aligns with the intended data processing requirements in the Databricks environment.

Answer

The Delta Lake History logs.

Answer

The Cluster Metrics dashboard.

Answer

The Data Explorer.

Question 11

A data engineer needs to monitor the costs associated with specific projects running on a shared Databricks workspace. Which feature should the engineer use to attribute these costs accurately?

Accepted Answer

Cluster Tags. Tags allow for granular cost tracking by applying metadata to Databricks resources like clusters. By using custom tags, organizations can categorize spending by project, department, or cost center. This is vital for financial accountability in multi-tenant environments. When billing records are exported, these tags are included, enabling precise allocation of expenditures back to the specific project owners or internal clients, ensuring transparency and efficient budget management across the organization.

Answer

Notebook Versioning

Answer

Delta Lake Audit Logs

Answer

Databricks SQL Alerts

Question 12

A data engineer is optimizing a Delta table that suffers from slow read performance due to small file sizes. Which command should the engineer execute to consolidate these small files into larger, more efficient files without altering the underlying table data?

Accepted Answer

OPTIMIZE table_name. The OPTIMIZE command is specifically designed for compacting small files into larger ones, which improves read performance by reducing metadata overhead and increasing scan efficiency. This operation is essential in Databricks environments where frequent streaming or batch writes create many small files, leading to 'small file syndrome.' By restructuring the data files while maintaining the table's logical state, OPTIMIZE ensures that downstream queries scan fewer, more optimized data blocks.

Answer

ALTER TABLE table_name REORGANIZE

Answer

VACUUM table_name

Answer

ANALYZE TABLE table_name COMPUTE STATISTICS

Question 13

A data engineer is optimizing a Databricks job that reads from a large Delta table and performs a join with a smaller table. The job is experiencing performance issues due to shuffling. The engineer wants to reduce the amount of data shuffled during the join. Which technique should the engineer use?

Accepted Answer

Broadcast the smaller table using a broadcast hint.. Broadcasting the smaller table eliminates the need to shuffle the larger table. The smaller table is replicated to all worker nodes, and the join is performed locally, which drastically reduces network I/O and improves performance. This is the most effective technique when one side of the join is small enough to fit in memory.

Answer

Increase the number of shuffle partitions to 2000.

Answer

Enable Adaptive Query Execution (AQE) and set spark.sql.adaptive.enabled to true.

Answer

Repartition the larger table on the join key before the join.

Question 14

A data engineer is configuring a storage credential in Unity Catalog to access an AWS S3 bucket. The engineer creates an IAM role with the necessary permissions and sets up the storage credential using the role ARN. What additional step is required to allow Databricks to assume the role?

Accepted Answer

Modify the role's trust policy to allow the Databricks AWS account to assume it.. For Databricks to assume an IAM role, the role's trust policy must explicitly allow the Databricks AWS account to assume it. This is a critical step in setting up a storage credential. The trust policy defines which principals can assume the role, and without it, the assumption fails regardless of the role's permissions.

Answer

Enable AWS IAM Identity Center and configure SAML federation for the role.

Answer

Create an access key and secret key for the IAM role and store them in Databricks secrets.

Answer

Attach an IAM policy to the role that grants sts:AssumeRole to the Databricks AWS account.

Question 15

A data engineer stores a Delta table in Unity Catalog at the managed location of the schema 'sales'. The table is later dropped using DROP TABLE. What happens to the underlying data files?

Accepted Answer

The data files are deleted from the managed storage location.. In Unity Catalog, managed tables have their data lifecycle controlled by the metastore. When you drop a managed table, the system deletes the data files from the managed storage location. This contrasts with external tables, where the data remains in the external location. Understanding this behavior is critical for data retention and cost management, as dropping a managed table permanently removes the data.

Answer

The data files remain in the managed storage location and can be re-registered later.

Answer

The data files are deleted after the default retention period of 30 days.

Answer

The data files are moved to a recycle bin and can be restored by an account admin.