Databricks-DE-Assoc Practice Test 16 — 15 Questions

Question 1

A data engineer manages a Lakeflow Job that runs a long-running notebook task on a job cluster. The task occasionally fails due to transient cloud storage errors, and the engineer wants the task to retry automatically without failing the entire job on the first attempt. The engineer also wants to be alerted only if all retries are exhausted. Which configuration should the engineer apply?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set the task's retry count to a value greater than zero and configure a notification for job failure.. Task-level retries re-execute a failed task automatically, so transient storage errors can be absorbed without failing the job immediately. The job is only marked failed after retries are exhausted, at which point a job failure notification alerts the team. This matches the requirement to retry silently and alert only on final failure, unlike cluster or continuous-mode settings that do not target task retries.

Answer

Enable continuous mode on the job and configure a notification for run start.

Answer

Set the cluster's autoscaling to a higher maximum and add a notification for cluster termination.

Answer

Configure a job-level timeout and enable email notifications on every task start.

Question 2

A data engineer needs to ensure that all queries against a Unity Catalog table are logged for audit purposes. The logs must include the user identity, the query text, and the timestamp. Which Databricks feature should be enabled to capture this information?

Accepted Answer

Audit logs. Audit logs are the Databricks feature designed to capture detailed activity logs, including query text, user identity, and timestamps. They are essential for security auditing and compliance. Enabling audit logs and configuring delivery to a central system meets the requirement to log all queries against Unity Catalog tables.

Answer

Query history

Answer

Unity Catalog data lineage

Answer

Delta Live Tables event log

Question 3

A data engineer is asked to ensure that all queries against a Unity Catalog table are recorded for auditing, including the identity of the user and the query text. Which Databricks feature should the engineer enable to capture this information?

Accepted Answer

Enable audit logs at the workspace level and configure delivery to a cloud storage location.. Audit logs in Databricks capture detailed events including query execution, user identity, and access to Unity Catalog objects. Delivering these logs to cloud storage or a SIEM allows security teams to review and retain records for compliance. This is the appropriate feature for recording queries against Unity Catalog tables.

Answer

Enable Unity Catalog metastore-level lineage tracking to record all query statements.

Answer

Configure a cluster log delivery to a DBFS path and parse the logs for query text.

Answer

Enable table access control (legacy) on the cluster to log all table reads.

Question 4

A data engineer is optimizing a Databricks job that processes a large dataset. The job performs a join between a large Delta table and a small dimension table, then writes the result to a Delta table. The engineer notices that the join is causing a large shuffle and wants to reduce shuffle overhead. Which two actions should the engineer take to improve performance? (Choose two.)

Accepted Answer

Enable Adaptive Query Execution (AQE) to automatically convert the join to a broadcast join if applicable.. Broadcasting the small dimension table eliminates the need to shuffle the large table, as the small table is replicated to each executor for local joins. Enabling Adaptive Query Execution allows Spark to automatically convert sort-merge joins to broadcast joins when one side is small, further reducing shuffle. Together, these actions minimize shuffle overhead and improve join performance.

Answer

Repartition the large Delta table on the join key before the join.

Answer

Increase the number of shuffle partitions to distribute the join workload more evenly.

Answer

Cache the large Delta table in memory before the join to speed up reading.

Question 5

A data engineer is using Databricks Jobs to run a nightly ETL pipeline. The job occasionally fails due to a transient network error when writing to an external database. The engineer wants to automatically retry the job a few times before marking it as failed. What is the most efficient way to configure this in Databricks?

Accepted Answer

Configure the job to retry on failure with a specified number of retries and interval.. Databricks Jobs offer a built-in retry policy that automatically re-runs a failed job a specified number of times with a defined interval. This is ideal for handling transient errors like network issues. Configuring retries at the job level is efficient, requires no code changes, and integrates with job monitoring and alerting.

Answer

Use a Databricks notebook to catch exceptions and loop until the write succeeds.

Answer

Set the job's timeout to a higher value to give the write operation more time to complete.

Answer

Set the job's maximum concurrent runs to 3 to allow multiple attempts.

Question 6

A data engineer is investigating why a Databricks job that writes to a Delta table is experiencing performance degradation over time. The job performs frequent small appends. Which TWO actions should the engineer take to improve write performance? (Choose two.)

Accepted Answer

Run OPTIMIZE on the Delta table to compact small files.. Frequent small appends create many small files, which degrade performance. Running OPTIMIZE compacts these files into larger ones, and enabling auto compaction automates this process. Together, they reduce the number of files and improve write and read efficiency. These are standard Delta Lake maintenance practices.

Answer

Use partitioning on a high-cardinality column.

Answer

Increase the number of shuffle partitions to 2000.

Answer

Set spark.sql.adaptive.enabled to true.

Question 7

A data engineer is managing a Unity Catalog table that contains sensitive financial data. The table is owned by the 'finance' group, and the data engineer needs to allow the 'auditors' group to read the table but not modify it. Additionally, the data engineer wants to ensure that the 'auditors' group can see the table's metadata (e.g., column names and types) but cannot access the underlying data files directly. Which TWO actions should the data engineer take to meet these requirements? (Choose two.)

Accepted Answer

Grant USAGE on the schema containing the table to the 'auditors' group.. To allow read-only access, the data engineer must grant SELECT on the table and USAGE on the containing schema. SELECT provides read access and metadata visibility, while USAGE is required to access any object within the schema. Granting MODIFY or ALL PRIVILEGES would allow modifications, and READ FILES would bypass table-level security. These two privileges together satisfy the requirements without over-provisioning.

Answer

Grant READ FILES on the external location to the 'auditors' group.

Answer

Grant MODIFY on the table to the 'auditors' group.

Answer

Grant ALL PRIVILEGES on the table to the 'auditors' group.

Question 8

A data engineer is monitoring a Databricks job that runs a Structured Streaming query. The engineer notices that the query's input rate is high, but the processing rate is low, and the batch duration is increasing over time. The query uses a Delta table as a source and writes to another Delta table. Which action should the engineer take to improve the streaming query's performance?

Accepted Answer

Tune the source Delta table by running OPTIMIZE to compact small files.. For a Structured Streaming query reading from a Delta table, many small files can cause high read overhead per micro-batch, leading to low processing rates and increasing batch durations. Compacting the source table with OPTIMIZE reduces the number of files, improving read efficiency. Other options either do not address the root cause or could exacerbate the issue.

Answer

Increase the trigger interval to process larger batches less frequently.

Answer

Enable Delta Lake optimized writes by setting spark.databricks.delta.optimizeWrite.enabled to true.

Answer

Increase the number of shuffle partitions to improve parallelism during processing.

Question 9

A data engineer is configuring Unity Catalog to allow a service principal to run a Databricks job that writes to a Delta table in an external location. The service principal has been granted USAGE on the catalog and schema, and MODIFY on the table. However, the job fails with an error indicating that the service principal cannot access the external location. What is the most likely missing privilege?

Accepted Answer

The service principal needs WRITE FILES on the external location.. For an external table, writing data requires both MODIFY on the table and WRITE FILES on the external location. The service principal has MODIFY but lacks WRITE FILES, causing the access error. Granting WRITE FILES on the external location will allow the job to write the underlying data files. Other privileges like SELECT, READ FILES, or CREATE do not address the write requirement.

Answer

The service principal needs SELECT on the table.

Answer

The service principal needs READ FILES on the external location.

Answer

The service principal needs CREATE on the external location.

Question 10

A data engineer has a Delta table named `sales` with columns `sale_id`, `customer_id`, `amount`, and `sale_date`. They need to create a new table that contains only the `customer_id` and the total `amount` per customer for all sales in 2023. Which SQL statement correctly creates this aggregated table?

Accepted Answer

CREATE TABLE customer_totals AS SELECT customer_id, SUM(amount) AS total_amount FROM sales WHERE YEAR(sale_date) = 2023 GROUP BY customer_id;. The correct approach uses a CREATE TABLE AS SELECT statement with a WHERE clause filtering for 2023 sales before aggregation, followed by GROUP BY customer_id and SUM(amount). This materializes the desired summary table. The other options either misuse HAVING, omit GROUP BY, or add an extra grouping column that changes the granularity of the result.

Answer

CREATE TABLE customer_totals AS SELECT customer_id, SUM(amount) AS total_amount FROM sales WHERE YEAR(sale_date) = 2023 GROUP BY customer_id, sale_date;

Answer

CREATE TABLE customer_totals AS SELECT customer_id, SUM(amount) AS total_amount FROM sales WHERE YEAR(sale_date) = 2023;

Answer

CREATE TABLE customer_totals AS SELECT customer_id, SUM(amount) AS total_amount FROM sales GROUP BY customer_id HAVING YEAR(sale_date) = 2023;

Question 11

A data engineer needs to ingest a large CSV file from cloud storage into a Delta table using Databricks SQL. The engineer wants to perform a one-time load and ensure that the operation is atomic. Which command should be used?

Accepted Answer

`COPY INTO`. `COPY INTO` is the recommended command for loading files from cloud storage into Delta tables. It provides atomicity, idempotency, and file-level tracking, making it perfect for one-time or incremental batch ingestion. It also supports schema evolution and validation.

Answer

`CREATE TABLE AS SELECT` from the CSV path

Answer

`INSERT INTO` with a `SELECT * FROM csv.` path ``

Answer

`MERGE INTO` using the CSV as source

Question 12

A data engineer is using Databricks Auto Loader to ingest JSON files from a directory. The stream is configured with `cloudFiles.schemaLocation` set to a specific path. The engineer notices that the stream fails when a new file contains an additional column. What is the most likely reason for the failure?

Accepted Answer

The `cloudFiles.schemaEvolutionMode` is set to `failOnNewColumns`.. Auto Loader's `schemaEvolutionMode` controls how new columns are handled. Setting it to `failOnNewColumns` causes the stream to fail when a new column is encountered, requiring manual intervention. To allow automatic schema evolution, the mode should be set to `addNewColumns`.

Answer

The `cloudFiles.inferColumnTypes` option is set to `false`, causing schema inference to fail.

Answer

The `cloudFiles.useNotifications` option is disabled, so new files are not detected.

Answer

The schema location is not writable, so Auto Loader cannot update the schema.

Question 13

A data engineer needs to ensure that only members of the 'finance' group can view a specific column containing credit card numbers in a Unity Catalog table. Which feature should be used?

Accepted Answer

Column masks. Column masks are designed to control access to specific columns by applying a masking function that evaluates the user's identity or group membership. This allows sensitive data like credit card numbers to be visible only to authorized groups. Other features like row-level security or dynamic views address different aspects of data access and are not the direct solution for column-level control.

Answer

Column-level privileges with GRANT SELECT ON COLUMN

Answer

Row-level security with filter conditions

Answer

Dynamic views with CASE statements

Question 14

A data engineer is designing a Delta Live Tables (DLT) pipeline. They need to ensure that records with missing values in the 'customer_id' column are dropped during the ingestion process. Which constraint syntax should be used?

Accepted Answer

@expect_or_drop(customer_id IS NOT NULL). The EXPECT DROP VIOLATION constraint is a native DLT feature designed specifically for data quality enforcement. By applying this, the pipeline automatically discards rows that fail the specified predicate while allowing valid records to proceed. This approach is essential in production data engineering to maintain data integrity and prevent downstream errors caused by null values, ensuring only high-quality data enters the silver or gold tables.

Answer

@expect_or_fail(customer_id IS NOT NULL)

Answer

@expect(customer_id IS NOT NULL)

Answer

@filter(customer_id IS NOT NULL)

Question 15

Refer to the exhibit. A security administrator applies this Unity Catalog policy. What is the impact on users in the 'analyst-group'?

Accepted Answer

They can view the contents of the revenue table.. This Unity Catalog JSON policy grants the 'analyst-group' the SELECT privilege on the 'main.sales.revenue' table. Consequently, any user belonging to this group will be able to read data from this specific table. This is a standard governance practice in Databricks, providing a fine-grained, centralized method to control access to sensitive data assets across the entire organization, ensuring security and compliance without managing individual permissions for each user.

Answer

They can drop the revenue table.

Answer

They gain administrative rights over the entire catalog.

Answer

They can write new data into the revenue table.