Databricks-DE-Assoc Practice Test 15 — 15 Questions

Question 1

A data engineer needs to allow a service principal to read data from a Unity Catalog table `sales.orders` and also write to a volume `sales.raw_data`. Which set of privileges should be granted to the service principal?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

SELECT on table sales.orders, and WRITE VOLUME on volume sales.raw_data.. To read from a Unity Catalog table, the service principal needs SELECT on the table. To write to a volume, it needs WRITE VOLUME on the volume. These privileges are the minimal required to perform the specified actions, adhering to the principle of least privilege. Other options either grant insufficient access to the volume or excessive access to the table.

Answer

ALL PRIVILEGES on table sales.orders, and WRITE VOLUME on volume sales.raw_data.

Answer

SELECT on table sales.orders, and READ VOLUME on volume sales.raw_data.

Answer

MODIFY on table sales.orders, and WRITE VOLUME on volume sales.raw_data.

Question 2

A data engineer has a Lakeflow Job with two tasks: Task1 and Task2. Task2 must run only if Task1 succeeds. The engineer also wants Task2 to be skipped if Task1 fails, but the overall job status should be marked as failed. Which configuration should the engineer use for the dependency between Task1 and Task2?

Accepted Answer

Set Task2 to depend on Task1 with the 'All succeeded' condition.. The 'All succeeded' condition ensures Task2 runs only when Task1 succeeds. If Task1 fails, Task2 is skipped, and the job is marked as failed because a task failed. This precisely meets the requirement of conditional execution and overall job failure.

Answer

Set Task2 to depend on Task1 with the 'All done' condition.

Answer

Set Task2 to depend on Task1 with the 'At least one succeeded' condition.

Answer

Set Task2 to depend on Task1 with the 'None failed' condition.

Question 3

A data engineer is building a Lakeflow Job that must run a sequence of tasks across different compute types. The ingest task must run on a job cluster with a specific Spark configuration, the transform task must run as a Delta Live Tables pipeline, and the report task must run on a separate SQL warehouse. Which TWO statements about task-level compute configuration in Lakeflow Jobs are correct? (Choose two.)

Accepted Answer

A Delta Live Tables pipeline task and a SQL warehouse task can be combined in the same Lakeflow Job with dependencies between them.. Lakeflow Jobs are designed to orchestrate heterogeneous tasks, and each task can specify its own compute. This means a job can include a notebook task on a job cluster, a Delta Live Tables pipeline task, and a SQL warehouse task, with dependencies that control execution order. The two correct statements reflect that per-task compute is supported and that tasks with different compute types can be combined with dependencies.

Answer

A SQL warehouse task can only run if the job uses an all-purpose cluster for all other tasks.

Answer

Task-level compute configuration is only available for notebook tasks, not for DLT pipeline or SQL tasks.

Answer

All tasks in a Lakeflow Job must share the same compute; task-level compute is not supported.

Question 4

A data engineer is configuring a Lakeflow Job that processes sensitive customer data. The job must notify the on-call team when a run fails and must also capture the run's output for auditing. Which TWO actions should the engineer take in the Lakeflow Jobs configuration? (Choose two.)

Accepted Answer

Add an email notification for job failure that targets the on-call distribution list.. Failure notifications deliver timely alerts to the on-call team, and preserving run history with exported logs provides the audit trail. Together these native Lakeflow Jobs capabilities meet both the alerting and auditing requirements without adding unnecessary concurrency or retry complexity that would not notify or record anything on its own.

Answer

Configure a retry policy with three retries to ensure the on-call team is notified.

Answer

Add a task-level timeout so the job fails faster and generates an alert.

Answer

Set the job's maximum concurrent runs to 1 to prevent overlapping audit records.

Question 5

A data engineer has a Lakeflow Job with three tasks: bronze_ingest, silver_transform, and gold_aggregate. The silver_transform task must run only if bronze_ingest succeeds, and gold_aggregate must run only if silver_transform succeeds. The engineer also wants gold_aggregate to run even if silver_transform fails, so that partial results can be published. Which configuration should the engineer apply to gold_aggregate?

Accepted Answer

Set gold_aggregate's depends_on to include silver_transform and set run_if to ALL_DONE.. Ordering and run conditions are separate controls. depends_on places gold_aggregate after silver_transform, and run_if set to ALL_DONE allows the task to execute once upstream tasks reach a terminal state, whether they succeeded or failed. This is the standard pattern for publishing partial results or running cleanup tasks after a failure.

Answer

Remove depends_on from gold_aggregate and set run_if to ALL_DONE.

Answer

Set gold_aggregate's depends_on to include bronze_ingest and set run_if to AT_LEAST_ONE_SUCCESS.

Answer

Set gold_aggregate's depends_on to include silver_transform and set run_if to ALL_SUCCESS.

Question 6

A data engineer is investigating a Databricks job that failed overnight. The job's status in the Jobs UI shows 'Failed', and the engineer needs to view the error message and stack trace to determine the cause. Where should the engineer look to find the detailed error information for the failed run?

Accepted Answer

In the driver logs, accessible from the run's detail page in the Jobs UI.. Driver logs are the definitive source for application-level errors in Databricks jobs. They capture stdout and stderr from the driver, including full stack traces. The Jobs UI provides direct access to these logs from the run's detail page, making it the first place to check when a job fails due to an exception.

Answer

In the cluster's event log, which records all cluster scaling and termination events.

Answer

In the Databricks audit logs, which record user actions and API calls.

Answer

In the Spark UI's SQL tab, which shows the query plan and execution metrics.

Question 7

A data engineer notices that a Databricks job processing a Delta table with 10,000 partitions runs slowly. The job filters on a column that is not the partition column, and the query plan shows that all partitions are being scanned. The engineer wants to improve performance without repartitioning the table. Which feature should be used?

Accepted Answer

Z-ORDER BY on the filter column during OPTIMIZE. When a filter is on a non-partition column, Delta Lake can still skip files if it has min/max statistics for that column and the data is clustered. Z-ORDER BY reorganizes data so that related values are stored together, improving the effectiveness of data skipping. Running OPTIMIZE with Z-ORDER BY on the filter column allows the query to read only the relevant files, avoiding a full scan of all partitions.

Answer

Enabling Change Data Feed on the table

Answer

Setting delta.dataSkippingNumIndexedCols to 0

Answer

Increasing the number of shuffle partitions

Question 8

A data engineer is configuring audit logging for a Databricks workspace that uses Unity Catalog. The security team requires that all access to data in Unity Catalog be logged and available for analysis in a centralized location. The data engineer wants to enable the delivery of audit logs to an AWS S3 bucket. Which configuration should the data engineer use?

Accepted Answer

Configure an audit log delivery in the account console, specifying the S3 bucket and IAM role for delivery.. Databricks audit logs, including Unity Catalog data access, can be delivered to an AWS S3 bucket through account-level configuration. This involves setting up a delivery mechanism with an IAM role that has write permissions to the bucket. The account console provides the interface for this setup, ensuring centralized log collection.

Answer

Enable diagnostic logging in the Databricks workspace and configure it to send logs to S3.

Answer

Enable Unity Catalog audit logs in the workspace settings and specify an S3 path for storage.

Answer

Use the Databricks REST API to stream audit logs to an S3 bucket via a custom script.

Question 9

A data engineer is configuring a Unity Catalog storage credential to access an AWS S3 bucket. The S3 bucket policy grants access to an IAM role. The data engineer creates a storage credential with that IAM role's ARN. However, when attempting to create an external location using this storage credential, the operation fails with an error indicating insufficient permissions. The data engineer verifies that the IAM role has the correct S3 permissions. What is the most likely cause of the failure?

Accepted Answer

The IAM role's trust policy does not allow the Databricks AWS account to assume the role.. When creating a storage credential, Databricks must be able to assume the IAM role. This requires a trust relationship between the IAM role and the Databricks AWS account. Even if the role has correct S3 permissions, without a proper trust policy, Databricks cannot assume the role, causing external location creation to fail. The trust policy is a separate configuration from permissions.

Answer

The S3 bucket policy does not grant the necessary permissions to the IAM role.

Answer

The storage credential was created without the `READ FILES` privilege on the S3 bucket.

Answer

The storage credential was created without the `CREATE EXTERNAL LOCATION` privilege on the metastore.

Question 10

A data engineer configures a Lakeflow Job to run a notebook task on a job cluster. The notebook reads a parameter named run_date using the widget API. During a manual run, the engineer wants to supply a specific date without editing the notebook. The job also runs on a nightly schedule where the date should default to the current day. Which approach correctly supplies the parameter for both the manual and scheduled runs?

Accepted Answer

Define a job parameter with a default value and pass it to the notebook task, then override it when starting a manual run.. Job parameters with default values let a task receive a value automatically on schedule, while a manual run can override the same parameter at submission. The notebook consumes it via the widget API, so no code changes are needed between run modes. This is the standard Lakeflow Jobs pattern for dynamic per-run inputs and avoids hard-coding or external stores.

Answer

Hard-code the date inside the notebook and create a second notebook for manual runs.

Answer

Configure the notebook to read the date from a Unity Catalog table that the engineer updates before manual runs.

Answer

Use a cluster environment variable to store the date and change it before each manual run.

Question 11

A data engineer has a Unity Catalog table `prod.sales.orders` containing a column `customer_email`. Company policy requires that users in the `analyst_group` see only the domain part of the email (e.g., `***@example.com`), while members of `pii_admin_group` must see the full email. The engineer wants to enforce this at query time without creating separate views. Which approach should the engineer use?

Accepted Answer

Create a column mask function that returns the full email for `pii_admin_group` and a masked email otherwise, then apply it to the `customer_email` column.. Column masks in Unity Catalog let you attach a user-defined function to a specific column so that different users see different values for the same data. The function can inspect `is_account_group_member()` to return the raw email for the privileged group and a masked string for everyone else. This enforces the policy dynamically at query time and avoids creating separate views or duplicating the table.

Answer

Apply a table-level tag `pii` to `prod.sales.orders` and configure a tag-based policy that automatically masks all string columns for non-admin users.

Answer

Grant `SELECT` on `prod.sales.orders` only to `pii_admin_group` and grant `SELECT` on a view that excludes `customer_email` to `analyst_group`.

Answer

Create a row filter function that returns TRUE for `pii_admin_group` and FALSE for others, then apply it to `prod.sales.orders`.

Question 12

A data engineer has a Lakeflow Job that runs daily. They want to receive an email only when the job fails, not on every run. Which notification configuration should they set?

Accepted Answer

Add an email notification with the condition 'On failure' for the job.. Notification conditions in Lakeflow Jobs determine when alerts are sent. To receive an email only when the job fails, the condition must be set to 'On failure'. Other conditions such as 'On success', 'On start', or 'On all events' would send notifications at times that do not match the failure-only requirement, leading to unnecessary alerts.

Answer

Add an email notification with the condition 'On all events' for the job.

Answer

Add an email notification with the condition 'On success' for the job.

Answer

Add an email notification with the condition 'On start' for the job.

Question 13

A data engineer runs a nightly Databricks job that reads from a large Delta table and writes results to another Delta table. The job has been taking progressively longer each night. The engineer examines the Spark UI and sees that the 'SQL' tab shows a single stage with many small tasks, each processing very few records, and the 'Storage' tab shows that the source table has thousands of small files. Which action should the engineer take to improve performance?

Accepted Answer

Run OPTIMIZE on the source Delta table to compact small files.. The Spark UI reveals many small tasks and thousands of small files in the source Delta table, which is a small-file problem. OPTIMIZE compacts these small files into larger ones, reducing task overhead and improving read performance. Other options either do not address the root cause or could exacerbate the issue by creating more partitions.

Answer

Enable Adaptive Query Execution (AQE) by setting spark.sql.adaptive.enabled to true.

Answer

Increase the number of shuffle partitions by setting spark.sql.shuffle.partitions to a higher value.

Answer

Repartition the source DataFrame to a smaller number of partitions before writing.

Question 14

A data engineer is monitoring a Databricks job and notices that the job's tasks are spending a significant amount of time in garbage collection (GC). The job processes large amounts of data with many small objects. Which action should the engineer take to reduce GC overhead?

Accepted Answer

Use the Kryo serializer and optimize data structures to reduce object creation.. High garbage collection overhead often results from creating many short-lived objects. Using the Kryo serializer reduces the memory footprint of serialized data, and optimizing data structures to minimize object creation directly reduces the number of objects that the GC must manage. This leads to less frequent and shorter GC pauses.

Answer

Set spark.executor.extraJavaOptions to -XX:+UseG1GC.

Answer

Decrease the number of partitions to reduce task overhead.

Answer

Increase the executor memory and enable off-heap memory.

Question 15

A data engineer is investigating why a Databricks job that reads from a Delta table is slow. The job performs a simple SELECT with a filter on a partition column. The engineer suspects that the table has many small files. Which Spark UI tab should be examined to confirm the number of files read?

Accepted Answer

Stages tab. The Stages tab provides per-stage metrics including the number of tasks, input size, and records read. For a scan stage, the number of tasks is often proportional to the number of files or partitions. A high task count with small input size per task suggests many small files. This makes the Stages tab the most direct place to confirm the small-file issue.

Answer

SQL tab

Answer

Environment tab

Answer

Storage tab