Databricks-DE-Assoc Practice Test 19 — 15 Questions

Question 1

A data engineer is configuring a Databricks Auto Loader stream to ingest CSV files from a cloud storage location into a Delta table. The CSV files have a header row, and the engineer wants to automatically infer the schema and store the inferred schema in a specified location for consistency across restarts. Which Auto Loader option should be used to persist the inferred schema?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set cloudFiles.schemaLocation to a directory path.. Auto Loader requires a schemaLocation to persist the inferred schema and support schema evolution. Without it, the stream may re-infer the schema on each restart, causing inconsistencies. The other options control schema evolution behavior or listing optimizations but do not provide a storage location for the schema.

Answer

Set cloudFiles.inferColumnTypes to 'true'.

Answer

Set cloudFiles.schemaEvolutionMode to 'addNewColumns'.

Answer

Set cloudFiles.useIncrementalListing to 'true'.

Question 2

A data engineering team runs a nightly Lakeflow Job that ingests files from cloud storage, transforms them with a notebook, and then runs a SQL task. The team wants the SQL task to execute only after the notebook transform succeeds, but they do not want the SQL task to wait for a fixed delay. Which Lakeflow Jobs feature should they configure on the SQL task?

Accepted Answer

Set the SQL task's depends_on property to reference the notebook transform task.. The depends_on property is the native Lakeflow Jobs mechanism for expressing task-level dependencies. By setting depends_on on the SQL task to point at the notebook transform task, the SQL task is scheduled only after the transform completes successfully. This avoids fixed delays, avoids polling, and keeps the pipeline deterministic and observable within a single job run.

Answer

Configure a time-based trigger on the SQL task using a cron expression that starts two minutes after the job run begins.

Answer

Add a run_if condition to the SQL task that checks whether the transform task's output path exists.

Answer

Place the SQL task and the notebook transform task in separate jobs and chain them using a job-level trigger.

Question 3

A data engineer needs to grant the `analyst` group the ability to read data from a Unity Catalog table `sales.fact_orders`. They also want to ensure that members of `analyst` can see the table in the catalog explorer but cannot modify it. Which privilege should be granted to the `analyst` group on the table?

Accepted Answer

GRANT SELECT ON TABLE sales.fact_orders TO `analyst`;. To allow read-only access to a Unity Catalog table, the SELECT privilege is granted. It permits querying and viewing metadata without modification rights. MODIFY would allow changes, ALL PRIVILEGES is excessive, and USAGE is not applicable to tables. SELECT is the least privilege that meets the requirement.

Answer

GRANT USAGE ON TABLE sales.fact_orders TO `analyst`;

Answer

GRANT MODIFY ON TABLE sales.fact_orders TO `analyst`;

Answer

GRANT ALL PRIVILEGES ON TABLE sales.fact_orders TO `analyst`;

Question 4

A team is designing a CI/CD pipeline using Databricks Asset Bundles (DABs). Which TWO of the following are primary benefits of using DABs for managing Databricks projects?

Accepted Answer

DABs enable consistent deployment of jobs and pipelines across different environments.. Databricks Asset Bundles represent the modern standard for orchestrating Databricks projects as code. By codifying infrastructure and application configuration into YAML, teams achieve environment parity across development, staging, and production. This approach reduces manual configuration errors, simplifies deployment automation via CLI-based workflows, and provides a version-controlled foundation for managing complex data assets, which is critical for large-scale enterprise data engineering environments.

Answer

DABs automatically provision cloud-native IAM roles for all users.

Answer

DABs provide built-in visual drag-and-drop tools for building ETL pipelines.

Answer

DABs bypass the need for any authentication tokens during deployment.

Question 5

A data engineer is designing a pipeline using Structured Streaming to ingest data into Delta Lake. Which THREE benefits are provided by using checkpoints in this scenario?

Accepted Answer

Enabling the stream to resume from the exact point of failure.. Checkpoints are a fundamental feature of Structured Streaming that enable fault tolerance and exactly-once processing. They store the current state and progress of a stream, allowing it to recover from failures without data loss or duplication. In the context of Delta Lake, checkpoints ensure that the ingestion process remains reliable and consistent across various execution cycles.

Answer

Providing the ability to recover the previous version of the table.

Answer

Automatically cleaning up old data files in the target directory.

Question 6

A job is failing with a 'Disk Space' error on the worker nodes. The code performs several large joins. Which configuration should the engineer adjust to mitigate the disk space usage?

Accepted Answer

Increase spark.sql.shuffle.partitions.. Large joins often require spilling to disk when the shuffle data exceeds the available executor memory. Adjusting the spark.sql.shuffle.partitions configuration can reduce the size of individual partitions, potentially fitting them within memory. Alternatively, increasing the instance type memory or optimizing the join condition reduces the reliance on local disk storage. Managing shuffle partitions is a standard practice to balance memory pressure against the risk of disk overflow during shuffles.

Answer

Decrease spark.driver.memory.

Answer

Set spark.databricks.io.cache.enabled to false.

Answer

Increase the number of task retries.

Question 7

When implementing a Medallion Architecture, what is the primary purpose of the 'Silver' layer?

Accepted Answer

To provide a cleaned and integrated source of truth for business reporting.. The Silver layer acts as the validated, cleaned, and enriched integration layer. It transforms raw Bronze data into a usable state by applying data quality checks, schema enforcement, and deduplication. This layer serves as the foundation for downstream modeling, allowing data scientists and analysts to work with reliable data without needing to handle the noise and structural inconsistencies inherent in raw data sources.

Answer

To store raw, unmodified data in its original format from source systems.

Answer

To act as a sandbox environment for ad-hoc exploration and training.

Answer

To aggregate data into pre-calculated metrics for high-performance dashboards.

Question 8

A data engineer is using Delta Live Tables to build a pipeline. They need to create a table that contains the latest record for each customer based on a `last_updated` timestamp. The source is a streaming table with append-only data. Which Delta Live Tables operation should be used to achieve this?

Accepted Answer

Use the APPLY CHANGES INTO operation with a streaming source and specify customer_id as the key and last_updated as the sequencing column.. The correct operation is APPLY CHANGES INTO, which is designed for change data capture and deduplication. By specifying the key and sequencing column, Delta Live Tables automatically retains the latest record for each customer. Other approaches either do not support streaming updates or require inefficient full reprocessing.

Answer

Use a streaming live table with a GROUP BY customer_id and MAX(last_updated) to get the latest timestamp, then join back to the source.

Answer

Use a batch live table that reads the entire source and uses a MERGE statement to upsert records.

Answer

Use a streaming live table with a window function to rank records by last_updated and filter for rank 1.

Question 9

A data engineer is using Auto Loader and wants to handle a situation where a column 'user_id' is sometimes an integer and sometimes a string in the source JSON files. Which TWO strategies can be used to manage this schema conflict?

Accepted Answer

Providing a schema hint to force 'user_id' to be treated as a string.. Handling type mismatches is a common challenge in data ingestion. Auto Loader provides schema hints to force a specific type and the rescued data column to capture records that fail to meet that type. These tools allow the engineer to maintain data integrity and prevent the ingestion pipeline from failing due to inconsistent source data.

Answer

Using the 'mergeSchema' option to create two separate columns for the types.

Answer

Setting 'cloudFiles.inferColumnTypes' to false to ignore all types.

Answer

Deleting the checkpoint and restarting the stream to re-infer the types.

Question 10

Which THREE of the following are benefits of using Delta Live Tables (DLT) for managing your data pipelines?

Accepted Answer

Automated data quality monitoring via expectations.. Delta Live Tables simplifies the complexity of building reliable data pipelines by handling infrastructure orchestration, quality checks, and dependency management automatically. Understanding these benefits is essential for modern data engineering, as DLT reduces the burden of manual configuration, ensures data reliability through expectations, and provides declarative syntax that allows engineers to focus on business logic rather than low-level Spark optimization or cluster management tasks.

Answer

Manual management of cluster provisioning.

Answer

Ability to use non-Delta file formats.

Question 11

A data engineer is designing an access control model in Unity Catalog for a new catalog `finance`. The team wants to follow the principle of least privilege while still enabling collaboration. Which TWO of the following practices best align with Unity Catalog's privilege model? (Choose two.)

Accepted Answer

Use account-level groups as principals for grants rather than individual users, so membership changes automatically update effective permissions.. Least privilege in Unity Catalog is achieved by granting `USE CATALOG` broadly only where navigation is needed, limiting `USE SCHEMA` to required schemas, and using account-level groups as principals so membership changes propagate automatically. Broad grants such as `ALL PRIVILEGES`, `SELECT` to `public`, or `MANAGE` to large groups violate least privilege and should be avoided.

Answer

Grant `SELECT` on all tables in `finance` to the `public` group to ensure no one is blocked during onboarding.

Answer

Grant `ALL PRIVILEGES` on the `finance` catalog to the engineering group to simplify administration.

Answer

Grant `MANAGE` on the `finance` catalog to the data engineering group so they can delegate privileges to others.

Question 12

What is the primary function of a Delta Lake 'Vacuum' operation?

Accepted Answer

It removes old data files that are no longer needed by the Delta table.. Vacuum is an essential maintenance task for Delta tables, as it removes physical data files that are no longer referenced by the transaction log and are past the retention period. This is critical for data governance and cost management, preventing storage bloat and ensuring compliance with data privacy regulations like GDPR, which may require the physical deletion of data beyond a certain timeframe.

Answer

It optimizes the data for faster reading by sorting files.

Answer

It compresses the Delta table logs to improve metadata performance.

Answer

It rebuilds the Delta table's transaction log to fix corruption.

Question 13

A data engineer is monitoring a Databricks job and notices that the job's duration has gradually increased over the past week. The job reads a large Delta table, performs aggregations, and writes results to another Delta table. The engineer wants to identify the stage that is taking the most time. Which Spark UI tab should the engineer use to quickly identify the slowest stage?

Accepted Answer

Stages tab. The Spark UI Stages tab provides a detailed list of all stages, including their duration, number of tasks, and shuffle read/write metrics. By examining this tab, the engineer can sort stages by duration and identify the one consuming the most time. This is the most efficient way to locate the bottleneck stage and then investigate further.

Answer

Environment tab

Answer

Storage tab

Answer

Jobs tab

Question 14

A data engineer needs to share a table in Unity Catalog with a partner organization using a different Databricks account. Which feature should be used to provide secure access without moving the data?

Accepted Answer

Delta Sharing. Delta Sharing is the industry-standard open protocol for secure data sharing across organizational boundaries in Databricks. By utilizing Unity Catalog, Delta Sharing allows data providers to share live data directly from the storage layer without replicating files. This ensures governance, auditing, and real-time data access while maintaining strict security boundaries, which is critical for compliance and data sovereignty in modern enterprise data architectures.

Answer

Databricks File System (DBFS) mount points

Answer

Unity Catalog External Locations

Answer

Cross-account IAM Role Assumption

Question 15

A data engineer has a Unity Catalog table `prod.sales.orders` that contains a column `customer_email`. They need to allow analysts in the `marketing` group to query the table but only see a masked version of `customer_email` (e.g., `a***@example.com`). The masking logic is implemented as a SQL user-defined function `prod.security.mask_email`. Which statement should the engineer execute to apply the mask?

Accepted Answer

ALTER TABLE prod.sales.orders ALTER COLUMN customer_email SET MASK prod.security.mask_email;. Unity Catalog column masks are applied using ALTER TABLE ... ALTER COLUMN ... SET MASK <function>. This associates a user-defined function with the column. Users without the UNMASK privilege automatically see the function's output, while privileged users see raw data. The other options either use invalid syntax or fail to enforce masking at the column level.

Answer

CREATE VIEW prod.sales.orders_masked AS SELECT customer_email, prod.security.mask_email(customer_email) AS customer_email FROM prod.sales.orders;

Answer

ALTER TABLE prod.sales.orders SET TBLPROPERTIES ('mask.customer_email' = 'prod.security.mask_email');

Answer

GRANT SELECT ON TABLE prod.sales.orders TO `marketing` WITH MASK prod.security.mask_email ON customer_email;