Databricks-DE-Assoc Hard Questions — 25 Difficult Questions

Question 1

A data engineer is using Auto Loader to ingest data from a directory that receives thousands of files every hour. They are considering switching from the default directory listing mode to file notification mode. What is the primary reason for making this change?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

To reduce the cost and latency of discovering new files as the bucket grows.. As the number of files in a cloud storage bucket grows, the time and cost required to list all files recursively increase significantly. File notification mode solves this by using cloud services to push events about new files directly to Auto Loader, making it far more efficient for high-volume, long-term ingestion projects.

Answer

To enable the use of schema evolution in the ingestion pipeline.

Answer

To ensure that files are processed in the exact order they arrived.

Answer

To allow the stream to process multiple file formats simultaneously.

Question 2

A data engineer is designing a secure architecture using the Databricks Intelligence Platform. Which TWO of the following statements accurately describe the role and capabilities of Unity Catalog within this platform? (Choose TWO)

Accepted Answer

Unity Catalog provides centralized access control for tables, views, volumes, and AI models across multiple workspaces.. Unity Catalog is the fine-grained governance solution for data and AI across the Databricks platform, centralizing access control for tables, volumes, and models. It operates across multiple workspaces in a metastore, ensuring consistent security policies and audit logging regardless of which workspace or compute resource executes the user queries.

Answer

Unity Catalog replaces the underlying cloud provider IAM roles for all compute resources executing queries.

Answer

Unity Catalog automatically handles the physical partitioning of Delta tables for optimal query performance.

Answer

Unity Catalog requires all data to be migrated into proprietary binary formats before governance policies can be applied.

Question 3

A data engineer is designing a Delta Lake Bronze-to-Silver pipeline using Delta Live Tables (DLT) in Python. The engineer wants to ensure that incoming records are validated for data quality, dropping records that fail null checks on critical ID fields while failing the pipeline for invalid business logic. Which TWO DLT expectation decorators should be used to accomplish this?

Accepted Answer

@dlt.expect_or_drop("valid_id", "id IS NOT NULL"). Delta Live Tables expectations enforce data quality directly within transformation pipelines by specifying constraints on streaming datasets. The expectation modifier drop row allows the pipeline to silently discard invalid records while keeping the stream healthy, whereas fail update immediately halts execution when critical business invariants are violated. Selecting these precise decorators allows data engineers to balance fault tolerance against strict compliance rules.

Answer

@dlt.expect("valid_id", "id IS NOT NULL")

Answer

@dlt.drop_where("id IS NOT NULL")

Answer

@dlt.fail_update("valid_id", "id IS NOT NULL")

Question 4

A team is using Databricks Asset Bundles (DABs) to deploy a job to multiple environments (dev, staging, prod). They need to ensure that the job uses different cluster sizes and schedules per environment. Which DABs feature should they use?

Accepted Answer

Define separate bundle configuration files for each environment and use the --target flag with databricks bundle deploy.. Databricks Asset Bundles use targets to define environment-specific overrides in the databricks.yml file. When deploying, the --target flag selects the target, applying the corresponding cluster size and schedule. This keeps a single source of truth while allowing differences per environment, which is essential for promoting bundles through dev, staging, and prod.

Answer

Use the Databricks REST API to update the job configuration after deployment with environment-specific values.

Answer

Create a separate Databricks workspace for each environment and hardcode the cluster size and schedule in the job definition.

Answer

Use Jinja2 templating in the job definition to inject environment-specific values from environment variables.

Question 5

A data engineer is designing a workflow that requires running a series of tasks with dependencies. The workflow must be able to retry failed tasks automatically and send notifications on failure. The engineer also needs to ensure that the workflow can be triggered on a schedule and via an API. Which Databricks feature should the engineer use?

Accepted Answer

Databricks Jobs with task dependencies and retry policies.. Databricks Jobs are the primary orchestration feature for workflows with dependencies, retries, and notifications. They support multi-task DAGs, task-level retry policies, and alerting on failure. Jobs can be scheduled and triggered via API, making them the correct choice. The other options lack the necessary orchestration capabilities.

Answer

Delta Live Tables pipelines with continuous mode.

Answer

Databricks SQL dashboards with scheduled refresh.

Answer

Databricks Repos with Git hooks.

Question 6

A data engineer is configuring Unity Catalog row-level security on a table `prod.finance.transactions`. They want to ensure that users in the `us_team` group can only see rows where `region = 'US'`, while users in the `eu_team` group can only see rows where `region = 'EU'`. They create a row filter function `prod.security.region_filter`. Which TWO statements accurately describe how to implement and manage this row-level security? (Choose two.)

Accepted Answer

The row filter function must accept the column(s) as parameters and return a BOOLEAN value indicating whether the row should be visible.. Row-level security in Unity Catalog is implemented by attaching a row filter function to a table using ALTER TABLE ... SET ROW FILTER. The function receives column values and returns a boolean, often using identity functions to filter based on group membership. Both managed and external tables support row filters, and bypassing them requires the BYPASS ROW FILTER privilege.

Answer

Row filters can only be applied to external tables, not managed tables, because managed tables do not support fine-grained access control.

Answer

The row filter function must be defined in the same schema as the table it protects.

Answer

Users with the MODIFY privilege on the table automatically bypass the row filter and see all rows.

Question 7

Which of the following best describes the purpose of 'Credential Passthrough' in Databricks?

Accepted Answer

It enables users to access cloud storage using their own identity rather than the cluster's service principal.. Credential Passthrough allows users to authenticate to cloud storage using their own identity from the Databricks environment. By passing the user's credentials to the storage layer, Databricks ensures that the storage provider enforces access policies, creating a seamless audit trail and simplifying security management in environments that require strict alignment between user access in Databricks and direct cloud storage access.

Answer

It allows a Databricks cluster to automatically scale based on storage latency.

Answer

It encrypts data in transit between the Databricks workspace and the cloud storage.

Answer

It forces all notebook users to use a single shared service principal for security.

Question 8

A team is transitioning to the Databricks Intelligence Platform. Which TWO actions are required to successfully register a table in Unity Catalog using the three-level namespace?

Accepted Answer

Grant the 'USE CATALOG' privilege on the target catalog.. The three-level namespace (catalog.schema.table) is a fundamental feature of Unity Catalog. To register a table, an engineer must have sufficient privileges in the specific catalog and schema, and the table must be created using the correct fully qualified path. Understanding this structure is crucial because it enables unified data discovery and governance across different environments, preventing namespace collisions and allowing for standardized cross-database operations in the platform.

Answer

Configure a local Hive Metastore on the cluster.

Answer

Manually update the cluster's Spark configuration to point to the metastore URL.

Answer

Set the default database to 'default' before executing the CREATE statement.

Question 9

A data engineer maintains a Delta table where each row represents a customer record, and updates arrive continuously as change data capture events. The engineer needs to apply inserts, updates, and deletes from a staging table into the target table in a single atomic operation, matching records on customer_id. Which Delta Lake operation should be used?

Accepted Answer

MERGE INTO the target table using the staging table as the source, with matched update/delete clauses and a not-matched insert clause.. MERGE INTO is the Delta Lake primitive designed for upsert and delete semantics in one atomic transaction. By matching on customer_id, the engineer can update or delete matched rows and insert new ones from the staging table, applying CDC events idempotently. This avoids data gaps, prevents duplicate versions, and keeps the target table representing current state for downstream consumers.

Answer

Run a DELETE followed by an INSERT on the target table inside the same notebook cell.

Answer

Append the staging rows to the target table and deduplicate later with a window function during reads.

Answer

Use INSERT OVERWRITE to replace all partitions touched by the staging table with the staged rows.

Question 10

A data engineering team must implement dynamic row-level filtering on a customer analytics table in Unity Catalog so that regional analysts only view records corresponding to their assigned territory. Which TWO steps are required to achieve this using Unity Catalog features? (Choose 2)

Accepted Answer

Create a SQL user-defined function (UDF) that returns a boolean expression evaluating user identity and territory.. Implementing row-level security in Unity Catalog requires creating a SQL user-defined function that evaluates the current user's identity or group membership against a territory mapping table, and then applying that function to the target table using an ALTER TABLE command.

Answer

Modify the underlying cloud storage bucket IAM policy to restrict read access based on user session tags.

Answer

Create separate physical views for each regional analyst group and grant SELECT privileges exclusively on those views.

Answer

Configure cluster-level Spark configurations to automatically inject WHERE clauses into user queries.

Question 11

Refer to the exhibit. A data engineer receives this error when collecting data from a large transformation back to the driver node. Which approach should be used to fix this issue?

Accepted Answer

Rewrite the job to write the results to a Delta table.. The error occurs because the result set being pulled to the driver exceeds the configured limit. Collecting massive data to the driver is an anti-pattern in distributed computing as it bypasses the cluster's parallel processing capabilities. Instead of forcing data into the driver's memory, the engineer should write the output to cloud storage or a Delta table, allowing downstream processes to handle the data in a distributed, scalable manner without memory pressure.

Answer

Increase the spark.driver.maxResultSize setting.

Answer

Reduce the number of tasks in the Spark job.

Answer

Enable Spark dynamic allocation.

Question 12

Refer to the exhibit. An engineer needs to ensure that the cluster can handle sudden increases in data volume while minimizing costs during idle periods. Given the JSON configuration shown, what will happen when the workload demand spikes?

Accepted Answer

The cluster will add up to 8 additional worker nodes.. In Databricks terminology, adding more nodes to a cluster is referred to as 'scaling out' (or autoscaling up to the maximum number of workers), not 'scaling up' (which typically refers to resizing a node to a larger instance type with more CPU/RAM). Option C uses 'scale up' incorrectly.

Answer

The cluster will terminate automatically after 10 minutes of inactivity.

Answer

The cluster will scale up to a maximum of 8 worker nodes.

Answer

The cluster will lock at 2 workers to prevent cost overruns.

Question 13

You are auditing a Databricks environment and notice that the Silver tables contain massive amounts of historical data that is no longer needed. Which TWO operations should be used to safely remove this data while keeping the Delta Lake ACID consistency?

Accepted Answer

Use the VACUUM command on the Delta table.. The DELETE command is used to remove specific rows from a table, which triggers a rewrite of the affected data files and creates a new version in the transaction log. VACUUM is then required to physically remove the old, unreferenced data files from storage to reclaim space. Together, they ensure data is removed while maintaining ACID consistency.

Answer

Manually delete Parquet files from the underlying cloud storage (S3/ADLS).

Answer

Run an ALTER TABLE DROP PARTITION command to remove old data files instantly.

Answer

Recreate the table by overwriting it with a subset of the original data.

Question 14

Which DLT feature should a data engineer use to ensure that the Silver layer remains highly available and that data lineage is automatically tracked?

Accepted Answer

Define tables using 'CREATE OR REFRESH STREAMING TABLE' syntax.. The explanation mentions 'expectations' (which are for data quality) rather than explaining why 'CREATE OR REFRESH STREAMING TABLE' is the correct syntax for DLT pipelines to ensure high availability and lineage tracking. The explanation should focus on how DLT pipelines manage infrastructure and metadata automatically.

Answer

Use standard Spark SQL 'CREATE TABLE' statements.

Answer

Manually create a pipeline using 'dbutils.notebook.run' for each stage.

Answer

Execute the transformation using a Databricks Workflow job.

Question 15

Which THREE of the following are valid ways to define schemas in a Databricks data engineering pipeline?

Accepted Answer

Providing a DDL-formatted string to the 'schema' parameter in a read operation.. Option A (DDL strings) and Option B (StructType) are standard Spark/Databricks methods. Option D is a valid architectural pattern where schemas are stored as metadata (JSON) and converted to StructType at runtime. Option C is incorrect because schema inference is discouraged in production due to performance overhead and potential schema drift. Option E is incorrect because ALTER TABLE modifies existing table metadata rather than defining the schema for an ingestion operation.

Answer

Relying on 'spark.read.inferSchema=true' for all production tables.

Answer

Using the 'ALTER TABLE' command to define the schema before ingestion.

Question 16

Refer to the exhibit. An engineer applies these configurations to a cluster. What is the primary benefit of enabling the Databricks IO Cache for a workload that involves repeatedly reading the same Delta tables?

Accepted Answer

It speeds up repeated reads by caching data on local SSDs.. The Databricks IO Cache (also known as the disk cache) accelerates data reads by caching remote data on the local SSDs of the worker nodes. For workloads that frequently query the same tables, this eliminates the latency and network overhead of repeatedly fetching data from cloud object storage. This is particularly effective for read-heavy analytical workloads, enabling significantly faster query execution times by leveraging high-speed local disk I/O.

Answer

It enables ACID transactions for non-Delta tables.

Answer

It automatically scales the cluster based on disk usage.

Answer

It forces the cluster to store all data in memory.

Question 17

If an organization wants to ensure that no data can be accessed unless it is explicitly granted, which Unity Catalog setting should be enforced?

Accepted Answer

Enable Unity Catalog for all workspaces.. Unity Catalog enforces a 'deny-by-default' security model by design once a workspace is enabled for Unity Catalog. There is no specific setting called 'Enable Unity Catalog for all workspaces' that acts as a toggle for this behavior; rather, it is the default state of the Unity Catalog metastore. The question is poorly phrased as it implies a configuration setting that doesn't exist in the Databricks documentation.

Answer

Enable external locations.

Answer

Use the 'Account Admin' role.

Answer

Define a default catalog.

Question 18

A streaming job using Structured Streaming is lagging significantly behind the 'current time'. Which THREE of the following could be the root cause of this processing latency?

Accepted Answer

The trigger interval is set to a time shorter than the processing time of the batch.. Streaming latency often stems from resource contention, inefficient state management, or micro-batch trigger configurations. When the input rate exceeds the processing rate, lag accumulates. Identifying these bottlenecks requires analyzing the Spark UI for processing time vs. batch time. Understanding these factors is vital for maintaining low-latency pipelines and ensuring that SLAs are met in production-grade streaming environments deployed on Databricks.

Answer

Using a fixed-size cluster with no auto-scaling enabled.

Answer

Using too few partitions in the input source.

Question 19

Which of the following describes the correct order of operations to configure a new external location in Unity Catalog?

Accepted Answer

Create credential, create location, grant privileges.. The process begins with creating a storage credential, which holds the cloud-specific security details (like an IAM role or service account). This credential is then used to define the external location, which points to a specific path in cloud storage. Finally, you grant the user or service the 'READ FILES' or 'WRITE FILES' privilege on the external location to permit interaction with the underlying data files within the Unity Catalog framework.

Answer

Grant privileges, create location, create credential.

Answer

Create location, grant privileges, create credential.

Answer

Create credential, grant privileges, create location.

Question 20

Which TWO of the following are true regarding the use of Personal Access Tokens (PATs) versus Service Principals for automated jobs in Unity Catalog?

Accepted Answer

Service principals are independent of individual user accounts.. Service Principals are independent of individual user accounts, which prevents service disruption when employees leave. Additionally, they do not require a standard interactive Databricks user license for API/automated job access. PATs are tied to users, making them a security risk upon offboarding and harder to audit globally compared to centralized service principals.

Answer

PATs are recommended for production job automation.

Answer

PATs have higher security than service principals.

Answer

PATs are easier to audit than service principals.