Courseiva

Databricks Certified Data Engineer Associate (Databricks-DE-Assoc) — Questions 1–75

276 questions total · 4 pages · All types, answers revealed

Page 1 of 4

Page 2
1
MCQmedium

A data engineer runs a nightly Databricks job that reads from a large Delta table and writes results to another Delta table. The job has been taking progressively longer each night. The engineer examines the Spark UI and sees that the 'SQL' tab shows a single stage with many small tasks, each processing very few records, and the 'Storage' tab shows that the source table has thousands of small files. Which action should the engineer take to improve performance?

A.Enable Adaptive Query Execution (AQE) by setting spark.sql.adaptive.enabled to true.
B.Increase the number of shuffle partitions by setting spark.sql.shuffle.partitions to a higher value.
C.Run OPTIMIZE on the source Delta table to compact small files.
D.Repartition the source DataFrame to a smaller number of partitions before writing.
AnswerC

OPTIMIZE compacts many small files into larger ones, reducing the number of tasks and improving read throughput. In this scenario, the Spark UI shows thousands of small files and many small tasks, which is a classic small-file problem. Compaction directly addresses this by merging files, leading to fewer, larger files that are more efficient to read.

Why this answer

The Spark UI reveals many small tasks and thousands of small files in the source Delta table, which is a small-file problem. OPTIMIZE compacts these small files into larger ones, reducing task overhead and improving read performance. Other options either do not address the root cause or could exacerbate the issue by creating more partitions.

Exam trap

The trap here is assuming that increasing shuffle partitions or enabling AQE will solve small-file problems, when in fact compaction is required.

2
MCQmedium

Which of the following is a fundamental principle of implementing effective CI/CD for Databricks workflows?

A.Deploying code from a local laptop directly to the production cluster.
B.Using environment variables to manage service endpoints and authentication secrets.
C.Merging all development code into the main branch every few months.
D.Ignoring unit testing until the code is fully deployed in production.
AnswerB

Environment variables decouple code from the underlying infrastructure, allowing developers to manage configuration dynamically. This is a best practice in CI/CD as it enables the same codebase to run in any environment by simply injecting the appropriate parameters, reducing the risk of hardcoding secrets or environment-specific values.

Why this answer

Effective CI/CD requires separating code logic from environment-specific configurations. By parameterizing job settings and using environment-specific variables, teams ensure that the same code can be deployed safely across development, testing, and production. This practice minimizes errors during promotion and allows for automated testing cycles, which are essential for maintaining a robust, stable, and scalable data platform in any enterprise Databricks environment.

Exam trap

Candidates frequently select answers related to manual UI configuration or hardcoding values in code, ignoring that CI/CD best practices require environment abstraction through parameterization and secret management.

3
Multi-Selectmedium

A data engineer is tasked with securing sensitive PII data in Unity Catalog. Which THREE actions are recommended to ensure robust security and compliance?

Select 3 answers
A.Grant SELECT access to all users to ensure data accessibility.
B.Apply Dynamic Data Masking to columns containing PII.
C.Enable Unity Catalog audit logs to monitor data access.
D.Use table-level permissions to restrict access to sensitive datasets.
E.Store all PII data in the root metastore directory for easy access.
AnswersB, C, D

Dynamic Data Masking is a primary control for PII. By masking sensitive columns, you ensure that analysts can work with the data for aggregate statistics without actually seeing raw, identifiable information, which helps satisfy data privacy requirements while maintaining the utility of the dataset for authorized users.

Why this answer

Implementing a defense-in-depth strategy is crucial for PII. Using Unity Catalog's fine-grained access controls, dynamic masking, and audit logs provides a layered approach to security. These tools allow organizations to restrict access, obscure sensitive values, and maintain a verifiable trail of who accessed what data, which is essential for meeting regulatory requirements and minimizing the risk of unauthorized data breaches.

Exam trap

Candidates often select single-layer security approaches like only masking columns, ignoring that robust PII compliance requires a defense-in-depth strategy combining masking, auditing, and permissions.

4
MCQhard

Which of the following describes the correct order of operations to configure a new external location in Unity Catalog?

A.Grant privileges, create location, create credential.
B.Create credential, create location, grant privileges.
C.Create location, grant privileges, create credential.
D.Create credential, grant privileges, create location.
AnswerB

This is the correct sequence. A storage credential encapsulates the cloud permissions required to access the files. Then, the location maps that credential to a specific path. Finally, granting privileges allows users to interact with the data at that location, ensuring a secure and ordered governance workflow.

Why this answer

The process begins with creating a storage credential, which holds the cloud-specific security details (like an IAM role or service account). This credential is then used to define the external location, which points to a specific path in cloud storage. Finally, you grant the user or service the 'READ FILES' or 'WRITE FILES' privilege on the external location to permit interaction with the underlying data files within the Unity Catalog framework.

Exam trap

Candidates often attempt to create an external location before setting up the underlying storage credential, forgetting that the location explicitly depends on a pre-existing credential object.

5
MCQmedium

A data engineer is designing a Delta Live Tables (DLT) pipeline. They need to ensure that records with missing values in the 'customer_id' column are dropped during the ingestion process. Which constraint syntax should be used?

A.@expect_or_fail(customer_id IS NOT NULL)
B.@expect(customer_id IS NOT NULL)
C.@expect_or_drop(customer_id IS NOT NULL)
D.@filter(customer_id IS NOT NULL)
AnswerC

This specific DLT decorator instructs the pipeline to evaluate the expression and drop any rows that return false. It is the correct mechanism for filtering out invalid data silently while allowing the pipeline to continue processing subsequent batches of data without interruption or manual intervention.

Why this answer

The EXPECT DROP VIOLATION constraint is a native DLT feature designed specifically for data quality enforcement. By applying this, the pipeline automatically discards rows that fail the specified predicate while allowing valid records to proceed. This approach is essential in production data engineering to maintain data integrity and prevent downstream errors caused by null values, ensuring only high-quality data enters the silver or gold tables.

Exam trap

Candidates frequently confuse DLT expectations like 'expect_or_drop' with standard Spark SQL filter clauses or constraint keywords from traditional relational databases.

6
MCQmedium

A data engineer is preparing a notebook that must authenticate to cloud storage using a short-lived token that is automatically rotated by Databricks and is never written into the notebook source. The engineer wants the least administrative overhead while keeping secrets out of the code. Which approach should the engineer use?

A.Hard-code the storage account access key into a widget and pass it at runtime.
B.Attach an instance profile or managed identity to the cluster and access the storage directly.
C.Create a personal access token for the workspace and use it to call the storage REST API.
D.Store the storage key in a Databricks secret scope and reference it with dbutils.secrets.get in the notebook.
AnswerB

Attaching a cloud identity (instance profile on AWS, managed identity on Azure) to the cluster lets the notebook obtain short-lived credentials from the cloud metadata service. Databricks rotates them automatically, no secret appears in code, and access is scoped by the identity's IAM or RBAC policy. This is the lowest-overhead pattern for storage access from notebooks.

Why this answer

Attaching a cloud identity such as an instance profile or managed identity to the cluster is the standard Databricks pattern for credential-free storage access. The cluster's metadata service issues short-lived, automatically rotated credentials scoped to that identity, so nothing sensitive is stored in the notebook or job definition. Secret scopes and widgets still require manual credential handling and rotation, and workspace tokens do not authorize cloud storage access at all.

Exam trap

The trap here is assuming that a Databricks secret scope automatically rotates the underlying cloud credential, when in fact scopes only store and redact whatever value the engineer manually maintains.

7
MCQmedium

Your organization requires that all data processing pipelines enforce a strict schema to prevent corrupt data from landing in the Silver layer. Which feature should be configured to ensure that only data matching the expected schema is written?

A.Enable 'spark.databricks.delta.schema.autoMerge.enabled'.
B.Rely on the default Delta Lake schema enforcement mechanism.
C.Use a JSON schema file during the read process to discard invalid records.
D.Set the table property 'delta.columnMapping.mode' to 'name'.
AnswerB

Delta Lake, by default, enforces the schema of the target table. Any write that includes columns not in the schema or data types that do not match will fail. This provides the necessary guardrails to ensure that only valid, predictable data enters the table, which is essential for data integrity.

Why this answer

Schema enforcement (often called schema validation) is the standard mechanism in Delta Lake to protect downstream tables. By setting 'mergeSchema' to false or relying on default behavior, you ensure that any incoming data that deviates from the predefined target schema will cause the pipeline to fail. This is critical for data quality in production, preventing unexpected data structures from breaking analytical models and downstream dashboards.

Exam trap

Candidates often incorrectly assume they must manually write complex validation logic or custom UDFs to enforce schemas, forgetting that Delta Lake provides built-in schema enforcement as a native, default feature.

8
MCQhard

A CI/CD pipeline uses the Databricks CLI to deploy a job to production. The pipeline must ensure that the job configuration is identical across environments except for the cluster size, which differs between staging and production. Which approach best supports this requirement?

A.Define the job in a JSON file with placeholders, and use environment variables in the CI/CD pipeline to substitute cluster size values before deployment.
B.Maintain separate JSON files for staging and production, and manually update both when the job logic changes.
C.Store the job definition in a Databricks notebook and use widgets to parameterize the cluster size at runtime.
D.Use the Databricks REST API to create the job in production, then manually edit the cluster size in the Databricks UI after deployment.
AnswerA

Using a templated JSON file with environment-specific variables allows the pipeline to inject the correct cluster size at deployment time. This keeps the job definition consistent across environments while allowing controlled differences. It is a common pattern for infrastructure-as-code and aligns with CI/CD best practices for Databricks jobs.

Why this answer

Templating the job JSON with environment variables allows the pipeline to substitute environment-specific values like cluster size while keeping the rest of the configuration identical. This ensures consistency and reduces drift, as the same template is used for all environments, and differences are explicitly managed through variables.

Exam trap

The trap here is confusing runtime parameters (like widgets) with deployment-time configuration, which must be injected before the job is created or updated.

9
MCQhard

Refer to the exhibit. A security administrator applies this Unity Catalog policy. What is the impact on users in the 'analyst-group'?

A.They can drop the revenue table.
B.They can view the contents of the revenue table.
C.They gain administrative rights over the entire catalog.
D.They can write new data into the revenue table.
AnswerB

The 'SELECT' privilege explicitly authorizes users to read data from the table. By granting this to the 'analyst-group', all members can now execute SELECT queries against the table, enabling them to analyze the data for business purposes while maintaining security protocols through the centralized Unity Catalog governance model.

Why this answer

This Unity Catalog JSON policy grants the 'analyst-group' the SELECT privilege on the 'main.sales.revenue' table. Consequently, any user belonging to this group will be able to read data from this specific table. This is a standard governance practice in Databricks, providing a fine-grained, centralized method to control access to sensitive data assets across the entire organization, ensuring security and compliance without managing individual permissions for each user.

Exam trap

Test-takers often misinterpret explicit SELECT privileges in Unity Catalog policies as restrictive blocks, confusing granting read access with revoking permissions.

10
Multi-Selectmedium

Which THREE of the following are core components of the Databricks Intelligence Platform?

Select 3 answers
A.Delta Lake
B.Unity Catalog
C.Managed Spark Clusters
D.Proprietary File System Format
E.Hardware-level Storage Controllers
AnswersA, B, C

Delta Lake is the core storage format providing ACID transactions and schema enforcement. It is an essential component that allows the platform to maintain data reliability at scale, providing the foundation for the Lakehouse architecture by ensuring that data stored in cloud object storage is consistent, performant, and easily queryable.

Why this answer

The Databricks Intelligence Platform is built on a foundation that includes high-performance compute, unified governance, and open storage. These components work together to provide a seamless experience for data engineering, science, and analytics. Recognizing these elements helps engineers understand the platform's capabilities to scale horizontally, secure data assets centrally, and maintain interoperability through open standards, which are essential for modern data architecture requirements.

Exam trap

Candidates often include legacy components like 'Hive Metastore' or 'Databricks SQL' as core platform components. They fail to identify the foundational pillars: Delta Lake, Unity Catalog, and Managed Spark.

11
MCQmedium

A data engineer notices that a production Delta Lake table is experiencing slow read performance during concurrent write operations. The table contains millions of small files. Which action should the engineer take to resolve this performance degradation?

A.Run the VACUUM command on the table.
B.Increase the number of worker nodes in the cluster.
C.Execute the OPTIMIZE command on the table.
D.Enable Z-Ordering on all columns in the table.
AnswerC

OPTIMIZE compacts small files into larger, optimally sized files, which significantly improves read performance. This process is essential for maintaining high-performance Delta tables that experience frequent streaming writes or high-frequency batch inserts, as it reduces the number of files the query engine must track and scan.

Why this answer

Compacting small files into larger files using the OPTIMIZE command reduces metadata overhead and improves I/O efficiency. This is a critical maintenance task in Databricks because small files cause excessive object storage requests and metadata listing latency. By consolidating these files, the engine can scan data much more effectively during read operations, even while concurrent writes are occurring, as Delta Lake ensures ACID transactions remain consistent and performant.

Exam trap

Candidates often suggest 'dropping and recreating the table' or 'increasing the cluster size', which are destructive or costly workarounds for a simple file management issue that OPTIMIZE is designed to solve.

12
Multi-Selecthard

A data engineer is optimizing a Databricks job that performs a join between a large fact table and a small dimension table. The job is slow, and the engineer suspects that the join strategy is not optimal. Which TWO actions should the engineer take to improve performance? (Choose two.)

Select 2 answers
A.Enable Adaptive Query Execution (AQE) by setting spark.sql.adaptive.enabled to true
B.Broadcast the small dimension table using a broadcast hint
C.Cache the large fact table in memory
D.Increase spark.sql.shuffle.partitions to a very high value
E.Repartition the large fact table by the join key before the join
AnswersA, B

AQE can automatically convert a sort-merge join to a broadcast join at runtime if it detects that one side is small enough. It also optimizes shuffle partitions and handles skew. Enabling AQE allows Spark to adapt the join strategy based on actual data sizes, which can improve performance without manual hints. This is a recommended best practice in Databricks.

Why this answer

Broadcasting the small dimension table avoids shuffling the large fact table, which is a major performance win. Enabling Adaptive Query Execution allows Spark to automatically choose a broadcast join at runtime if the dimension table is small enough, and to optimize other aspects of the query. Together, these actions address the inefficient join strategy and improve performance.

Exam trap

The trap here is assuming that increasing shuffle partitions or caching always helps, when the real issue is the join strategy and the solution is to avoid the shuffle altogether.

13
MCQmedium

What is the primary benefit of using Data Live Tables (DLT) for managing dependencies between tables in a pipeline?

A.It allows for manual control of the Spark cluster settings for every single step.
B.It automatically manages the dependency graph and ensures execution order.
C.It eliminates the need for any data transformation logic to be written in SQL.
D.It forces all data to be stored in the Gold layer for final reporting.
AnswerB

DLT tracks the dependencies between tables defined in your code and automatically orchestrates the execution flow. If Table B depends on Table A, DLT ensures Table A is processed first. This eliminates the need for manual orchestration tools and makes the pipeline much more resilient to changes.

Why this answer

DLT automatically manages the Directed Acyclic Graph (DAG) of the entire data pipeline. By defining table relationships through code, DLT handles the execution order and ensures dependencies are met. This simplifies pipeline management and maintenance, reducing the need for manual orchestration and ensuring that data is always processed in the correct order across the entire Medallion architecture.

Exam trap

Candidates often believe manual notebook scheduling or external orchestrators are required to sequence table refreshes, ignoring DLT's native dependency graph capabilities.

14
MCQmedium

A data engineering team uses Databricks Asset Bundles (DABs) to manage a job that must deploy to both a staging and a production workspace. The team wants to avoid hardcoding workspace-specific values such as the cluster ID and the storage path in databricks.yml. Which approach should they use?

A.Run databricks bundle deploy once, then use the Databricks REST API to patch the job's cluster ID and storage path after each deployment.
B.Define variables in the bundle and set target-specific values in databricks.yml under targets, then deploy with databricks bundle deploy -t <target>.
C.Store the cluster ID and storage path in a Databricks secret scope and reference them with dbutils.secrets.get inside the job's notebook code.
D.Create a separate Git branch for each workspace and manually edit the cluster ID and storage path in databricks.yml before each deployment.
AnswerB

DABs support variables and targets in databricks.yml. You declare variables once and override their values per target (for example, staging and prod), so the same bundle definition deploys correctly to each workspace. Running databricks bundle deploy -t <target> selects the target and applies its variable values, eliminating hardcoded cluster IDs and paths while keeping one source of truth.

Why this answer

Databricks Asset Bundles use a declarative databricks.yml with variables and targets. Declaring variables once and overriding them per target lets one bundle deploy consistently to staging and production without hardcoded workspace-specific values. Selecting the target at deploy time applies the correct values, keeps a single source of truth in Git, and avoids drift from manual edits or post-deployment patching.

Exam trap

The trap here is assuming environment-specific values must be embedded in the bundle file or patched afterward, when DABs targets and variables are designed precisely to parameterize them at deploy time.

15
MCQeasy

Which feature in Unity Catalog is primarily used to track data movement and transformation history for compliance and auditing?

A.Audit Logs
B.Data Lineage
C.Delta Sharing
D.Catalog Explorer
AnswerB

Data lineage captures the dependencies between tables, views, and notebooks. It shows the flow of data through transformations, which is essential for auditability. By tracking these relationships, data engineers and security officers can verify the provenance of data and fulfill regulatory requirements regarding data processing and handling.

Why this answer

Data lineage provides a graphical and programmatic representation of data flow from source tables to downstream targets. It is vital for governance as it allows administrators to see exactly where data came from and how it was processed. This transparency is critical for compliance audits, allowing teams to quickly identify the impact of changes or potential data leakage points within the entire data ecosystem.

Exam trap

Candidates often mistake 'Data Lineage' for 'Audit Logs' or 'Query History'. While audit logs track access, only Data Lineage specifically tracks the transformation and flow of data objects.

16
MCQmedium

When considering the Databricks Intelligence Platform, what is the primary role of the 'Lakehouse' architecture?

A.To replace cloud object storage with a proprietary proprietary binary format.
B.To unify the best features of data warehouses and data lakes.
C.To isolate streaming and batch data processing into distinct storage layers.
D.To mandate the use of SQL as the only supported programming language.
AnswerB

The Lakehouse unifies data warehousing and data lake architectures. It offers the performance and transactional consistency of a warehouse for BI workloads while providing the scalability and support for unstructured data, machine learning, and streaming workloads that are typical of data lakes, all within a single unified platform.

Why this answer

The Lakehouse architecture combines the data management, schema support, and ACID transactions of a data warehouse with the scale, flexibility, and low-cost storage of a data lake. This is important for engineers because it eliminates the need to maintain separate systems for BI and AI/ML, allowing teams to use a single platform for all data processing tasks, thereby reducing data duplication and architectural complexity.

Exam trap

Candidates often describe the Lakehouse as just a 'better data lake' or 'cloud warehouse'. They miss the core definition of unifying the strengths of both architectures into one platform.

17
Multi-Selectmedium

A data engineer is investigating a slow-running SQL query. Which TWO metrics from the Spark UI should the engineer examine to identify data skew?

Select 2 answers
A.Task duration
B.Executor CPU usage
C.Task input size
D.Shuffle read size
E.Driver memory usage
AnswersA, C

Task duration highlights significant variations in processing time across different executors. If most tasks complete in seconds but a few take minutes, it is a clear indicator of data skew or resource contention, prompting further investigation into the specific data distribution of the underlying partition.

Why this answer

Data skew occurs when one partition is significantly larger than others, causing one task to take much longer than the rest. By examining task duration and task input size, an engineer can spot outliers where a single executor handles a disproportionate amount of data. Identifying these metrics allows the engineer to apply remediation techniques like salting or repartitioning to balance the workload across the cluster effectively.

Exam trap

Candidates often choose metrics like 'Shuffle read size' or 'JVM memory usage', which are related to performance but do not directly expose the uneven distribution of data that characterizes skew.

18
MCQmedium

A data engineer has a Delta table named `sales` with columns `sale_id`, `customer_id`, `amount`, and `sale_date`. They need to create a new table that contains only the `customer_id` and the total `amount` per customer for all sales in 2023. Which SQL statement correctly creates this aggregated table?

A.CREATE TABLE customer_totals AS SELECT customer_id, SUM(amount) AS total_amount FROM sales WHERE YEAR(sale_date) = 2023 GROUP BY customer_id;
B.CREATE TABLE customer_totals AS SELECT customer_id, SUM(amount) AS total_amount FROM sales WHERE YEAR(sale_date) = 2023 GROUP BY customer_id, sale_date;
C.CREATE TABLE customer_totals AS SELECT customer_id, SUM(amount) AS total_amount FROM sales WHERE YEAR(sale_date) = 2023;
D.CREATE TABLE customer_totals AS SELECT customer_id, SUM(amount) AS total_amount FROM sales GROUP BY customer_id HAVING YEAR(sale_date) = 2023;
AnswerA

This statement uses CREATE TABLE AS SELECT (CTAS) to create a new table from the query. It filters rows for 2023 using WHERE YEAR(sale_date) = 2023, groups by customer_id, and sums the amount. The result is a new Delta table with the required columns, meeting the scenario's goal. It is the correct and efficient way to materialize the aggregated data.

Why this answer

The correct approach uses a CREATE TABLE AS SELECT statement with a WHERE clause filtering for 2023 sales before aggregation, followed by GROUP BY customer_id and SUM(amount). This materializes the desired summary table. The other options either misuse HAVING, omit GROUP BY, or add an extra grouping column that changes the granularity of the result.

Exam trap

The trap here is confusing the order of filtering and aggregation, such as using HAVING instead of WHERE for row-level filtering, or forgetting that GROUP BY must include all non-aggregated columns.

19
Multi-Selecthard

A platform team is setting up a CI/CD pipeline that deploys Databricks jobs and notebooks from a Git repository. They must ensure deployments are secure and auditable. (Choose two.)

Select 2 answers
A.Grant the deployment identity only the permissions required to manage the specific jobs and notebooks it deploys.
B.Authenticate the pipeline with a service principal and store its OAuth secret in the CI/CD system's secret store.
C.Disable workspace audit logs during deployments to reduce log volume and improve pipeline speed.
D.Use a different engineer's personal account for each environment so responsibility is clear.
E.Commit the workspace personal access token to the repository so all pipeline runs use the same credential.
AnswersA, B

Least-privilege permissions limit the blast radius if the pipeline credential is compromised and keep audit logs focused on the resources the pipeline should touch. This directly supports the security requirement while preserving traceability of what the deployment identity changed.

Why this answer

Secure, auditable deployments rely on a dedicated non-human identity with narrowly scoped permissions. A service principal whose secret lives in the CI/CD secret store provides traceable, rotatable credentials, and least-privilege grants limit what a compromised pipeline could do while keeping audit records meaningful.

Exam trap

The trap here is treating any working credential, such as a committed personal access token, as acceptable for automation when the requirement is specifically secure and auditable deployment.

20
MCQmedium

A data engineering team is migrating a legacy data warehouse to Databricks. They want to ensure that their raw data is ingested into a 'Bronze' table in its original format. Which approach is most recommended for this ingestion layer?

A.Use a standard Spark read with schema inference enabled
B.Use Auto Loader with cloudFiles format
C.Perform a manual copy of files to a local cluster before processing
D.Load the data into a Pandas DataFrame and write it to Delta
AnswerB

Auto Loader with cloudFiles is the optimal choice for incremental ingestion. It manages state, handles schema evolution automatically, and processes files as they arrive, making it highly efficient. It also allows for the inclusion of metadata columns that help track the provenance of the raw data for auditing purposes.

Why this answer

Using Auto Loader is the best practice for ingesting raw data from cloud object storage. It provides efficient, scalable, and incremental ingestion using cloud-native file notifications. By capturing the file as-is and adding metadata columns (like _rescued_data), the Bronze layer preserves the original record integrity.

This ensures that if business logic changes, the team can reprocess the raw data from scratch, which is a foundational principle of the Medallion architecture.

Exam trap

Candidates often select standard batch read methods like spark.read.json for streaming ingestion, missing the scalability and incremental file notification benefits of Auto Loader.

21
Multi-Selectmedium

Which TWO of the following scenarios are valid use cases for utilizing Delta Lake's Change Data Feed (CDF)?

Select 2 answers
A.Auditing data changes for compliance reporting.
B.Performing full table overwrites in a batch job.
C.Incremental propagation of changes to downstream systems.
D.Compressing large files into smaller partitions.
E.Querying the table history using the VERSION AS OF syntax.
AnswersA, C

CDF provides a detailed log of all changes, making it ideal for compliance and audit logs. It records exactly what was changed and when, allowing engineers to reconstruct the state of a row at any point in time without complex manual tracking.

Why this answer

Change Data Feed allows users to track row-level changes (inserts, updates, deletes) in a Delta table. It is essential for incremental processing, auditing, and syncing data between systems. By enabling CDF, engineers can decouple the processing logic from the source table history, simplifying complex pipelines that need to respond to specific data mutations rather than re-scanning entire datasets periodically.

Exam trap

Candidates often assume CDF is a general-purpose logging tool, failing to realize it is specifically optimized for capturing row-level changes to propagate state changes to downstream systems incrementally.

22
MCQmedium

A data engineer is configuring a Databricks job that must run on a schedule and send an email notification if the job fails. They want to minimize manual intervention. Which feature should they use to define the schedule and failure notification?

A.Cluster policies with auto-termination settings
B.Databricks SQL alerts with a custom schedule
C.Databricks Jobs with a cron schedule and email notifications on failure
D.Delta Live Tables with expectations for failure handling
AnswerC

Databricks Jobs support scheduling with cron expressions and can be configured to send email notifications when a run fails. This allows the engineer to automate the job and receive alerts without manual monitoring. This is the standard way to schedule and monitor jobs in Databricks.

Why this answer

Databricks Jobs provide built-in scheduling via cron and support email notifications on failure. This directly meets the need to automate job runs and alert the engineer when something goes wrong, reducing manual intervention. Other features like SQL alerts or cluster policies serve different purposes and cannot replace job scheduling and failure notifications.

Exam trap

The trap here is confusing monitoring features like SQL alerts with job orchestration and notification capabilities of Databricks Jobs.

23
MCQhard

A data engineer needs to run a nightly transformation that reads a large Parquet dataset, writes a curated Delta table, and then immediately runs OPTIMIZE and VACUUM on that table. The engineer wants each step to be observable, retryable, and to avoid data loss if VACUUM fails. Which orchestration approach best meets these requirements?

A.A single Databricks notebook that performs the read, write, OPTIMIZE, and VACUUM in one cell so the steps always run together.
B.A Databricks Job with separate tasks for the write, OPTIMIZE, and VACUUM, chained with task dependencies and configured retries.
C.An external cron job on a VM that calls the Databricks REST API to submit each step as a separate one-time run.
D.A Databricks Job that runs one notebook task containing the write and OPTIMIZE, followed by a separate job that runs VACUUM on a trigger.
AnswerB

A multi-task Databricks Job gives each step its own task, so the write, OPTIMIZE, and VACUUM run as distinct units with individual logs, durations, and retry policies. Dependencies ensure ordering, and a failed VACUUM can be retried without rerunning the write, preserving the committed Delta table. This directly satisfies observability, retryability, and data-loss avoidance.

Why this answer

Modeling the pipeline as a Databricks Job with discrete tasks lets each stage be independently logged, retried, and ordered through dependencies. The write commits first, OPTIMIZE then compacts files, and VACUUM runs only after the others succeed, so a VACUUM failure can be retried without recomputing the curated table. Single-cell notebooks and externally triggered jobs sacrifice that granular control and coordinated retry behavior.

Exam trap

The trap here is treating VACUUM as just another statement in the same notebook, which hides the fact that it is a separate, retryable unit whose failure should not force a full recomputation.

24
MCQmedium

A data engineer is building a Databricks job that processes millions of small JSON files landed in cloud storage each hour. The job currently spends most of its runtime on file listing and task scheduling overhead. The engineer wants to improve throughput without changing the downstream table schema. Which change should be made to the ingestion step?

A.Convert the source JSON files to Parquet with a separate Spark batch job before each hourly run.
B.Enable Auto Compaction on the target Delta table so small files are merged after each write.
C.Use Auto Loader with cloudFiles and enable file notification mode to ingest new files incrementally.
D.Repartition the DataFrame to one partition per input file before writing to the target table.
AnswerC

Auto Loader with cloudFiles tracks newly arrived files using a scalable discovery mechanism, and file notification mode uses cloud queue notifications instead of repeated directory listings. This directly removes the listing and scheduling overhead described, while schema inference and evolution keep the downstream Delta table schema stable.

Why this answer

The bottleneck is file discovery and task scheduling for many small files, not the target table layout or columnar format. Auto Loader with cloudFiles incrementally discovers new files, and file notification mode avoids full directory listings by using cloud storage event notifications, which scales to high file counts. This preserves the existing schema while improving ingestion throughput.

Exam trap

The trap here is assuming that a target-side optimization such as Auto Compaction solves a source-side file listing bottleneck.

25
MCQmedium

A data engineer wants to run an incremental ingestion job every six hours. They want to ensure that each run processes all available data and then shuts down the cluster to save costs. Which Trigger should be used in the Structured Streaming code?

A.Trigger.ProcessingTime('6 hours')
B.Trigger.Once
C.Trigger.AvailableNow
D.Trigger.Continuous('1 minute')
AnswerC

Trigger.AvailableNow is designed for this exact use case. It processes all currently available data from the source, potentially splitting it into multiple smaller micro-batches for better performance and stability. Once all data is processed, the query terminates, allowing the cluster to be shut down automatically.

Why this answer

Trigger.AvailableNow is the modern replacement for Trigger.Once. It provides better scalability by processing all available data in multiple micro-batches if necessary, while still allowing the job to terminate once all data is processed. This is ideal for cost-effective, periodic batch-style processing of streaming data sources.

Exam trap

Candidates frequently choose Trigger.Once, failing to recognize that Trigger.AvailableNow is the modern, scalable replacement for batch-style streaming jobs.

26
MCQhard

Refer to the exhibit. A data engineer is running these commands before performing heavy write operations into a Delta table. What is the primary benefit of enabling these configurations?

A.It increases the number of concurrent write operations allowed on the table.
B.It mitigates the small file problem and improves read performance.
C.It forces the data to be written in Parquet format only.
D.It disables the need for periodic VACUUM maintenance.
AnswerB

By optimizing the writing process and automatically compacting files into larger, more efficient sizes, these settings prevent the creation of many small files that degrade read performance. This results in a cleaner, more readable table structure that allows the Spark engine to scan data more efficiently during analytical queries.

Why this answer

Enabling 'optimizeWrite' and 'autoCompact' is a common strategy to mitigate the 'small file problem' in Delta Lake. 'OptimizeWrite' rearranges data before writing to ensure consistent, larger file sizes, while 'autoCompact' merges small files post-write. Together, they ensure that the table remains highly performant for read operations by maintaining an optimal file layout, which is critical for reducing metadata overhead and maximizing query scan efficiency in large datasets.

Exam trap

Candidates often think these settings are for data compression or security, overlooking their primary purpose: addressing the 'small file problem' to improve metadata and scan performance.

27
MCQmedium

A data engineering team needs to restrict access to a sensitive customer table in Unity Catalog so that junior data engineers can only view non-PII columns, while senior engineers can view all columns. Which approach should be used to implement this requirement securely and efficiently?

A.Create separate physical copies of the customer table for junior and senior engineers, applying access control lists directly on the underlying files in cloud object storage using IAM policies.
B.Revoke ALL PRIVILEGES on the customer table from the junior engineers group and grant SELECT privilege only after they sign a data access request form stored in a shared workspace notebook.
C.Define a SQL user-defined function combined with a dynamic view utilizing the IS_MEMBER function to conditionally mask sensitive columns based on the querying user's group membership.
D.Apply table ACLs directly on the raw Delta table by executing GRANT SELECT ON TABLE customer TO `junior-engineers` while writing custom Spark UDFs in every notebook to drop PII columns.
AnswerC

A dynamic view calling a SQL UDF that evaluates IS_MEMBER returns masked or full column values per group, so junior engineers see non-PII only while senior engineers see everything. This enforces row-and-column-level control centrally in Unity Catalog without duplicating tables.

Why this answer

Row and column-level security in Unity Catalog is implemented by defining SQL user-defined functions and applying dynamic views or masking policies. Creating a dynamic view using the IS_MEMBER function allows the database engine to evaluate user group membership at query execution time and conditionally mask or exclude sensitive columns without duplicating the underlying physical dataset, ensuring optimal storage and compliance.

Exam trap

Candidates often assume they need to create separate physical tables for different user groups, which violates data governance best practices and creates maintenance overhead and data duplication risks.

28
MCQmedium

A data engineer is using Structured Streaming to ingest data from a Kafka topic. They want to ensure that if the pipeline fails, it can resume exactly where it left off, without processing duplicate data. Which component enables this functionality?

A.The Delta transaction log
B.A persistent checkpoint location
C.An external database to store row IDs
D.The cluster's memory state
AnswerB

Checkpoint locations store the progress of the streaming query. By writing this information to durable storage, Spark can recover from any failure by reading the last saved state, ensuring that it only processes new data. This is critical for preventing duplicate ingestion and maintaining the integrity of the streaming pipeline.

Why this answer

Checkpointing is the fundamental mechanism in Spark Structured Streaming that persists state information, including the offset of the processed records, to reliable cloud storage. When a job restarts after a failure, it reads the checkpoint location to determine which records were already successfully processed. This ensures fault tolerance and enables exactly-once processing semantics, which is a requirement for reliable and accurate data engineering pipelines in production environments.

Exam trap

Candidates often assume Spark handles state recovery automatically in memory, forgetting that without a persistent checkpoint location, the system loses the offset upon cluster restart.

29
Multi-Selecthard

Which TWO of the following statements are true regarding the behavior and capabilities of Databricks Jobs parameters and values?

Select 2 answers
A.Task values can be used to pass small amounts of data, such as counts or status codes, from one task to a downstream task.
B.Job parameters defined at the workflow level can be overridden when triggering a run via the Databricks CLI or REST API.
C.Job parameters are automatically persisted in a Unity Catalog volume after the workflow completes successfully.
D.Notebook tasks cannot accept parameter values passed from the parent job orchestration configuration.
E.Task values support passing large dataframes containing millions of rows between dependent tasks.
AnswersA, B

The dbutils.jobs.taskValues API allows tasks to set key-value pairs that downstream tasks can retrieve. This facilitates dynamic branching and parameter passing based on runtime metrics generated during the execution of upstream workflow steps.

Why this answer

Databricks Jobs support parameterized runs, enabling dynamic workflows where values are passed at runtime or configured statically. Understanding how parameters propagate to tasks and how task values pass outputs between tasks is essential for building modular, reusable data engineering pipelines within the Lakeflow framework.

Exam trap

Candidates often think job-level parameters are immutable and cannot be altered at runtime, missing the flexibility of the Databricks API and CLI overrides.

30
Multi-Selecthard

A data engineer is configuring Unity Catalog governance for a multi-department organization. Which TWO actions require the metastore admin or catalog owner to have a workspace-independent metastore assigned? (Choose TWO)

Select 2 answers
A.Creating a new catalog inside the Unity Catalog metastore.
B.Binding a workspace to a Unity Catalog metastore to enable catalog-level data access.
C.Granting the SELECT privilege on a specific schema to a group of data analysts.
D.Configuring the root storage location for the metastore using a secure cloud storage container.
E.Defining a custom dynamic view with column-level masking functions for auditing.
AnswersB, D

Workspace binding enforces security isolation by mapping specific workspaces to a Unity Catalog metastore. This administrative configuration ensures that data assets governed by the metastore are only accessible through authorized workspaces, requiring proper metastore administration.

Why this answer

A Unity Catalog metastore is the top-level container for metadata, governing three levels of namespacing (catalog, schema, table). Certain administrative tasks like assigning a metastore to workspaces or managing root storage locations require specific privileges and architectural scopes, ensuring centralized security governance across multiple workspaces.

Exam trap

Test-takers frequently assume routine table-level operations or basic user grants require a metastore-level scope, confusing everyday data engineering tasks with account-level administrative actions.

31
MCQmedium

Refer to the exhibit. A data engineer is configuring an Auto Loader stream with the provided options. What will happen if a new JSON file arrives containing a field that is not currently in the target table schema?

A.The stream will fail and require a manual schema update.
B.The new field will be added to the target table automatically.
C.The new field will be dropped and only existing columns are kept.
D.The data for the new field will be moved to a _rescued_data column.
AnswerB

The addNewColumns mode enables Auto Loader to update the table schema dynamically. When a new column is detected in the source files, it is added to the table's metadata and the data is successfully ingested. This allows the pipeline to adapt to upstream changes without stopping or losing data.

Why this answer

The schemaEvolutionMode configuration determines how Auto Loader reacts to changes in the source data structure. By setting this to addNewColumns, the engineer ensures that the pipeline remains operational when new fields appear. This is a key part of the Medallion architecture where bronze tables must capture all incoming raw data without strict validation rules.

Exam trap

Candidates often assume schema evolution requires manual table alterations or that it will crash the stream, failing to realize that Auto Loader can dynamically update the metastore schema automatically when configured correctly.

32
Multi-Selecthard

A data engineer is migrating legacy tables to Unity Catalog. Which TWO of the following statements regarding the transition to three-level namespace (catalog.schema.table) are true?

Select 2 answers
A.Three-level namespace applies to all Unity Catalog objects.
B.Hive Metastore objects are automatically migrated to Unity Catalog.
C.Permissions at the catalog level cascade to all child schemas and tables.
D.External locations are not required for three-level namespace tables.
E.Three-level namespace is optional in Unity Catalog.
AnswersA, C

Unity Catalog enforces a strict hierarchy consisting of catalog, schema, and table or view. This structure provides a unified way to manage permissions and lineage across different workspaces. Every managed or external object must adhere to this naming convention to be correctly registered within the Unity Catalog metadata store.

Why this answer

Unity Catalog introduces a hierarchical namespace that forces a clear separation between data assets. Understanding this structure is critical for governance, as permissions are inherited down from the catalog and schema levels. By adopting this structure, organizations can enforce consistent policies across environments, simplify cross-workspace data sharing, and ensure that all metadata is discoverable through a centralized catalog that replaces the legacy Hive Metastore structure.

Exam trap

Candidates often assume permissions must be explicitly assigned at the table level, overlooking the inheritance model where catalog-level grants automatically cascade down to schemas and tables.

33
MCQmedium

When ingesting data from cloud storage, what is the most important reason to use a Service Principal or IAM Role instead of individual user credentials?

A.To increase the maximum file size that can be ingested by Spark.
B.To enable the use of the COPY INTO command in SQL warehouses.
C.To ensure the pipeline continues to run if the user leaves the company.
D.To allow Auto Loader to use directory listing instead of notifications.
AnswerC

Service Principals and IAM Roles provide persistent, non-human identities for automation. If a pipeline is tied to a user's account and that user's access is disabled, the production pipeline will fail. Using a service identity ensures the longevity and reliability of critical data engineering workflows over time.

Why this answer

Using service-level authentication is a security best practice that ensures ingestion pipelines are not tied to specific individuals. This prevents production jobs from failing when a user leaves the organization or has their permissions revoked. It also provides a more secure and auditable way to manage access to sensitive data in cloud environments.

Exam trap

Candidates often choose answers focusing solely on convenience or performance, missing that operational continuity and preventing pipeline failures upon staff departures is the core security advantage.

34
MCQhard

A data engineer notices that a Databricks job processing a Delta table with 10,000 partitions runs slowly. The job filters on a column that is not the partition column, and the query plan shows that all partitions are being scanned. The engineer wants to improve performance without repartitioning the table. Which feature should be used?

A.Enabling Change Data Feed on the table
B.Setting delta.dataSkippingNumIndexedCols to 0
C.Increasing the number of shuffle partitions
D.Z-ORDER BY on the filter column during OPTIMIZE
AnswerD

Z-ORDER BY co-locates related data in the same set of files, allowing Delta Lake to skip files based on min/max statistics for the Z-ordered column. When the filter column is Z-ordered, the query can skip many files even if it is not the partition column, dramatically reducing the amount of data scanned and improving performance without changing the partitioning scheme.

Why this answer

When a filter is on a non-partition column, Delta Lake can still skip files if it has min/max statistics for that column and the data is clustered. Z-ORDER BY reorganizes data so that related values are stored together, improving the effectiveness of data skipping. Running OPTIMIZE with Z-ORDER BY on the filter column allows the query to read only the relevant files, avoiding a full scan of all partitions.

Exam trap

The trap here is assuming that partitioning is the only way to achieve data skipping, overlooking Z-ORDER BY as a complementary technique for non-partition columns.

35
Multi-Selectmedium

A data engineer is configuring a Lakeflow Job that processes sensitive customer data. The job must notify the on-call team when a run fails and must also capture the run's output for auditing. Which TWO actions should the engineer take in the Lakeflow Jobs configuration? (Choose two.)

Select 2 answers
A.Add an email notification for job failure that targets the on-call distribution list.
B.Configure a retry policy with three retries to ensure the on-call team is notified.
C.Add a task-level timeout so the job fails faster and generates an alert.
D.Enable the job's run history and export logs to a durable storage location for auditing.
E.Set the job's maximum concurrent runs to 1 to prevent overlapping audit records.
AnswersA, D

Email notifications configured for the failure event send an alert to specified recipients when the job run fails. Targeting the on-call distribution list ensures the team is paged without manual monitoring. This directly satisfies the requirement to notify the team on failure and is a native Lakeflow Jobs notification option.

Why this answer

Failure notifications deliver timely alerts to the on-call team, and preserving run history with exported logs provides the audit trail. Together these native Lakeflow Jobs capabilities meet both the alerting and auditing requirements without adding unnecessary concurrency or retry complexity that would not notify or record anything on its own.

Exam trap

The trap here is conflating reliability settings such as retries and concurrency limits with notification and audit capabilities, which are separate configuration areas.

36
MCQmedium

Which security feature should a data engineer configure to ensure that audit logs from all Databricks workspaces are captured and stored in a single, centralized location?

A.Cluster event logs.
B.Diagnostic logs.
C.System schema (system.access).
D.Workspace-local storage.
AnswerB

Diagnostic logs are the official channel for exporting audit data from Databricks. By streaming these to a workspace-external destination, administrators can aggregate audit data from multiple workspaces into one place, enabling a unified view of all user activity, configuration changes, and data access events for audit and security analysis.

Why this answer

Databricks diagnostic logs are the primary mechanism for collecting security, governance, and operational audit information. By configuring diagnostic log streaming to a centralized location, such as a cloud-native log repository (e.g., Azure Log Analytics or AWS S3), organizations can perform centralized monitoring, anomaly detection, and compliance reporting across their entire Databricks estate, ensuring no audit trail is lost or siloed in individual workspaces.

Exam trap

Candidates frequently confuse standard cluster log delivery with audit logs, missing that security events and governance tracking across workspaces require configuring diagnostic logs.

37
MCQhard

A data engineer is using Auto Loader to ingest data from a Kafka topic into a Delta table. The engineer wants to ensure that the ingestion handles late-arriving data and provides exactly-once semantics. Which combination of features should the engineer use?

A.Auto Loader with cloudFiles.useNotifications = true and checkpointing
B.Structured Streaming with Kafka source, watermarking, and checkpointing
C.COPY INTO with Kafka connector and FORCE = true
D.Delta Live Tables with Kafka source and APPLY CHANGES
AnswerB

Structured Streaming provides a Kafka connector that supports exactly-once semantics via checkpointing and offset management. Watermarking allows handling late-arriving data by specifying a threshold for how late data can arrive. This combination is the correct approach for ingesting from Kafka with exactly-once and late data handling.

Why this answer

For Kafka ingestion with exactly-once semantics and late data handling, Structured Streaming with the Kafka source, watermarking, and checkpointing is the correct approach. Checkpointing ensures exactly-once by tracking offsets, and watermarking allows processing of late-arriving data up to a specified threshold.

Exam trap

The trap here is mixing Auto Loader or COPY INTO with Kafka, as those are for file-based sources, and overlooking the need for watermarking to handle late data.

38
MCQmedium

A data engineering team is building a medallion architecture in Databricks. In the Silver layer, streaming data from Kafka must be cleaned, deduplicated, and written into a Delta table. Which Structured Streaming output mode should the engineer select to ensure append-only storage of fully processed, stateful deduplicated records?

A.Complete output mode
B.Update output mode
C.Append output mode
D.Overwrite output mode
AnswerC

Append output mode is the default and mandatory output mode when executing stateful operations like dropDuplicates() on streaming DataFrames. It ensures that only new, non-duplicate rows resulting from each micro-batch are appended to the target Delta table sink.

Why this answer

When performing stateful operations such as deduplication with dropDuplicates() on a streaming DataFrame, Structured Streaming requires the output mode to be set to append. This mode allows newly emitted unique rows to be written continuously to the target Delta table without requiring complete table rewrites, fitting the streaming append-only nature of the Silver medallion layer.

Exam trap

Candidates often mistakenly choose 'Complete' output mode when dealing with stateful deduplication, forgetting that complete mode requires rewriting the entire table on every trigger.

39
MCQmedium

Refer to the exhibit. A data engineer is reviewing the configuration of a Delta table that is frequently queried by 'customer_id'. Given the current metadata, which action will provide the most significant improvement to query performance?

A.Execute VACUUM to remove old files
B.Run OPTIMIZE with ZORDER BY (customer_id)
C.Re-partition the table by 'customer_id'
D.Increase the 'order_date' partition size
AnswerB

Z-Ordering by 'customer_id' reorganizes the data into files that share similar customer IDs. This allows the query engine to skip entire files that do not contain the requested ID, making the 'orders' table queries significantly faster, especially when combined with the file compaction that occurs during the optimization process.

Why this answer

The table is currently suffering from poor data layout due to the small file size and lack of clustering on the frequently queried 'customer_id' column. Performing an OPTIMIZE operation with Z-Ordering on 'customer_id' will cluster the data physically on disk based on that key. This enables the engine to perform data skipping, reducing the amount of data scanned and significantly speeding up point lookups and aggregations by customer.

Exam trap

Candidates often choose partitioning or indexing, but in Delta Lake, Z-Ordering is the specific technique to improve data skipping for high-cardinality columns like 'customer_id'.

40
MCQeasy

A data engineer needs to store structured data in a cloud object storage location while maintaining full ACID guarantees. Which storage format is the foundation of the Databricks Lakehouse architecture that enables this functionality?

A.JSON
B.Delta Lake
C.CSV
D.Avro
AnswerB

Delta Lake provides the necessary ACID transactional capabilities, scalable metadata handling, and time travel features required for a robust Lakehouse. It sits on top of existing object storage, ensuring data integrity across complex pipelines while allowing high-performance concurrent access for both streaming and batch data processing workloads.

Why this answer

Delta Lake is the open-source storage layer that brings reliability and ACID transactions to data lakes. By utilizing transaction logs and Parquet files, it ensures data consistency for concurrent read and write operations. Understanding Delta Lake is fundamental for Databricks engineers, as it underpins virtually all data operations within the platform, enabling features like time travel, schema enforcement, and efficient upserts on massive datasets.

Exam trap

Candidates often confuse Delta Lake with the underlying storage format (Parquet) or the compute engine (Spark). They fail to realize that Delta Lake is specifically the ACID layer built on top of Parquet files.

41
MCQhard

A data engineer has a Lakeflow Job with two tasks: Task1 and Task2. Task2 must run only if Task1 succeeds. The engineer also wants Task2 to be skipped if Task1 fails, but the overall job status should be marked as failed. Which configuration should the engineer use for the dependency between Task1 and Task2?

A.Set Task2 to depend on Task1 with the 'All done' condition.
B.Set Task2 to depend on Task1 with the 'At least one succeeded' condition.
C.Set Task2 to depend on Task1 with the 'None failed' condition.
D.Set Task2 to depend on Task1 with the 'All succeeded' condition.
AnswerD

The 'All succeeded' condition means Task2 runs only if Task1 completes successfully. If Task1 fails, Task2 is skipped, and the job is marked as failed because a task failed. This matches the requirement: Task2 runs only on success, and the job fails if Task1 fails.

Why this answer

The 'All succeeded' condition ensures Task2 runs only when Task1 succeeds. If Task1 fails, Task2 is skipped, and the job is marked as failed because a task failed. This precisely meets the requirement of conditional execution and overall job failure.

Exam trap

The trap here is confusing 'All succeeded' with 'None failed' or 'All done', which have different semantics for skip and failure propagation.

42
MCQmedium

A data engineer is configuring a Databricks cluster to run a Spark job that processes large datasets. The job requires high memory and will run for several hours. The engineer wants to minimize costs while ensuring the job completes successfully. Which cluster configuration should the engineer choose?

A.Use an all-purpose cluster with a fixed number of high-memory workers and enable autoscaling to reduce costs during peak hours.
B.Use a job cluster with a fixed number of high-memory workers and enable auto-termination after 30 minutes of inactivity.
C.Use a job cluster with autoscaling enabled, high-memory instance types, and a minimum of 2 workers and a maximum of 10 workers.
D.Use an all-purpose cluster with autoscaling enabled and a minimum of 2 workers and a maximum of 10 workers, with high-memory instance types.
AnswerC

Job clusters are designed for automated workloads and are more cost-effective than all-purpose clusters. Autoscaling dynamically adjusts the number of workers based on workload, optimizing resource usage and cost. High-memory instances address the memory requirement. This configuration balances performance and cost for a long-running job with varying demands.

Why this answer

For long-running batch jobs, job clusters are more cost-effective than all-purpose clusters. Autoscaling allows the cluster to scale up or down based on workload, ensuring resources are used efficiently. High-memory instances meet the memory requirement.

Setting a minimum and maximum worker count provides boundaries for autoscaling, preventing over-provisioning. This combination minimizes cost while ensuring the job has sufficient resources.

Exam trap

The trap here is assuming that all-purpose clusters are always better because they can be reused, but for a single long-running job, a job cluster with autoscaling is more cost-effective.

43
MCQmedium

A data engineer needs to pass the execution date to a job task dynamically. Which feature should they use?

A.Hardcoding the date in the notebook.
B.Defining job parameters in the task configuration.
C.Creating a new cluster for each date.
D.Using global workspace variables.
AnswerB

Task-level parameters allow values to be passed to notebooks or scripts at execution time. These parameters can be referenced within the code using standard widget APIs or environment variable lookups, providing a clean and programmatic way to inject dynamic information into the task execution without altering the underlying logic.

Why this answer

Using Job Parameters allows engineers to inject dynamic values into tasks at runtime. This capability is crucial for backfilling data or running incremental pipelines where the processing logic depends on specific dates or data partitions. By parameterizing tasks, engineers create reusable jobs that can handle varying datasets without requiring manual code changes, thereby improving the maintainability and scalability of the entire data pipeline architecture.

Exam trap

Candidates often confuse task parameters with cluster-level environment variables or hardcoded notebook widgets, failing to use the native job task configuration feature.

44
MCQmedium

Refer to the exhibit. A data engineer attempts to run the GRANT statement above in a Databricks SQL query editor. What is the most likely reason for failure?

A.The syntax of the GRANT statement is incorrect.
B.The table does not exist.
C.The table must be referenced using a three-level namespace.
D.The group 'data_scientists' does not exist.
AnswerC

Unity Catalog requires the fully qualified name (catalog.schema.table). Providing only the table name causes the operation to fail because the metastore cannot resolve the resource context. This forces explicit identification of the data, which is a security best practice to prevent unauthorized or unintended access to tables.

Why this answer

The command fails because Unity Catalog requires the full three-level namespace (catalog.schema.table). Without specifying 'main.prod.sales_data', the engine cannot locate the table within the hierarchical metadata store. This strict requirement ensures that all data access is explicitly scoped, preventing ambiguity in multi-catalog or multi-schema environments, which is a fundamental tenet of Unity Catalog's governance model for enterprise-grade security and asset management.

Exam trap

Candidates often assume that the current active catalog and schema context apply to GRANT statements, forgetting that Unity Catalog strictly mandates the full three-level namespace.

45
Multi-Selectmedium

A team is setting up a CI/CD pipeline that deploys Databricks Asset Bundles to production. They want the pipeline to be secure and to fail fast before any production resources are changed. Which TWO practices should they implement? (Choose two.)

Select 2 answers
A.Run the deployment on every push to any branch so problems are discovered as early as possible in feature development.
B.Authenticate the deployment using a service principal with credentials injected from the CI system's secret manager at runtime.
C.Run databricks bundle validate in an earlier pipeline stage so configuration errors are caught before the deploy step executes.
D.Store the service principal secret as a plaintext variable in the repository's CI configuration file so all pipeline stages can read it.
E.Grant the service principal workspace admin rights so it can deploy any resource without encountering permission errors.
AnswersB, C

A service principal provides non-interactive, auditable automation identity, and injecting its secret from the CI secret manager at runtime keeps credentials out of the repository and logs. This is the recommended way to authenticate deployments in CI/CD. It satisfies the security requirement while allowing the pipeline to deploy to production reliably across runs.

Why this answer

Failing fast and staying secure means validating the bundle before deploying and authenticating with a service principal whose secret comes from the CI secret manager. Validation catches configuration errors before production is touched, and the secret manager keeps credentials out of the repository. Plaintext secrets, admin-level permissions, and branch-agnostic production deploys all increase risk without meeting the stated goals.

Exam trap

The trap here is treating broad permissions or early deployment as safety measures, when least privilege and pre-deploy validation are what actually protect production.

46
Multi-Selecthard

A data engineer is working with a large, partitioned table and needs to perform a complex transformation. Which THREE of the following strategies will optimize query performance for this transformation?

Select 3 answers
A.Use Z-Ordering on columns frequently used in WHERE clauses.
B.Partition the table by every column in the dataset.
C.Filter data using the partition columns in the query.
D.Always use a full table scan to ensure data integrity.
E.Enable Auto-Optimize to automatically compact small files.
AnswersA, C, E

Z-Ordering co-locates related information in the same set of files. This significantly improves data skipping when queries filter by the Z-Ordered columns, as the engine can quickly ignore files that do not contain relevant data, reducing total I/O consumption.

Why this answer

Performance optimization in Databricks relies on intelligent data layout and compute utilization. Techniques like Z-Ordering, proper partitioning, and effective filtering are essential for reducing the amount of data scanned. By selecting the right combinations of these strategies, engineers can significantly reduce the I/O overhead and compute costs, which is paramount for large-scale production ETL/ELT workflows.

Exam trap

Candidates often select 'Repartitioning' or 'Increasing Cluster Size' as primary optimization strategies, failing to recognize that Z-Ordering and file compaction are the specific, targeted solutions for file-level performance issues in Delta Lake.

47
MCQmedium

When promoting code from a development workspace to a production workspace, what is the primary risk of using manual notebook exports?

A.The notebooks will run significantly slower in the production environment.
B.The process creates an inconsistent state between development and production.
C.The Databricks workspace will automatically delete the old notebooks.
D.Manual exports are prohibited by Databricks security policies.
AnswerB

Manual processes lack the rigor of automated CI/CD pipelines. Differences in libraries, environment settings, or notebook versions often occur when files are manually copied. These inconsistencies lead to code that works in development but fails in production, causing difficult-to-debug errors and potential downtime for critical data tasks.

Why this answer

Manual notebook exports lack version integrity and metadata synchronization, leading to 'code drift' where the production code doesn't match the source of truth in Git. This manual process is prone to human error, such as importing the wrong version or missing library dependencies. Standardizing on Git-integrated workflows or Databricks Asset Bundles mitigates this risk by automating the deployment and ensuring that exactly what was tested in development reaches production.

Exam trap

Candidates often view manual notebook exports as harmless, overlooking how easily they introduce code drift and lack metadata synchronization across environments.

48
MCQeasy

A data engineer needs to create a Silver Delta table that contains only distinct, non-null `customer_id` values from a Bronze table, and the result must be refreshed idempotently each night. Which statement best satisfies the requirement?

A.CREATE OR REPLACE TABLE silver AS SELECT DISTINCT customer_id FROM bronze WHERE customer_id IS NOT NULL
B.INSERT OVERWRITE silver SELECT DISTINCT customer_id FROM bronze WHERE customer_id IS NOT NULL
C.MERGE INTO silver AS t USING bronze AS s ON t.customer_id = s.customer_id WHEN NOT MATCHED THEN INSERT (customer_id) VALUES (s.customer_id)
D.INSERT INTO silver SELECT DISTINCT customer_id FROM bronze WHERE customer_id IS NOT NULL
AnswerA

CREATE OR REPLACE TABLE atomically replaces the table contents on each run, so the result is always the current distinct set of non-null customer IDs. Re-running the job produces the same final state, satisfying idempotency. It also keeps the output as a Delta table with transactional guarantees, which is appropriate for a Silver layer.

Why this answer

CREATE OR REPLACE TABLE with a DISTINCT and null-filtered SELECT atomically rebuilds the Silver table each night, guaranteeing the same result on every run. This satisfies both the distinct non-null requirement and idempotent refresh. Appending via INSERT INTO, incremental MERGE, or relying on a pre-existing target for INSERT OVERWRITE does not meet all the stated conditions.

Exam trap

The trap here is choosing INSERT INTO or MERGE for a full-refresh snapshot, when those patterns accumulate rows and break the idempotent nightly rebuild.

49
MCQmedium

Which Spark configuration property can be used to enable Adaptive Query Execution (AQE) in Databricks?

A.spark.sql.adaptive.enabled
B.spark.sql.autoBroadcastJoinThreshold
C.spark.databricks.adaptive.execution
D.spark.sql.shuffle.partitions
AnswerA

Setting spark.sql.adaptive.enabled to 'true' is the required configuration to turn on the Adaptive Query Execution framework. Once enabled, Spark will dynamically optimize query execution plans based on runtime statistics, leading to more efficient processing and faster performance for complex SQL queries in Spark 3.x environments.

Why this answer

Adaptive Query Execution (AQE) is a critical optimization framework in Spark that re-optimizes query plans at runtime based on statistics collected during shuffle stages. It enables features like coalescing shuffle partitions and dynamically switching join strategies. Enabling AQE is a best practice for modern Databricks workloads as it provides significant performance gains without manual intervention, automatically adapting to varying data distributions in production pipelines.

Exam trap

Candidates often confuse AQE with general Spark configurations like executor memory or shuffle partitions. They might pick settings that control static partitioning instead of the dynamic runtime optimization framework.

50
Multi-Selectmedium

A data engineer is working on a Bronze-to-Silver transformation. Which THREE of the following practices are recommended to optimize performance and data quality during this stage?

Select 3 answers
A.Apply data quality checks to filter out or quarantine malformed records before writing to Silver.
B.Always use 'SELECT *' to ensure the entire schema is captured in the Silver table.
C.Implement streaming writes to ensure data is processed in micro-batches.
D.Perform heavy joins and aggregations directly on the Bronze table.
E.Use Z-Ordering on frequently filtered columns to speed up downstream queries.
AnswersA, C, E

Validation at the Bronze layer is critical. By filtering malformed data, engineers prevent 'garbage-in-garbage-out' scenarios in downstream reporting. Quarantine patterns, where invalid records are moved to a separate error table, allow for auditability and manual correction without disrupting the main production data flow.

Why this answer

Effective Bronze-to-Silver transformations are the foundation of a robust Lakehouse. By combining data validation, efficient partitioning, and incremental processing, engineers can minimize resource consumption while ensuring the data is reliable. These practices ensure the pipeline remains performant as volume scales and that the downstream Gold tables contain clean, query-optimized data, which is critical for business intelligence and data science use cases.

Exam trap

Candidates often select 'Drop all malformed records' as a best practice, failing to realize that quarantining or filtering via validation is superior for auditability and data recovery.

51
MCQmedium

Refer to the exhibit. A user who is a member of 'data_analysts_group' reports they cannot see the 'orders' table in the Catalog Explorer. What is the most likely cause?

A.The user lacks the 'SELECT' privilege.
B.The user lacks the 'USAGE' privilege on the parent objects.
C.The table is not registered in the Hive Metastore.
D.The group does not exist in the workspace.
AnswerB

Unity Catalog requires USAGE on the catalog and schema to navigate to the table. If these permissions were missing, the user would not be able to traverse the path to the table in Catalog Explorer, even if they had SELECT permissions on the table itself. This is a common security configuration error.

Why this answer

In Unity Catalog, the 'USAGE' privilege is a mandatory prerequisite for navigating any parent object. Even if the user has SELECT access on the table, they must also hold USAGE on the parent catalog and schema to traverse the hierarchy and discover the table. Without these permissions, the object remains invisible to the user in the UI, even if they have technical access to the underlying data records.

Exam trap

Candidates focus solely on table-level permissions and overlook the requirement for 'USAGE' privileges on the parent catalog and schema, which are mandatory for object discovery in the UI.

52
MCQeasy

What is the primary purpose of a 'feature branch' in a Git-based workflow for Databricks?

A.To allow multiple users to edit the same file simultaneously without sync.
B.To isolate changes so they can be tested before merging to production.
C.To bypass the authentication requirements for the workspace.
D.To automatically deploy changes to the production environment.
AnswerB

Branching provides a safe sandbox for development. By working on a feature branch, developers can test their code in a dedicated workspace environment without endangering the current production code. This separation is fundamental to CI/CD, as it ensures that only verified, stable code is eventually merged into the main line.

Why this answer

Feature branches allow developers to build and test new functionality in isolation, without affecting the main production code. This workflow is essential for CI/CD because it facilitates peer reviews, automated integration testing, and safe integration. By keeping the main branch stable and only merging validated features, teams maintain a high standard of quality and avoid breaking the production environment with incomplete or untested code changes, which is a core benefit of modern DevOps practices.

Exam trap

Candidates often confuse feature branches with production deployment branches, missing the core purpose of isolation for testing and peer review prior to merging into the main branch.

53
MCQmedium

A data engineer notices that a scheduled Delta Lake maintenance pipeline is running significantly slower than expected. Upon checking the table history, they see that hundreds of tiny, fragmented data files have accumulated due to frequent streaming micro-batches. Which specific optimization command should the data engineer run first to resolve this file-size bottleneck?

A.Run VACUUM table_name RETAIN 0 HOURS to instantly purge all historical file references and force file compaction across the entire dataset.
B.Execute REFRESH TABLE table_name to clear the local metastore cache and force Spark to rebuild the underlying file index from scratch.
C.Invoke the OPTIMIZE table_name command to compact small files into larger, optimized data files and improve subsequent query execution efficiency.
D.Execute ALTER TABLE table_name SET TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true) to rewrite past historical micro-batches retroactively.
AnswerC

OPTIMIZE compacts the hundreds of small fragmented files produced by frequent streaming micro-batches into larger, right-sized data files. This directly resolves the file-size bottleneck, reducing per-file overhead and improving subsequent query execution efficiency without altering the underlying data.

Why this answer

Running OPTIMIZE reorganizes the layout of Delta Lake data by compacting small files into larger, uniform files of roughly 1 GB in size. This significantly reduces metadata overhead and improves scan performance for subsequent read queries. While VACUUM removes old physical files, it does not compact active files.

OPTIMIZE directly addresses the core symptom of small file proliferation caused by frequent streaming micro-batches.

Exam trap

Candidates often confuse OPTIMIZE with VACUUM, believing that cleaning up old versions will automatically combine active small files into larger ones, which is not what VACUUM does.

54
MCQmedium

A data engineer needs to configure a Databricks Job containing multiple tasks where downstream tasks should only execute if all upstream parent tasks complete successfully. Which task dependency setting should be configured?

A.Configure a retry policy with exponential backoff on the parent task to guarantee eventual success.
B.Set up a global workspace webhook to trigger downstream tasks manually after the parent completes.
C.Add the upstream tasks as parents of the downstream task directly within the job configuration graph.
D.Enable concurrent run suppression on the job level to prevent overlapping workflow executions.
AnswerC

Declaring upstream tasks as parents of the downstream task in the job graph creates explicit dependencies, so Databricks triggers the downstream task only after every parent completes successfully. This directly enforces the all-upstream-success condition the scenario requires.

Why this answer

To ensure tasks execute conditionally based on the success of parent tasks, you configure task dependencies within the Databricks Jobs UI or JSON definition. By default, adding a parent task creates a strict dependency where downstream tasks trigger only upon successful completion. This mechanism orchestrates complex directed acyclic graphs for robust, reliable data pipelines.

Exam trap

Candidates sometimes try to manage task sequencing using external orchestration tools instead of native Databricks dependency graphs within the job configuration.

55
Multi-Selecthard

A data engineering team is setting up a CI/CD pipeline for Databricks notebooks using Databricks Repos and a Git provider. They want to ensure that changes are tested before being merged and that production deployments are controlled. Which TWO practices should they implement? (Choose two.)

Select 2 answers
A.Store production credentials in the Repo's notebooks as widgets so the CI job can read them during tests.
B.Run automated tests in a CI job on a feature branch using a dedicated test workspace before merging to main.
C.Merge changes only after the CI pipeline passes and use a separate deployment job to update the production Repo to a tagged release.
D.Allow developers to commit directly to the main branch in the production workspace to reduce merge conflicts.
E.Configure the production Repo to track the main branch and enable automatic pull on every commit.
AnswersB, C

Running tests on feature branches in an isolated test workspace validates changes without affecting production data or users. This practice catches integration issues early and aligns with the CI principle of fast feedback. Using a dedicated workspace also prevents accidental writes to production catalogs and keeps test credentials separate from production credentials.

Why this answer

A robust CI/CD process for Databricks Repos runs automated tests on feature branches in a dedicated test workspace, then gates merges on CI success. Production deployment should be a separate, controlled step that updates the production Repo to a tagged release, providing an immutable and rollback-friendly reference. These two practices together enforce quality gates and controlled promotion, which are core to CI/CD with Databricks Repos.

Exam trap

The trap here is confusing continuous synchronization of the main branch with proper CI/CD, when production should instead be updated through a controlled release step.

56
MCQeasy

A data engineer is creating a Lakeflow Job that must run a Python script stored in DBFS. The engineer wants to ensure the script is executed with the correct dependencies and environment. Which task type should be used?

A.JAR task
B.Notebook task
C.Pipeline task
D.Python script task
AnswerD

The Python script task type in Lakeflow Jobs allows you to specify a Python file stored in DBFS, Unity Catalog volumes, or cloud storage. It executes the script with the specified libraries and environment, making it ideal for running standalone Python scripts with dependencies. This directly matches the requirement.

Why this answer

Lakeflow Jobs offer a dedicated Python script task type that runs a Python file from DBFS or other storage locations. It supports specifying libraries and environment settings, ensuring the script runs with the correct dependencies. Notebook, JAR, and pipeline tasks serve different purposes and cannot directly execute a standalone Python script with its own environment.

Exam trap

The trap here is assuming a notebook task can run any Python file; notebook tasks specifically require a notebook format and do not execute standalone .py scripts.

57
MCQmedium

A data engineer needs to share a subset of a table with an external partner who does not have access to the internal Databricks workspace. What is the most appropriate method to achieve this?

A.Create a service principal with restricted access.
B.Export data to CSV and email it.
C.Use Delta Sharing to share a table.
D.Grant the partner access to a shared S3 bucket.
AnswerC

Delta Sharing provides a secure and governed way to share data with third parties. It does not require the recipient to be a user in the provider's environment, thereby keeping the internal workspace secure while allowing controlled access to specific datasets through signed URLs and the Delta protocol.

Why this answer

Delta Sharing is the secure, open-standard solution for sharing data externally. It allows the recipient to access data directly from cloud storage without needing to provision an account in the provider's Databricks workspace. This process is secure because it uses short-lived tokens and does not require complex credential management, ensuring that data stays governed and that the provider retains control over what is being shared.

Exam trap

Candidates often choose 'exporting to CSV' or 'creating a shared volume,' failing to realize that Delta Sharing is the only native, secure, and governance-compliant method for sharing data outside the Databricks ecosystem.

58
MCQhard

A data engineer is working with a Delta table that contains a column 'timestamp' of type timestamp. The table is partitioned by date. The engineer needs to run a query that filters on a specific date range and also on a high-cardinality column 'user_id'. The query is performing poorly. Which optimization technique should the engineer apply to improve query performance?

A.Use Z-ORDER BY on the 'user_id' column when writing the Delta table.
B.Convert the Delta table to Parquet format to leverage predicate pushdown.
C.Enable Delta Lake caching by running CACHE TABLE on the Delta table.
D.Repartition the Delta table by the 'timestamp' column.
AnswerA

Z-ORDER BY is a technique that colocates related data in the same set of files, improving data skipping for queries that filter on the Z-ordered columns. By Z-Ordering on 'user_id', the engineer can reduce the amount of data scanned when filtering on that high-cardinality column. This is especially effective when combined with partitioning on date, as it further optimizes within each partition. It is a recommended best practice for Delta Lake performance tuning.

Why this answer

Z-ORDER BY is designed to optimize data skipping for high-cardinality columns by clustering data with similar values in the same files. This reduces the number of files scanned when filtering on that column. Partitioning on date already helps with date-range filters, but Z-Ordering on user_id addresses the high-cardinality filter.

Caching, repartitioning, or converting to Parquet do not provide the same targeted benefit for this scenario.

Exam trap

The trap here is assuming that caching or repartitioning will solve selective query performance, when the real issue is data layout for skipping on a high-cardinality column.

59
Multi-Selecthard

A data engineer is designing a secure architecture using the Databricks Intelligence Platform. Which TWO of the following statements accurately describe the role and capabilities of Unity Catalog within this platform? (Choose TWO)

Select 2 answers
A.Unity Catalog provides centralized access control for tables, views, volumes, and AI models across multiple workspaces.
B.Unity Catalog replaces the underlying cloud provider IAM roles for all compute resources executing queries.
C.Unity Catalog maintains a single, unified metastore that can be shared across multiple workspaces within the same account.
D.Unity Catalog automatically handles the physical partitioning of Delta tables for optimal query performance.
E.Unity Catalog requires all data to be migrated into proprietary binary formats before governance policies can be applied.
AnswersA, C

Unity Catalog centralizes governance by defining permissions once in an account-level metastore and applying them consistently across all attached workspaces. This architecture simplifies auditing and ensures uniform security standards for both structured data and machine learning artifacts.

Why this answer

Unity Catalog is the fine-grained governance solution for data and AI across the Databricks platform, centralizing access control for tables, volumes, and models. It operates across multiple workspaces in a metastore, ensuring consistent security policies and audit logging regardless of which workspace or compute resource executes the user queries.

Exam trap

A common misconception is that Unity Catalog operates locally within a single workspace or manages underlying cloud virtual machines. In reality, a Unity Catalog metastore can be shared across multiple workspaces in an account.

60
MCQmedium

A data engineer is building a Lakeflow Job that must process a parameterized date range. The engineer wants to pass start_date and end_date values into a notebook task at runtime and have those values available as widget-like parameters inside the notebook. Which approach should the engineer use?

A.Define job parameters and reference them in the notebook using the task's base parameters, then read them with dbutils.widgets.get in the notebook.
B.Store the dates in a Delta table and have the notebook query that table at startup.
C.Use the notebook's %run magic to pass arguments from a parent notebook.
D.Set environment variables on the job cluster and read them with os.environ in the notebook.
AnswerA

Job parameters and task base parameters are passed to notebook tasks as widget values. The notebook can retrieve them with dbutils.widgets.get using the parameter key. This is the native way to parameterize a Lakeflow Job task and makes start_date and end_date available at runtime without hardcoding values.

Why this answer

Lakeflow Jobs parameters are surfaced to notebook tasks as widget values, so the notebook can call dbutils.widgets.get with the parameter name. Defining start_date and end_date as job or task base parameters makes them available per run and visible in the job UI, enabling dynamic date-range processing without hardcoding or external lookups.

Exam trap

The trap here is assuming that cluster environment variables or external tables are the primary way to pass runtime values, when Lakeflow Jobs exposes parameters to notebooks as widgets.

61
MCQmedium

A data engineer is configuring audit logging for a Databricks workspace that uses Unity Catalog. The security team requires that all access to data in Unity Catalog be logged and available for analysis in a centralized location. The data engineer wants to enable the delivery of audit logs to an AWS S3 bucket. Which configuration should the data engineer use?

A.Configure an audit log delivery in the account console, specifying the S3 bucket and IAM role for delivery.
B.Enable diagnostic logging in the Databricks workspace and configure it to send logs to S3.
C.Enable Unity Catalog audit logs in the workspace settings and specify an S3 path for storage.
D.Use the Databricks REST API to stream audit logs to an S3 bucket via a custom script.
AnswerA

Unity Catalog audit logs are delivered via a system table or to a cloud storage location configured at the account level. The account console allows setting up audit log delivery to an S3 bucket with an IAM role that grants write access. This centralizes logs for security analysis.

Why this answer

Databricks audit logs, including Unity Catalog data access, can be delivered to an AWS S3 bucket through account-level configuration. This involves setting up a delivery mechanism with an IAM role that has write permissions to the bucket. The account console provides the interface for this setup, ensuring centralized log collection.

Exam trap

The trap here is confusing workspace-level diagnostic logging with account-level audit log delivery, or assuming audit logs are configured per workspace.

62
MCQhard

A data engineer is using Auto Loader to stream data from Kafka into a Delta table. The Kafka topic receives messages in Avro format, and the schema is stored in a Confluent Schema Registry. The engineer wants Auto Loader to automatically fetch the schema from the registry and evolve it as new versions are registered. Which configuration should the engineer use?

A.Use the Kafka source with the option kafka.schema.registry.url and set the value format to 'avro'.
B.Set the option cloudFiles.format to 'avro' and provide the schema registry URL via cloudFiles.schemaRegistryUrl.
C.Set the option cloudFiles.useNotifications to true and configure Kafka as a notification source.
D.Auto Loader cannot ingest from Kafka; use Structured Streaming with the Kafka source and Schema Registry integration.
AnswerD

Auto Loader is specifically designed for ingesting files from cloud storage into Delta Lake. It does not support Kafka as a source. To ingest from Kafka with Avro and Schema Registry, the engineer should use Spark Structured Streaming's Kafka source, which provides built-in support for Schema Registry via options like kafka.schema.registry.url and value.deserializer. This allows schema fetching and evolution. Therefore, the correct approach is to use Structured Streaming, not Auto Loader.

Why this answer

Auto Loader is a file ingestion tool and does not support Kafka as a source. For Kafka ingestion with Avro and Schema Registry, the appropriate method is to use Spark Structured Streaming's Kafka source, which can integrate with Schema Registry to fetch and evolve schemas. The engineer should not attempt to use Auto Loader for this purpose.

Therefore, the correct answer is to use Structured Streaming with the Kafka source and the necessary Schema Registry configurations.

Exam trap

The trap here is assuming that Auto Loader can handle any streaming source, when it is limited to file-based sources in cloud storage.

63
MCQhard

Refer to the exhibit. A job fails with the provided error message. What is the most likely cause of this failure in a Databricks Delta Lake environment?

A.The Delta table has reached the maximum file limit.
B.The VACUUM retention threshold was set too low.
C.The cluster running the job lacks proper IAM permissions.
D.The transaction log is corrupted and requires manual deletion.
AnswerB

Setting the VACUUM retention period shorter than the time a long-running process needs to access old data leads to failures. If a process is mid-read while a concurrent VACUUM deletes files, the reader will be unable to locate the necessary data files, resulting in the reported error.

Why this answer

This error occurs when the Delta log entries point to files that no longer exist, typically caused by a VACUUM operation running with a retention period shorter than the time required for long-running streaming jobs to finish. When a file is physically deleted by VACUUM, but a reader still references it via the transaction log, the read operation fails because the file object is inaccessible in the underlying storage.

Exam trap

Candidates often assume the error is related to network connectivity or cluster permissions. They overlook the relationship between long-running jobs and the aggressive deletion of files by the VACUUM command.

64
MCQmedium

An organization is adopting the Databricks Intelligence Platform and wants to leverage Mosaic AI for building custom machine learning models. Which feature allows data engineers to track machine learning experiments, log parameters, and manage model artifacts reliably?

A.Databricks SQL dashboards
B.Unity Catalog volumes
C.MLflow
D.Delta Live Tables
AnswerC

MLflow is natively integrated into Databricks to provide robust experiment tracking, parameter and metric logging, and a centralized model registry. It is the primary tool used within Mosaic AI to manage machine learning lifecycles and artifact governance.

Why this answer

MLflow is the open-source platform integrated natively into Databricks Mosaic AI for managing the end-to-end machine learning lifecycle. It provides comprehensive tracking of experiments, parameters, metrics, and registered model versions, ensuring reproducibility and smooth transition from development to production environments within the data lakehouse architecture.

Exam trap

Candidates often look for a custom Databricks-specific tool and overlook MLflow because it is an open-source project. However, MLflow is deeply embedded as the standard tracking mechanism in Databricks Mosaic AI.

65
MCQmedium

A data engineer is configuring a Databricks Auto Loader stream to ingest CSV files from a cloud storage location into a Delta table. The CSV files have a header row, and the engineer wants to automatically infer the schema and store the inferred schema in a specified location for consistency across restarts. Which Auto Loader option should be used to persist the inferred schema?

A.Set cloudFiles.inferColumnTypes to 'true'.
B.Set cloudFiles.schemaEvolutionMode to 'addNewColumns'.
C.Set cloudFiles.useIncrementalListing to 'true'.
D.Set cloudFiles.schemaLocation to a directory path.
AnswerD

This option correctly specifies the directory where Auto Loader stores the inferred schema and its evolution. By providing a schemaLocation, the stream maintains schema consistency across restarts and allows schema evolution to be tracked. This is the recommended approach when using schema inference with Auto Loader, as it avoids re-inferring the schema on each run and supports adding new columns over time.

Why this answer

Auto Loader requires a schemaLocation to persist the inferred schema and support schema evolution. Without it, the stream may re-infer the schema on each restart, causing inconsistencies. The other options control schema evolution behavior or listing optimizations but do not provide a storage location for the schema.

Exam trap

The trap here is confusing schema evolution settings with schema storage configuration; only schemaLocation persists the schema.

66
Multi-Selectmedium

A data engineer is managing a Unity Catalog table that contains sensitive financial data. The table is owned by the 'finance' group, and the data engineer needs to allow the 'auditors' group to read the table but not modify it. Additionally, the data engineer wants to ensure that the 'auditors' group can see the table's metadata (e.g., column names and types) but cannot access the underlying data files directly. Which TWO actions should the data engineer take to meet these requirements? (Choose two.)

Select 2 answers
A.Grant READ FILES on the external location to the 'auditors' group.
B.Grant MODIFY on the table to the 'auditors' group.
C.Grant USAGE on the schema containing the table to the 'auditors' group.
D.Grant ALL PRIVILEGES on the table to the 'auditors' group.
E.Grant SELECT on the table to the 'auditors' group.
AnswersC, E

USAGE on the schema is required for any user to access objects within that schema. Without USAGE, even if SELECT is granted on the table, the auditors cannot resolve the table's fully qualified name. This is a necessary prerequisite for table access in Unity Catalog.

Why this answer

To allow read-only access, the data engineer must grant SELECT on the table and USAGE on the containing schema. SELECT provides read access and metadata visibility, while USAGE is required to access any object within the schema. Granting MODIFY or ALL PRIVILEGES would allow modifications, and READ FILES would bypass table-level security.

These two privileges together satisfy the requirements without over-provisioning.

Exam trap

The trap here is forgetting that USAGE on the schema is a prerequisite for table access, or mistakenly granting file-level permissions that circumvent governance.

67
MCQmedium

A data engineer needs to allow a service principal to read data from a Unity Catalog table `sales.orders` and also write to a volume `sales.raw_data`. Which set of privileges should be granted to the service principal?

A.ALL PRIVILEGES on table sales.orders, and WRITE VOLUME on volume sales.raw_data.
B.SELECT on table sales.orders, and WRITE VOLUME on volume sales.raw_data.
C.SELECT on table sales.orders, and READ VOLUME on volume sales.raw_data.
D.MODIFY on table sales.orders, and WRITE VOLUME on volume sales.raw_data.
AnswerB

SELECT on the table grants read access to its data. WRITE VOLUME on the volume grants the ability to write files to that volume. Together, these privileges meet the requirement for the service principal to read from the table and write to the volume. This combination is precise and follows the principle of least privilege.

Why this answer

To read from a Unity Catalog table, the service principal needs SELECT on the table. To write to a volume, it needs WRITE VOLUME on the volume. These privileges are the minimal required to perform the specified actions, adhering to the principle of least privilege.

Other options either grant insufficient access to the volume or excessive access to the table.

Exam trap

The trap here is confusing READ VOLUME with WRITE VOLUME, or granting MODIFY instead of SELECT on the table, leading to over-privilege.

68
MCQmedium

A data engineering team runs a nightly Lakeflow Job that ingests files from cloud storage, transforms them with a notebook, and then runs a SQL task. The team wants the SQL task to execute only after the notebook transform succeeds, but they do not want the SQL task to wait for a fixed delay. Which Lakeflow Jobs feature should they configure on the SQL task?

A.Configure a time-based trigger on the SQL task using a cron expression that starts two minutes after the job run begins.
B.Set the SQL task's depends_on property to reference the notebook transform task.
C.Add a run_if condition to the SQL task that checks whether the transform task's output path exists.
D.Place the SQL task and the notebook transform task in separate jobs and chain them using a job-level trigger.
AnswerB

The depends_on property on a task creates an explicit dependency on another task in the same job. When the SQL task lists the notebook transform task in depends_on, the SQL task will not start until the transform task completes successfully. This enforces the required ordering without introducing a fixed delay or polling mechanism.

Why this answer

The depends_on property is the native Lakeflow Jobs mechanism for expressing task-level dependencies. By setting depends_on on the SQL task to point at the notebook transform task, the SQL task is scheduled only after the transform completes successfully. This avoids fixed delays, avoids polling, and keeps the pipeline deterministic and observable within a single job run.

Exam trap

The trap here is assuming that scheduling a task later in time or checking for an output file creates a dependency, when Lakeflow Jobs requires an explicit depends_on relationship to order tasks.

69
MCQeasy

A data engineer needs to grant the `analyst` group the ability to read data from a Unity Catalog table `sales.fact_orders`. They also want to ensure that members of `analyst` can see the table in the catalog explorer but cannot modify it. Which privilege should be granted to the `analyst` group on the table?

A.GRANT USAGE ON TABLE sales.fact_orders TO `analyst`;
B.GRANT MODIFY ON TABLE sales.fact_orders TO `analyst`;
C.GRANT ALL PRIVILEGES ON TABLE sales.fact_orders TO `analyst`;
D.GRANT SELECT ON TABLE sales.fact_orders TO `analyst`;
AnswerD

The SELECT privilege allows users to read data from the table, which includes querying it and viewing its metadata in Catalog Explorer. It does not grant the ability to modify data or table structure. This is the minimum privilege required for read-only access, aligning with the requirement.

Why this answer

To allow read-only access to a Unity Catalog table, the SELECT privilege is granted. It permits querying and viewing metadata without modification rights. MODIFY would allow changes, ALL PRIVILEGES is excessive, and USAGE is not applicable to tables.

SELECT is the least privilege that meets the requirement.

Exam trap

The trap here is confusing USAGE with SELECT; USAGE is for catalogs and schemas, not tables, and does not grant data access.

70
MCQmedium

A team is designing a CI/CD pipeline using Databricks Asset Bundles (DABs). Which TWO of the following are primary benefits of using DABs for managing Databricks projects?

A.DABs automatically provision cloud-native IAM roles for all users.
B.DABs enable consistent deployment of jobs and pipelines across different environments.
C.DABs provide built-in visual drag-and-drop tools for building ETL pipelines.
D.DABs simplify version control for infrastructure and job configurations.
E.DABs bypass the need for any authentication tokens during deployment.
AnswerB, D

DABs use a declarative YAML structure that allows developers to define jobs, pipelines, and workspace objects once and deploy them consistently across multiple workspaces. This ensures that development, staging, and production environments remain synchronized, preventing the 'works on my machine' syndrome that often plagues manual configuration efforts.

Why this answer

Databricks Asset Bundles represent the modern standard for orchestrating Databricks projects as code. By codifying infrastructure and application configuration into YAML, teams achieve environment parity across development, staging, and production. This approach reduces manual configuration errors, simplifies deployment automation via CLI-based workflows, and provides a version-controlled foundation for managing complex data assets, which is critical for large-scale enterprise data engineering environments.

Exam trap

Candidates often confuse DABs with older manual deployment methods or generic Terraform scripts, failing to recognize that DABs are specifically optimized for Databricks project orchestration, environment parity, and CLI-based automation.

71
Multi-Selectmedium

A data engineer is designing a pipeline using Structured Streaming to ingest data into Delta Lake. Which THREE benefits are provided by using checkpoints in this scenario?

Select 3 answers
A.Enabling the stream to resume from the exact point of failure.
B.Providing the ability to recover the previous version of the table.
C.Ensuring exactly-once processing semantics for the ingestion.
D.Automatically cleaning up old data files in the target directory.
E.Storing the state of aggregations across streaming batches.
AnswersA, C, E

Checkpoints record the offsets of the data that has been successfully processed. If the streaming job fails or is manually stopped, Spark uses these offsets to determine where to restart the processing. This ensures that no data is skipped and that the pipeline maintains its continuity across different runs.

Why this answer

Checkpoints are a fundamental feature of Structured Streaming that enable fault tolerance and exactly-once processing. They store the current state and progress of a stream, allowing it to recover from failures without data loss or duplication. In the context of Delta Lake, checkpoints ensure that the ingestion process remains reliable and consistent across various execution cycles.

Exam trap

Many candidates incorrectly believe checkpoints only store the data itself, rather than understanding they store the metadata, offsets, and state information required to ensure exactly-once processing and fault recovery.

72
MCQmedium

A data engineer is designing a Bronze-to-Silver transformation pipeline using Delta Lake. They need to ensure that the Silver table contains only records where the 'transaction_id' is not null and the 'amount' is positive. Which technique best ensures data quality at this stage?

A.Apply a post-load DELETE statement to the Silver table after every micro-batch.
B.Define a separate table for rejected records and use manual SQL queries to move them later.
C.Use a filter transformation in the Spark Structured Streaming query before writing to the target table.
D.Set the table property 'delta.constraints.check' to filter incoming null values.
AnswerC

Filtering during the stream transformation ensures that invalid data is dropped before the commit happens. This is the most efficient method because it eliminates invalid records without needing extra write operations. It keeps the Silver table clean from the beginning, adhering to best practices for production ETL pipelines.

Why this answer

Implementing Delta Lake expectations or inline 'where' clauses during the write process is critical for maintaining high-quality Silver layers. By filtering records before they are committed, you prevent corrupt data from propagating to downstream analytical tables. This pattern is fundamental to the Medallion Architecture, ensuring the Silver layer serves as a reliable, cleaned source of truth for downstream consumption and complex modeling tasks.

Exam trap

Candidates often suggest using complex post-write cleanup jobs or external scripts, ignoring that filtering data during the streaming write is the most efficient and standard practice.

73
MCQmedium

A job is failing with a 'Disk Space' error on the worker nodes. The code performs several large joins. Which configuration should the engineer adjust to mitigate the disk space usage?

A.Increase spark.sql.shuffle.partitions.
B.Decrease spark.driver.memory.
C.Set spark.databricks.io.cache.enabled to false.
D.Increase the number of task retries.
AnswerA

Increasing the number of shuffle partitions creates smaller, more manageable chunks of data. By reducing the size of individual partitions being joined, it becomes more likely that the operation can complete in memory, thereby avoiding the need to spill data to the local disk of worker nodes.

Why this answer

Large joins often require spilling to disk when the shuffle data exceeds the available executor memory. Adjusting the spark.sql.shuffle.partitions configuration can reduce the size of individual partitions, potentially fitting them within memory. Alternatively, increasing the instance type memory or optimizing the join condition reduces the reliance on local disk storage.

Managing shuffle partitions is a standard practice to balance memory pressure against the risk of disk overflow during shuffles.

Exam trap

Candidates often suggest 'increasing executor memory' as the first step, which is a costly infrastructure change compared to the performance tuning of shuffle partitions for memory-intensive joins.

74
MCQeasy

When implementing a Medallion Architecture, what is the primary purpose of the 'Silver' layer?

A.To store raw, unmodified data in its original format from source systems.
B.To provide a cleaned and integrated source of truth for business reporting.
C.To act as a sandbox environment for ad-hoc exploration and training.
D.To aggregate data into pre-calculated metrics for high-performance dashboards.
AnswerB

The Silver layer is the trusted source of record. It consolidates data from multiple Bronze sources, enforces quality constraints, and provides a structured schema. This level of refinement ensures that analysts and data science models are based on reliable information, minimizing the time spent on data preparation for downstream tasks.

Why this answer

The Silver layer acts as the validated, cleaned, and enriched integration layer. It transforms raw Bronze data into a usable state by applying data quality checks, schema enforcement, and deduplication. This layer serves as the foundation for downstream modeling, allowing data scientists and analysts to work with reliable data without needing to handle the noise and structural inconsistencies inherent in raw data sources.

Exam trap

Test-takers often confuse the Silver layer's refined integrated data purpose with the raw landing zone of the Bronze layer or the highly aggregated business view of the Gold layer.

75
MCQmedium

A data engineer is using Delta Live Tables to build a pipeline. They need to create a table that contains the latest record for each customer based on a `last_updated` timestamp. The source is a streaming table with append-only data. Which Delta Live Tables operation should be used to achieve this?

A.Use a streaming live table with a GROUP BY customer_id and MAX(last_updated) to get the latest timestamp, then join back to the source.
B.Use a batch live table that reads the entire source and uses a MERGE statement to upsert records.
C.Use a streaming live table with a window function to rank records by last_updated and filter for rank 1.
D.Use the APPLY CHANGES INTO operation with a streaming source and specify customer_id as the key and last_updated as the sequencing column.
AnswerD

APPLY CHANGES INTO (formerly Delta Live Tables CDC) is designed to handle updates and deduplication from a streaming source. By specifying customer_id as the key and last_updated as the sequence column, Delta Live Tables will automatically keep the latest record for each customer based on the timestamp. This is the correct and efficient way to maintain a latest-state table.

Why this answer

The correct operation is APPLY CHANGES INTO, which is designed for change data capture and deduplication. By specifying the key and sequencing column, Delta Live Tables automatically retains the latest record for each customer. Other approaches either do not support streaming updates or require inefficient full reprocessing.

Exam trap

The trap here is assuming that a simple streaming aggregation or window function can maintain the latest record per key. In Delta Live Tables, APPLY CHANGES INTO is the purpose-built operation for this pattern.

Page 1 of 4

Page 2

All pages