Courseiva

Databricks Certified Data Engineer Associate (Databricks-DE-Assoc) — Questions 226–276

276 questions total · 4 pages · All types, answers revealed

Page 3

Page 4 of 4

226
Multi-Selectmedium

When configuring a Databricks Job, which TWO factors most directly influence the choice between a 'Job Cluster' and an 'All-Purpose Cluster'?

Select 2 answers
A.The need for interactive debugging and notebook testing.
B.The need for minimal compute cost for scheduled production workflows.
C.The requirement for high availability during query execution.
D.The number of users concurrently using the cluster.
E.The type of data source (e.g., S3 vs. ADLS).
AnswersA, B

All-Purpose clusters are designed for interactive use, allowing engineers to run code snippets, inspect data, and debug notebooks in real-time. They offer faster startup times for subsequent commands, making them the appropriate choice for development environments where quick iteration and constant developer feedback are required.

Why this answer

Choosing the right cluster type is vital for balancing cost and performance. Job Clusters are ephemeral, optimized for production tasks, and significantly cheaper, whereas All-Purpose Clusters are persistent and intended for interactive development. Understanding the cost implications and lifecycle management of these clusters is fundamental for any Databricks engineer responsible for managing operational budgets and ensuring efficient resource utilization across various environments.

Exam trap

Candidates often select All-Purpose clusters for production tasks to save time, ignoring the significantly higher costs and lack of proper lifecycle isolation compared to ephemeral Job clusters.

227
MCQhard

A data engineer is designing a workflow that requires running a series of tasks with dependencies. The workflow must be able to retry failed tasks automatically and send notifications on failure. The engineer also needs to ensure that the workflow can be triggered on a schedule and via an API. Which Databricks feature should the engineer use?

A.Delta Live Tables pipelines with continuous mode.
B.Databricks SQL dashboards with scheduled refresh.
C.Databricks Jobs with task dependencies and retry policies.
D.Databricks Repos with Git hooks.
AnswerC

Databricks Jobs support multi-task workflows with dependencies, allowing tasks to run in sequence or parallel based on upstream task completion. Jobs can be configured with retry policies for individual tasks and notifications on failure. They can be scheduled and triggered via API, meeting all requirements.

Why this answer

Databricks Jobs are the primary orchestration feature for workflows with dependencies, retries, and notifications. They support multi-task DAGs, task-level retry policies, and alerting on failure. Jobs can be scheduled and triggered via API, making them the correct choice.

The other options lack the necessary orchestration capabilities.

Exam trap

The trap here is confusing Delta Live Tables pipelines with general-purpose workflow orchestration; DLT is optimized for data pipelines but does not offer the same task-level control and API triggering as Jobs.

228
Multi-Selecthard

A data engineer is evaluating whether to use Auto Loader or the COPY INTO command for a new ingestion pipeline. Which TWO features are unique to Auto Loader compared to COPY INTO?

Select 2 answers
A.The ability to perform incremental loads by tracking processed files.
B.Native support for schema evolution through schemaEvolutionMode.
C.Support for ingesting data from cloud object storage like S3 or ADLS.
D.Integration with cloud-native file notification services for discovery.
E.Requirement for a Delta Lake table as the final destination.
AnswersB, D

Auto Loader provides a dedicated schema evolution feature that can automatically add new columns to the target table when they are detected in the source. This is a significant advantage for handling semi-structured data where the structure might change over time without breaking the primary ingestion pipeline.

Why this answer

Auto Loader and COPY INTO both offer idempotent loading, but Auto Loader is built on Structured Streaming. This allows it to provide more advanced features like automatic schema evolution and a notification mode that uses cloud services to detect new files. Understanding these differences is crucial for selecting the right tool based on volume and schema complexity.

Exam trap

Candidates frequently assume COPY INTO supports automatic schema evolution and cloud notifications natively, confusing it with Auto Loader capabilities.

229
MCQmedium

A data engineer is working with a large Delta table and notices that queries filtering by 'region_id' are performing slowly. The table is currently partitioned by 'date'. Which strategy should the engineer use to optimize query performance for 'region_id' filtering without increasing the number of partitions?

A.Change the partition column to 'region_id'
B.Execute 'OPTIMIZE table_name ZORDER BY (region_id)'
C.Enable 'auto-compaction' on the table
D.Convert the table to a Liquid Clustering table
AnswerB

Z-Ordering co-locates data with similar values in the same files, which enables data skipping. This is the standard best practice for optimizing filter performance on non-partitioned columns, as it provides a structured way for the query engine to ignore irrelevant data blocks during scan operations.

Why this answer

Z-Ordering is the most effective technique for multidimensional clustering in Delta Lake, especially when partitioning by date. By Z-Ordering on 'region_id', the data engineer colocates related information within the same set of files. This significantly reduces the amount of data scanned during queries, improving performance without the overhead associated with high-cardinality partitioning, which could otherwise lead to the small file problem.

Exam trap

Candidates often choose partitioning instead of Z-Ordering for high-cardinality columns, misunderstanding that partitioning creates too many small files, whereas Z-Ordering clusters data within existing partitions efficiently.

230
MCQmedium

A data engineer needs to configure a Databricks Job to orchestrate a data pipeline that includes a Python task, a SQL task, and a notebook task. The pipeline requires passing a dynamic run identifier from the Python task to the subsequent SQL and notebook tasks. Which mechanism should the data engineer use to achieve this task-to-task dependency parameter passing?

A.Write the dynamic identifier to a shared Unity Catalog volume and read it inside the downstream tasks.
B.Use the dbutils.jobs.taskValues.set method in the Python task and reference it downstream using the task value syntax.
C.Store the identifier as an environment variable in the cluster configuration shared by all three tasks.
D.Modify the global job parameters dynamically via the Databricks REST API from within the running Python script.
AnswerB

The dbutils.jobs.taskValues API allows tasks within a Databricks workflow to securely pass small payloads like run identifiers to downstream tasks. Downstream tasks retrieve these values via task value interpolation, enabling seamless dynamic parameter passing and robust pipeline orchestration.

Why this answer

Databricks workflows natively support passing values between tasks using task values. A Python task can set a task value using dbutils.jobs.taskValues.set, which downstream tasks can then reference using standard task value interpolation syntax. This eliminates external storage dependencies, ensuring robust, serverless orchestration directly managed by the Databricks control plane.

Exam trap

Candidates often suggest writing task variables to external storage or global variables, forgetting Databricks provides a native task values API for inter-task communication.

231
MCQhard

Refer to the exhibit. The merge operation is failing in your pipeline. What is the root cause of this error, and how should it be resolved?

A.The target table does not have a primary key defined.
B.The source DataFrame must be aggregated or deduplicated to ensure one match per target ID.
C.The target table is too large to perform a MERGE and should be replaced with a full overwrite.
D.The cluster configuration needs more memory to handle the join operation.
AnswerB

This is the classic resolution for 'multiple matches'. By using a window function or 'dropDuplicates' on the source DataFrame, you ensure that for every ID, there is only one source record. This removes the ambiguity that causes the MERGE operation to fail during the execution phase.

Why this answer

The error occurs because the join condition in the MERGE statement is not unique relative to the source data. When the source contains multiple records with the same identifier that exists in the target, the engine cannot decide which source record to use for the update. To resolve this, you must deduplicate the source data before performing the merge, ensuring each key has exactly one entry.

Exam trap

Candidates often attempt to resolve MERGE errors by changing the join condition or adding more columns to the ON clause, rather than addressing the root cause: duplicate source keys.

232
MCQhard

When utilizing Databricks Asset Bundles, how should secrets (e.g., API keys, database credentials) be handled to ensure security during CI/CD?

A.Hardcode the secrets in the bundle's YAML configuration files.
B.Store secrets as environment variables in the Git repository.
C.Use the bundle's secret reference syntax to map variables to a secret scope.
D.Ask all developers to manually enter secrets in the production UI.
AnswerC

Referencing secret scopes allows the CI/CD pipeline to inject credentials at execution time without exposing the actual values in the code. This is the industry-standard approach to managing sensitive information, ensuring that credentials remain secure while allowing the automation engine to authenticate successfully to external services during job execution.

Why this answer

Hardcoding secrets in configuration files is a critical security vulnerability. Databricks Asset Bundles support referencing secrets through the 'secrets' scope, which securely retrieves credentials at runtime. By decoupling sensitive data from the version-controlled YAML files, teams maintain security, avoid leaking credentials in Git, and simplify secret rotation.

This practice is essential for enterprise security and ensures that sensitive infrastructure data is managed in alignment with the principle of least privilege.

Exam trap

Candidates frequently suggest storing secrets in environment variables or configuration files directly, ignoring the security risk of leaking credentials and the availability of Databricks' built-in secret reference syntax.

233
Multi-Selecthard

A data engineer is configuring Unity Catalog row-level security on a table `prod.finance.transactions`. They want to ensure that users in the `us_team` group can only see rows where `region = 'US'`, while users in the `eu_team` group can only see rows where `region = 'EU'`. They create a row filter function `prod.security.region_filter`. Which TWO statements accurately describe how to implement and manage this row-level security? (Choose two.)

Select 2 answers
A.Row filters can only be applied to external tables, not managed tables, because managed tables do not support fine-grained access control.
B.The row filter function must accept the column(s) as parameters and return a BOOLEAN value indicating whether the row should be visible.
C.The row filter function must be defined in the same schema as the table it protects.
D.Apply the row filter using: ALTER TABLE prod.finance.transactions SET ROW FILTER prod.security.region_filter ON (region);
E.Users with the MODIFY privilege on the table automatically bypass the row filter and see all rows.
AnswersB, D

Unity Catalog row filter functions are SQL UDFs that take one or more column values as input and return a boolean. The function evaluates to TRUE for rows the user is allowed to see. It can reference the current user or group via functions like is_account_group_member(), enabling dynamic filtering based on identity.

Why this answer

Row-level security in Unity Catalog is implemented by attaching a row filter function to a table using ALTER TABLE ... SET ROW FILTER. The function receives column values and returns a boolean, often using identity functions to filter based on group membership.

Both managed and external tables support row filters, and bypassing them requires the BYPASS ROW FILTER privilege.

Exam trap

The trap here is thinking that MODIFY privilege bypasses row filters or that row filters are limited to external tables, when in fact a dedicated BYPASS ROW FILTER privilege is required and both table types support filters.

234
MCQhard

Refer to the exhibit. A data engineer is configuring an IAM policy for a Databricks storage credential. Which of the following is the most significant security risk in this configuration?

A.The policy restricts access to only a single file.
B.The policy lacks an explicit 'Deny' statement for all other resources.
C.The policy allows access to all objects in the bucket rather than using specific prefixes.
D.The policy is missing a condition to require multi-factor authentication (MFA).
AnswerC

Granting access to the entire bucket via 'arn:aws:s3:::my-bucket/*' is a significant risk. If multiple datasets or environments share a bucket, any cluster using this policy could potentially read or list data it is not authorized to access. Scoping policies to specific prefixes is a mandatory security best practice.

Why this answer

The policy uses overly permissive resource definitions and potentially excessive actions if not scoped correctly. In production environments, adhering to the principle of least privilege is critical. Granting broad 's3:ListBucket' or 's3:GetObject' access to an entire bucket without specific prefix scoping can lead to data exfiltration risks or unauthorized access to non-production data stored in the same bucket.

Exam trap

Candidates often ignore the scope of the IAM policy, failing to realize that granting access to an entire bucket rather than a specific prefix violates the principle of least privilege.

235
MCQmedium

Why should developers avoid using 'notebook' references in production pipelines that point to the 'Shared' folder for shared development work?

A.Databricks jobs cannot execute notebooks from the 'Shared' directory.
B.The 'Shared' directory is deleted automatically every 24 hours.
C.It lacks the version control and access isolation required for production.
D.Notebooks in the 'Shared' directory are limited to a smaller file size.
AnswerC

Production assets require strict access controls and a clear audit trail. The 'Shared' folder usually has loose permissions, allowing any user to edit the code. This creates a high risk of production instability. Proper CI/CD processes dictate that production assets are deployed to hardened, versioned folders via automated pipelines.

Why this answer

The 'Shared' folder is meant for collaborative development, not for hosting production-ready code. Pipelines referencing this folder are susceptible to unpredictable changes, as any developer with access can modify the code. In CI/CD, production code should reside in a controlled, versioned, and immutable location.

Using the 'Shared' folder for production pipelines violates the separation of concerns and increases the risk of unauthorized or accidental changes breaking critical data workflows.

Exam trap

Candidates often underestimate the risks of the 'Shared' folder, incorrectly believing it is a valid location for production code if permissions are restricted, ignoring the need for immutable version control.

236
MCQeasy

A data engineer working in a Databricks workspace needs to create a new notebook that will be shared with teammates in the same workspace. They want the notebook to be organized in a folder named 'team_project' and be visible to all workspace users. Which action should the data engineer take to accomplish this in the Databricks workspace?

A.Create the notebook in the Workspace folder, then use the Databricks CLI to set the notebook's permission to 'Can Manage' for the 'users' group.
B.Create the notebook in the Workspace folder under their user directory, then move it to a shared folder and grant 'Can Edit' permission to all workspace users.
C.Navigate to the Workspace folder, create a new folder named 'team_project', then create the notebook inside it and set its permissions to 'Can View' for all workspace users.
D.Use the Databricks Repos feature to create a new repository, add a notebook, and set the repository permissions to 'Can Read' for all workspace users.
AnswerC

The Workspace folder is the default location for shared content in Databricks. Creating a folder there and then a notebook inside it allows organization. Setting permissions to 'Can View' for all workspace users makes it visible to everyone, satisfying the requirement. This is the straightforward, native approach without extra tools.

Why this answer

In Databricks, the Workspace folder is the shared area where notebooks and folders can be organized and accessed by workspace users. Creating a folder named 'team_project' there and placing the notebook inside, then setting permissions to 'Can View' for all workspace users, directly meets the requirement. This method uses native workspace features without introducing unnecessary complexity.

Exam trap

The trap here is assuming that Databricks Repos is the default way to share notebooks, when in fact the Workspace folder with proper permissions is the simpler and more direct method for basic sharing.

237
MCQeasy

When integrating Databricks with a CI/CD tool like GitHub Actions, how should you securely manage the authentication token used for deployments?

A.Hardcode the Databricks Personal Access Token directly into the YAML pipeline file.
B.Store the token as an encrypted secret in the CI/CD provider's secret store.
C.Save the token in a public S3 bucket and download it when the pipeline starts.
D.Use your own password as the token for easier memorization and access.
AnswerB

Storing credentials in a secure, encrypted secret store provided by CI/CD platforms like GitHub Actions or GitLab is the standard security practice. This prevents exposure in logs or source control while ensuring the pipeline can securely access the Databricks API during the execution of deployment tasks.

Why this answer

Managing secrets securely is vital for CI/CD. Storing tokens directly in code or pipeline definitions exposes them to unauthorized access. By using a secrets manager, you decouple the sensitive credentials from the pipeline logic.

This ensures that the CI/CD platform can authenticate with Databricks dynamically during the execution phase, maintaining a high security posture while allowing for easy rotation of credentials without needing to refactor the entire codebase.

Exam trap

Candidates often suggest hardcoding tokens in the repo or using insecure plain-text files, failing to recognize that CI/CD providers offer dedicated encrypted secret stores for this exact purpose.

238
MCQhard

Which of the following best describes the purpose of 'Credential Passthrough' in Databricks?

A.It allows a Databricks cluster to automatically scale based on storage latency.
B.It enables users to access cloud storage using their own identity rather than the cluster's service principal.
C.It encrypts data in transit between the Databricks workspace and the cloud storage.
D.It forces all notebook users to use a single shared service principal for security.
AnswerB

This feature maps the identity of the user running a notebook or job to the storage access request. Instead of the cluster using a single shared service principal, each user's individual permissions are applied, ensuring that the cloud provider's access logs accurately reflect individual user actions during query execution.

Why this answer

Credential Passthrough allows users to authenticate to cloud storage using their own identity from the Databricks environment. By passing the user's credentials to the storage layer, Databricks ensures that the storage provider enforces access policies, creating a seamless audit trail and simplifying security management in environments that require strict alignment between user access in Databricks and direct cloud storage access.

Exam trap

Candidates often confuse Credential Passthrough with instance profiles, mistakenly believing it allows the cluster to use a single shared administrative service principal for all users instead of individual user identities.

239
MCQhard

A data engineer is using Auto Loader to ingest JSON files from a cloud storage directory into a Delta table. The directory receives files continuously, and the engineer wants to ensure that the ingestion process can handle schema drift where new columns are added to the JSON files over time. The engineer also wants to minimize the need to reprocess all data when the schema changes. Which configuration should the engineer use?

A.Set the option 'cloudFiles.schemaEvolutionMode' to 'failOnNewColumns' and enable 'mergeSchema' on the write stream.
B.Set the option 'cloudFiles.schemaEvolutionMode' to 'addNewColumns' and set 'mergeSchema' to 'false' on the write stream.
C.Set the option 'cloudFiles.schemaEvolutionMode' to 'addNewColumns' and enable 'mergeSchema' on the write stream.
D.Set the option 'cloudFiles.schemaEvolutionMode' to 'rescue' and enable 'mergeSchema' on the write stream.
AnswerC

Auto Loader's 'cloudFiles.schemaEvolutionMode' controls how schema changes are handled. Setting it to 'addNewColumns' allows new columns to be added to the schema automatically. Additionally, enabling 'mergeSchema' on the write operation ensures that the Delta table schema is updated to include these new columns. This combination effectively manages schema drift without reprocessing all data.

Why this answer

To handle schema drift with Auto Loader, the 'cloudFiles.schemaEvolutionMode' should be set to 'addNewColumns' to automatically add new columns to the schema. Additionally, the write stream must have 'mergeSchema' enabled to update the Delta table schema. The other modes either fail or rescue data without adding columns, and without 'mergeSchema' on write, the new columns won't be persisted.

Exam trap

The trap here is assuming that setting the schema evolution mode alone is sufficient, without also enabling 'mergeSchema' on the write operation to propagate schema changes to the Delta table.

240
MCQmedium

A data engineer manages a Lakeflow Job that runs a long-running notebook task on a job cluster. The task occasionally fails due to transient cloud storage errors, and the engineer wants the task to retry automatically without failing the entire job on the first attempt. The engineer also wants to be alerted only if all retries are exhausted. Which configuration should the engineer apply?

A.Set the task's retry count to a value greater than zero and configure a notification for job failure.
B.Enable continuous mode on the job and configure a notification for run start.
C.Set the cluster's autoscaling to a higher maximum and add a notification for cluster termination.
D.Configure a job-level timeout and enable email notifications on every task start.
AnswerA

Task-level retries allow the task to re-execute automatically when it fails, and the job only reports failure after retries are exhausted. A job failure notification then fires only when the task ultimately fails, matching the alerting requirement. This combination handles transient errors gracefully while avoiding noisy alerts on each retry attempt.

Why this answer

Task-level retries re-execute a failed task automatically, so transient storage errors can be absorbed without failing the job immediately. The job is only marked failed after retries are exhausted, at which point a job failure notification alerts the team. This matches the requirement to retry silently and alert only on final failure, unlike cluster or continuous-mode settings that do not target task retries.

Exam trap

The trap here is assuming that a job-level timeout or continuous mode provides retries; only task-level retry settings re-execute a failed task within the same run.

241
MCQmedium

A data engineer is implementing a Type 2 slowly changing dimension in Delta Lake for a customers table. The table has columns customer_id, name, address, effective_date, end_date, and is_current. When a customer's address changes, the engineer wants to expire the existing current row and insert a new current row in a single atomic operation. Which Delta Lake feature should the engineer use?

A.Use OPTIMIZE with ZORDER BY customer_id to reorganize the table so that new versions are placed next to old versions
B.Use INSERT OVERWRITE to replace the entire customers table with a new snapshot that includes the expired and new rows
C.MERGE INTO the customers table using a source of changed records, with WHEN MATCHED UPDATE to set end_date and is_current on the existing row and WHEN NOT MATCHED INSERT to add the new version
D.Use DELETE followed by INSERT in two separate statements to expire the old row and add the new one
AnswerC

MERGE INTO supports updating matched rows and inserting unmatched rows in one atomic transaction, which is exactly what a Type 2 dimension requires: expire the current row and add the new version. Because Delta Lake provides ACID guarantees, readers never see a state where the old row is expired but the new row is missing.

Why this answer

A Type 2 dimension change requires atomically expiring the current row and inserting a new current row. Delta Lake MERGE INTO expresses both actions in one statement, and its ACID transaction guarantees that readers see either the old state or the new state, never a partial update. INSERT OVERWRITE, separate DELETE and INSERT, and OPTIMIZE do not provide the same incremental, atomic behavior.

Exam trap

The trap here is assuming that separate DELETE and INSERT statements or a full table overwrite are equivalent to a single MERGE, when only MERGE provides atomic expire-and-insert semantics.

242
MCQmedium

Which action is recommended to resolve a scenario where a Databricks Job is failing due to excessive metadata operations on a Delta table with millions of files?

A.Increase the driver node instance size.
B.Run OPTIMIZE to consolidate files.
C.Disable the Delta transaction log.
D.Add more worker nodes to the cluster.
AnswerB

Running OPTIMIZE reduces the number of files by merging small files into larger ones. This directly reduces the number of entries in the Delta log and the number of metadata calls required to resolve the table state, significantly improving the performance of subsequent queries and avoiding the metadata bottleneck.

Why this answer

When a Delta table contains millions of small files, the transaction log and metadata operations become a bottleneck. The 'list' operations required to build the state of the table consume significant time and driver memory. Implementing partition pruning or using Delta Lake's table property 'delta.enableChangeDataFeed' are not the primary solutions here.

Instead, running OPTIMIZE to consolidate files is the standard way to reduce metadata overhead and improve table performance.

Exam trap

Candidates often suggest partitioning the table as a fix. While partitioning helps with data skipping, it does not fix the metadata overhead caused by having millions of small files.

243
Multi-Selecthard

A team is transitioning to the Databricks Intelligence Platform. Which TWO actions are required to successfully register a table in Unity Catalog using the three-level namespace?

Select 2 answers
A.Grant the 'USE CATALOG' privilege on the target catalog.
B.Configure a local Hive Metastore on the cluster.
C.Define the table using the catalog.schema.table naming convention.
D.Manually update the cluster's Spark configuration to point to the metastore URL.
E.Set the default database to 'default' before executing the CREATE statement.
AnswersA, C

The 'USE CATALOG' privilege is a prerequisite for interacting with any assets contained within that catalog. Without this permission, a user cannot traverse the catalog to locate or create the schema and tables inside, effectively preventing any table registration attempts within that specific catalog scope.

Why this answer

The three-level namespace (catalog.schema.table) is a fundamental feature of Unity Catalog. To register a table, an engineer must have sufficient privileges in the specific catalog and schema, and the table must be created using the correct fully qualified path. Understanding this structure is crucial because it enables unified data discovery and governance across different environments, preventing namespace collisions and allowing for standardized cross-database operations in the platform.

Exam trap

Candidates often forget that explicit catalog-level privileges must be granted before referencing objects within a three-level namespace structure.

244
MCQhard

A CI/CD pipeline must run unit tests on Python transformation code before deploying a Databricks job. The tests should execute quickly without starting a cluster and should validate the transformation logic in isolation. Which approach best meets these requirements?

A.Deploy the job to a development workspace and run an integration test against production data to confirm the transformations produce expected results.
B.Configure the pipeline to run databricks bundle validate and rely on schema validation to confirm the transformation logic is correct.
C.Use pytest in the CI runner to test the transformation functions directly, mocking Spark or using a local SparkSession only where needed.
D.Run the tests as a Databricks job task on an all-purpose cluster so the tests use the same runtime as production.
AnswerC

pytest runs in the CI runner with no cluster, so it is fast and isolated. Transformation logic written as pure functions can be tested directly, and a local SparkSession can be created only for tests that need Spark APIs. This keeps the feedback loop short, avoids cloud compute costs, and validates logic before deployment, which is exactly what a pre-deploy unit test stage should do.

Why this answer

Unit tests for transformation logic belong in the CI runner, not on a cluster. Using pytest with pure functions and a local SparkSession where necessary keeps tests fast, isolated, and free of cloud compute. This catches logic errors before deployment, whereas bundle validation only checks configuration and integration tests against production data are slow and risky.

Exam trap

The trap here is conflating bundle validation with unit testing, assuming a successful databricks bundle validate means the transformation logic has been verified.

245
MCQmedium

A data engineer needs to share a Delta table with an external partner who does not have a Databricks account. The engineer wants to provide read-only access to the table for a limited time, ensuring the partner cannot access any other data. Which Databricks feature should the engineer use?

A.Unity Catalog delta sharing with a recipient and expiration.
B.Generate a personal access token (PAT) for the partner to query the table via JDBC.
C.Export the table to a CSV file and share it via a secure file transfer.
D.Create a Databricks SQL endpoint and provide the partner with a SQL query URL.
AnswerA

Delta Sharing in Unity Catalog allows secure data sharing with external recipients, even those without Databricks accounts. The engineer can create a share, add the table, and grant access to a recipient. The recipient can be configured with an expiration date, and access is limited to the shared data only, meeting the requirements.

Why this answer

Delta Sharing is Databricks' open protocol for secure data sharing with external parties, including those without Databricks accounts. With Unity Catalog, you can create a share containing specific tables and grant access to a recipient. The recipient can be given an expiration date, and they can access the data using any Delta Sharing client.

This provides granular, read-only access limited to the shared data.

Exam trap

The trap here is assuming that a personal access token or SQL endpoint can be securely shared with an external partner without granting broader access; Delta Sharing is purpose-built for this use case with fine-grained controls.

246
MCQhard

A data engineer maintains a Delta table where each row represents a customer record, and updates arrive continuously as change data capture events. The engineer needs to apply inserts, updates, and deletes from a staging table into the target table in a single atomic operation, matching records on customer_id. Which Delta Lake operation should be used?

A.Run a DELETE followed by an INSERT on the target table inside the same notebook cell.
B.MERGE INTO the target table using the staging table as the source, with matched update/delete clauses and a not-matched insert clause.
C.Append the staging rows to the target table and deduplicate later with a window function during reads.
D.Use INSERT OVERWRITE to replace all partitions touched by the staging table with the staged rows.
AnswerB

MERGE INTO supports exactly this pattern: matched rows can be updated or deleted, and unmatched source rows can be inserted, all within one ACID transaction. Matching on customer_id applies CDC changes idempotently, and the operation is atomic so readers never see a partially applied batch.

Why this answer

MERGE INTO is the Delta Lake primitive designed for upsert and delete semantics in one atomic transaction. By matching on customer_id, the engineer can update or delete matched rows and insert new ones from the staging table, applying CDC events idempotently. This avoids data gaps, prevents duplicate versions, and keeps the target table representing current state for downstream consumers.

Exam trap

The trap here is treating a delete-plus-insert sequence or an append pattern as equivalent to an atomic key-based upsert.

247
MCQmedium

A data engineer needs to ingest data from a legacy SQL Server database into a Delta Lake bronze table. The ingestion must be performant and support parallel reads from the source table. What is the best practice for configuring the JDBC connection in this scenario?

A.Use a single executor to ensure data consistency during the read.
B.Configure 'partitionColumn', 'lowerBound', 'upperBound', and 'numPartitions'.
C.Rely on the default JDBC settings for automatic parallelization.
D.Use the COPY INTO command to read directly from the JDBC source.
AnswerB

These four parameters allow Spark to split the JDBC query into multiple smaller queries that can be executed in parallel. By defining the range and number of partitions, the data engineer enables multiple workers to fetch data at the same time, which is critical for high-performance data ingestion from external databases.

Why this answer

Parallelizing JDBC reads is essential for moving large datasets from relational databases to Databricks. By providing partitioning columns and boundaries, Spark can spawn multiple executors to read different segments of the data simultaneously. This significantly reduces the time required for the initial load and improves the overall throughput of the ingestion pipeline.

Exam trap

Candidates often forget the required set of four parameters ('partitionColumn', 'lowerBound', 'upperBound', 'numPartitions') and mistakenly think a single parameter is enough to enable parallel reads.

248
Multi-Selectmedium

When designing an ingestion strategy for a Delta Lake architecture, which TWO advantages does Delta Lake provide over traditional Parquet tables for incoming data?

Select 2 answers
A.ACID transactions ensure partial writes do not occur during failures.
B.Native support for sub-second streaming latency from any source.
C.Automatic indexing of all string columns for faster search queries.
D.Schema enforcement prevents the ingestion of incorrectly structured data.
E.Support for data encryption at rest using proprietary algorithms.
AnswersA, D

In traditional Parquet tables, a failed write can leave partial data files in storage, leading to inconsistent results. Delta Lake uses a transaction log to ensure that a write operation is either completely successful or not applied at all, maintaining the integrity of the table even if the ingestion job crashes.

Why this answer

Delta Lake enhances the standard Parquet format by adding a transaction log that enables ACID compliance and efficient metadata handling. These features are vital for ingestion as they prevent data corruption and allow for advanced operations like schema evolution and time travel, which are not possible with standard Parquet files on cloud storage.

Exam trap

Candidates frequently confuse storage optimizations like compression with functional ingestion guarantees like ACID transactions and strict schema enforcement.

249
MCQeasy

Which security feature in Databricks allows administrators to mask sensitive data, such as email addresses or social security numbers, in query results based on user-defined functions?

A.Row-Level Security
B.Dynamic Data Masking
C.Table ACLs
D.Workspace-level isolation
AnswerB

Dynamic Data Masking allows you to apply a masking function to columns, ensuring that users only see redacted or transformed versions of sensitive data. This is the precise feature designed to protect PII, such as emails or IDs, while still allowing data analysis on the non-sensitive parts.

Why this answer

Dynamic Data Masking allows you to define a masking function that hides or transforms data based on the user's role. By applying these functions to specific columns in a view or table, sensitive information is obscured in the results returned to the user, providing an effective way to maintain compliance with data privacy regulations like GDPR or HIPAA.

Exam trap

Candidates confuse static table partitioning or column dropping with Dynamic Data Masking, which obscures sensitive field values on-the-fly based on user identity.

250
MCQeasy

A data engineer needs to run a Databricks notebook that processes data stored in an external ADLS Gen2 location. The notebook must access the data securely without embedding credentials in the notebook code. The engineer has already configured a Unity Catalog external location with a storage credential. Which method should the engineer use to read the data?

A.Reference the external location path directly in the notebook using the Unity Catalog external location URL.
B.Use a personal access token (PAT) stored in Databricks secrets to authenticate to ADLS Gen2.
C.Set Spark configuration spark.hadoop.fs.azure.account.oauth2.client.id and related properties in the notebook to authenticate.
D.Mount the ADLS Gen2 container to DBFS using a service principal and then read the data from the mount point.
AnswerA

Unity Catalog external locations allow direct access to cloud storage using the storage credential configured for the location. The engineer can reference the path (e.g., abfss://container@storage.dfs.core.windows.net/path) in Spark read operations, and Unity Catalog will handle authentication and authorization, ensuring secure access without embedding credentials.

Why this answer

Unity Catalog external locations are designed to provide secure access to cloud storage using storage credentials managed by Unity Catalog. Once an external location is configured, data engineers can reference the path directly in their notebooks, and Unity Catalog handles authentication and authorization. This approach avoids embedding credentials in code, aligns with security best practices, and ensures that access is governed by Unity Catalog permissions.

Exam trap

The trap here is assuming that mounting to DBFS or manually configuring OAuth properties is necessary, when Unity Catalog external locations already provide a seamless and secure way to access data without additional credential management.

251
MCQeasy

What is the primary function of an 'Access Connector' for Azure Databricks when using Unity Catalog?

A.It acts as a firewall between the Databricks workspace and the public internet.
B.It allows Databricks to access storage without managing account keys directly.
C.It automatically scales the compute resources of the cluster.
D.It provides a user interface for browsing the storage account content.
AnswerB

The Access Connector enables managed identity authentication, which is the most secure way to connect Databricks to ADLS. By leveraging Azure Active Directory (now Microsoft Entra ID) identities, it removes the need for managing sensitive storage keys, which are prone to leaks and difficult to rotate securely across large teams.

Why this answer

An Access Connector is a dedicated Azure resource that provides a secure, identity-based bridge between Unity Catalog and Azure Data Lake Storage (ADLS). It manages the identity used by Databricks to read and write data, ensuring that Unity Catalog can enforce access policies without requiring the storage account keys to be shared directly with users or hard-coded into notebooks.

Exam trap

Test-takers frequently assume the Access Connector is used for user authentication, confusing it with credential passthrough rather than recognizing it as a secure system-level bridge for Unity Catalog.

252
MCQmedium

Refer to the exhibit. A data engineer is attempting to ingest JSON data into a Delta table. Based on the error log, what is the most appropriate transformation step to implement before loading this data into the production table?

A.Use the 'DROP TABLE' command to clear the target and recreate it with nullable constraints.
B.Apply a 'coalesce' function or a 'filter' transformation on the DataFrame before writing to the Delta table.
C.Modify the target table schema using 'ALTER TABLE' to change the column to nullable.
D.Set the 'spark.sql.execution.nullValue' configuration to 'ignore' for the session.
AnswerB

By applying a 'coalesce' to supply a default value or a 'filter' to remove records with null IDs, the engineer ensures that the data meets the target table's schema requirements. This proactive cleaning prevents the NullPointerException and maintains the integrity of the data stream without failing the job.

Why this answer

The exhibit shows a failure due to null values in a required field during ingestion. In production pipelines, failing at the ingestion layer stops data flow. Implementing a transformation step to handle nulls, either by providing default values or filtering, ensures the pipeline is resilient.

This is a core Data Engineering principle: handle dirty source data before it reaches the target, preventing ingestion failures.

Exam trap

Candidates often attempt to alter the target Delta table configuration to accept bad data rather than cleaning and transforming the incoming stream beforehand.

253
MCQeasy

A junior data engineer writes a PySpark transformation that reads a Parquet dataset, filters out inactive users, and appends the resulting DataFrame to an existing Delta Lake table. However, the data engineer notices duplicate records appearing in the target table after multiple runs. Which technique should be implemented to ensure idempotency?

A.Use DataFrameWriter's mode("overwrite") with replaceWhere option matching the updated partition keys.
B.Implement a Delta Lake MERGE operation using a unique identifier to conditionally insert or update records.
C.Increase the shuffle partition count using spark.sql("SET spark.sql.shuffle.partitions = 200").
D.Execute the VACUUM command immediately after every write operation completes.
AnswerB

Delta Lake MERGE provides a robust mechanism to perform atomic upserts based on matching keys. By checking whether a record already exists before inserting or updating, pipelines can be executed repeatedly with identical source data without introducing duplicate rows, thereby establishing pipeline idempotency.

Why this answer

Using Delta Lake's MERGE INTO operation allows data engineers to perform upserts by matching incoming records against existing table rows using a unique business key. This prevents duplicates during repeated pipeline executions, guaranteeing idempotency. Traditional append operations simply add records without checking for prior existence, which is a common root cause of duplicate data issues in incremental batch pipelines.

Exam trap

Candidates often try to solve duplicates using 'overwrite' modes or manual deduplication steps, failing to realize that Delta Lake's MERGE command is the native, idempotent solution for upserting data.

254
MCQhard

A team is using Databricks Asset Bundles to manage a job that writes to a Unity Catalog table. The bundle is deployed to a staging workspace for testing and then to a production workspace. The team wants the job to use different catalog and schema names in each environment without duplicating the entire bundle. Which approach should they use?

A.Define bundle variables for the catalog and schema, and set different values for each variable in the staging and production targets within the databricks.yml file.
B.Store the catalog and schema names in a Databricks secret scope and reference them in the job's notebook using dbutils.secrets.get.
C.Use a single target and pass the catalog and schema names as job parameters at runtime through the Databricks Jobs API.
D.Create two separate bundle configuration files, one for staging and one for production, and manually copy the job definition into each.
AnswerA

Databricks Asset Bundles support variables that can be overridden per target. By defining catalog and schema variables and assigning environment-specific values under each target in databricks.yml, the same job definition can deploy to staging and production with the correct Unity Catalog names. This is the intended way to parameterize bundles across environments.

Why this answer

Databricks Asset Bundles allow a single job definition to be deployed to multiple environments by using variables that are resolved differently per target. Defining catalog and schema variables and setting their values under the staging and production targets in databricks.yml keeps the bundle DRY and ensures each workspace uses the correct Unity Catalog objects.

Exam trap

The trap here is thinking that runtime job parameters or secret scopes are the right place for environment-specific catalog names, when bundle variables and targets are designed exactly for this purpose.

255
Multi-Selectmedium

Which TWO of the following capabilities are native to the Databricks Unity Catalog?

Select 2 answers
A.Row and column-level security
B.Auto-scaling of compute clusters
C.Centralized audit logging of data access
D.Automated ML model hyperparameter tuning
E.Direct hardware driver installation
AnswersA, C

Unity Catalog supports fine-grained access controls, allowing administrators to filter rows and redact columns based on user identity or group membership. This capability is essential for compliance requirements, ensuring that sensitive data is only visible to authorized users while maintaining a single, consistent security policy across all Databricks workspaces.

Why this answer

Unity Catalog centralizes governance by providing a unified interface for managing permissions across workspaces. It enables fine-grained access control down to the row and column level and provides a centralized audit log. These capabilities are vital for organizations needing to manage data security at scale, ensuring that data engineers and scientists have consistent access control policies across the entire Lakehouse regardless of the specific compute resource or workspace utilized.

Exam trap

Test-takers frequently select obsolete workspace-level access control mechanisms or features from external cloud providers, forgetting that Unity Catalog natively provides centralized governance across multiple workspaces.

256
Multi-Selectmedium

Which TWO of the following are benefits of using Unity Catalog over the legacy Hive Metastore?

Select 2 answers
A.Unified governance across all workspaces.
B.Support for more SQL dialects.
C.Automatic data lineage for all operations.
D.Increased query performance.
E.Lower storage costs.
AnswersA, C

Unity Catalog provides a single metadata store for all workspaces in an account. This eliminates the need to synchronize metadata manually or manage disparate security configurations across different environments. It ensures a consistent view of data, which is essential for scaling data operations securely and efficiently across a large enterprise.

Why this answer

Unity Catalog represents a modern approach to data governance, solving the fragmentation caused by having separate metastores in every workspace. By providing a centralized, account-level view of all data, it enables enterprise-wide discovery, consistent security policies, and deep integration with lineage and audit features. This shift simplifies the data engineer's job by reducing configuration overhead while significantly improving the organization's security posture and audit readiness for compliance requirements.

Exam trap

Candidates often select 'improved compute performance' or 'automatic data partitioning' as benefits of Unity Catalog, which are incorrect; the primary benefits are governance, security, and centralized management.

257
MCQmedium

A data engineering team needs to ingest millions of small JSON files from an S3 bucket into a Delta Lake table. The solution must provide incremental loading, support schema evolution, and automatically scale to handle increasing file volumes without manual tracking of processed files. Which tool is best suited for this requirement?

A.The standard Apache Spark DataFrame reader using the .load() method.
B.The COPY INTO SQL command with the mergeSchema option enabled.
C.Auto Loader using the cloudFiles source in Structured Streaming.
D.A Python loop that iterates through filenames and uses INSERT INTO.
AnswerC

Auto Loader efficiently handles incremental data loading by tracking the ingestion state through checkpoints. It supports schema inference and evolution, allowing the pipeline to adapt to changes automatically. This makes it the ideal choice for ingesting millions of files from cloud object storage with minimal configuration and maintenance.

Why this answer

Auto Loader is the recommended tool for incremental ingestion from cloud storage. It scales to millions of files using either directory listing or file notifications. Unlike standard Spark sources, it tracks processed files in a checkpoint, ensuring exactly-once semantics.

This automation reduces operational overhead when managing unpredictable data volumes and evolving schema structures in production environments.

Exam trap

Candidates often choose standard Spark read methods with directory paths, ignoring that Auto Loader is required for automated incremental state tracking.

258
MCQmedium

A data engineer is configuring Unity Catalog to allow a service principal to run a Databricks job that writes to a Delta table in an external location. The service principal has been granted USAGE on the catalog and schema, and MODIFY on the table. However, the job fails with an error indicating that the service principal cannot access the external location. What is the most likely missing privilege?

A.The service principal needs SELECT on the table.
B.The service principal needs READ FILES on the external location.
C.The service principal needs CREATE on the external location.
D.The service principal needs WRITE FILES on the external location.
AnswerD

When a table is stored in an external location, writing to it requires WRITE FILES on that external location, in addition to MODIFY on the table. Without WRITE FILES, the service principal cannot write the data files, resulting in an access error. This privilege is often overlooked when granting table-level permissions.

Why this answer

For an external table, writing data requires both MODIFY on the table and WRITE FILES on the external location. The service principal has MODIFY but lacks WRITE FILES, causing the access error. Granting WRITE FILES on the external location will allow the job to write the underlying data files.

Other privileges like SELECT, READ FILES, or CREATE do not address the write requirement.

Exam trap

The trap here is assuming that table-level MODIFY is sufficient for writes, overlooking the need for file-level privileges on the external location.

259
Multi-Selecthard

A data engineer is designing a Delta Lake table that will be used for both batch and streaming reads. The table must support upserts from a streaming source and maintain ACID transactions. The engineer wants to ensure that the table can be efficiently queried by downstream consumers using SQL while minimizing storage costs. Which TWO actions should the engineer take to meet these requirements? (Choose two.)

Select 2 answers
A.Use the MERGE INTO command to perform upserts from the streaming source into the Delta table.
B.Enable change data feed on the Delta table to allow downstream consumers to read change events.
C.Partition the Delta table by a high-cardinality column to improve query performance.
D.Run OPTIMIZE with Z-ORDER on the Delta table regularly to compact small files and improve data skipping.
E.Convert the Delta table to a Parquet table to reduce storage costs.
AnswersA, D

MERGE INTO is the correct SQL command to perform upserts (inserts, updates, deletes) in a single atomic operation on a Delta table. It supports ACID transactions and is efficient for streaming upserts when used with foreachBatch. This meets the requirement for upserts from a streaming source while maintaining ACID guarantees.

Why this answer

To support streaming upserts with ACID transactions and efficient SQL queries while minimizing storage costs, the engineer should use MERGE INTO for upserts and regularly run OPTIMIZE with Z-ORDER. MERGE INTO ensures atomic upserts, and OPTIMIZE with Z-ORDER compacts files and improves data skipping, reducing storage and query costs. Other options either add overhead, degrade performance, or remove essential Delta Lake features.

Exam trap

The trap here is assuming that partitioning by a high-cardinality column always improves performance, when it often leads to small file problems and higher costs.

260
MCQmedium

A data engineer needs to automate the ingestion and transformation of data within Databricks with minimal manual intervention. Which feature is most appropriate for orchestrating these data workflows?

A.Delta Live Tables (DLT)
B.Databricks Workflows
C.Unity Catalog UI
D.Databricks SQL Warehouse
AnswerB

Databricks Workflows is the native orchestration tool for the platform. It enables the creation of DAGs (directed acyclic graphs) to manage dependencies between various jobs, such as notebooks or SQL tasks. It provides robust monitoring, alerting, and retry capabilities, making it the standard choice for automating complex data engineering pipelines.

Why this answer

Databricks Workflows is a fully managed orchestration service that allows users to create, schedule, and monitor data pipelines. It is essential because it integrates seamlessly with the platform, allowing engineers to chain together notebooks, SQL queries, and Delta Live Tables pipelines into complex, multi-step workflows. This automation reduces manual toil and improves the reliability of data production environments in the Lakehouse.

Exam trap

Candidates often confuse Databricks Workflows with 'Jobs' or 'Delta Live Tables' exclusively. They fail to realize Workflows is the overarching orchestration tool that ties all these tasks together.

261
MCQmedium

A data engineer has a Delta table named silver_events with columns event_id (string), event_ts (timestamp), and payload (string). The table is partitioned by event_date (derived from event_ts). The engineer needs to update the payload column for all events that occurred on '2024-06-01' based on a mapping table named updates (event_id, new_payload). Which PySpark operation should be used to perform this update efficiently while preserving Delta Lake ACID guarantees?

A.Use DataFrame.write.mode('overwrite').partitionBy('event_date').saveAsTable('silver_events') with the updated DataFrame containing only the rows for '2024-06-01'.
B.Perform a MERGE INTO operation on silver_events using updates as the source, matching on event_id and filtering both source and target to event_date = '2024-06-01'.
C.Read the entire silver_events table into a DataFrame, apply a join with updates, and then use DataFrame.write.mode('overwrite').saveAsTable('silver_events') to persist the changes.
D.Use the DeltaTable API's update method with a condition on event_date = '2024-06-01' and a join to the updates table.
AnswerB

MERGE INTO is the correct Delta Lake operation for updating existing rows based on a source table. By matching on event_id and filtering both source and target to the specific event_date partition, the operation only scans and updates the relevant partition, which is efficient. It also preserves ACID guarantees, including atomicity and isolation, ensuring that concurrent readers see a consistent snapshot.

Why this answer

The correct approach is to use MERGE INTO, which is designed for upserts and updates based on a source table. Filtering both the target and source to the specific event_date partition ensures that only the relevant data is processed, optimizing performance. MERGE INTO also provides ACID guarantees, ensuring that the update is atomic and isolated, which is critical for maintaining data integrity in a Delta Lake table.

Exam trap

The trap here is assuming that overwriting a partition or the entire table is equivalent to an update, but it lacks the atomicity and efficiency of MERGE INTO.

262
MCQmedium

When a Data Engineer uses a 'Repair and Rerun' functionality on a failed Databricks Job, what happens?

A.It deletes all previous successful task logs.
B.It restarts from the failed task.
C.It resets all task statuses to 'Pending'.
D.It requires a new cluster to be provisioned.
AnswerB

The Repair and Rerun functionality intelligently resumes the workflow starting from the point of failure. It uses the output of previously successful tasks, ensuring that only the failed task and its downstream dependencies are executed, which optimizes compute usage and speeds up the time to recovery for pipelines.

Why this answer

Repair and Rerun is a critical feature for managing complex DAG workflows. It allows engineers to restart a job from the specific failed task, inheriting the successful results of prior tasks. This saves significant time and compute costs by avoiding redundant processing of already completed data, which is essential for maintaining efficient pipelines and preventing unnecessary data re-computation in production environments.

Exam trap

Candidates often assume 'Repair and Rerun' restarts the entire job from the very beginning, failing to realize it is designed specifically to resume only from the point of failure.

263
MCQeasy

A team wants to ensure that code changes to their Databricks notebooks are reviewed before being deployed to production. They use a Git repository and Databricks Repos. Which practice should they implement in their CI/CD process?

A.Allow developers to commit directly to the main branch and deploy automatically to production on every commit.
B.Use Databricks Repos to create a separate repository for each developer and merge changes manually in the production workspace.
C.Require pull requests with at least one approval before merging into the main branch, and trigger production deployment only after the merge.
D.Configure the production workspace to allow only workspace admins to edit notebooks, and have developers request admin assistance for each change.
AnswerC

Branch protection rules that require pull requests and approvals enforce peer review before code reaches main. Triggering production deployment after the merge ensures that only reviewed and approved code is deployed. This is a foundational CI/CD practice for controlled, auditable releases.

Why this answer

Enforcing pull requests with approvals through branch protection rules ensures that changes are reviewed before merging. Deploying to production only after the merge ties deployment to reviewed code. This practice is essential for maintaining code quality and auditability in a CI/CD pipeline.

Exam trap

The trap here is confusing access control (who can edit) with change control (who can approve and merge), when code review requires a pull request workflow.

264
MCQmedium

Which identity management approach is mandatory for the implementation of Unity Catalog within a Databricks environment?

A.Workspace-local users
B.Account-level identities
C.Local SQL-only credentials
D.Personal access tokens for every user
AnswerB

Unity Catalog relies on account-level identities to manage access. This enables a single source of truth for user and group definitions, allowing administrators to apply permissions once and have them respected across all workspaces. This centralization is essential for maintaining consistent security posture and simplifying identity lifecycle management.

Why this answer

Unity Catalog requires users to be managed at the account level, rather than at the individual workspace level. This synchronization ensures that permissions and identities are consistent across all workspaces in the account, which is a fundamental requirement for the unified governance model that Unity Catalog provides to the Databricks platform.

Exam trap

Candidates often assume legacy workspace-level user management is sufficient, overlooking Unity Catalog's strict requirement for centralized account-level identities.

265
MCQhard

A CI/CD pipeline runs unit tests against transformation logic before deploying to production. The tests must run on a Databricks cluster and produce a pass/fail result that fails the pipeline when assertions do not hold. Which implementation best fits this requirement?

A.Run pytest locally in the CI runner against a sample of production data downloaded from cloud storage.
B.Use the Databricks SQL Statement Execution API to run `SELECT` statements that compare row counts between source and target tables.
C.Configure the cluster's init script to run the test suite automatically whenever the cluster starts.
D.Create a Databricks Job that runs a notebook containing assertions, and have the CI/CD pipeline trigger the job and check its run result.
AnswerD

A Databricks Job that executes assertion notebooks on a real cluster exercises the production runtime and returns a run result the pipeline can inspect. If the job fails, the pipeline fails, giving the required pass/fail gate with the same Spark environment used in production.

Why this answer

Tests must run where the production code runs. Triggering a Databricks Job that executes assertion notebooks lets the CI/CD pipeline inspect the run result and fail on assertion errors. This validates behavior on the actual Databricks runtime rather than in a disconnected local environment.

Exam trap

The trap here is treating local pytest runs or SQL row-count checks as equivalent to executing assertions on the Databricks runtime.

266
MCQmedium

A data engineer needs to perform an upsert on a target Delta table using a source DataFrame. Which operation provides the most robust mechanism to handle duplicates and updates in a single pass?

A.Using 'INSERT INTO' combined with a 'DELETE' operation.
B.Using the 'MERGE INTO' command.
C.Using 'overwrite' mode in the DataFrameWriter.
D.Using 'append' mode with custom filter logic.
AnswerB

The MERGE command provides an atomic upsert operation that handles both new records and updates to existing ones in a single, safe transaction. It is the most robust and performant way to manage state changes in a Delta table, ensuring data consistency even in high-concurrency environments.

Why this answer

The MERGE operation is the standard industry practice for upsert patterns in Delta Lake. It allows for atomic updates, inserts, and deletions in a single transaction. This prevents partial writes and maintains the integrity of the data stream, which is crucial in production ELT workflows where data often arrives out of order or contains overlapping records that must be reconciled.

Exam trap

Candidates sometimes choose separate 'DELETE' and 'INSERT' statements, which lack atomicity and can leave the table in an inconsistent state if the job fails mid-execution.

267
MCQeasy

A data engineer needs to create a Delta table in Unity Catalog that will be used by multiple teams. The engineer wants to ensure that the table is governed by Unity Catalog and that access can be controlled using SQL GRANT statements. What is the correct way to create the table?

A.CREATE TABLE my_catalog.my_schema.my_table (id INT, name STRING) USING DELTA;
B.CREATE EXTERNAL TABLE my_catalog.my_schema.my_table (id INT, name STRING) USING DELTA LOCATION '/mnt/data/my_table';
C.CREATE TABLE my_catalog.my_schema.my_table AS SELECT * FROM parquet.`/mnt/data/my_table`;
D.CREATE TABLE my_table (id INT, name STRING) USING DELTA LOCATION '/mnt/data/my_table';
AnswerA

This statement creates a managed Delta table in Unity Catalog under the specified catalog and schema. The table is fully managed by Unity Catalog, and access can be controlled via SQL GRANT statements. This is the correct syntax for creating a Unity Catalog managed table.

Why this answer

To create a managed Delta table in Unity Catalog, use the CREATE TABLE statement with a three-part name (catalog.schema.table) and specify USING DELTA. This ensures the table is governed by Unity Catalog and can be secured with GRANT statements.

Exam trap

The trap here is using a two-part table name or specifying a raw path without registering an external location, which would result in a Hive metastore table or an error, not a Unity Catalog managed table.

268
MCQmedium

Refer to the exhibit. A data engineer is deploying an automated job. Based on the provided configuration, what is the primary benefit of using the 'autoscale' attribute in this cluster definition?

A.It forces the cluster to always use 8 nodes to ensure maximum performance.
B.It enables the cluster to adjust resources dynamically based on current job load.
C.It ensures that the cluster uses specific instance types defined in the policy.
D.It mandates the use of Spot Instances to reduce total cost of ownership.
AnswerB

Autoscaling provides the flexibility to increase or decrease worker node counts based on the volume of data being processed. This capability is essential for balancing job execution speed and resource costs, as it allows the platform to react automatically to the varying demands of Spark tasks.

Why this answer

The 'autoscale' configuration allows the Databricks Intelligence Platform to dynamically adjust the number of worker nodes based on the workload demands. By setting a minimum and maximum range, the cluster automatically adds nodes during intensive processing and removes them during idle periods. This is critical for optimizing costs and performance, ensuring that compute resources are consumed only when necessary without requiring manual intervention by the engineer.

Exam trap

Candidates often mistake autoscaling for spot instance management or automated software patching rather than dynamic resource allocation based on load.

269
MCQmedium

When a data engineer needs to automate a recurring ETL job, which Databricks tool is most appropriate for orchestrating tasks and handling dependencies?

A.Databricks Notebooks
B.Databricks Workflows
C.Delta Live Tables
D.MLflow Models
AnswerB

Databricks Workflows provides a fully managed orchestration service that allows users to define, schedule, and monitor multi-task pipelines. It supports sophisticated dependency graphs, job status notifications, and automated retries, which are essential for building resilient ETL processes that must run reliably in a production data environment.

Why this answer

Databricks Workflows allows engineers to create, run, and monitor multi-task jobs. It supports dependencies, retries, and notifications, making it the standard choice for orchestrating complex data pipelines. By using Workflows, engineers can ensure that data transformation tasks occur in the correct sequence, with robust error handling and monitoring, which is critical for maintaining reliable production data pipelines in the Databricks Intelligence Platform.

Exam trap

Candidates often choose Apache Airflow or Delta Live Tables for general workflow scheduling, overlooking Databricks Workflows as the native orchestration tool for managing multi-task jobs.

270
MCQeasy

A data engineer needs to run a Databricks notebook on a schedule every day at 8:00 AM UTC. The notebook performs data transformations and writes results to a Delta table. The engineer wants to ensure the job runs reliably and can be monitored. Which Databricks feature should the engineer use to schedule and monitor the notebook?

A.Databricks Jobs with a cron schedule.
B.Databricks Delta Live Tables with a continuous pipeline.
C.Databricks Repos with a CI/CD pipeline.
D.Databricks SQL Alerts.
AnswerA

Databricks Jobs allows you to schedule notebooks, JARs, or Python scripts with a cron expression. It provides monitoring, alerting, and retry capabilities. This is the native and recommended way to schedule and monitor recurring tasks in Databricks, making it the correct choice for this scenario.

Why this answer

Databricks Jobs is the correct feature for scheduling and monitoring notebooks. It supports cron-based schedules, provides a UI for monitoring runs, and can send alerts on failure. Other options like SQL Alerts, Repos, or Delta Live Tables serve different purposes and do not provide the required scheduling and monitoring for a notebook.

Therefore, Databricks Jobs is the appropriate choice.

Exam trap

The trap here is confusing Delta Live Tables with Databricks Jobs; Delta Live Tables is for pipelines, not for scheduling arbitrary notebooks.

271
Multi-Selectmedium

A Databricks job is failing due to 'Out of Memory' (OOM) errors during a join operation on two large datasets. Which TWO actions could help mitigate this issue?

Select 2 answers
A.Increase the cluster's worker node instance type with more memory.
B.Decrease the number of partitions in the Spark cluster.
C.Disable the spark.sql.autoBroadcastJoinThreshold configuration.
D.Enable dynamic allocation for the cluster.
E.Use the cache() method on every DataFrame in the pipeline.
AnswersA, C

Scaling up to worker nodes with higher memory capacity provides the Spark executors with more heap space. This allows them to process larger partitions and handle complex shuffle operations without spilling to disk or triggering OOM exceptions. This is the most direct hardware-level fix for memory-intensive join operations.

Why this answer

OOM errors during joins often occur because the driver or worker nodes lack enough memory to handle the shuffle or broadcast operations. Increasing the cluster memory allows for larger data partitions to reside in memory, while adjusting the broadcast join threshold prevents the optimizer from attempting to broadcast datasets that exceed available node capacity. These configurations are essential for stabilizing large-scale ETL pipelines that process high-volume data.

Exam trap

Candidates often try to optimize joins by enabling broadcast thresholds on massive datasets, accidentally triggering Out of Memory errors instead of disabling the threshold.

272
MCQeasy

A data engineer needs to ensure that only members of the 'finance' group can view a specific column containing credit card numbers in a Unity Catalog table. Which feature should be used?

A.Column-level privileges with GRANT SELECT ON COLUMN
B.Row-level security with filter conditions
C.Dynamic views with CASE statements
D.Column masks
AnswerD

Column masks in Unity Catalog allow you to apply a masking function to a column so that users see either the actual value or a masked value based on their group membership. This is the precise feature for restricting access to specific columns. You can define a mask that returns the actual value for the 'finance' group and a masked value for others.

Why this answer

Column masks are designed to control access to specific columns by applying a masking function that evaluates the user's identity or group membership. This allows sensitive data like credit card numbers to be visible only to authorized groups. Other features like row-level security or dynamic views address different aspects of data access and are not the direct solution for column-level control.

Exam trap

The trap here is confusing column-level security with row-level security, or assuming that column-level privileges can be granted directly.

273
Multi-Selecthard

A streaming job using Structured Streaming is lagging significantly behind the 'current time'. Which THREE of the following could be the root cause of this processing latency?

Select 3 answers
A.The trigger interval is set to a time shorter than the processing time of the batch.
B.Lack of proper watermarking on stateful aggregations.
C.Using a fixed-size cluster with no auto-scaling enabled.
D.Inefficient shuffling due to data skew.
E.Using too few partitions in the input source.
AnswersA, B, D

If the time taken to process a batch exceeds the trigger interval, the streaming job cannot keep up. This leads to a queue of pending batches, causing the watermark and processing time to drift further behind, effectively indicating that the cluster is undersized for the current workload volume.

Why this answer

Streaming latency often stems from resource contention, inefficient state management, or micro-batch trigger configurations. When the input rate exceeds the processing rate, lag accumulates. Identifying these bottlenecks requires analyzing the Spark UI for processing time vs. batch time.

Understanding these factors is vital for maintaining low-latency pipelines and ensuring that SLAs are met in production-grade streaming environments deployed on Databricks.

Exam trap

Candidates often overlook trigger interval misconfigurations or forget watermarking requirements, mistakenly assuming streaming lag is always caused solely by insufficient cluster sizing.

274
MCQeasy

Which Databricks compute resource is specifically optimized for running BI dashboards and SQL queries?

A.All-Purpose Cluster
B.Job Cluster
C.SQL Warehouse
D.Delta Live Tables Pipeline
AnswerC

SQL Warehouses are specifically optimized for SQL analytics. They provide features like intelligent query caching, rapid cluster startup, and auto-scaling to handle high concurrency. This makes them the primary compute choice for BI tools, SQL editors, and dashboards where low latency and high reliability are essential for end-user interaction.

Why this answer

Databricks SQL Warehouses are compute environments designed to deliver high-performance SQL execution, making them ideal for BI tools and dashboarding. Understanding the difference between SQL Warehouses and traditional data engineering clusters is important, as it helps engineers choose the right compute resource to match the workload, ensuring better query performance and cost efficiency for end-users who need to interact with data via SQL.

Exam trap

Candidates often confuse standard all-purpose clusters with SQL Warehouses, assuming general compute handles BI traffic equally well, ignoring the specialized query optimization and serverless scaling built specifically for dashboards.

275
MCQmedium

A data engineer maintains a Lakeflow Job with a scheduled trigger set to run every day at 08:00. The job's source table is refreshed by an upstream process that sometimes finishes later than expected, causing the job to process stale data. The engineer wants the job to start only after the upstream refresh completes, regardless of the clock time, while still preserving the existing 08:00 schedule as a fallback. Which trigger configuration should the engineer implement?

A.Convert the scheduled trigger to a continuous trigger so the job restarts immediately after each run.
B.Add a table update trigger on the upstream table while keeping the scheduled trigger, so the job runs when the table refreshes or at 08:00.
C.Add a file arrival trigger that monitors the upstream table's storage location and remove the scheduled trigger.
D.Add a run-duration timeout to the scheduled trigger so the job fails if the upstream has not finished by 08:00.
AnswerB

Lakeflow Jobs support table update triggers that start a run when a specified table receives a commit, which directly signals upstream completion. Keeping the scheduled trigger provides the fallback at 08:00 in case no update is detected. This combination satisfies both the event-driven and time-based requirements precisely without extra orchestration code.

Why this answer

The requirement is a job that starts when upstream data is ready, yet still runs at 08:00 if that event has not occurred. A table update trigger detects upstream commits and starts the job promptly, while the retained scheduled trigger supplies the fallback. Together they deliver event-driven orchestration with a time-based safety net, which neither a pure schedule nor a polling file trigger provides.

Exam trap

The trap here is assuming that a time-based schedule can be replaced by a file arrival trigger, when the requirement actually calls for a table update trigger combined with the existing schedule.

276
MCQmedium

A data engineer needs to join a small 'lookup' table with a large 'transactions' table in a Spark job. Which transformation strategy will provide the best performance in a cluster environment?

A.Increase the 'spark.sql.shuffle.partitions' value to 1000.
B.Use the 'broadcast()' function from 'pyspark.sql.functions' on the small table.
C.Sort both tables by the join key before performing the join.
D.Repartition both tables to the same number of partitions.
AnswerB

The broadcast function provides a hint to the Spark optimizer that the small table should be replicated to all executors. This avoids the cost of shuffling the large transaction table, leading to much faster join performance. It is the best practice for small-to-large table joins in Databricks.

Why this answer

In distributed computing, joining a large table with a small one causes a shuffle if not handled correctly. By using a broadcast join, the engineer forces the small table to be copied to every node, eliminating the need for a network-heavy shuffle of the large table. This technique is a standard optimization for improving query latency and overall throughput in data pipelines.

Exam trap

Candidates often select standard join operations or attempt to sort datasets manually, forgetting that standard joins on mismatched dataset sizes cause expensive and unnecessary network shuffles across executors.

Page 3

Page 4 of 4

All pages