Databricks-DE-Assoc Practice Questions

Question 1

A data engineer is using Auto Loader to ingest data from a directory that receives thousands of files every hour. They are considering switching from the default directory listing mode to file notification mode. What is the primary reason for making this change?

Accepted Answer

To reduce the cost and latency of discovering new files as the bucket grows.. As the number of files in a cloud storage bucket grows, the time and cost required to list all files recursively increase significantly. File notification mode solves this by using cloud services to push events about new files directly to Auto Loader, making it far more efficient for high-volume, long-term ingestion projects.

Answer

To enable the use of schema evolution in the ingestion pipeline.

Answer

To ensure that files are processed in the exact order they arrived.

Answer

To allow the stream to process multiple file formats simultaneously.

Question 2

A data engineer has a Lakeflow Job with a notebook task that occasionally fails due to transient network errors when reading from an external REST API. The engineer wants the task to automatically retry up to three times, but only for this specific task, without affecting other tasks in the job. What should the engineer do?

Accepted Answer

Set the task's Retries to 3 in the task configuration.. The correct approach is to set the Retries field on the specific task to 3. This is a built-in feature of Lakeflow Jobs that applies per task, so other tasks remain unaffected. It is the simplest and most direct way to handle transient failures without modifying code or job-level settings.

Answer

Configure the task's Retries setting to 3 and add a retry condition using the task's timeout.

Answer

Use a notebook %run magic command to wrap the API call in a loop that retries three times.

Answer

Set the job-level maximum concurrent runs to 3 and enable retries.

Question 3

While ingesting CSV files using Auto Loader, a data engineer notices that some records have malformed data that does not match the inferred schema. How can the engineer capture these records without failing the entire ingestion stream?

Accepted Answer

By enabling the '_rescued_data' column in the Auto Loader configuration.. The rescued data column is a powerful feature in Auto Loader that prevents data loss during ingestion. By capturing malformed or unexpected data in a dedicated column, engineers can ensure that the main pipeline continues to run while providing a way to audit and correct data quality issues after the fact.

Answer

By setting the 'mode' option to 'FAILFAST' in the reader.

Answer

By using a TRY_CAST function in a transformation after the read.

Answer

By increasing the 'maxFilesPerTrigger' to handle more errors.

Question 4

Refer to the exhibit. An organization needs to ensure that members of the 'analyst_group' can only view data where the region column equals 'US'. Based on the provided configuration, what is the best approach to implement this in Unity Catalog?

Accepted Answer

Create a Dynamic View with a WHERE clause and grant SELECT access to the view.. To implement row-level security in Databricks, developers use Dynamic Views. By defining a view that contains a filter clause, you can restrict which rows are returned to the user based on their context or group membership. This ensures that sensitive data is hidden at query time without physically duplicating data or creating multiple siloed tables for different regions.

Answer

Apply the policy directly to the table using an ALTER TABLE command.

Answer

Use the GRANT FILTER command to apply the condition to the analyst_group.

Answer

Enable Column-Level Security in the Unity Catalog metastore settings.

Question 5

A data engineer is designing a secure architecture using the Databricks Intelligence Platform. Which TWO of the following statements accurately describe the role and capabilities of Unity Catalog within this platform? (Choose TWO)

Accepted Answer

Unity Catalog provides centralized access control for tables, views, volumes, and AI models across multiple workspaces.. Unity Catalog is the fine-grained governance solution for data and AI across the Databricks platform, centralizing access control for tables, volumes, and models. It operates across multiple workspaces in a metastore, ensuring consistent security policies and audit logging regardless of which workspace or compute resource executes the user queries.

Answer

Unity Catalog replaces the underlying cloud provider IAM roles for all compute resources executing queries.

Answer

Unity Catalog automatically handles the physical partitioning of Delta tables for optimal query performance.

Answer

Unity Catalog requires all data to be migrated into proprietary binary formats before governance policies can be applied.

Question 6

A team is using Databricks Asset Bundles (DABs) to deploy a job to multiple environments (dev, staging, prod). They need to ensure that the job uses different cluster sizes and schedules per environment. Which DABs feature should they use?

Accepted Answer

Define separate bundle configuration files for each environment and use the --target flag with databricks bundle deploy.. Databricks Asset Bundles use targets to define environment-specific overrides in the databricks.yml file. When deploying, the --target flag selects the target, applying the corresponding cluster size and schedule. This keeps a single source of truth while allowing differences per environment, which is essential for promoting bundles through dev, staging, and prod.

Answer

Use the Databricks REST API to update the job configuration after deployment with environment-specific values.

Answer

Create a separate Databricks workspace for each environment and hardcode the cluster size and schedule in the job definition.

Answer

Use Jinja2 templating in the job definition to inject environment-specific values from environment variables.

Question 7

A data engineer is using Auto Loader to ingest files from an S3 bucket into a Delta table. The files are partitioned by date in the path, e.g., s3://bucket/data/2023-01-01/file1.json. The engineer wants to automatically add a column 'date' to the ingested data based on the file path. Which Auto Loader feature should be used?

Accepted Answer

Use the cloudFiles.partitionColumns option to specify 'date'.. Auto Loader can automatically extract partition columns from directory structures using the cloudFiles.partitionColumns option. When you specify the column names, Auto Loader parses the file path and creates corresponding columns in the DataFrame, populating them with the values from the path. This is the standard method for incorporating path-based partitioning into the ingested data without manual parsing. Other options like schema hints or format settings do not provide this capability, and incremental listing is unrelated.

Answer

Use the cloudFiles.useIncrementalListing option to parse the path.

Answer

Use the cloudFiles.format option set to 'json' and then use a SQL expression to extract the date.

Answer

Use the cloudFiles.schemaHints option to define 'date' as a string column.

Question 8

Which feature of the Databricks Intelligence Platform allows users to manage fine-grained access control across workspaces for tables, files, and machine learning models?

Accepted Answer

Unity Catalog. Unity Catalog is the centralized governance layer within the Databricks Intelligence Platform. It provides a single interface to manage permissions across the entire data estate. This is vital for security and compliance because it eliminates the need for siloed permission management in individual workspaces, ensuring that security policies are consistently applied and auditable across all users, data assets, and compute resources in the platform.

Answer

Cluster Policies

Answer

Workspace Admin Settings

Answer

Secret Scopes

Question 9

An organization requires that all data access logs across multiple Databricks workspaces be captured and sent to a centralized security information and event management (SIEM) system. Which Databricks feature should be configured to capture these audit events?

Accepted Answer

Configure diagnostic log delivery to stream account and workspace audit logs to cloud storage or event hubs.. Databricks audit logs capture workspace-level and account-level user activities, including data access, administrative actions, and permission changes. Organizations configure audit log delivery to route these JSON event logs directly to cloud storage buckets, where SIEM tools like Splunk or Datadog can ingest and analyze them for compliance.

Answer

Enable Spark event logs in the cluster configuration UI for every running compute cluster.

Answer

Run a nightly notebook that queries the system.information schema for active user sessions.

Answer

Install a Python logging package on every cluster driver node to intercept SQL queries.

Question 10

What is the primary function of the 'Retries' setting in a Databricks Job task?

Accepted Answer

To automatically recover from transient failures.. The 'Retries' setting is a robust mechanism for handling transient failures, such as network timeouts or temporary cloud provider issues, without manual intervention. By configuring retries, engineers increase the resilience of their pipelines. This is a critical best practice in production environments where external dependencies may be unstable, as it minimizes the need for on-call support and ensures that jobs eventually succeed despite minor, non-permanent infrastructure hiccups.

Answer

To increase the task execution speed.

Answer

To allow tasks to run in parallel.

Answer

To change the cluster node type.

Question 11

A team stores Databricks notebooks and job definitions in a Git repository and wants every merge to the main branch to automatically deploy to production. Which combination of practices should the pipeline implement to achieve this safely?

Accepted Answer

Trigger the deployment workflow on pushes to the main branch, authenticate with a service principal whose credentials are stored in the CI system's secret manager, and run databricks bundle deploy.. Safe continuous deployment requires two things: a trigger scoped to the main branch so only reviewed, merged code reaches production, and non-interactive authentication via a service principal stored in the CI secret manager. Running databricks bundle deploy from the merged commit then applies the bundle definition reproducibly. Scheduling, branch-agnostic triggers, and personal tokens all undermine control or automation.

Answer

Trigger the deployment workflow on a nightly schedule and use a personal access token for authentication so the deployment uses the developer's existing permissions.

Answer

Trigger the deployment workflow on pushes to any branch and use the Databricks CLI with interactive browser login to authenticate to the production workspace.

Answer

Trigger the deployment workflow on pull requests and use the developer's personal access token stored in the repository as a plaintext secret.

Question 12

When configuring a Databricks Job, which TWO factors most directly influence the choice between a 'Job Cluster' and an 'All-Purpose Cluster'?

Accepted Answer

The need for interactive debugging and notebook testing.. Choosing the right cluster type is vital for balancing cost and performance. Job Clusters are ephemeral, optimized for production tasks, and significantly cheaper, whereas All-Purpose Clusters are persistent and intended for interactive development. Understanding the cost implications and lifecycle management of these clusters is fundamental for any Databricks engineer responsible for managing operational budgets and ensuring efficient resource utilization across various environments.

Answer

The requirement for high availability during query execution.

Answer

The number of users concurrently using the cluster.

Answer

The type of data source (e.g., S3 vs. ADLS).

Question 13

A data engineer is designing a workflow that requires running a series of tasks with dependencies. The workflow must be able to retry failed tasks automatically and send notifications on failure. The engineer also needs to ensure that the workflow can be triggered on a schedule and via an API. Which Databricks feature should the engineer use?

Accepted Answer

Databricks Jobs with task dependencies and retry policies.. Databricks Jobs are the primary orchestration feature for workflows with dependencies, retries, and notifications. They support multi-task DAGs, task-level retry policies, and alerting on failure. Jobs can be scheduled and triggered via API, making them the correct choice. The other options lack the necessary orchestration capabilities.

Answer

Delta Live Tables pipelines with continuous mode.

Answer

Databricks SQL dashboards with scheduled refresh.

Answer

Databricks Repos with Git hooks.

Question 14

Which TWO of the following capabilities are native to the Databricks Unity Catalog?

Accepted Answer

Row and column-level security. Unity Catalog centralizes governance by providing a unified interface for managing permissions across workspaces. It enables fine-grained access control down to the row and column level and provides a centralized audit log. These capabilities are vital for organizations needing to manage data security at scale, ensuring that data engineers and scientists have consistent access control policies across the entire Lakehouse regardless of the specific compute resource or workspace utilized.

Answer

Auto-scaling of compute clusters

Answer

Automated ML model hyperparameter tuning

Answer

Direct hardware driver installation

Question 15

A data engineer is configuring Unity Catalog row-level security on a table `prod.finance.transactions`. They want to ensure that users in the `us_team` group can only see rows where `region = 'US'`, while users in the `eu_team` group can only see rows where `region = 'EU'`. They create a row filter function `prod.security.region_filter`. Which TWO statements accurately describe how to implement and manage this row-level security? (Choose two.)

Accepted Answer

The row filter function must accept the column(s) as parameters and return a BOOLEAN value indicating whether the row should be visible.. Row-level security in Unity Catalog is implemented by attaching a row filter function to a table using ALTER TABLE ... SET ROW FILTER. The function receives column values and returns a boolean, often using identity functions to filter based on group membership. Both managed and external tables support row filters, and bypassing them requires the BYPASS ROW FILTER privilege.

Answer

Row filters can only be applied to external tables, not managed tables, because managed tables do not support fine-grained access control.

Answer

The row filter function must be defined in the same schema as the table it protects.

Answer

Users with the MODIFY privilege on the table automatically bypass the row filter and see all rows.

Databricks Certified Data Engineer Associate Databricks-DE-Assoc practice test

Three ways to study

Study Databricks-DE-Assoc by domain

Study Databricks-DE-Assoc by topic

Data Transformation and Modeling practice questions

Databricks Intelligence Platform practice questions

Implementing CI/CD practice questions

Data Ingestion and Loading practice questions

Troubleshooting, Monitoring, and Optimization practice questions

Working with Lakeflow Jobs practice questions

Governance and Security practice questions

Databricks-DE-Assoc fundamentals practice questions

Databricks-DE-Assoc scenario practice questions

Databricks-DE-Assoc troubleshooting practice questions

Databricks Certified Data Engineer Associate practice questions