Databricks-DE-Assoc Quick Quiz — 10 Questions

Question 1

A data engineer is using Auto Loader to ingest data from a directory that receives thousands of files every hour. They are considering switching from the default directory listing mode to file notification mode. What is the primary reason for making this change?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

To reduce the cost and latency of discovering new files as the bucket grows.. As the number of files in a cloud storage bucket grows, the time and cost required to list all files recursively increase significantly. File notification mode solves this by using cloud services to push events about new files directly to Auto Loader, making it far more efficient for high-volume, long-term ingestion projects.

Answer

To enable the use of schema evolution in the ingestion pipeline.

Answer

To ensure that files are processed in the exact order they arrived.

Answer

To allow the stream to process multiple file formats simultaneously.

Question 2

A data engineer has a Lakeflow Job with a notebook task that occasionally fails due to transient network errors when reading from an external REST API. The engineer wants the task to automatically retry up to three times, but only for this specific task, without affecting other tasks in the job. What should the engineer do?

Accepted Answer

Set the task's Retries to 3 in the task configuration.. The correct approach is to set the Retries field on the specific task to 3. This is a built-in feature of Lakeflow Jobs that applies per task, so other tasks remain unaffected. It is the simplest and most direct way to handle transient failures without modifying code or job-level settings.

Answer

Configure the task's Retries setting to 3 and add a retry condition using the task's timeout.

Answer

Use a notebook %run magic command to wrap the API call in a loop that retries three times.

Answer

Set the job-level maximum concurrent runs to 3 and enable retries.

Question 3

While ingesting CSV files using Auto Loader, a data engineer notices that some records have malformed data that does not match the inferred schema. How can the engineer capture these records without failing the entire ingestion stream?

Accepted Answer

By enabling the '_rescued_data' column in the Auto Loader configuration.. The rescued data column is a powerful feature in Auto Loader that prevents data loss during ingestion. By capturing malformed or unexpected data in a dedicated column, engineers can ensure that the main pipeline continues to run while providing a way to audit and correct data quality issues after the fact.

Answer

By setting the 'mode' option to 'FAILFAST' in the reader.

Answer

By using a TRY_CAST function in a transformation after the read.

Answer

By increasing the 'maxFilesPerTrigger' to handle more errors.

Question 4

Refer to the exhibit. An organization needs to ensure that members of the 'analyst_group' can only view data where the region column equals 'US'. Based on the provided configuration, what is the best approach to implement this in Unity Catalog?

Accepted Answer

Create a Dynamic View with a WHERE clause and grant SELECT access to the view.. To implement row-level security in Databricks, developers use Dynamic Views. By defining a view that contains a filter clause, you can restrict which rows are returned to the user based on their context or group membership. This ensures that sensitive data is hidden at query time without physically duplicating data or creating multiple siloed tables for different regions.

Answer

Apply the policy directly to the table using an ALTER TABLE command.

Answer

Use the GRANT FILTER command to apply the condition to the analyst_group.

Answer

Enable Column-Level Security in the Unity Catalog metastore settings.

Question 5

A data engineer is designing a secure architecture using the Databricks Intelligence Platform. Which TWO of the following statements accurately describe the role and capabilities of Unity Catalog within this platform? (Choose TWO)

Accepted Answer

Unity Catalog provides centralized access control for tables, views, volumes, and AI models across multiple workspaces.. Unity Catalog is the fine-grained governance solution for data and AI across the Databricks platform, centralizing access control for tables, volumes, and models. It operates across multiple workspaces in a metastore, ensuring consistent security policies and audit logging regardless of which workspace or compute resource executes the user queries.

Answer

Unity Catalog replaces the underlying cloud provider IAM roles for all compute resources executing queries.

Answer

Unity Catalog automatically handles the physical partitioning of Delta tables for optimal query performance.

Answer

Unity Catalog requires all data to be migrated into proprietary binary formats before governance policies can be applied.

Question 6

A team is using Databricks Asset Bundles (DABs) to deploy a job to multiple environments (dev, staging, prod). They need to ensure that the job uses different cluster sizes and schedules per environment. Which DABs feature should they use?

Accepted Answer

Define separate bundle configuration files for each environment and use the --target flag with databricks bundle deploy.. Databricks Asset Bundles use targets to define environment-specific overrides in the databricks.yml file. When deploying, the --target flag selects the target, applying the corresponding cluster size and schedule. This keeps a single source of truth while allowing differences per environment, which is essential for promoting bundles through dev, staging, and prod.

Answer

Use the Databricks REST API to update the job configuration after deployment with environment-specific values.

Answer

Create a separate Databricks workspace for each environment and hardcode the cluster size and schedule in the job definition.

Answer

Use Jinja2 templating in the job definition to inject environment-specific values from environment variables.

Question 7

A data engineer is using Auto Loader to ingest data from a directory that receives thousands of files every hour. They are considering switching from the default directory listing mode to file notification mode. What is the primary reason for making this change?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the cloudFiles.partitionColumns option to specify 'date'.. Auto Loader can automatically extract partition columns from directory structures using the cloudFiles.partitionColumns option. When you specify the column names, Auto Loader parses the file path and creates corresponding columns in the DataFrame, populating them with the values from the path. This is the standard method for incorporating path-based partitioning into the ingested data without manual parsing. Other options like schema hints or format settings do not provide this capability, and incremental listing is unrelated.

Answer

Use the cloudFiles.useIncrementalListing option to parse the path.

Answer

Use the cloudFiles.format option set to 'json' and then use a SQL expression to extract the date.

Answer

Use the cloudFiles.schemaHints option to define 'date' as a string column.

Question 8

Which feature of the Databricks Intelligence Platform allows users to manage fine-grained access control across workspaces for tables, files, and machine learning models?

Accepted Answer

Unity Catalog. Unity Catalog is the centralized governance layer within the Databricks Intelligence Platform. It provides a single interface to manage permissions across the entire data estate. This is vital for security and compliance because it eliminates the need for siloed permission management in individual workspaces, ensuring that security policies are consistently applied and auditable across all users, data assets, and compute resources in the platform.

Answer

Cluster Policies

Answer

Workspace Admin Settings

Answer

Secret Scopes

Question 9

An organization requires that all data access logs across multiple Databricks workspaces be captured and sent to a centralized security information and event management (SIEM) system. Which Databricks feature should be configured to capture these audit events?

Accepted Answer

Configure diagnostic log delivery to stream account and workspace audit logs to cloud storage or event hubs.. Databricks audit logs capture workspace-level and account-level user activities, including data access, administrative actions, and permission changes. Organizations configure audit log delivery to route these JSON event logs directly to cloud storage buckets, where SIEM tools like Splunk or Datadog can ingest and analyze them for compliance.

Answer

Enable Spark event logs in the cluster configuration UI for every running compute cluster.

Answer

Run a nightly notebook that queries the system.information schema for active user sessions.

Answer

Install a Python logging package on every cluster driver node to intercept SQL queries.

Question 10

What is the primary function of the 'Retries' setting in a Databricks Job task?

Accepted Answer

To automatically recover from transient failures.. The 'Retries' setting is a robust mechanism for handling transient failures, such as network timeouts or temporary cloud provider issues, without manual intervention. By configuring retries, engineers increase the resilience of their pipelines. This is a critical best practice in production environments where external dependencies may be unstable, as it minimizes the need for on-call support and ensures that jobs eventually succeed despite minor, non-permanent infrastructure hiccups.

Answer

To increase the task execution speed.

Answer

To allow tasks to run in parallel.

Answer

To change the cluster node type.