Databricks-DE-Assoc Databricks Intelligence Platform — 15 Questions

Question 1

A data engineering team is migrating legacy Hive metastore tables to Unity Catalog. They want to ensure proper governance, data lineage tracking, and secure external access. Which TWO actions must be performed to successfully manage these tables in Unity Catalog? (Choose two)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Reference the migrated tables using the three-level namespace format: catalog_name.schema_name.table_name.. Migrating to Unity Catalog requires updating table references to the three-level namespace and managing data access through external locations rather than legacy storage credentials. These changes ensure unified governance, automated lineage tracking, and secure isolation across workspaces within the Databricks Intelligence Platform.

Answer

Continue using legacy dbfs:/my_table paths directly in Spark read commands without defining an external location.

Answer

Configure each user's personal access token as the default storage credential for reading managed table data.

Answer

Disable automatic data lineage tracking in the metastore settings to improve write performance.

Question 2

A data engineer is designing an incremental data ingestion pipeline using Delta Live Tables (DLT). The source files arrive continuously in an object storage bucket, but some files occasionally contain corrupted records that must be isolated rather than failing the entire pipeline. Which DLT feature should the data engineer implement to handle these invalid records?

Accepted Answer

Apply expectations with the ON VIOLATION DROP ROW clause to automatically filter out invalid records from the target table.. While 'ON VIOLATION DROP ROW' satisfies the requirement to not fail the pipeline, it discards the data. To 'isolate' records as requested in the stem, the engineer would typically use a combination of expectations and a separate table (quarantine) or simply acknowledge that DROP ROW is the mechanism to prevent pipeline failure, even if it doesn't technically 'isolate' (save) the data. The explanation is slightly misleading regarding the 'isolation' aspect.

Answer

Wrap the ingestion code in a traditional try-except block and write failed batches to a delta log file.

Answer

Configure the cluster to automatically restart whenever an out-of-memory or parsing exception occurs.

Answer

Use Spark Structured Streaming's maxFilesPerTrigger configuration to restrict processing to one clean file at a time.

Question 3

Refer to the exhibit. An engineer needs to ensure that the cluster can handle sudden increases in data volume while minimizing costs during idle periods. Given the JSON configuration shown, what will happen when the workload demand spikes?

Accepted Answer

The cluster will add up to 8 additional worker nodes.. In Databricks terminology, adding more nodes to a cluster is referred to as 'scaling out' (or autoscaling up to the maximum number of workers), not 'scaling up' (which typically refers to resizing a node to a larger instance type with more CPU/RAM). Option C uses 'scale up' incorrectly.

Answer

The cluster will terminate automatically after 10 minutes of inactivity.

Answer

The cluster will scale up to a maximum of 8 worker nodes.

Answer

The cluster will lock at 2 workers to prevent cost overruns.

Question 4

Which Databricks feature allows users to query data using standard SQL directly in a notebook environment?

Accepted Answer

Databricks SQL. While Databricks SQL is a product offering, the ability to run SQL in a notebook is a feature of the Databricks Runtime (specifically the Spark SQL engine), not the 'Databricks SQL' product itself. Databricks SQL is a separate compute environment (SQL Warehouses). The question asks for a feature that allows SQL in a notebook, which is technically the Spark SQL engine accessed via magic commands, not the Databricks SQL product.

Answer

Databricks File System (DBFS)

Answer

Delta Sharing

Answer

MLflow Tracking

Question 5

A data engineering team wants to allow business analysts to query processed data tables securely using natural language. Which component of the Databricks Intelligence Platform should the team implement to directly support natural language querying over curated tables?

Accepted Answer

Databricks Assistant. While Databricks Assistant helps write code, the specific feature designed for business analysts to query data using natural language in a dashboard/BI context is Databricks AI/BI (formerly Lakeview) or the SQL AI features. Databricks Assistant is a developer productivity tool, not a business analyst querying interface for curated tables.

Answer

Unity Catalog external locations

Answer

Delta Live Tables expectation monitors

Answer

Databricks Git integration

Question 6

A data engineering team at a retail company uses Azure Databricks with Unity Catalog. They need to grant a group of analysts read-only access to all tables in the schema analytics.sales without granting access to future tables added to that schema. Which SQL statement should they execute?

Accepted Answer

GRANT SELECT ON SCHEMA analytics.sales TO analysts;. To provide read-only access to all current and future tables in a schema, the correct privilege to grant is SELECT on the schema. This grants the SELECT privilege on all tables within the schema and automatically applies to any tables created later. Granting USAGE alone does not allow data access, changing ownership is excessive, and wildcard table grants are not supported.

Answer

ALTER SCHEMA analytics.sales OWNER TO analysts;

Answer

GRANT SELECT ON TABLE analytics.sales.* TO analysts;

Answer

GRANT USAGE ON SCHEMA analytics.sales TO analysts;

Question 7

A data engineer is using Databricks SQL to create a dashboard that visualizes sales data. They need to ensure that the dashboard refreshes automatically every hour and that users see the latest data. Which approach should they take?

Accepted Answer

Create a scheduled query that runs every hour and configure the dashboard to use that query's results.. Scheduled queries in Databricks SQL allow you to run queries on a schedule and have dashboards display the results. This provides automatic hourly refresh and ensures users see the latest data from the last run. Other options either do not schedule refreshes or require manual intervention.

Answer

Configure the dashboard to query live data directly from the source tables without caching.

Answer

Enable auto-refresh on the dashboard with a 1-hour interval.

Answer

Use a Databricks job to run the dashboard queries and update a materialized view.

Question 8

A data engineer is using Databricks Workflows to orchestrate a data pipeline. The pipeline consists of a series of tasks that must run sequentially, and the engineer wants to ensure that if a task fails, the entire job stops and does not proceed to subsequent tasks. Which configuration should the engineer set for the job?

Accepted Answer

Set the task dependencies to linear and configure the job to fail fast.. Databricks Workflows allows defining task dependencies to control execution order. With linear dependencies, tasks run sequentially. The fail-fast behavior ensures that if a task fails, the job is immediately stopped, and no further tasks are executed. This is the correct configuration to meet the requirement.

Answer

Set the job to use a job cluster and enable autoscaling.

Answer

Set the maximum concurrent runs to 1.

Answer

Set the job to run continuously with a retry policy.

Question 9

A data engineer is using Databricks Delta Live Tables (DLT) to build a pipeline. They need to ensure that the pipeline processes data incrementally and handles late-arriving data by updating existing records. Which DLT feature should they use?

Accepted Answer

APPLY CHANGES INTO with a specified sequence column. APPLY CHANGES INTO is the DLT feature for handling updates and late-arriving data. It uses a sequence column to order changes and applies them incrementally. Other options either do not update existing records or are not valid DLT syntax.

Answer

CREATE LIVE TABLE with a MERGE statement

Answer

CREATE STREAMING LIVE TABLE with a watermark

Answer

Use AUTO CDC with a sequence column

Question 10

A data engineer is working with a Delta table that contains sensitive customer data. The table is stored in a Unity Catalog managed location. The engineer needs to ensure that only members of the 'data_analysts' group can query the table, while members of the 'data_engineers' group can also modify it. Other users should have no access. Which approach should the engineer use to meet these requirements?

Accepted Answer

Grant USAGE on the schema and SELECT on the table to 'data_analysts', and grant USAGE on the schema, SELECT, and MODIFY on the table to 'data_engineers'. Ensure no other grants exist.. Unity Catalog requires USAGE on the schema for any access to objects within it. To query a table, SELECT is needed; to modify, MODIFY is required. Granting these specific privileges to the appropriate groups and ensuring no other grants (including inherited ones) enforces the least privilege principle. This approach precisely meets the requirement that analysts can query and engineers can modify, while others have no access.

Answer

Grant SELECT to 'data_analysts' and ALL PRIVILEGES to 'data_engineers' on the table, and revoke all privileges from other users.

Answer

Create a view that filters sensitive columns, grant SELECT on the view to 'data_analysts', and grant MODIFY on the table to 'data_engineers'.

Answer

Grant SELECT to 'data_analysts' and MODIFY to 'data_engineers' on the table using Unity Catalog SQL commands.

Question 11

A data engineer needs to configure a Databricks Job to orchestrate a data pipeline that includes a Python task, a SQL task, and a notebook task. The pipeline requires passing a dynamic run identifier from the Python task to the subsequent SQL and notebook tasks. Which mechanism should the data engineer use to achieve this task-to-task dependency parameter passing?

Accepted Answer

Use the dbutils.jobs.taskValues.set method in the Python task and reference it downstream using the task value syntax.. Databricks workflows natively support passing values between tasks using task values. A Python task can set a task value using dbutils.jobs.taskValues.set, which downstream tasks can then reference using standard task value interpolation syntax. This eliminates external storage dependencies, ensuring robust, serverless orchestration directly managed by the Databricks control plane.

Answer

Write the dynamic identifier to a shared Unity Catalog volume and read it inside the downstream tasks.

Answer

Store the identifier as an environment variable in the cluster configuration shared by all three tasks.

Answer

Modify the global job parameters dynamically via the Databricks REST API from within the running Python script.

Question 12

A data engineer needs to store structured data in a cloud object storage location while maintaining full ACID guarantees. Which storage format is the foundation of the Databricks Lakehouse architecture that enables this functionality?

Accepted Answer

Delta Lake. Delta Lake is the open-source storage layer that brings reliability and ACID transactions to data lakes. By utilizing transaction logs and Parquet files, it ensures data consistency for concurrent read and write operations. Understanding Delta Lake is fundamental for Databricks engineers, as it underpins virtually all data operations within the platform, enabling features like time travel, schema enforcement, and efficient upserts on massive datasets.

Answer

JSON

Answer

CSV

Answer

Avro

Question 13

A data engineer is designing a pipeline and needs to ensure that data remains consistent during concurrent read and write operations. Which Databricks feature provides the mechanism to track and validate these operations?

Accepted Answer

Delta Lake Transaction Log. The Delta Lake transaction log, often referred to as the _delta_log, is a centralized record of all changes made to a table. It ensures atomicity and durability by serializing transactions. This mechanism is critical because it allows Databricks to manage concurrent access without locks, providing a consistent view of the data even when multiple users or processes are interacting with the same underlying storage files simultaneously.

Answer

Unity Catalog

Answer

Databricks SQL

Answer

Cluster Manager

Question 14

Which TWO of the following statements accurately describe the functionality of Unity Catalog within the Databricks Intelligence Platform?

Accepted Answer

It provides centralized access control, auditing, and lineage for all data and AI assets.. Unity Catalog provides a unified governance layer for data and AI assets across a Databricks account. It simplifies security by providing a single place to define access control policies, and it enables cross-workspace data sharing. Understanding this is essential for engineers responsible for managing multi-workspace environments, as it replaces the older, workspace-specific Hive Metastore with a centralized, cloud-agnostic governance model for better auditability and compliance.

Answer

It requires a separate Hive Metastore for each individual Databricks workspace.

Answer

It is an execution engine used to run heavy machine learning training jobs.

Answer

It only supports files stored in Amazon S3 buckets.

Question 15

When considering the Databricks Intelligence Platform, what is the primary role of the 'Lakehouse' architecture?

Accepted Answer

To unify the best features of data warehouses and data lakes.. The Lakehouse architecture combines the data management, schema support, and ACID transactions of a data warehouse with the scale, flexibility, and low-cost storage of a data lake. This is important for engineers because it eliminates the need to maintain separate systems for BI and AI/ML, allowing teams to use a single platform for all data processing tasks, thereby reducing data duplication and architectural complexity.

Answer

To replace cloud object storage with a proprietary proprietary binary format.

Answer

To isolate streaming and batch data processing into distinct storage layers.

Answer

To mandate the use of SQL as the only supported programming language.